710
muskrat's data eng expert's hard drive overheats while processing 60k rows
(lemmy.dbzer0.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 16 Mar 2025
710 points (98.9% liked)
Programmer Humor
21568 readers
2032 users here now
Welcome to Programmer Humor!
This is a place where you can post jokes, memes, humor, etc. related to programming!
For sharing awful code theres also Programming Horror.
Rules
- Keep content in english
- No advertisements
- Posts must be related to programming or programmer topics
founded 2 years ago
MODERATORS
I'm a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?
Some interesting facts about excel I learned the hard way.
Not really related to what you said, but I'm still sore about the bad data import that caused me days of work to clean up.
The row limitation seems, to me, like an actually-good thing. Excel is for data where you might conceivably scroll up and down looking at it and 1M is definitely beyond the ability of a human even to just skim looking for something different.
An older version of Excel could only handle 64k rows and I had a client who wanted large amounts of data in Excel format. "Oh sorry, it's a Microsoft limitation," I was thrilled to say. "I have no choice but to give you a useful summarization of the data instead of 800k rows (each 1000 columns wide) of raw data."