Elon Musk is the walking talking embodiment of the Dunning-Kruger effect.

Its because the comments he made are inconsistent with common conventions in data engineering.

1. It is very common not to deduplicate data and instead just append rows, The current value is the most recent and all the old ones are simply historical. That way you don't risk losing data and you have an entire history.
   • whilst you could do some trickery to deduplicate the data it does create more complexity. There's an old saying with ZFS: "Friends don't let friends dedupe" And it's much the same here.
   • compression is usually good enough. It will catch duplicated data and deal with it in a fairly efficient way, not as efficient as deduplication but it's probably fine and it's definitely a lot simpler
2. Claiming the government does not use SQL
   • It's possible they have rolled their own solution or they are using MongoDB Or something but this would be unlikely and wouldn't really refute the initial claim
   • I believe many other commenters noted that it probably is MySQL anyway.

Basically what he said is ~~incoherent~~ inconsistent with typical practices among data engineers ~~to anybody who has worked with larger data.~~

In terms of using SQL, it's basically just a more reliable and better Excel that doesn't come with a default GUI.

If you need to store data, It's almost always best throw it into a SQLite database Because it keeps it structured. It's standardised and it can be used from any programming language.

However, many people use excel because they don't have experience with programming languages.

Get chatGpt to help you write a PyQT GUI for a SQLite database and I think you would develop a high level understanding for how the pieces fit together

Edit: @zalgotext made a good point.

As a data engineer for the past 20+ years: There is absolutely no fucking way that the us gov doesnt use sql. This is what shows that he’s stupid not only in sql but in data science in general.

Regarding duplications: its more nuanced than those statements each side put. There can be duplications in certain situations. In some situations there shouldnt be. And I dont really see how duplications in a db is open to fraud.

Because a simple query would have shown that SSN was a compound key with another column (birth date, I think), and not the identifier he thinks it is.

He is saying the US government doesn't use structured databases.

At least 90% of all databases have a structure.

It’s so basic that documentation is completely unnecessary.

“De-duping” could mean multiple things, depending on what you mean by “duplicate”.

It could mean that the entire row of some table is the same. But that has nothing to do with the kind of fraud he’s talking about. Two people with the same SSN but different names wouldn’t be duplicates by that definition, so “de-duping” wouldn’t remove it.

It can also mean that a certain value shows up more than once (eg just the SSN). But that’s something you often want in database systems. A transaction log of SSN contributions would likely have that SSN repeated hundreds of times. It has nothing to do with fraud, it’s just how you record that the same account has multiple contributions.

A database system as large as the SSA has needs to deal with all kinds of variations in data (misspellings, abbreviations, moves, siblings, common names, etc). Something as simplistic as “no dupes anywhere” would break immediately.

Because of course the government uses SQL. It's as stupid as saying the government doesn't use electricity or something equally stupid. The government is myriad agencies running myriad programs on myriad hardware with myriad people. My damned computers at home are using at least 2-3 SQL databases for some of the programs I run.

SQL is damn near everywhere where data sets are found.

He's just a permanent petulant child.

TL;DR de-deuplication in that form is used to refer a technique where you reference two different pieces of data in the file system, with one single piece of data on the drive, the intention being to optimize file storage size, and minimize fragmentation.

You can imagine this would be very useful when taking backups for instance, we call this a "Copy on Write" approach, since generally it works by copying the existing file to a second reference point, where you can then add an edit on top of the original file, while retaining 100% of the original file size, and both copies of the file (its more complicated than this obviously, but you get the idea)

now just to be clear, if you did implement this into a DB, which you could do fairly trivially, this would change nothing about how the DB operates, it wouldn't remove "duplicates" it would only coalesce duplicate data into one single tree to optimize disk usage. I have no clue what elon thinks it does.

The problem here, as a non programmer, is that i don't understand why you would ever de-duplicate a database. Maybe there's a reason to do it, but i genuinely cannot think of a single instance where you would want to delete one entry, and replace it with a reference to another, or what elon is implying here (remove "duplicate" entries, however that's supposed to work)

Elon doesn't know what "de-duplication" is, and i don't know why you would ever want that in a DB, seems like a really good way to explode everything,

139 comments and no one addresses his use of a slur.

I think a lot of comments here miss the mark, it's not really just about stating the gov does not use SQL or speculation regarding keys.

Deduplication is generally part of a compression strategy and has nothing to do with SQL. If we're being generous he may have been talking about normalization, but no one I have ever met has confused the two terms (they are distinctly different from an engineering perspective).

There are degrees of normalization too, so it may make total sense to normalize 3NF (third normal form) rather than say 6NF depending on the data.