1721
I love it when I have to scream at a computer
(lemmy.world)
Post funny things about programming here! (Or just rant about your favourite programming language.)
Your knowledge of data engineering may be limited. SQL is predominant in data processing nowadays. FOSS tools such as DBT allows to write efficient data processing pipelines with SQL and some YAML config without the need for a general purpose coding language.
Why would anyone want that? Because SQL has the interesting property of describing the result you want rather than describing how to compute it. So you can put inside the database, a query engine with decades of optimizations, that will make a much better job at finding the best execution plan than the average developer.
It also means it's easier to train people for data processing nowadays.
Learning DBT was pretty easy for me as a data analyst. Now I’m contributing to my company’s data warehouse instead of just pulling existing data.