Is SQL a good language for data transformations? (discuss.tchncs.de)

submitted 1 year ago by Mattol@discuss.tchncs.de to c/programming@beehaw.org

1 comments fedilink hide all child comments

I always wish I wouldn't have to pull in python/pandas to do simple data transformations on smaller local datasets/csv files and stick with sqlite. But then I tend to quickly hit a problem that seems to require dynamic SQL. Anything in between SQL and a general programming language that is made for transforming tabular data?

top 1 comments

sorted by: hot top controversial new old

[-] selawdivad@lemm.ee 3 points 1 year ago

Can I suggest duckdb? You can start out writing SQL directly on top of CSV and Parquet files. But then if you want/need to do something more complicated, you can import duckdb into Python, keep running the SQL you already wrote, convert it to a Pandas or Polars dataframe, transform it, then query the result in SQL.

this post was submitted on 14 Jun 2023

2 points (100.0% liked)

Programming

13361 readers

4 users here now

All things programming and coding related. Subcommunity of Technology.

This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 1 year ago

MODERATORS

alyaza@beehaw.org

Hexorg@beehaw.org

remington@beehaw.org