15
How would you design parallel grep for huge JSONL files?
(lemmy.world)
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev
How large is very large? Would it be something that
jqcan’t do? Is it purely string search or JSON-tree search?Generally you would want to get file size, split it into ranges which can be read as valid UTF-8. Feed each range into reader thread. Can be inefficient for HDDs because each thread will try to access random location on disk forcing needle to jump back and forth. Also you’ll need reread ranges at split point with some positive and negative offset in case desired content got split. Things are getting much more complicated if you want JSON-tree grep. Branches may get split from parent nodes across multiple ranges.