[coolworker378]

The database is huge and in JSON format. So far I've only got 33,000 analytical data for the US and 11,000 for Australia. I don't think these are complete figures, I want parsing data for each country

[ygrek]

why not simply split it on country ? do a python script with matching on country and write in a different file for each country, ask any good and free AI bot if you are new, give him a sample with 3-4 line of your original file and run it on your hardware, it's not that hard

[shuiyazi]

why not simply split it on country ? do a python script with matching on country and write in a different file for each country, ask any good and free AI bot if you are new, give him a sample with 3-4 line of your original file and run it on your hardware, it's not that hard

even for a 400+gb file?

[r00f3rr]

why not simply split it on country ? do a python script with matching on country and write in a different file for each country, ask any good and free AI bot if you are new, give him a sample with 3-4 line of your original file and run it on your hardware, it's not that hard

even for a 400+gb file?

Even for terabytes-big files, all you'll need is time Just run 1gb sample first, and count time to estimate it on big file processing

[ygrek]

for files larger than the ram you have you have to work by "streaming" with a reader, that way you load in memory only the lines you are working with, also do some multihread work to speed it up, or with premises, you also have to keep in a variable the "last line" treated, for some big files it can take a week of 100% of all the cores. You can also simply fragment the files and then work on them, you can firstly do a loop to split the big file to 4gb file each and then work on each file, this way you can interupt and suffer less loss in case of power issue or needing you computing power for something else. those are all tips of how to handle that kind of situation and is relatif to global knowledge on how to handle very big files