[countdown1945]

What is everyone's preferred method of exploring large datasets and dumps? So far I've had luck using linux terminal (grepping, awk'ing, etc.,).

Loading really huge dumps will inevitably make my shit freeze.

What's your preferred method of exploring datasets?

[zastound]

For medium sized data a simple Python Pandas solution (or even excel) is enough but for large data you can’t process it in a single computer, a cluster is needed.

[just1m]

I like, first, analyze what type of data it contains and, once you figured out, put in a local database. For me is simple and easy dump all the data in a database/table (with the proper format) and do selects or whatever you need to read the data. Although depends on the size of the dump, usually I could use grep/awk/cli commands to extract the data, but if the dump is large, the best option is throwing to a database

[user512]

This thread is really interesting and useful to me because I'm just trying to figure things out. So far my solution was to buy more RAM and open a large (15gb?) .sql in notepad++. It worked but I'm sure there are better methods.

I've been looking at postgres but I don't understand the process of importing a .sql file into it and then running queries. I learn best by hands on but it's been rough having no real direction.

[countdown1945]

This thread is really interesting and useful to me because I'm just trying to figure things out. So far my solution was to buy more RAM and open a large (15gb?) .sql in notepad++. It worked but I'm sure there are better methods.

I've been looking at postgres but I don't understand the process of importing a .sql file into it and then running queries. I learn best by hands on but it's been rough having no real direction.

So as far as I'm aware, the process to importing an sql dump in postgresql is fairly straightforward:

1. Create a database 
2. Sign in to database
3. Use: \i /path/to/your/dump_file.sql;
4. Use queries to find relevant data

[user512]

This thread is really interesting and useful to me because I'm just trying to figure things out. So far my solution was to buy more RAM and open a large (15gb?) .sql in notepad++. It worked but I'm sure there are better methods.

I've been looking at postgres but I don't understand the process of importing a .sql file into it and then running queries. I learn best by hands on but it's been rough having no real direction.

So as far as I'm aware, the process to importing an sql dump in postgresql is fairly straightforward:

1. Create a database 
2. Sign in to database
3. Use: \i /path/to/your/dump_file.sql;
4. Use queries to find relevant data

Amazing, thank you. I'll start trying to play around with that. 

Generally, what's considered a large data set? I expected 15 gb to be pretty huge but was able to navigate reasonably well with notepad++

[19689p]

emeditor to the rescue. That is what is mostly use by people here

[breachxyz]

for windows i use Select-String at powershell and for linux grep and cut