Skimming through a large file
by lnonymous - Wednesday October 16, 2024 at 09:37 PM
#1
What's the best/fastest way to skim through or do a CTRL+F on a 60GB .txt or .json file?
Reply
#2
When having a file this big, your regular options are gonna struggle. You can't open a file this big in regular software like Excel or notepad because you (I doubt) don't have 60+GB of RAM.

You can create a simple python script that opens the file, reads it line by line, then prints any matches.

If you want to go the proper way, you need to import your file in an SQL database. MySQL workbench has everything you need to get going. Databases engines are incredibly fast for going through large amounts of data like you have.
Reply
#3
(Oct 16, 2024, 09:37 PM)lnonymous Wrote: What's the best/fastest way to skim through or do a CTRL+F on a 60GB .txt or .json file?

use grep and search keyword what you are looking for. thats way easy
Reply
#4
ripgrep if you can figure out how to get that running, you can find it on github
Reply
#5
grep that shitttttt
Reply
#6
bash: cat file.txt | grep "search term"
windows powershell: get-content file.txt | findstr -i "search term"

Although if it is very large and you have some programming skill, write a parser in C, or just search using regular expressions in chunks of the file.

This forum account is currently banned. Ban Length: Permanent (N/A Remaining)
Ban Reason: Self-Ban | Retired |http://c66go4clkqodr7tdjfu76jztjs7w7d3fajdeypxn73v4ju3dt7g5yyyd.onion/Forum-Ban-Appeals if you wish to be unbanned in the future.
Reply
#7
REgular expressions man
"Universal appeal is poison masquerading as medicine. Horror is not meant to be universal. It's meant to be personal, private, animal"
Reply
#8
(Oct 17, 2024, 07:20 AM)GenFe1 Wrote:
(Oct 16, 2024, 09:37 PM)lnonymous Wrote: What's the best/fastest way to skim through or do a CTRL+F on a 60GB .txt or .json file?

use grep and search keyword what you are looking for. thats way easy

the king's answer Cool
windows also has alternative - FINDSTR /?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  NEW USERS READ - how to avoid malware on the forum Sukob 104 12,857 6 hours ago
Last Post: Elowyn
  A collection of deepweb sites [2025] dg7ka 112 3,429 Yesterday, 07:57 PM
Last Post: IsItReal
  Hacking forums and their links 2026 onionlinks 1 274 Yesterday, 05:01 PM
Last Post: phas3lock
  What is your most efficent way to gain initial access? likju 1 202 Yesterday, 04:47 PM
Last Post: phas3lock
  FREE 3 UNCENSORED HACKING LLM QaboosbinSaidAlSaid 69 2,015 Yesterday, 04:37 PM
Last Post: phas3lock

Forum Jump:


 Users browsing this forum: 1 Guest(s)