[ygrek]

Hi everyone, 
since their is more and more way to get url:login:password file i was wandering what was the best way to sort them, remove duplicate and then analyse them 
I mean it's ok to sort -u for file lower than 20gb but once their is more, let's say 500gb+ it's way harder and take so much time, i would have wanted to be able to gest a count by domain/mail/password simply for some stats. anyone have that kind of sorter? it's okay to also create a sort of flow to split by domain every digested files so each folder named after the domain contain a file or a bunch of files size limited inside without dup
thank you

[gagola]

if you want to sort and remove duplicates from +500gb you need to use linux sort

[ygrek]

the sort -u is not realy the best way, for now i am using some sql converter for the simples data and multi threaded grep for the files but that's not the best option

[gagola]

if you sort all the lines in one file its easy to remove duplicate from you just spilt it into 20 gb files and remove duplicates from each file
i have removed duplicate from 350gb file like this

---

also you need to use merge sort
sort each file and then use merge sort command to get one big sorted file