[vague]

I got bored and found out it was really easy to scrape Bangladesh sites for some data it's in no way useful or worth downloading but the following data includes: Fullname,Designation,Department,BCS Batch,Cadre ID,First Station & Date,Present Station & Date,Contact Number,E-Mail,Present Address,Father's Name,Mother's Name,Date of Birth,Gender,Religion,Blood Group,NID,Education,Permanent Address

List of sites:

laxmipurgovtcollege.edu.bd
www.akcc.gov.bd.csv
www.noakhalicoll.gov.bd

How I did it:
```#!/bin/sh

counter=1

table=("ID No." "NID." "Full Name" "Sex" "Religion" "Designation" "Date of Birth" "Blood Group" "Note" "Father Name" "Mother Name" "Education" "Address" "Contact Number" "E-Mail Address" "Jonining in This job" "Present Station")

while true; do
page="$(curl -s "https://laxmipurgovtcollege.edu.bd/teacher_staff_profile.php?id=$counter")"

# Check if the page is not empty
if [ -n "$page" ]; then
entry=""
for QUERY in "${table[@]}"; do
value="$(echo -n "$page" | grep -A 2 "$QUERY" | tr '\n' '|' | cut -d '|' -f3 | tr -d ' ' | tr -d '\n' | tr -d '\r')"

# Check if the value is not empty before appending
if [ -n "$value" ]; then
entry+="$(printf $value| tr -d '\n'),"
fi
done

# Remove <td></td> and similar patterns
entry="$(printf "$entry" | sed -e 's|<td[^>]*>||g' -e 's|</td>||g' -e "s|<aclass=\"dropdown-item\"href='students_list.php'>StudentsList</a>||g")"

echo "$entry"
# Remove trailing comma and echo the formatted entry
echo "${entry%,}" >> "$1"
else
echo "Failed to fetch data for ID $counter"
fi

((counter++))
done
```
Not many entries but who cares
here is the download link: https://gofile.io/d/3wTPgz

[ricky_bobby]

great post, thank you

[Addka72424]

The link in this thread is dead. Please reply to the PM you were sent to get your thread moved back to the Databases section.