Oh geez. Webcrawler, and robots.txt. There was no SEO, just "don't overwhelm my ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		insulanus on June 26, 2018 \| parent \| context \| favorite \| on: Old Internet Files Oh geez. Webcrawler, and robots.txt. There was no SEO, just "don't overwhelm my modem / ISDN connection / T1". It's been a while!

abritinthebay on June 27, 2018 [–]

A lot more interesting protocols too! I miss gopher & finger

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact