CRAWLER

If a request from crawler/… or crawler-unrestricted/… with a link to this page turned up in your logs, it came from my web crawler. It is an experimental, personal crawler: I'm building an index of the web for my own use, the way a search engine would, one site at a time. It is run by me, Carter Cole, not by a company.

Something wrong? Email me

abuse@cartercole.com reaches me directly. Tell me the site, roughly when, and the User-Agent you saw, and I'll stop crawling it.

How it identifies itself

• crawler/<version> (+https://cartercole.net/bot): the ordinary crawler. It reads and obeys robots.txt, including Crawl-delay.
• crawler-unrestricted/<version> (+https://cartercole.net/bot): the same crawler on runs where I have chosen not to read robots.txt, such as my own sites or a small, specific set of pages. Every such run has to be given a written reason before it starts, and it is paced exactly like the ordinary one. If you see it and would rather not, email me and it stops.

It never pretends to be a browser or another company's bot.

How it behaves

• One request at a time per site, with a pause between them (a quarter of a second by default, and longer if your robots.txt asks), measured from when your server answered, not from when it asked. Sites that share a server address share that budget too.
• It backs off. A 429 or 5xx doubles the pause, up to half an hour, and a Retry-After becomes the pause from then on.
• It takes no for an answer. If a site refuses it several times in a row (a 403, a rate limit, a bot challenge or a captcha), it stops asking that site, leaves it alone for hours, and comes back more slowly, slower again each time. Every refusal is logged so I can see where it isn't welcome.
• It reads pages, not everything. It fetches HTML and only rarely images, video or fonts. It doesn't fill in or submit forms, log in, or make POST requests. Now and then it loads a page in a headless browser when the page is empty without JavaScript.
• It avoids traps. Endless calendars, session IDs in URLs and pages that repeat themselves are detected and dropped.

How to keep it out

Add this to your robots.txt:

User-agent: crawler
Disallow: /

Or slow it down with Crawl-delay: 10 in the same group. For anything else, including the unrestricted runs, email me.

What it keeps

The pages it fetches and their headers, so the index can be searched and compared over time. It only fetches what anyone could see without signing in: it sends no cookies and never logs in.