crawler

A fast, polite, queue-driven web crawler. It runs on its own from one binary, or splits into a coordinator and stateless workers that talk over RabbitMQ. Politeness is decided per host, across every worker at once, so adding machines widens the crawl without making it any harder on a site. Extraction is CSS selectors in the style of BeautifulSoup and Scrapy, and one JSON object per page comes out on stdout.

Polite by construction

Per-host pacing is not a setting a worker can forget. A host gate answers "may this host be fetched right now?" and the coordinator is the only thing that asks it. It holds at most one request in flight per host unless a job raises that, and it measures the delay from when the last result came back, not from when the last task was sent, so a backed-up queue cannot release a burst at one site. A 429 or 5xx doubles the delay, up to six times and thirty minutes; a robots Crawl-delay is honoured. The gate runs in memory for one process or in Redis for many, and leases are timestamps, so a crashed worker cannot wedge a host. A Redis wipe holds every host for one delay rather than guessing.

robots.txt is fetched as a task like any other, so the coordinator never makes a request from its own address. The crawler has two honest User-Agents, one that obeys robots and one that does not, and ignoring robots requires a written authorization on the command line. A task carries which identity to use and never a string, so it cannot pose as a browser.

Crawler traps

Layered from cheapest to dearest, in the coordinator, where the whole picture is visible. URLs are canonicalized first: tracking and session parameters stripped, queries sorted, dot segments resolved. Shape limits come next: length, depth, parameter count, and any run of path segments repeated three times back to back, the classic relative-link loop. Each URL is then mapped to a template (/cal/{n}, ?page={n}) and each host's queue round-robins across templates, so one endless calendar cannot starve the rest of a site.

Once a template has been fetched enough, novelty decides. Every page is fingerprinted by a simhash of its main text with navigation and footers removed, and a template whose recent pages are nearly all near-duplicates is demoted, then stopped, with the evidence recorded. A paginated catalogue passes; an empty calendar that serves the same page for every month does not. Page, depth and redirect-chain caps sit behind all of it.

Workers that are not trusted

A task is a URL and a set of selectors, never code. The worker binary fetches, extracts and answers; it holds no queue state, no robots cache and no scope. On a machine it does not control it refuses anything but public addresses: no loopback, private, link-local or cloud-metadata ranges, no URL credentials, only ports 80, 443, 8080 and 8443. The check is in a custom DNS resolver, so the address that was vetted is the one connected to, and DNS rebinding has nothing to swap. Every redirect hop is checked again. --allow-private-addresses exists for crawling a server on your own machine and says so loudly.

Replies go to a private queue per coordinator, so two coordinators cannot see each other's results. A worker acknowledges a task only after it has answered, so one that dies mid-fetch costs a retry and not the crawl.

Stated honestly

• This is a design that is built up to its second phase. The host gate, robots policy, trap detection and RabbitMQ transport are done and tested; 71 tests run with no services, and the Redis and RabbitMQ suites join in when their URLs are set. The public gateway, worker registration and signing, result verification, Postgres persistence and a Homebrew client are specified in docs/distributed-design.md and not built.
• Workers are for machines you own. Handing public volunteers broker credentials would let any of them attack the broker, and RabbitMQ has had enough advisories this year that the design requires 4.3.7 or later; the gateway that fixes this is the unbuilt part.
• Messages are not signed yet, so a worker is believed on the strength of the network it sits on.
• Per-host pacing caps one site at about one page per delay however many workers there are. The swarm helps with breadth, bandwidth and geography, and does nothing for a deep crawl of a single site.
• The trap detector's near-duplicate test is a Hamming distance of 3 on a 64-bit simhash, so a trap that varies the main text of a short page can slip past it; the hard backstops still stop the crawl.
• Private repository, not yet committed anywhere.