grep-the-internet

Distributed grep over Common Crawl. Give it a regex and a crawl, or a date range, and it slices the job into one task per crawl file, fans them out over a queue to whatever machines are running a worker, and streams the hits back into a web UI or a JSON API as they land. It prices the job before it runs and can kill it while it does.

One task, one file

A Common Crawl snapshot is about 90,000 files. A job becomes one task per file on a shared RabbitMQ queue, and each worker pulls as many tasks at a time as it has cores to give. Nothing assigns work to machines, so a fast box simply acknowledges more often, and a new one can join mid-job by starting the binary.

A task never touches disk. The file is downloaded in chunks, stream-decoded through gzip, walked record by record as WARC, and scanned with regex::bytes in constant memory. A 1 GB file costs the same memory as a small one. Which records get scanned depends on the tier: extracted text (WET), raw HTTP responses (WARC, roughly ten times the bytes) or metadata JSON (WAT).

A job names crawl ids such as CC-MAIN-2026-43, or a time range that is mapped to crawls through Common Crawl's own catalog. max_files caps the job by sampling evenly across the manifest, so a hundred-file test touches the whole crawl and not just its first segment.

Priced before it runs

POST /v1/estimate costs nothing and creates nothing. The pattern is parsed and checked for a literal prefix the engine can prefilter on, then compiled and timed against a synthetic megabyte of web-shaped text. That gives a measured MB/s for this exact pattern, which sorts it into trivial, moderate, heavy or pathological, and the figure is capped at the rate gzip decoding allows. Multiplied out by the bytes the job will download and decompress, that is core-hours and dollars.

Warnings come with it: a pattern that matches the empty string, one with no literal start that forces every byte to be inspected, one that fires so often on ordinary text that the results would be noise. Rust's regex engine cannot backtrack catastrophically, so the worst case is slow and not stuck, but a slow pattern is the expensive kind and worth hearing about first.

Live, and stoppable

Workers publish every progress tick, batch of hits and task transition to a per-job Redis channel, and the API turns that into server-sent events. The web UI tails it with EventSource, drawing per-file progress bars and a fleet panel from worker heartbeats. A customer gets the same stream from curl -N.

Terminating a job flips one state key. Workers check it in the middle of a file, about every 32 MB scanned, not between files, so a kill does not have to wait for a gigabyte of WARC to finish. Stored results are capped per job and the counters keep counting past the cap, so a pattern that matches half the web fills a list and not a disk.

Stated honestly

• It is built to become a paid service and is not one yet. There are no API keys, and nothing meters real usage back to an account; the estimate is computed and stored per job and that is as far as billing goes.
• Cost uses average file sizes and decompression ratios per tier, not the sizes of the files in a given job. It is an estimate and says so.
• Results live in Redis, capped. There is no export to S3 or a downloadable NDJSON yet.
• Every task downloads a whole file. Querying Common Crawl's columnar URL index first, so a URL-constrained job could fetch only the matching records by byte range, is planned and not built.
• No autoscaling. The fleet is the machines somebody started a worker on.