LIVE cartercole.net --:--:-- UTC

// 01 ACTIVE SYSTEMS

carterlitics

PYTHON LAMBDA DYNAMODB REPO

First-party analytics, self-hosted. One collector counts every request to this site and cartercole.com, from CloudFront's own access logs: people, crawlers, AI bots, and scanners probing for /wp-login.php. One script adds page views, visits, events and JavaScript errors without a cookie. Only a visitor who clicks Accept gets a visitor ID and a session recording. It's stdlib Python on Lambda with one DynamoDB table and two S3 buckets, and it costs cents.

> The touron YouTube1:23
The carterlitics console on its demo data, not real traffic: cartercole.net over 30 days, signed in as local admin. Tiles read 1,535 page views, 795 visitors, 795 visits at 1.9 pages each, a 45.4% bounce rate, 52 s engaged time, 3,225 requests at the edge, 50.9% bots (1,643 bot requests) and 47 errors in 3 distinct groups. Below, a bar chart of page views per day from c.js with one tall day, and a stacked chart of requests at the edge from CloudFront's logs, people in blue and bots in orange, then the top of the Pages, Sources and Countries tables 01 People and bots, counted at the edge The console's Bots tab on its demo data, not real traffic: 1,643 bot requests, 10 distinct bots, 0 running JavaScript, 109 probes, 667 verified crawlers and 184 fake crawlers. Lists of bots (curl, Googlebot (fake), ClaudeBot, GPTBot, AhrefsBot, Feedly, Googlebot, Bingbot, Facebook, probe), their kinds (ai, search, library, impostor, feed, seo, social, scanner) and the networks that own their addresses; below, verified crawlers (GPTBot, AhrefsBot, Googlebot, Bingbot) beside one impostor, Googlebot with 184 requests, and the host domains from reverse DNS 02 The real Googlebot, and the other one The console's Funnels tab for cartercole.com on its demo data, not real visitors: 'Home to a finished post' counts 157 of 198 consented sessions at /, 90 on /blog/* and 45 (28.7%) reaching event:post_read, with each step's share that went on, how many stopped, the median time to the next step and links to watch sessions that stopped; then 'A post, read to the end', 120 to 45 (37.5%), and 'Home to a link off the site', 157 to 3 (1.9%) 03 Where each funnel loses people The address panel over the Bots tab, on the console's demo data: 45.148.10.20, placed in NL on AS48090 Techoff SRV Limited, connection hosting or cloud, datacenter, reverse DNS none, on the lists 'hosting: datacenter' and, in red, 'abuse: blocklist.de'. Its one logged request is GET /rollopolis, tagged Googlebot (fake), with Googlebot's user agent 04 Says Googlebot; the address disagrees The console's Errors tab for cartercole.net on its demo data, not real errors: 47 errors in 3 distinct groups. The first group, TypeError: Cannot read properties of undefined (reading 'width'), seen 17 times, is open to its stack, at layout and at render in app-3f9a1c2e.js, and to its latest occurrences, each with its time, browser, place and user agent, and a Replay link on the two from visitors who consented 05 Forty-seven errors, three bugs

Two sources, one count

The server side sees everything, cache hits included, because CloudFront delivers its access logs to the collector: address, user agent, path, status, referrer, country and network. The client side, /_c/c.js, adds what only a browser knows: engaged time, single-page navigations, where a visit came from, custom events, and errors (uncaught exceptions, unhandled rejections, failed resources and CSP violations), grouped by bug. A page written for Google Analytics works unchanged, since gtag(), dataLayer.push(), ga() and _gaq.push() are all understood.

Every site routes /_c/* to the same Lambda through its own CloudFront distribution, so all of it is first-party. The cookies are the site's, CloudFront supplies the real viewer address, and blockers aimed at third-party analytics hosts have nothing to block. On the way in, CloudFront adds a shared secret and the site's ID as headers and overwrites any header of the same name a viewer sent, so neither can be forged.

Unique without a cookie

A unique visitor is sha256(daily salt, site, address, user agent). The salt is random and deleted after two days, so once it's gone a hash can't be traced back to anyone. Visits, bounces, landing pages and exit pages are counted on the collector from the same hash as they happen, so the numbers cover everyone who hasn't opted out, whether they consented or not. Do Not Track and Global Privacy Control count as opting out.

What an address is

Every address the collector keeps is classified without being sent to a lookup service: reverse DNS from AWS's own resolver, and one binary search over about thirty published lists. Those lists cover the ranges Google, Bing, OpenAI, Apple and others crawl from, the big clouds, iCloud Private Relay's egress, Tor exits, VPN providers, Starlink, Spamhaus DROP, and addresses seen attacking in the last few days. Each search takes about four microseconds. A request that says it's Googlebot is checked against Google's published ranges and forward-confirmed DNS, and one that fails is counted as "Googlebot (fake)".

With consent

A visitor who clicks Accept gets a visitor cookie, a session per tab, and an rrweb recording the console can replay. Inputs are masked, and anything marked [data-private] is never recorded. Funnels count steps in order over those sessions, with the drop-off at each step and replays of the sessions that stopped. What is collected, and for how long has a page of its own.

The console

One console at /_c/admin/ covers every site: overview, traffic, Google Search, bots, visitors, events, funnels, errors and replays. Search Console is read through a Google service account that the collector reaches with no stored key. Sign-in is OpenID Connect through Kith, and only admins get a session.

Stated honestly

Cookieless uniques are an estimate. A visit ends at midnight UTC, when the salt changes, and whenever the visitor's address changes, so one person whose phone leaves Wi-Fi counts twice. The GeoLite2 databases it reads are licensed and are not in the repository.

About these pictures

figures/carterlitics.sh in this site's repository draws all five from the console itself, served by carterlitics' own dev server running its demo: 30 days of generated traffic, with made-up visitors, crawlers, an impostor Googlebot, scanners and errors, fed through the same storage code the Lambda uses. None of the numbers are this site's real ones. The demo is laid out relative to the day it runs, so a rerun can move a digit. The run stays offline: the address panel is told not to ask the registry, which would send the address to ARIN and show a real holder.

The tour is filmed from the same console, a real pointer in headless Chrome, once as a desktop window and once at a phone's width for a vertical cut. Its log lines and its visitor are made up for it, from the addresses set aside for documentation, and run through carterlitics' own log reader, classifier and visitor hash; the two requests from c.js are what it sent from a page that loaded it through the dev server. The voice is Orpheus, run locally.