dj-acme

A self-improving AI DJ that plays the music, paints the visuals and puppets a VTuber character, all on one beat clock in one browser tab. It live-codes Strudel, writes its own WebGL2 shaders, and lets you take the character's face or arms from it mid-phrase. The room votes from phones and Twitch chat, telemetry records which bar they reacted to, and every few tracks it breeds a new generation of patterns. It succeeds strudel-ai-dj, which became its audio brain when it was merged with the vdj visual engine.

Ten roles, three latency budgets

The DJ is not one model with one prompt. Ten roles each have their own prompt, output schema, cadence and time budget, and each is bound to whichever model can meet it: a reflex tier under 1.5 s for the character's motion, its one-liners and the next-track pick; a smart tier under 20 s for the director, the critic and character design; a creative tier under 300 s, Claude by default, for new patterns and shaders. Any OpenAI-compatible endpoint can serve a tier.

The budget is a hard timeout, because a late answer lands on the wrong bar. A turn over budget is dropped and a procedural layer covers the phrase: breathing, a blink timer, a beat bounce, lip-sync from the pattern's own events. A reply with one value out of range is dropped whole, never half-applied. Each role shows its live p50 and drop rate with the reason (timeout, a reasoning model that spent its budget thinking, unparseable, wrong shape), so a laggy character reads as "bind the puppeteer to something smaller". The MC is told that saying nothing is a valid and common answer: a small model that feels obliged to fill every phrase turns into a chatterbox in about ninety seconds.

A puppet you can grab

The character has eighteen channels across head, body, face, arms and glow, and every frame each one is lerp(ai, human, authority) plus idle motion scaled by the share the AI still holds. Touching a slider is the takeover: that channel snaps to you, and when a release timer runs out (four seconds by default) it eases back over one second and the AI picks the limb up mid-motion. There is no manual mode. Expressions and mouth shapes switch rather than blend, because a half-interpolated mouth shape is a smear, not a phoneme.

New characters come from a prompt. A designer role writes a name, a persona and one image prompt per rig part, and ComfyUI renders eleven parts on flat magenta that the server keys out in plain JavaScript. With no ComfyUI the character is still made, drawn procedurally in canvas 2D from its own colors, and every channel still moves it.

Which bar the room liked

A vote stamped with where in the track it landed says the break was good, not just the track. Reactions fold into 64 bins per track beside the density of notes played, and the composer gets one line per parent pattern, such as peak reaction around cycle 18.5 of 32 (bd, supersaw playing); dead zone cycles 0-4. With too few reactions it says nothing rather than something confident built from noise.

Fitness adds emoji weights, valence, half-weighted energy and a 0–10 critic review of the code, discounted when a pattern has only a few votes. Each generation the DJ writes itself a one- or two-sentence lesson about the room, and the last ten go into every later prompt. Twitch chat is read anonymously, with no OAuth and no key, at one vote per user per 10 s and a weight that keeps ten thousand people in chat from drowning out twenty in the room. Every generation is written to music/ as .strudel files with a one-click strudel.cc link.

Stated honestly

• A phone tap lags the moment by 0.4–1.2 s. A fixed 700 ms correction moves the mean back and leaves the spread, and the heatmap says so under it: read a region, not a bin.
• No model in the loop hears the music; the critic reviews code. A local classifier served by cev was tried as ears and scored digital silence and a real mix at the same energy. It is used only where it measured well: whether the projected frame is dark, flat or busy, and whether a person is in view of the opt-in room camera, a verdict that is not fed into fitness.
• Measured on the development laptop, nothing installed meets the puppeteer's 1.2 s budget. A local 27B reasoning model ignored both switches for turning its thinking off; a JSON-schema grammar cut that thinking from 253 tokens and 18.4 s to under ten tokens and about 2 s, enough for the director and the critic but not for the reflex tier. The procedural layer carries the character.
• It runs locally from npm start and is not deployed. The first load needs the internet for Strudel and its sample banks. Generated shaders and characters stay on disk, stamped unreviewed; only CC0, public-domain or original work is committed.