octoturge
b42cbac299
The other end of the `stream:camera_frames` drift. This side wrote `farmId`, `printerId`, `printJobId`, `r2ObjectKey` and `capturedAt`; `continuum-ai-worker` read `printer_id` and nothing else. Every task was delivered and acknowledged, and every frame was attributed to a printer called "unknown", with no error on either side - a stream entry is a flat string map, so there is nothing to parse and nothing to fail. The field names are now snake_case, from `schemas/continuum/v1/ai/frame_task.schema.json`. That is the convention this repo already follows for anything shared with a Rust consumer (see `/ws/edge/v1`), and it is not an inconsistency with the camelCase HTTP routes: this is not an HTTP route. `print_job_id` is omitted from the XADD when there is no job rather than written as `""`. An empty string is a value: a consumer that believes it looks up job id `""`, or puts it in a uuid column. A stream entry has no null, so "no value" has to mean "no field". The producer checks each task against the shared schema before writing it. The check is not for the database's sake - a stream accepts any field name - it is to make this exact class of drift loud. It earned its keep immediately by failing on every task with `Unknown format 'uuid'`: TypeBox ships no format validators and treats an *unregistered* format as a failure rather than skipping it, so a schema using `format: "uuid"` checked outside Elysia rejects everything. `src/validation/formats.ts` registers the two this codebase uses, guarded by `Has` so Elysia's own registrations still win for the route validators. (The HTTP routes were never affected - importing Elysia registers ~24 formats including `uuid`, which I confirmed by asserting a non-uuid body gets a 422.) A `frame_batch` now goes onto the stream in one pipelined round trip instead of one per frame, which was a round trip each while holding up the WebSocket message handler - and a farm produces roughly a frame per printer per second. ioredis reports per-command pipeline errors in the result tuple rather than throwing, so those are checked explicitly; a pipeline where one XADD failed otherwise looks like success. `frameTaskToStreamFields` and the compiled check live in their own module because `src/queue/index.ts` opens a Valkey connection on import. Keeping the wire format pure means `tests/frame_task_contract.test.ts` asserts it with nothing running. Tests, mirroring the worker's own so neither side can agree only with itself: the exact key set, values carried through unchanged, `print_job_id` present when there is a job and the key absent when there is not, no camelCase key ever emitted, and the uuid and date-time formats actually enforced - that last one fails if the format registration is lost (everything rejected) or becomes a no-op (nothing rejected). Plus `PrintAlertSchema`, which this repo will have to parse once the fan-out route exists. `scripts/e2e-ai-frames.sh` runs this producer against the real worker binary over real Valkey and asserts the field names by reading them back out of Valkey - rather than from either side's source, which is what made the original bug invisible. Two faults in the first draft of that script, both worth naming because a test that passes for the wrong reason is worse than no test: it stranded *every* entry on a dead consumer, so the ordinary delivery path was never exercised at all; and it grepped the subscriber output for "print_alert", which matched `redis-cli`'s own subscription confirmation line and reported a published alert that did not exist.
continuum-backend
Cloud control plane for Continuum. Bun + ElysiaJS, PostgreSQL (via pgCat/PgBouncer, with pgvector for frame embeddings), Valkey Streams, and Cloudflare R2 storage.
Run
bun install
scripts/dev-env.sh up # Valkey + Postgres/pgvector, and print the URLs
scripts/dev-env.sh schema # apply drizzle/ - same as bun run db:migrate
cp .env.example .env # fill in the URLs it printed, plus R2_*, SUPABASE_JWT_SECRET
bun run dev
scripts/dev-env.sh uses docker-compose.yml when a Docker daemon is
answering, and falls back to the same two services from apt when there is no
usable container runtime - some sandboxes cannot run Docker at all, because
dockerd needs CAP_NET_ADMIN for its bridge and user namespaces to unpack an
image, and neither can be granted from inside. scripts/dev-env.sh status
says which backend is in use.
Schema changes go through migrations: edit src/db/schema/, run
bun run db:generate, and commit the file it writes to drizzle/ along with
the schema change. scripts/dev-env.sh push reshapes a local database straight
from the schema with no migration file, which is faster while a table is still
moving but records nothing and will drop a column to make things match - so
generate before committing. drizzle/0000_*.sql creates the vector extension
itself; that line is hand-added, since drizzle-kit does not emit extension
statements and the migration would otherwise only apply to a database somebody
had already prepared.
Test
bun test # unit + protocol contract tests, no services needed
scripts/e2e-edge.sh # the whole edge path: real database, real printers
scripts/e2e-edge.sh seeds a farm, claims its printers, mints an edge token,
runs the real continuum-proxy against this backend, and asserts that live
telemetry reached Postgres and that a command reached a printer. It only ever
sends a fan speed, and only to a printer reporting IDLE.
scripts/capture-gateway-frames.js is the smaller tool: it stands in for this
backend and records what a gateway sends, for refreshing the fixtures in
tests/fixtures/.
Layout
src/db/- Drizzle schema (printers,farms,print_jobs,frame_embeddings) and client, pointed at pgCat.src/db/validators.tsderives Elysia/TypeBox request validators straight from the Drizzle tables viadrizzle-typebox, so the Postgres schema and the API validators come from one definition instead of two hand-synced ones.src/s3/- R2 presigned URL helpers (PUTfor client uploads,GETfor the edge proxy).src/queue/- Valkey client (ioredis) +XADDhelper pushing ontostream:camera_framesforcontinuum-ai-worker.
Valkey, not Redis: Redis 7.4+ is RSALv2/SSPL and 8.0+ adds AGPLv3, none of
which a commercial product can ship. Valkey is BSD-3-Clause and speaks the
same RESP protocol, so ioredis is unchanged - only the server differs. See
continuum-common/docs/LICENSES.md. docker-compose.yml here brings up
Valkey and Postgres+pgvector for local development.
src/routes/-/auth,/farms,/printers,/jobs, and the edge-gateway WebSocket at/ws/edge/v1. That route's wire format is not defined here: it isschemas/continuum/v1/edge/in continuum-schemas, validated through its TypeBox mirror in@continuum/ts-types, because both ends of it are separate programs in different languages and they had already drifted apart once.printers.gateway_printer_idis what maps a gateway's own printer ids ("p1p") onto rows here; telemetry for an unclaimed id is logged and skipped, never inserted.
frame_embeddings.embedding requires the vector extension. The first
migration creates it, and scripts/dev-env.sh up also does so for the push
path, which does not run migrations.
Dependency pins
elysia, @sinclair/typebox and the drizzle-* packages are pinned tightly
on purpose, not out of caution. Caret ranges here resolved into a combination
that could not boot: elysia: ^1.1.26 installed 1.4.30, which calls
t.Module and so needs TypeBox >=0.34, while drizzle-typebox: ^0.2.1
installed a version whose peers demand drizzle-orm >=0.36 against the
^0.33.0 pinned here. Typechecking passed throughout - only starting the
server failed. tests/dependency_identity.test.ts guards the part of this that
can silently come back.