octoturge
d827b465cf
`drizzle/` was gitignored, so every environment re-derived the schema from src/db/schema and nothing recorded what a given release expected. That is survivable while nothing is deployed and a problem the moment something is. The generated migration is now committed: four tables, three enums, five foreign keys, and the per-farm unique index on gateway_printer_id. It needed one hand-added line to be usable on its own. drizzle-kit does not emit extension statements, but frame_embeddings.embedding is vector(512) with an ivfflat index, so as generated the migration only applied to a database somebody had already run `CREATE EXTENSION vector` on - which was true locally precisely because dev-env.sh does that, and would not have been true on a fresh deploy. `CREATE EXTENSION IF NOT EXISTS vector` now leads the file, and it survives regeneration: drizzle only appends new migrations, and the diffing snapshot lives in meta/0000_snapshot.json rather than in the SQL. Verified by dropping the database, recreating it with no extensions at all (`pg_extension` count 0), and running db:migrate: the extension and all four tables were created, `db:generate` then reported no pending changes, and the e2e script still passed against both live printers on the result. `dev-env.sh schema` now runs `migrate` instead of `push`, so a local database and a deployed one get the schema by the same route - testing `push` locally while production runs `migrate` tests two different things. `push` is still available as its own command for iterating on a table that is still moving, with a comment about what it does not record.
88 lines
4.3 KiB
Markdown
88 lines
4.3 KiB
Markdown
# continuum-backend
|
|
|
|
Cloud control plane for Continuum. Bun + ElysiaJS, PostgreSQL (via pgCat/PgBouncer,
|
|
with pgvector for frame embeddings), Valkey Streams, and Cloudflare R2 storage.
|
|
|
|
## Run
|
|
|
|
```sh
|
|
bun install
|
|
scripts/dev-env.sh up # Valkey + Postgres/pgvector, and print the URLs
|
|
scripts/dev-env.sh schema # apply drizzle/ - same as bun run db:migrate
|
|
cp .env.example .env # fill in the URLs it printed, plus R2_*, SUPABASE_JWT_SECRET
|
|
bun run dev
|
|
```
|
|
|
|
`scripts/dev-env.sh` uses `docker-compose.yml` when a Docker daemon is
|
|
answering, and falls back to the same two services from apt when there is no
|
|
usable container runtime - some sandboxes cannot run Docker at all, because
|
|
dockerd needs `CAP_NET_ADMIN` for its bridge and user namespaces to unpack an
|
|
image, and neither can be granted from inside. `scripts/dev-env.sh status`
|
|
says which backend is in use.
|
|
|
|
Schema changes go through migrations: edit `src/db/schema/`, run
|
|
`bun run db:generate`, and commit the file it writes to `drizzle/` along with
|
|
the schema change. `scripts/dev-env.sh push` reshapes a local database straight
|
|
from the schema with no migration file, which is faster while a table is still
|
|
moving but records nothing and will drop a column to make things match - so
|
|
generate before committing. `drizzle/0000_*.sql` creates the `vector` extension
|
|
itself; that line is hand-added, since drizzle-kit does not emit extension
|
|
statements and the migration would otherwise only apply to a database somebody
|
|
had already prepared.
|
|
|
|
## Test
|
|
|
|
```sh
|
|
bun test # unit + protocol contract tests, no services needed
|
|
scripts/e2e-edge.sh # the whole edge path: real database, real printers
|
|
```
|
|
|
|
`scripts/e2e-edge.sh` seeds a farm, claims its printers, mints an edge token,
|
|
runs the real `continuum-proxy` against this backend, and asserts that live
|
|
telemetry reached Postgres and that a command reached a printer. It only ever
|
|
sends a fan speed, and only to a printer reporting `IDLE`.
|
|
`scripts/capture-gateway-frames.js` is the smaller tool: it stands in for this
|
|
backend and records what a gateway sends, for refreshing the fixtures in
|
|
`tests/fixtures/`.
|
|
|
|
## Layout
|
|
|
|
- `src/db/` - Drizzle schema (`printers`, `farms`, `print_jobs`, `frame_embeddings`)
|
|
and client, pointed at pgCat. `src/db/validators.ts` derives Elysia/TypeBox
|
|
request validators straight from the Drizzle tables via `drizzle-typebox`,
|
|
so the Postgres schema and the API validators come from one definition
|
|
instead of two hand-synced ones.
|
|
- `src/s3/` - R2 presigned URL helpers (`PUT` for client uploads, `GET` for the
|
|
edge proxy).
|
|
- `src/queue/` - Valkey client (`ioredis`) + `XADD` helper pushing onto
|
|
`stream:camera_frames` for `continuum-ai-worker`.
|
|
|
|
Valkey, not Redis: Redis 7.4+ is RSALv2/SSPL and 8.0+ adds AGPLv3, none of
|
|
which a commercial product can ship. Valkey is BSD-3-Clause and speaks the
|
|
same RESP protocol, so `ioredis` is unchanged - only the server differs. See
|
|
`continuum-common/docs/LICENSES.md`. `docker-compose.yml` here brings up
|
|
Valkey and Postgres+pgvector for local development.
|
|
- `src/routes/` - `/auth`, `/farms`, `/printers`, `/jobs`, and the edge-gateway
|
|
WebSocket at `/ws/edge/v1`. That route's wire format is not defined here: it
|
|
is `schemas/continuum/v1/edge/` in continuum-schemas, validated through its
|
|
TypeBox mirror in `@continuum/ts-types`, because both ends of it are separate
|
|
programs in different languages and they had already drifted apart once.
|
|
`printers.gateway_printer_id` is what maps a gateway's own printer ids
|
|
(`"p1p"`) onto rows here; telemetry for an unclaimed id is logged and skipped,
|
|
never inserted.
|
|
|
|
`frame_embeddings.embedding` requires the `vector` extension. The first
|
|
migration creates it, and `scripts/dev-env.sh up` also does so for the `push`
|
|
path, which does not run migrations.
|
|
|
|
## Dependency pins
|
|
|
|
`elysia`, `@sinclair/typebox` and the `drizzle-*` packages are pinned tightly
|
|
on purpose, not out of caution. Caret ranges here resolved into a combination
|
|
that could not boot: `elysia: ^1.1.26` installed 1.4.30, which calls
|
|
`t.Module` and so needs TypeBox >=0.34, while `drizzle-typebox: ^0.2.1`
|
|
installed a version whose peers demand `drizzle-orm >=0.36` against the
|
|
`^0.33.0` pinned here. Typechecking passed throughout - only starting the
|
|
server failed. `tests/dependency_identity.test.ts` guards the part of this that
|
|
can silently come back.
|