cd34f7b5979bc932faaffd21c9b0e9063286e6b3
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 octoturge
|
847f7cb686
|
Raise alerts: failed or paused prints, and AI detections
Nothing subscribed to channel:print_alerts and nothing watched a printer fail, so an operator away from the screen found out when they came back to a ruined plate. Alerts are now rows in printer_alerts, listed at GET /alerts (open=true for what still needs a person) and cleared with POST /alerts/:id/acknowledge, which is operator-only and keeps the first acknowledger on the record. Two sources. Telemetry: a change into error is a failed print if the printer was printing or paused, otherwise a printer error; a print the printer paused itself (how an AMS fault looks) is announced too. Transitions, not levels - a printer sitting in error is one alert, not one per frame - and a connection's first sight of a printer raises nothing, so a gateway that reconnects does not re-announce known faults. AI: the worker's Valkey publications are stored when confidence is at least ALERT_MIN_CONFIDENCE (default 0.6); the worker deliberately does not threshold, so the policy lives here. A bad message costs one message, not the listener. e2e-authz.sh gains the scoping/ack cases; new e2e-alerts.sh drives a stand-in gateway over the real /ws/edge/v1 and the real Valkey channel (checked to fail when the derivation is broken). |
||
|
octoturge
|
24142d4603
|
Add the operator console's live telemetry socket
The only socket until now was the gateway-facing /ws/edge/v1, which
authenticates as a *farm* with a credential any of its gateways holds,
and which accepts telemetry. Pointing a browser at it would mean handing
the browser something that can drive printers. /ws/telemetry
authenticates as a *user* with a Supabase access token, and accepts
nothing at all: it is push-only, and a frame arriving from the browser is
ignored. Everything that changes a printer keeps going over HTTP, where
the role checks and the audit trail already are.
Auth is checked at the upgrade, as on the edge socket, so an
unauthenticated browser never gets a socket. The token rides in
`Sec-WebSocket-Protocol` - `new WebSocket(url, ["bearer", token])` -
because that field is the whole of what JavaScript controls about a
browser handshake, and the usual workaround of `?token=` would put a
bearer credential into every access and proxy log. "bearer" is first
because the runtime echoes the *first* offered value back in the
response header, so the token must not be in that position; asserted,
not assumed.
The payload is the printer resource itself, returned straight out of
`persistTelemetry`'s UPDATE through `printerProjection` - the same
projection the REST routes use. So a push is byte-for-byte what
`GET /printers/:id` would have answered, and a client applies it to the
row it already holds. Rebuilding it from the gateway's snapshot instead
would be a second translation of the same data, free to drift, and
missing the columns a gateway never reports at all like `name` and
`vendor`.
`maySee` is what keeps one organisation's telemetry out of another's
console, and it lives in `telemetry/broadcast.ts` with no imports but a
type so it can be tested without a socket or a database - the same reason
as `dispatch/job_status.ts`, `stream.ts` and `auth/edge_upgrade.ts`. It
refuses a printer with no farm rather than matching it against anything,
which no row can be today, because this is the one function where being
wrong leaks another org's machines.
**A latent bug this turned up:** the edge route's resolve was
`{ as: "scoped" }`, copied from `requireUser`. For `requireUser` that is
right - it is used inside each route file and must propagate `user` to
that file's routes. On a ws plugin it propagates the check to the parent
app, where it gates every route mounted *after* that plugin. Measured
with a two-plugin probe: the second plugin's own socket is refused
because the first one's check runs first and finds no edge token. Nothing
shipped broken only because /ws/telemetry is the first route mounted
after it - and it was refused outright until this was found. Both ws
resolves now use the default `local` scope.
Known limits, both written down rather than discovered later:
- **In-process fan-out.** A `Set` of callbacks in one process, so two
backend instances would each push only the telemetry their own
gateways are connected to and an operator would see half a fleet. The
fix is Valkey pub/sub, already a dependency; doing it now would mean
testing a message broker to deliver events between two objects in the
same process.
- **Visibility is resolved once, when the socket opens.** Re-deriving it
per message would put a database round trip on every push to every
viewer. The direction that matters: someone removed from an
organisation keeps receiving its telemetry until their socket drops.
Closing that needs a signal on membership change that this backend does
not have; a periodic re-resolve would narrow the window without closing
it. Telemetry is also the least sensitive thing here, and every HTTP
route re-checks on each request.
Verified against the real stack, not just mirrored in unit tests:
`scripts/e2e-edge.sh` now drives the actual route with a real
JWKS-verified token and asserts the handshake echoes "bearer", that a
`ready` frame arrives, and that a real push carries the printer resource
with live readings - "2 push(es); last was H2C state=FINISH nozzle=30
camera=h2c". A bogus token is refused before any socket exists.
`scripts/watch-telemetry.ts` is the client it uses, and is the quickest
way to see by hand whether a farm's telemetry reaches a console.
110 tests (was 95), `bunx tsc --noEmit` clean.
|
||
|
octoturge
|
57af710b1a
|
Verify the edge token at the upgrade, not in the first frame
`/ws/edge/v1` took its credential from a `token` field on the `hello` message. It now takes it from `Authorization: Bearer` on the upgrade request, verified before the socket exists, so an unauthenticated peer gets an HTTP 401 and no WebSocket: no entry in `connections`, no frame ever parsed, no state of any kind allocated for it. The old arrangement worked, and every message type refused to act without a verified farm. But "it cannot do anything" is a weaker guarantee than "it does not exist" - an anonymous peer still got a live socket and had its JSON run through `EdgeClientMessageSchema` and the message switch before proving anything, and it took a timeout to stop anyone holding sockets open for free. A `?token=` query parameter was never on the table: a bearer credential in a URL lands in access logs, proxy logs and error breadcrumbs, which is the thing the proxy's hand-written `Debug` for `PrinterCommand` already exists to prevent. The decision lives in `src/auth/edge_upgrade.ts`, a leaf module with no imports but the error types - the same reason as `dispatch/job_status.ts` and `stream.ts`. This is the authentication boundary for the entire gateway protocol, so it should be the last thing only reachable through a running database. It checks `scope` and not just the signature (the same secret could be made to sign something else one day), rejects a token whose farm claim is absent or not a string (that would otherwise produce a connection whose `farmId` is `undefined`, which reads as "every farm" to a lookup), and answers every failure with one message - which part of a forged token to fix next is not something to tell its holder. `conn.farmId` is therefore set from the verified token in `open`, and the `hello` handler no longer verifies anything. It still checks `message.farm_id` against the token's farm, so a gateway naming a farm its token was not minted for is refused rather than silently working against the wrong one. The handshake timer stays but changed meaning: it is no longer an auth timeout, just a bound on an authenticated socket that never identifies its gateway. Verified, not assumed. That a throwing `resolve` fails the upgrade is a framework behaviour the whole design rests on, so `tests/ edge_upgrade.test.ts` stands up a real Elysia app with a real JWT plugin and asserts a socket never opens without a valid edge token, and that a resolved value reaches `ws.data`. The real `continuum-proxy` was also run against this composition: valid token gives HTTP 101 with the farm taken from the token, forged token gives 401 at the handshake. The committed captures predate this and still contain `token: "dev-token"`. That validates - the schema is a liberal reader - and is harmless, because the field is now ignored and such a gateway is refused at the upgrade for having no header. A test says so. 95 tests (was 85), `bunx tsc --noEmit` clean. |
||
|
octoturge
|
1db96caa91
|
Stop falling back to a committed edge signing secret
`EDGE_JWT_SECRET` was read as `process.env.EDGE_JWT_SECRET ?? "dev-edge-secret-change-me"` in two places. Anything holding that string can mint an edge JWT for any `farm_id` it likes and open `/ws/edge/v1` as that farm's gateway: enough to feed a farm false telemetry and to receive the commands meant for its printers. It is in a git repository, so it is not a secret. The `??` made it worse rather than better. It only catches null and undefined, and the committed `.env.example` ships `EDGE_JWT_SECRET=` *empty* - so a deployment that copied the template and filled in everything else signed its edge JWTs with the empty string and never reached the fallback at all. That is exactly the case the fallback existed to cover. Both reads now come from `src/auth/edge_secret.ts`, which throws at module load if the variable is unset or empty. Same shape as `verifier.ts` and for the same reason: a backend that boots and then fails every gateway handshake looks like a network problem somewhere else. There is no development default on purpose - a secret that works out of the box is a secret that reaches production, and `openssl rand -hex 32` is a one-time cost. Tests: the empty-string case, since that is the one the template produced, and a grep for both shapes that reintroduce the hole - a literal passed to `jwt()`, and a defaulted read of the variable - plus an assertion that the variable is read in exactly one place. 60 pass. |
||
|
octoturge
|
d8f8653dee
|
Authenticate and authorize every route: org layer + Supabase JWKS
Until now every HTTP route was unauthenticated. A probe against the
running server returned a farm's `edgeProxyToken` and a printer's
`accessCode` to an anonymous caller, and that pair is enough to connect
as a farm's gateway and drive its printers. The previous commit stopped
the credentials appearing in responses; this one makes the routes ask
who is calling.
Authentication. Access tokens are verified against the Supabase
project's published JWKS via `createSupabaseJwtVerifier` from
`@continuum/ts-core`: RS256, public key only, no secret in this
process. It replaces the HS256 `@elysiajs/jwt` path, where
`SUPABASE_JWT_SECRET` was simultaneously the verification key and a
key that could mint tokens, present in the environment of every
instance. `src/auth/verifier.ts` builds it at module load so a
misconfigured deployment fails at boot rather than on first login.
The edge-gateway token stays symmetric and that is not an
inconsistency: this service is both its issuer and its only verifier,
so there is no second party to keep a secret from. It now carries an
`exp`, which it did not, despite the response saying
`expiresInSeconds: 3600`.
Tenancy. An org layer above farms - `organisations`,
`org_members(org_id, user_id, role)`, `farms.org_id` - so a farm can
have more than one operator without sharing an account. `farms.owner_id`
is gone. Authorization is one rule applied everywhere: resolve the
thing being acted on back to an organisation and check the caller's
role there, user -> org -> farm -> printer, with nothing in between
carrying permissions of its own. The ladder is
`viewer` < `operator` < `owner`; printer commands and job submission
need `operator`, because they move physical machines, and that is the
one boundary here that is not about data.
A non-member gets 404, not 403: a 403 would confirm that a given farm
or printer id exists and belongs to someone, which is an enumeration
oracle over the whole system. 403 is kept for a member who may see the
record but not do this to it.
List endpoints scope in the `where` clause, not after the query - a
filter applied to the result is one `return` away from being skipped.
They are also bounded now (`limit`/`offset`, default 50, max 200
refused rather than clamped).
`ownerId` and `submittedBy` no longer come from request bodies. Nor do
the printer telemetry columns: `POST /printers` is narrowed to the
columns a client owns, because a client that could POST
`status: "idle"` could make the scheduler send work to a printer that
is mid-print.
Also fills the CRUD gaps that made the API unusable - PATCH and DELETE
for farms, printers, jobs and orgs, and membership management - plus
one error shape for the whole API via `toErrorResponse`, so an
authorization failure is a throw that leaves the handler rather than a
status somebody has to remember to set.
Guards on the two changes that cannot be undone through the API:
removing or demoting an organisation's last owner, and deleting an
organisation while farms are still attached (it would cascade to every
printer, print job and stored frame).
Migration 0001 is hand-edited after generation. drizzle-kit emitted
`ADD COLUMN org_id uuid NOT NULL`, which cannot apply to a populated
table, and dropped `owner_id` without moving what it encodes anywhere.
It now adds the column nullable, creates one organisation per distinct
legacy owner with that user as its `owner` member, points each farm at
it, and only then tightens the column and drops `owner_id`. Verified
both ways: from empty via `drizzle-kit migrate`, and against a seeded
database where two farms shared an owner and a third did not.
Testing. `scripts/dev-auth.ts` is a local RS256 issuer publishing a
JWKS at the path the backend fetches, so the real verification path is
testable without a Supabase project. The alternative was a test-only
bypass in the backend - an env var making it trust an unsigned token -
which would be a permanent hole one misconfiguration from production,
and would mean the code under test is not the code that ships.
- `tests/authz.test.ts` (19 tests): the role ladder written out in full
rather than derived the same way the implementation derives it, and
the real JWKS path - a token signed by a key outside the JWKS, an
expired token and a token with `aud: "anon"` (Supabase's own anon key
is a valid project JWT) are all rejected, with the same message, so
probing cannot tell which part of a forged token to fix next.
- `scripts/e2e-authz.sh` (68 assertions, real Postgres, real routes):
four users across two organisations; anonymous 401s, cross-org 404s,
viewer-cannot-command, operator-cannot-rotate, a spoofed
`submittedBy` in the body ignored, no credential in any read, and the
rotated token still minting an edge JWT.
- `scripts/e2e-edge.sh` updated for the org layer and now authenticates
as a seeded `operator` - the weakest role that should work, so the
run would fail if the command boundary were not where it claims to
be. Re-verified against both live printers: telemetry from `h2c` and
`p1p` persisted, and an authenticated `set_fan_speed` went
HTTP -> WebSocket -> MQTT -> P1P and back `ok:true`, fan returned
to 0.
Two bugs found in the test scripts while writing this, both the same
shape as the earlier `pkill -f` incident: `kill "${PID:-0}"` with an
unset PID signals the whole process group, including the harness, and
`uuidgen` is not installed in this sandbox.
|