Dispatch queued jobs, and derive their state from the printers

`POST /jobs` ended at a `queued` row and an upload URL; nothing moved it.
This is the join, plus the two things that join turns out to need.

## The scheduler

`src/dispatch` ticks every 5s: for each idle, claimed printer on a farm with
a connected gateway, take the oldest unassigned queued job and send
`start_print`. A timer rather than an event, because a job becomes
dispatchable for two unrelated reasons - someone queued it, or a printer
became free - and the second has no event this backend owns. It arrives as a
telemetry snapshot, seconds after the fact. A scheduler that only ran on job
creation would leave a queue stuck forever behind one busy printer.

Claiming is an `UPDATE ... WHERE status = 'queued'` that returns the row, so
two ticks cannot both win the same job. Selecting and then updating would let
a tick and a nudge each believe they owned it and send two starts.

It checks the bucket before dispatching. `POST /jobs` hands back a presigned
PUT and returns, so the row exists before the file does; dispatching anyway
would have the gateway download R2's 403 error document and give it to a
printer as a project. A job whose file never arrives just stays queued, and a
zero-byte upload fails outright because it will never become valid. The
download URL is presigned at dispatch and not at creation - a queued job may
wait days.

## Job state comes from telemetry, not from the gateway's ok

This is the part that is not obvious and was learned the expensive way on
2026-10-15: a real print was accepted, uploaded byte-perfect, reached
RUNNING, and then the printer paused itself on a physical broken-filament
fault. A job marked `printing` on the gateway's reply would have claimed to
be printing for the rest of the day.

So `advanceJobsFrom` runs on every telemetry snapshot and the mapping lives
in `job_status.ts`, apart from the database so every pair can be tested
without a Postgres or a printer. The line that matters: an idle printer does
*not* complete a job that is still `uploading`. A printer is idle for the
entire transfer - minutes of it, at the 0.2 MiB/s measured over FTPS - so
reading that as completion would mark every job successful seconds after
dispatch, before the file reached the machine. 15 tests cover the table
exhaustively.

Because the gateway now acks a start immediately (it must - awaiting the
transfer would stop its telemetry for minutes, and telemetry is what this
depends on), a failure after the ack has no result message to travel on.
Hence the reaper: a job `uploading` for 20 minutes is failed, not requeued,
because the printer may actually be printing it and a requeue would put a
second job on a working machine.

## Unclaimed printers are visible now

`persistTelemetry` still refuses to create printer rows for gateway ids it
does not recognise - a typo in a `printers.toml` becoming a permanent phantom
printer is worse. But the refusal was silent, which made the worst first-run
experience in the platform: gateway connects, reports happily, fleet stays
empty, and the only evidence is in stderr. The observation is now kept and
served by `GET /farms/:id/unclaimed-printers`, with `gatewayOnline` alongside
it because an empty list otherwise cannot distinguish "all claimed" from "no
gateway ever connected". Claiming is still `POST /printers`, still owner-only.

77 tests pass.
This commit is contained in:
2026-10-15 22:54:44 +00:00
parent 1db96caa91
commit 564d01816b
9 changed files with 723 additions and 18 deletions
+42
View File
@@ -6,6 +6,7 @@ import { farms } from "../db/schema";
import { farmProjection } from "../db/projections";
import { requireFarm, requireOrg, requireUser, visibleFarms } from "../auth";
import { PaginationQuery, paginate } from "./pagination";
import { isGatewayOnline, listUnclaimedPrinters } from "./ws/edge";
const IdParams = t.Object({ id: t.String({ format: "uuid" }) });
@@ -168,4 +169,45 @@ export const farmRoutes = new Elysia({ prefix: "/farms" })
summary: "Rotate and return this farm's edge-proxy token (owner only)",
},
},
)
/**
* Printers this farm's gateway can see that no printer row claims.
*
* This exists because of the worst first-run experience in the platform.
* A gateway reports its fleet by the ids in its own `printers.toml`, and
* `persistTelemetry` refuses to create rows for ids it does not recognise -
* correctly, since a typo there would otherwise become a permanent phantom
* printer that nobody can explain. But the refusal used to be silent, so a
* new owner saw a connected gateway, an empty fleet, and no way to find out
* why. The answer was in the server's stderr.
*
* Now it is here. Claim one by creating it with `POST /printers`, passing
* the `gatewayPrinterId` from this list; the next telemetry batch fills in
* everything else and the entry disappears from here.
*
* `viewer`, not `owner`: this reports what hardware exists, which is the
* same class of fact as the printer list itself. Creating the row still
* needs `owner`.
*
* `gatewayOnline` is in the response because an empty list is ambiguous on
* its own - it means either "everything is claimed" or "no gateway has ever
* connected", and those need completely different things done about them.
*/
.get(
"/:id/unclaimed-printers",
async ({ user, params }) => {
await requireFarm(user.id, params.id, "viewer");
return {
farmId: params.id,
gatewayOnline: isGatewayOnline(params.id),
unclaimed: listUnclaimedPrinters(params.id),
};
},
{
params: IdParams,
detail: {
tags: ["farms"],
summary: "Printers the gateway reports that no printer row claims yet",
},
},
);