Dispatch queued jobs, and derive their state from the printers

`POST /jobs` ended at a `queued` row and an upload URL; nothing moved it.
This is the join, plus the two things that join turns out to need.

## The scheduler

`src/dispatch` ticks every 5s: for each idle, claimed printer on a farm with
a connected gateway, take the oldest unassigned queued job and send
`start_print`. A timer rather than an event, because a job becomes
dispatchable for two unrelated reasons - someone queued it, or a printer
became free - and the second has no event this backend owns. It arrives as a
telemetry snapshot, seconds after the fact. A scheduler that only ran on job
creation would leave a queue stuck forever behind one busy printer.

Claiming is an `UPDATE ... WHERE status = 'queued'` that returns the row, so
two ticks cannot both win the same job. Selecting and then updating would let
a tick and a nudge each believe they owned it and send two starts.

It checks the bucket before dispatching. `POST /jobs` hands back a presigned
PUT and returns, so the row exists before the file does; dispatching anyway
would have the gateway download R2's 403 error document and give it to a
printer as a project. A job whose file never arrives just stays queued, and a
zero-byte upload fails outright because it will never become valid. The
download URL is presigned at dispatch and not at creation - a queued job may
wait days.

## Job state comes from telemetry, not from the gateway's ok

This is the part that is not obvious and was learned the expensive way on
2026-10-15: a real print was accepted, uploaded byte-perfect, reached
RUNNING, and then the printer paused itself on a physical broken-filament
fault. A job marked `printing` on the gateway's reply would have claimed to
be printing for the rest of the day.

So `advanceJobsFrom` runs on every telemetry snapshot and the mapping lives
in `job_status.ts`, apart from the database so every pair can be tested
without a Postgres or a printer. The line that matters: an idle printer does
*not* complete a job that is still `uploading`. A printer is idle for the
entire transfer - minutes of it, at the 0.2 MiB/s measured over FTPS - so
reading that as completion would mark every job successful seconds after
dispatch, before the file reached the machine. 15 tests cover the table
exhaustively.

Because the gateway now acks a start immediately (it must - awaiting the
transfer would stop its telemetry for minutes, and telemetry is what this
depends on), a failure after the ack has no result message to travel on.
Hence the reaper: a job `uploading` for 20 minutes is failed, not requeued,
because the printer may actually be printing it and a requeue would put a
second job on a working machine.

## Unclaimed printers are visible now

`persistTelemetry` still refuses to create printer rows for gateway ids it
does not recognise - a typo in a `printers.toml` becoming a permanent phantom
printer is worse. But the refusal was silent, which made the worst first-run
experience in the platform: gateway connects, reports happily, fleet stays
empty, and the only evidence is in stderr. The observation is now kept and
served by `GET /farms/:id/unclaimed-printers`, with `gatewayOnline` alongside
it because an empty list otherwise cannot distinguish "all claimed" from "no
gateway ever connected". Claiming is still `POST /printers`, still owner-only.

77 tests pass.
This commit is contained in:
2026-10-15 22:54:44 +00:00
parent 1db96caa91
commit 564d01816b
9 changed files with 723 additions and 18 deletions
+76 -5
View File
@@ -9,6 +9,7 @@ import {
} from "@continuum/ts-types";
import { db } from "../../db";
import { printers } from "../../db/schema";
import { advanceJobsFrom } from "../../dispatch";
import { enqueueFrameIngestionBatch } from "../../queue";
import { edgeJwtSecret } from "../../auth/edge_secret";
@@ -59,6 +60,70 @@ const connections = new Map<string, EdgeConn>();
*/
const gatewayByFarm = new Map<string, string>();
/**
* farmId -> the printers its gateway is reporting that nothing in the cloud
* claims, keyed by the gateway's own id for them.
*
* `persistTelemetry` deliberately refuses to insert a printer row for an id
* it does not recognise - a typo in someone's `printers.toml` would otherwise
* become a permanent phantom printer, and a gateway should not be able to
* create records just by mentioning them. But the previous behaviour was to
* log a warning and drop the snapshot, which made the single worst first-run
* experience in the platform: the gateway connects, reports happily, and the
* fleet stays empty with the only evidence in the server's stderr.
*
* So the refusal stands and the *observation* is kept. A new farm's owner can
* list what their gateway can see and claim it with `POST /printers`, which
* is the one place printer rows are created.
*
* In memory on purpose: this is live gateway state, exactly like
* `connections` and `gatewayByFarm` beside it. It is rebuilt within one
* telemetry interval of a reconnect, and persisting it would mean inventing
* a second, staler source of truth about what a gateway can see.
*/
const unclaimedByFarm = new Map<string, Map<string, UnclaimedPrinter>>();
export interface UnclaimedPrinter {
/** The id in the gateway's `printers.toml` - what to put in `gatewayPrinterId`. */
gatewayPrinterId: string;
/** The gateway's own display name for it, to make a claim prompt legible. */
name: string;
/** Its vendor state, so an owner can tell a live printer from a stale entry. */
vendorState: string | null;
firstSeenAt: string;
lastSeenAt: string;
}
function noteUnclaimed(farmId: string, snapshot: PrinterTelemetry) {
let farm = unclaimedByFarm.get(farmId);
if (!farm) {
farm = new Map();
unclaimedByFarm.set(farmId, farm);
}
const now = new Date().toISOString();
const existing = farm.get(snapshot.printer_id);
farm.set(snapshot.printer_id, {
gatewayPrinterId: snapshot.printer_id,
name: snapshot.name,
vendorState: snapshot.state ?? null,
firstSeenAt: existing?.firstSeenAt ?? now,
lastSeenAt: now,
});
}
/**
* What this farm's gateway is reporting that no printer row claims.
*
* Empty is the healthy answer, and also the answer when no gateway has ever
* connected - a caller that wants to tell those apart should check
* `isGatewayOnline` too.
*/
export function listUnclaimedPrinters(farmId: string): UnclaimedPrinter[] {
return [...(unclaimedByFarm.get(farmId)?.values() ?? [])].sort((a, b) =>
a.gatewayPrinterId.localeCompare(b.gatewayPrinterId),
);
}
/** How long to wait for a gateway to answer a command before giving up. */
const COMMAND_TIMEOUT_MS = 10_000;
@@ -118,16 +183,16 @@ async function persistTelemetry(conn: EdgeConn, batch: PrinterTelemetry[]) {
});
if (!row) {
console.warn(
`[edge] gateway ${conn.gatewayId} reported unknown printer ` +
`'${snapshot.printer_id}'; no printer in farm ${conn.farmId} claims that ` +
`gateway_printer_id, skipping`,
);
// Still not inserted - see `unclaimedByFarm` for why - but no longer
// only a line in stderr. An owner can now see it and claim it.
noteUnclaimed(conn.farmId, snapshot);
continue;
}
printerId = row.id;
conn.printerIds.set(snapshot.printer_id, printerId);
// It is claimed now, so it is no longer waiting to be.
unclaimedByFarm.get(conn.farmId)?.delete(snapshot.printer_id);
}
const status = mapStatus(snapshot.state, snapshot.print_error);
@@ -156,6 +221,12 @@ async function persistTelemetry(conn: EdgeConn, batch: PrinterTelemetry[]) {
updatedAt: new Date(),
})
.where(eq(printers.id, printerId));
// The printer's row is the latest snapshot; the job's row is the history
// that outlives it. This is the only thing that moves a job out of
// `uploading` - see `advanceJobsFrom` for why the gateway's `ok` on a
// start command is explicitly not trusted to do it.
await advanceJobsFrom(printerId, status, snapshot.percent);
}
}