Dispatch queued jobs, and derive their state from the printers
`POST /jobs` ended at a `queued` row and an upload URL; nothing moved it. This is the join, plus the two things that join turns out to need. ## The scheduler `src/dispatch` ticks every 5s: for each idle, claimed printer on a farm with a connected gateway, take the oldest unassigned queued job and send `start_print`. A timer rather than an event, because a job becomes dispatchable for two unrelated reasons - someone queued it, or a printer became free - and the second has no event this backend owns. It arrives as a telemetry snapshot, seconds after the fact. A scheduler that only ran on job creation would leave a queue stuck forever behind one busy printer. Claiming is an `UPDATE ... WHERE status = 'queued'` that returns the row, so two ticks cannot both win the same job. Selecting and then updating would let a tick and a nudge each believe they owned it and send two starts. It checks the bucket before dispatching. `POST /jobs` hands back a presigned PUT and returns, so the row exists before the file does; dispatching anyway would have the gateway download R2's 403 error document and give it to a printer as a project. A job whose file never arrives just stays queued, and a zero-byte upload fails outright because it will never become valid. The download URL is presigned at dispatch and not at creation - a queued job may wait days. ## Job state comes from telemetry, not from the gateway's ok This is the part that is not obvious and was learned the expensive way on 2026-10-15: a real print was accepted, uploaded byte-perfect, reached RUNNING, and then the printer paused itself on a physical broken-filament fault. A job marked `printing` on the gateway's reply would have claimed to be printing for the rest of the day. So `advanceJobsFrom` runs on every telemetry snapshot and the mapping lives in `job_status.ts`, apart from the database so every pair can be tested without a Postgres or a printer. The line that matters: an idle printer does *not* complete a job that is still `uploading`. A printer is idle for the entire transfer - minutes of it, at the 0.2 MiB/s measured over FTPS - so reading that as completion would mark every job successful seconds after dispatch, before the file reached the machine. 15 tests cover the table exhaustively. Because the gateway now acks a start immediately (it must - awaiting the transfer would stop its telemetry for minutes, and telemetry is what this depends on), a failure after the ack has no result message to travel on. Hence the reaper: a job `uploading` for 20 minutes is failed, not requeued, because the printer may actually be printing it and a requeue would put a second job on a working machine. ## Unclaimed printers are visible now `persistTelemetry` still refuses to create printer rows for gateway ids it does not recognise - a typo in a `printers.toml` becoming a permanent phantom printer is worse. But the refusal was silent, which made the worst first-run experience in the platform: gateway connects, reports happily, fleet stays empty, and the only evidence is in stderr. The observation is now kept and served by `GET /farms/:id/unclaimed-printers`, with `gatewayOnline` alongside it because an empty list otherwise cannot distinguish "all claimed" from "no gateway ever connected". Claiming is still `POST /printers`, still owner-only. 77 tests pass.
This commit is contained in:
@@ -0,0 +1,72 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { nextJobStatus } from "../src/dispatch/job_status";
|
||||
|
||||
/**
|
||||
* The job-status mapping, tested exhaustively.
|
||||
*
|
||||
* This is the logic that decides whether a print happened. It exists because
|
||||
* a gateway accepting `start_print` says nothing about the outcome: a real
|
||||
* end-to-end run on a P1P (2026-10-15) was accepted, uploaded byte-perfect,
|
||||
* reached RUNNING and then the printer paused itself on a physical
|
||||
* broken-filament fault. Job state is therefore derived from the telemetry
|
||||
* that follows, and getting this table wrong means the API reports prints
|
||||
* that never ran - the one failure nobody would catch by looking.
|
||||
*/
|
||||
describe("nextJobStatus", () => {
|
||||
const currents = ["uploading", "printing", "paused"] as const;
|
||||
const printerStates = ["idle", "printing", "paused", "error"] as const;
|
||||
|
||||
// Written out rather than computed: a table derived the same way the
|
||||
// implementation derives it agrees with a wrong implementation.
|
||||
const expected: Record<string, string | undefined> = {
|
||||
"uploading/idle": undefined,
|
||||
"uploading/printing": "printing",
|
||||
"uploading/paused": "paused",
|
||||
"uploading/error": "failed",
|
||||
"printing/idle": "completed",
|
||||
"printing/printing": "printing",
|
||||
"printing/paused": "paused",
|
||||
"printing/error": "failed",
|
||||
"paused/idle": "completed",
|
||||
"paused/printing": "printing",
|
||||
"paused/paused": "paused",
|
||||
"paused/error": "failed",
|
||||
};
|
||||
|
||||
for (const current of currents) {
|
||||
for (const printerState of printerStates) {
|
||||
const key = `${current}/${printerState}`;
|
||||
test(`${key} -> ${expected[key] ?? "unchanged"}`, () => {
|
||||
expect(nextJobStatus(current, printerState)).toBe(expected[key] as never);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
test("an idle printer does NOT complete a job that was still uploading", () => {
|
||||
// The case worth its own test. A printer is idle for the entire transfer
|
||||
// - minutes of it, at the 0.2 MiB/s measured over FTPS - so "idle" there
|
||||
// means "has not started", not "finished". Reading it as completion
|
||||
// would mark every job successful seconds after dispatch, before the
|
||||
// file had even reached the machine.
|
||||
expect(nextJobStatus("uploading", "idle")).toBeUndefined();
|
||||
});
|
||||
|
||||
test("a printer error fails the job from any state, including mid-upload", () => {
|
||||
// The P1P's self-pause arrived while the job was still `uploading` by
|
||||
// the cloud's reckoning, because the printer faulted during its
|
||||
// pre-print stages. If `uploading` were excluded here, that job would
|
||||
// have sat claiming to upload until the reaper got it twenty minutes
|
||||
// later.
|
||||
for (const current of currents) {
|
||||
expect(nextJobStatus(current, "error")).toBe("failed");
|
||||
}
|
||||
});
|
||||
|
||||
test("completion is never reported from a state the print had not begun in", () => {
|
||||
// Guards the invariant rather than the table: the only route to
|
||||
// "completed" is through a state that means the printer had actually
|
||||
// been printing.
|
||||
const completable = currents.filter((c) => nextJobStatus(c, "idle") === "completed");
|
||||
expect(completable).toEqual(["printing", "paused"]);
|
||||
});
|
||||
});
|
||||
@@ -140,6 +140,19 @@ describe("backend -> gateway frames", () => {
|
||||
printer_id: "p1p",
|
||||
command: { action: "control_print", control: "stop" },
|
||||
},
|
||||
{
|
||||
type: "command",
|
||||
command_id: "9a7d2f15-0000-4000-8000-000000000000",
|
||||
printer_id: "p1p",
|
||||
command: {
|
||||
action: "start_print",
|
||||
job_id: "7c9e6679-0000-4000-8000-000000000000",
|
||||
file_name: "cube.gcode.3mf",
|
||||
// Exactly what src/dispatch sends: a presigned GET, minted at
|
||||
// dispatch time rather than at job creation.
|
||||
url: "https://example.r2.cloudflarestorage.com/farms/f/jobs/j/cube.gcode.3mf?X-Amz-Signature=x",
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
for (const frame of outbound) {
|
||||
@@ -165,18 +178,61 @@ describe("backend -> gateway frames", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("has no start-print command to send", () => {
|
||||
// Still deliberate, but no longer for the original reason: starting an
|
||||
// uploaded file *has* now been confirmed on a real P1P (2026-10-11).
|
||||
// What is missing is the path around it - nothing uploads a sliced file
|
||||
// to a gateway and nothing decides which printer runs which job. Delete
|
||||
// this test when there is something to start.
|
||||
it("rejects a start_print missing any of its three required fields", () => {
|
||||
// The variant exists now - the job scheduler in src/dispatch sends it.
|
||||
// What this guards is that none of its fields is optional. Each one is
|
||||
// load-bearing in a different way: without `url` the gateway has nothing
|
||||
// to fetch, without `file_name` it has nowhere to put it, and without
|
||||
// `job_id` nothing can line a gateway's logs up against a job. An
|
||||
// optional field here would fail at the gateway, minutes later, on a
|
||||
// machine nobody is watching.
|
||||
const complete = {
|
||||
action: "start_print",
|
||||
job_id: "7c9e6679-0000-4000-8000-000000000000",
|
||||
file_name: "plate_1.3mf",
|
||||
url: "https://example.test/signed",
|
||||
};
|
||||
check(EdgeServerMessageSchema, {
|
||||
type: "command",
|
||||
command_id: "x",
|
||||
printer_id: "p1p",
|
||||
command: complete,
|
||||
});
|
||||
|
||||
for (const omit of ["job_id", "file_name", "url"] as const) {
|
||||
const partial: Record<string, unknown> = { ...complete };
|
||||
delete partial[omit];
|
||||
expect(
|
||||
Value.Check(EdgeServerMessageSchema, {
|
||||
type: "command",
|
||||
command_id: "x",
|
||||
printer_id: "p1p",
|
||||
command: partial,
|
||||
}),
|
||||
).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("will not carry a filament or AMS mapping on a start_print", () => {
|
||||
// Deliberately absent from the contract, and the reason is expensive:
|
||||
// hardcoding the filament choice instead of reading it from the project
|
||||
// archive made a real P1P heat to 250C and pause itself at 1%. The
|
||||
// gateway derives it from the `.3mf`'s own slice_info.config, which is
|
||||
// the only thing that knows. A field here would invite a caller to guess
|
||||
// again - and because this schema is a closed union, an extra property
|
||||
// makes the whole command fail to validate rather than being ignored.
|
||||
expect(
|
||||
Value.Check(EdgeServerMessageSchema, {
|
||||
type: "command",
|
||||
command_id: "x",
|
||||
printer_id: "p1p",
|
||||
command: { action: "start_print", file_name: "plate_1.3mf" },
|
||||
command: {
|
||||
action: "start_print",
|
||||
job_id: "7c9e6679-0000-4000-8000-000000000000",
|
||||
file_name: "plate_1.3mf",
|
||||
url: "https://example.test/signed",
|
||||
ams_mapping: [3, -1, -1, -1, -1],
|
||||
},
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user