octoturge
564d01816b
`POST /jobs` ended at a `queued` row and an upload URL; nothing moved it. This is the join, plus the two things that join turns out to need. ## The scheduler `src/dispatch` ticks every 5s: for each idle, claimed printer on a farm with a connected gateway, take the oldest unassigned queued job and send `start_print`. A timer rather than an event, because a job becomes dispatchable for two unrelated reasons - someone queued it, or a printer became free - and the second has no event this backend owns. It arrives as a telemetry snapshot, seconds after the fact. A scheduler that only ran on job creation would leave a queue stuck forever behind one busy printer. Claiming is an `UPDATE ... WHERE status = 'queued'` that returns the row, so two ticks cannot both win the same job. Selecting and then updating would let a tick and a nudge each believe they owned it and send two starts. It checks the bucket before dispatching. `POST /jobs` hands back a presigned PUT and returns, so the row exists before the file does; dispatching anyway would have the gateway download R2's 403 error document and give it to a printer as a project. A job whose file never arrives just stays queued, and a zero-byte upload fails outright because it will never become valid. The download URL is presigned at dispatch and not at creation - a queued job may wait days. ## Job state comes from telemetry, not from the gateway's ok This is the part that is not obvious and was learned the expensive way on 2026-10-15: a real print was accepted, uploaded byte-perfect, reached RUNNING, and then the printer paused itself on a physical broken-filament fault. A job marked `printing` on the gateway's reply would have claimed to be printing for the rest of the day. So `advanceJobsFrom` runs on every telemetry snapshot and the mapping lives in `job_status.ts`, apart from the database so every pair can be tested without a Postgres or a printer. The line that matters: an idle printer does *not* complete a job that is still `uploading`. A printer is idle for the entire transfer - minutes of it, at the 0.2 MiB/s measured over FTPS - so reading that as completion would mark every job successful seconds after dispatch, before the file reached the machine. 15 tests cover the table exhaustively. Because the gateway now acks a start immediately (it must - awaiting the transfer would stop its telemetry for minutes, and telemetry is what this depends on), a failure after the ack has no result message to travel on. Hence the reaper: a job `uploading` for 20 minutes is failed, not requeued, because the printer may actually be printing it and a requeue would put a second job on a working machine. ## Unclaimed printers are visible now `persistTelemetry` still refuses to create printer rows for gateway ids it does not recognise - a typo in a `printers.toml` becoming a permanent phantom printer is worse. But the refusal was silent, which made the worst first-run experience in the platform: gateway connects, reports happily, fleet stays empty, and the only evidence is in stderr. The observation is now kept and served by `GET /farms/:id/unclaimed-printers`, with `gatewayOnline` alongside it because an empty list otherwise cannot distinguish "all claimed" from "no gateway ever connected". Claiming is still `POST /printers`, still owner-only. 77 tests pass.
65 lines
2.9 KiB
TypeScript
65 lines
2.9 KiB
TypeScript
import { S3Client, PutObjectCommand, GetObjectCommand, HeadObjectCommand } from "@aws-sdk/client-s3";
|
|
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
|
|
|
|
const accountId = process.env.R2_ACCOUNT_ID ?? "";
|
|
const bucket = process.env.R2_BUCKET ?? "continuum";
|
|
|
|
export const s3 = new S3Client({
|
|
region: "auto",
|
|
endpoint: process.env.R2_ENDPOINT ?? `https://${accountId}.r2.cloudflarestorage.com`,
|
|
credentials: {
|
|
accessKeyId: process.env.R2_ACCESS_KEY_ID ?? "",
|
|
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY ?? "",
|
|
},
|
|
});
|
|
|
|
/** Presigned PUT for clients uploading a G-code/3MF file directly to R2. */
|
|
export async function getUploadUrl(key: string, contentType = "application/octet-stream", expiresInSeconds = 900) {
|
|
const command = new PutObjectCommand({ Bucket: bucket, Key: key, ContentType: contentType });
|
|
return getSignedUrl(s3, command, { expiresIn: expiresInSeconds });
|
|
}
|
|
|
|
/** Presigned GET for the edge proxy to download a job file over LAN via FTPS relay or direct fetch. */
|
|
export async function getDownloadUrl(key: string, expiresInSeconds = 900) {
|
|
const command = new GetObjectCommand({ Bucket: bucket, Key: key });
|
|
return getSignedUrl(s3, command, { expiresIn: expiresInSeconds });
|
|
}
|
|
|
|
/**
|
|
* Whether the object is actually in the bucket, and how big it is.
|
|
*
|
|
* The scheduler needs this because `POST /jobs` returns a presigned PUT and
|
|
* the client has not used it yet: the row exists before the file does. Rather
|
|
* than trust a client to tell us it finished - a client that crashes
|
|
* mid-upload would never say so, and one that lies would have a gateway
|
|
* download an error document - this asks the bucket.
|
|
*
|
|
* Returns `undefined` for a missing object rather than throwing, because for
|
|
* this caller "not uploaded yet" is the ordinary case on every tick between
|
|
* queueing a job and the upload finishing.
|
|
*/
|
|
export async function headObject(key: string): Promise<{ size: number } | undefined> {
|
|
try {
|
|
const head = await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));
|
|
return { size: head.ContentLength ?? 0 };
|
|
} catch (err) {
|
|
// 404/NotFound is the expected answer for a job whose file has not
|
|
// arrived. Anything else - credentials, networking, a bucket that does
|
|
// not exist - is a real problem and must not be reported as "no file",
|
|
// or every job on a misconfigured deployment would sit queued forever
|
|
// with nothing saying why.
|
|
const name = (err as { name?: string }).name;
|
|
const status = (err as { $metadata?: { httpStatusCode?: number } }).$metadata?.httpStatusCode;
|
|
if (name === "NotFound" || name === "NoSuchKey" || status === 404) return undefined;
|
|
throw err;
|
|
}
|
|
}
|
|
|
|
export function jobFileKey(farmId: string, jobId: string, fileName: string) {
|
|
return `farms/${farmId}/jobs/${jobId}/${fileName}`;
|
|
}
|
|
|
|
export function frameKey(farmId: string, printerId: string, frameId: string) {
|
|
return `farms/${farmId}/printers/${printerId}/frames/${frameId}.jpg`;
|
|
}
|