Files
continuum-backend/src/routes/farms.ts
T
octoturge 564d01816b Dispatch queued jobs, and derive their state from the printers
`POST /jobs` ended at a `queued` row and an upload URL; nothing moved it.
This is the join, plus the two things that join turns out to need.

## The scheduler

`src/dispatch` ticks every 5s: for each idle, claimed printer on a farm with
a connected gateway, take the oldest unassigned queued job and send
`start_print`. A timer rather than an event, because a job becomes
dispatchable for two unrelated reasons - someone queued it, or a printer
became free - and the second has no event this backend owns. It arrives as a
telemetry snapshot, seconds after the fact. A scheduler that only ran on job
creation would leave a queue stuck forever behind one busy printer.

Claiming is an `UPDATE ... WHERE status = 'queued'` that returns the row, so
two ticks cannot both win the same job. Selecting and then updating would let
a tick and a nudge each believe they owned it and send two starts.

It checks the bucket before dispatching. `POST /jobs` hands back a presigned
PUT and returns, so the row exists before the file does; dispatching anyway
would have the gateway download R2's 403 error document and give it to a
printer as a project. A job whose file never arrives just stays queued, and a
zero-byte upload fails outright because it will never become valid. The
download URL is presigned at dispatch and not at creation - a queued job may
wait days.

## Job state comes from telemetry, not from the gateway's ok

This is the part that is not obvious and was learned the expensive way on
2026-10-15: a real print was accepted, uploaded byte-perfect, reached
RUNNING, and then the printer paused itself on a physical broken-filament
fault. A job marked `printing` on the gateway's reply would have claimed to
be printing for the rest of the day.

So `advanceJobsFrom` runs on every telemetry snapshot and the mapping lives
in `job_status.ts`, apart from the database so every pair can be tested
without a Postgres or a printer. The line that matters: an idle printer does
*not* complete a job that is still `uploading`. A printer is idle for the
entire transfer - minutes of it, at the 0.2 MiB/s measured over FTPS - so
reading that as completion would mark every job successful seconds after
dispatch, before the file reached the machine. 15 tests cover the table
exhaustively.

Because the gateway now acks a start immediately (it must - awaiting the
transfer would stop its telemetry for minutes, and telemetry is what this
depends on), a failure after the ack has no result message to travel on.
Hence the reaper: a job `uploading` for 20 minutes is failed, not requeued,
because the printer may actually be printing it and a requeue would put a
second job on a working machine.

## Unclaimed printers are visible now

`persistTelemetry` still refuses to create printer rows for gateway ids it
does not recognise - a typo in a `printers.toml` becoming a permanent phantom
printer is worse. But the refusal was silent, which made the worst first-run
experience in the platform: gateway connects, reports happily, fleet stays
empty, and the only evidence is in stderr. The observation is now kept and
served by `GET /farms/:id/unclaimed-printers`, with `gatewayOnline` alongside
it because an empty list otherwise cannot distinguish "all claimed" from "no
gateway ever connected". Claiming is still `POST /printers`, still owner-only.

77 tests pass.
2026-10-15 22:54:44 +00:00

214 lines
7.9 KiB
TypeScript

import { Elysia, t } from "elysia";
import { and, desc, eq } from "drizzle-orm";
import { NotFoundError } from "@continuum/ts-core";
import { db } from "../db";
import { farms } from "../db/schema";
import { farmProjection } from "../db/projections";
import { requireFarm, requireOrg, requireUser, visibleFarms } from "../auth";
import { PaginationQuery, paginate } from "./pagination";
import { isGatewayOnline, listUnclaimedPrinters } from "./ws/edge";
const IdParams = t.Object({ id: t.String({ format: "uuid" }) });
export const farmRoutes = new Elysia({ prefix: "/farms" })
.use(requireUser)
/**
* Farms in the caller's organisations. The scoping is in the `where`, not
* applied to the result afterwards: a filter that runs after the query is
* one `return` away from being skipped, and it also makes the database read
* rows it is not allowed to show.
*
* A projection rather than `select *`, because the table holds
* `edgeProxyToken`, which trades for an edge JWT - see src/db/projections.ts.
*/
.get(
"/",
async ({ user, query }) => {
const scope = query.orgId
? and(visibleFarms(user.id), eq(farms.orgId, query.orgId))
: visibleFarms(user.id);
return db.query.farms.findMany({
columns: farmProjection.columns,
where: scope,
orderBy: [desc(farms.createdAt)],
...paginate(query),
});
},
{
query: t.Object({
orgId: t.Optional(t.String({ format: "uuid" })),
...PaginationQuery,
}),
detail: { tags: ["farms"], summary: "Farms in the caller's organisations" },
},
)
.get(
"/:id",
async ({ user, params }) => {
const { role } = await requireFarm(user.id, params.id, "viewer");
const farm = await db.query.farms.findFirst({
columns: farmProjection.columns,
where: eq(farms.id, params.id),
});
if (!farm) throw new NotFoundError(`Farm ${params.id} not found`);
return { ...farm, role };
},
{ params: IdParams, detail: { tags: ["farms"] } },
)
.post(
"/",
async ({ user, body, set }) => {
// `orgId` comes from the body and is then authorized, which is the
// right way round: the caller says where the farm should go, and the
// check is whether they own that organisation. What must never come
// from the body is *who the caller is* - that is `user.id`, from a
// verified token, and it is why there is no `ownerId` field here any
// more.
await requireOrg(user.id, body.orgId, "owner", "creating a farm");
// The generated token is deliberately not returned. A farm is created
// far more often than a gateway is provisioned, and a credential that
// comes back from a routine create ends up in logs, browser history and
// screenshots. Fetch it when a gateway actually needs it, below.
const [farm] = await db
.insert(farms)
.values({
name: body.name,
orgId: body.orgId,
edgeProxyToken: crypto.randomUUID(),
location: body.location,
})
.returning(farmProjection.returning);
set.status = 201;
return farm;
},
{
body: t.Object({
name: t.String({ minLength: 1, maxLength: 200 }),
orgId: t.String({ format: "uuid" }),
location: t.Optional(t.String({ maxLength: 500 })),
}),
detail: { tags: ["farms"] },
},
)
.patch(
"/:id",
async ({ user, params, body }) => {
await requireFarm(user.id, params.id, "owner", "editing a farm");
// `orgId` is not patchable here on purpose: moving a farm between
// organisations changes who can see its print history and would need
// the `owner` role in *both*, so it wants its own endpoint rather than
// a field on a general update.
const [farm] = await db
.update(farms)
.set({ ...body, updatedAt: new Date() })
.where(eq(farms.id, params.id))
.returning(farmProjection.returning);
if (!farm) throw new NotFoundError(`Farm ${params.id} not found`);
return farm;
},
{
params: IdParams,
body: t.Object({
name: t.Optional(t.String({ minLength: 1, maxLength: 200 })),
location: t.Optional(t.String({ maxLength: 500 })),
settings: t.Optional(t.Record(t.String(), t.Unknown())),
}),
detail: { tags: ["farms"] },
},
)
.delete(
"/:id",
async ({ user, params }) => {
await requireFarm(user.id, params.id, "owner", "deleting a farm");
// Cascades to the farm's printers and print jobs, per the schema's
// foreign keys. Worth knowing before calling it.
await db.delete(farms).where(eq(farms.id, params.id));
return { deleted: params.id };
},
{ params: IdParams, detail: { tags: ["farms"] } },
)
/**
* Issues a *new* `edgeProxyToken` for this farm and returns it once.
*
* The only route that ever returns that credential, and it rotates on every
* call rather than revealing the stored one: a reveal endpoint returns the
* same secret to everyone who asks and leaves no trace of having been
* called, while a rotate endpoint makes the old value useless, so a
* gateway's token leaking has a bounded blast radius and reprovisioning is
* the fix.
*
* The cost is that rotating locks out the gateway currently using the old
* token until it is reconfigured, which is the right trade for a credential
* that drives physical machines - but it does mean this is not a "show me
* the token" button.
*/
.post(
"/:id/edge-proxy-token",
async ({ user, params }) => {
await requireFarm(user.id, params.id, "owner", "rotating a farm's gateway credential");
const token = crypto.randomUUID();
const [farm] = await db
.update(farms)
.set({ edgeProxyToken: token, updatedAt: new Date() })
.where(eq(farms.id, params.id))
.returning({ id: farms.id });
if (!farm) throw new NotFoundError(`Farm ${params.id} not found`);
return {
farmId: farm.id,
edgeProxyToken: token,
note: "Any gateway still using the previous token must be reconfigured.",
};
},
{
params: IdParams,
detail: {
tags: ["farms"],
summary: "Rotate and return this farm's edge-proxy token (owner only)",
},
},
)
/**
* Printers this farm's gateway can see that no printer row claims.
*
* This exists because of the worst first-run experience in the platform.
* A gateway reports its fleet by the ids in its own `printers.toml`, and
* `persistTelemetry` refuses to create rows for ids it does not recognise -
* correctly, since a typo there would otherwise become a permanent phantom
* printer that nobody can explain. But the refusal used to be silent, so a
* new owner saw a connected gateway, an empty fleet, and no way to find out
* why. The answer was in the server's stderr.
*
* Now it is here. Claim one by creating it with `POST /printers`, passing
* the `gatewayPrinterId` from this list; the next telemetry batch fills in
* everything else and the entry disappears from here.
*
* `viewer`, not `owner`: this reports what hardware exists, which is the
* same class of fact as the printer list itself. Creating the row still
* needs `owner`.
*
* `gatewayOnline` is in the response because an empty list is ambiguous on
* its own - it means either "everything is claimed" or "no gateway has ever
* connected", and those need completely different things done about them.
*/
.get(
"/:id/unclaimed-printers",
async ({ user, params }) => {
await requireFarm(user.id, params.id, "viewer");
return {
farmId: params.id,
gatewayOnline: isGatewayOnline(params.id),
unclaimed: listUnclaimedPrinters(params.id),
};
},
{
params: IdParams,
detail: {
tags: ["farms"],
summary: "Printers the gateway reports that no printer row claims yet",
},
},
);