Fix the frame producer's field names, and pipeline the batch
The other end of the `stream:camera_frames` drift. This side wrote `farmId`, `printerId`, `printJobId`, `r2ObjectKey` and `capturedAt`; `continuum-ai-worker` read `printer_id` and nothing else. Every task was delivered and acknowledged, and every frame was attributed to a printer called "unknown", with no error on either side - a stream entry is a flat string map, so there is nothing to parse and nothing to fail. The field names are now snake_case, from `schemas/continuum/v1/ai/frame_task.schema.json`. That is the convention this repo already follows for anything shared with a Rust consumer (see `/ws/edge/v1`), and it is not an inconsistency with the camelCase HTTP routes: this is not an HTTP route. `print_job_id` is omitted from the XADD when there is no job rather than written as `""`. An empty string is a value: a consumer that believes it looks up job id `""`, or puts it in a uuid column. A stream entry has no null, so "no value" has to mean "no field". The producer checks each task against the shared schema before writing it. The check is not for the database's sake - a stream accepts any field name - it is to make this exact class of drift loud. It earned its keep immediately by failing on every task with `Unknown format 'uuid'`: TypeBox ships no format validators and treats an *unregistered* format as a failure rather than skipping it, so a schema using `format: "uuid"` checked outside Elysia rejects everything. `src/validation/formats.ts` registers the two this codebase uses, guarded by `Has` so Elysia's own registrations still win for the route validators. (The HTTP routes were never affected - importing Elysia registers ~24 formats including `uuid`, which I confirmed by asserting a non-uuid body gets a 422.) A `frame_batch` now goes onto the stream in one pipelined round trip instead of one per frame, which was a round trip each while holding up the WebSocket message handler - and a farm produces roughly a frame per printer per second. ioredis reports per-command pipeline errors in the result tuple rather than throwing, so those are checked explicitly; a pipeline where one XADD failed otherwise looks like success. `frameTaskToStreamFields` and the compiled check live in their own module because `src/queue/index.ts` opens a Valkey connection on import. Keeping the wire format pure means `tests/frame_task_contract.test.ts` asserts it with nothing running. Tests, mirroring the worker's own so neither side can agree only with itself: the exact key set, values carried through unchanged, `print_job_id` present when there is a job and the key absent when there is not, no camelCase key ever emitted, and the uuid and date-time formats actually enforced - that last one fails if the format registration is lost (everything rejected) or becomes a no-op (nothing rejected). Plus `PrintAlertSchema`, which this repo will have to parse once the fan-out route exists. `scripts/e2e-ai-frames.sh` runs this producer against the real worker binary over real Valkey and asserts the field names by reading them back out of Valkey - rather than from either side's source, which is what made the original bug invisible. Two faults in the first draft of that script, both worth naming because a test that passes for the wrong reason is worse than no test: it stranded *every* entry on a dead consumer, so the ordinary delivery path was never exercised at all; and it grepped the subscriber output for "print_alert", which matched `redis-cli`'s own subscription confirmation line and reported a published alert that did not exist.
This commit is contained in:
Executable
+248
@@ -0,0 +1,248 @@
|
||||
#!/usr/bin/env bash
|
||||
# End-to-end test of the stream:camera_frames contract: the real
|
||||
# continuum-backend producer, real Valkey, and the real continuum-ai-worker
|
||||
# binary.
|
||||
#
|
||||
# tests/frame_task_contract.test.ts and continuum-ai-worker's task.rs tests
|
||||
# each prove one side agrees with the schema. Only this proves the two agree
|
||||
# with each other, which is the thing that was wrong: the backend wrote
|
||||
# `printerId` and the worker read `printer_id`, so every task was delivered,
|
||||
# acknowledged, and attributed to a printer called "unknown". Nothing failed,
|
||||
# because a stream entry is a flat string map and asking it for a field that
|
||||
# is not there is not an error.
|
||||
#
|
||||
# What it asserts:
|
||||
# 1. a task the backend enqueues is consumed and acknowledged
|
||||
# 2. an old-format camelCase entry is dead-lettered, not silently mishandled
|
||||
# 3. an entry left pending by a dead consumer is reclaimed and acknowledged
|
||||
# 4. nothing is published to channel:print_alerts - the inference stub
|
||||
# abstains, and used to fabricate a 0.91 "spaghetti" for every frame
|
||||
#
|
||||
# Usage:
|
||||
# scripts/dev-env.sh up
|
||||
# scripts/e2e-ai-frames.sh
|
||||
#
|
||||
# WORKER_DIR=../continuum-ai-worker
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
WORKER_DIR="${WORKER_DIR:-$REPO_ROOT/../continuum-ai-worker}"
|
||||
VALKEY_URL="${VALKEY_URL:-redis://localhost:6379}"
|
||||
VALKEY_PORT="${VALKEY_PORT:-6379}"
|
||||
STREAM="stream:camera_frames"
|
||||
DEAD="stream:camera_frames:dead"
|
||||
GROUP="ai-worker"
|
||||
ALERTS="channel:print_alerts"
|
||||
export VALKEY_URL
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
WORKER_PID=""
|
||||
SUB_PID=""
|
||||
FAILURES=0
|
||||
|
||||
cli() { redis-cli -p "$VALKEY_PORT" "$@"; }
|
||||
|
||||
new_uuid() {
|
||||
if [ -r /proc/sys/kernel/random/uuid ]; then
|
||||
cat /proc/sys/kernel/random/uuid
|
||||
else
|
||||
python3 -c 'import uuid; print(uuid.uuid4())'
|
||||
fi
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
# Never `kill $PID` with an unset PID: `kill 0` signals the whole process
|
||||
# group, which includes this script and whatever launched it.
|
||||
for pid in "$WORKER_PID" "$SUB_PID"; do
|
||||
[ -n "$pid" ] && kill -9 "$pid" 2>/dev/null || true
|
||||
done
|
||||
if [ -z "${KEEP_DATA:-}" ]; then
|
||||
cli DEL "$STREAM" "$DEAD" >/dev/null 2>&1 || true
|
||||
fi
|
||||
if [ $rc -ne 0 ] || [ "$FAILURES" -ne 0 ]; then
|
||||
echo
|
||||
echo "--- worker log ---"; cat "$WORK/worker.log" 2>/dev/null || true
|
||||
fi
|
||||
rm -rf "$WORK"
|
||||
exit $rc
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
fail() { echo "FAIL: $*" >&2; exit 1; }
|
||||
ok() { echo " ok: $*"; }
|
||||
bad() { echo " FAILED: $*" >&2; FAILURES=$((FAILURES + 1)); }
|
||||
|
||||
# --- 0. preconditions -----------------------------------------------------
|
||||
|
||||
echo "==> checking Valkey"
|
||||
cli ping >/dev/null 2>&1 || fail "no Valkey at $VALKEY_URL - run scripts/dev-env.sh up"
|
||||
|
||||
# This test owns and then deletes these two keys. Refuse to touch a Valkey
|
||||
# that has real frames queued on it.
|
||||
EXISTING="$(cli XLEN "$STREAM" 2>/dev/null || echo 0)"
|
||||
[ "$EXISTING" = "0" ] || fail "$STREAM already has $EXISTING entries; refusing to run against live data"
|
||||
cli DEL "$STREAM" "$DEAD" >/dev/null
|
||||
ok "Valkey up, $STREAM empty"
|
||||
|
||||
echo "==> building the worker"
|
||||
( cd "$WORKER_DIR" && cargo build --quiet 2>&1 | tail -5 ) || fail "the worker did not build"
|
||||
WORKER_BIN="$WORKER_DIR/target/debug/continuum-ai-worker"
|
||||
[ -x "$WORKER_BIN" ] || fail "no worker binary at $WORKER_BIN"
|
||||
ok "worker built"
|
||||
|
||||
FARM="$(new_uuid)"; PRINTER="$(new_uuid)"; JOB="$(new_uuid)"
|
||||
|
||||
# --- 1. an entry stranded by a dead consumer -----------------------------
|
||||
|
||||
# Done first, and with COUNT 1, so exactly one entry goes through recovery
|
||||
# and everything added afterwards goes through the ordinary `>` read. An
|
||||
# earlier version stranded every entry on the stream, which meant the normal
|
||||
# delivery path was never exercised at all.
|
||||
echo "==> stranding one entry on a consumer that never acknowledges it"
|
||||
cli XGROUP CREATE "$STREAM" "$GROUP" 0 MKSTREAM >/dev/null 2>&1 || true
|
||||
cli XADD "$STREAM" '*' \
|
||||
farm_id "$FARM" printer_id "$PRINTER" \
|
||||
r2_object_key "farms/$FARM/frames/stranded.jpg" captured_at "2026-10-11T12:00:03Z" >/dev/null
|
||||
# Read as a consumer that then "dies": the entry is delivered and pending, so
|
||||
# no other consumer is offered it. Without recovery it sits there forever.
|
||||
cli XREADGROUP GROUP "$GROUP" dead-consumer COUNT 1 STREAMS "$STREAM" '>' >/dev/null
|
||||
STRANDED="$(cli XPENDING "$STREAM" "$GROUP" | head -1)"
|
||||
[ "${STRANDED:-0}" = "1" ] && ok "1 entry pending on a consumer that is gone" \
|
||||
|| bad "expected exactly 1 pending entry, XPENDING says '$STRANDED'"
|
||||
|
||||
# --- 2. enqueue through the real producer --------------------------------
|
||||
|
||||
echo "==> enqueuing two tasks through the backend's own producer"
|
||||
# The real `enqueueFrameIngestionBatch`, not a hand-written XADD: the point is
|
||||
# to test what the backend actually writes.
|
||||
bun -e "
|
||||
const { enqueueFrameIngestionBatch, valkey } = await import('./src/queue/index.ts');
|
||||
const ids = await enqueueFrameIngestionBatch([
|
||||
{ farm_id: '$FARM', printer_id: '$PRINTER', r2_object_key: 'farms/$FARM/frames/a.jpg', captured_at: '2026-10-11T12:00:00Z' },
|
||||
{ farm_id: '$FARM', printer_id: '$PRINTER', print_job_id: '$JOB', r2_object_key: 'farms/$FARM/frames/b.jpg', captured_at: '2026-10-11T12:00:01Z' },
|
||||
]);
|
||||
console.log(ids.join(' '));
|
||||
valkey.disconnect();
|
||||
" > "$WORK/ids.txt" 2>"$WORK/producer.err" || fail "the producer failed: $(cat "$WORK/producer.err")"
|
||||
ok "2 tasks enqueued"
|
||||
|
||||
# The field names actually on the wire, read back from Valkey rather than from
|
||||
# either side's source. This is the assertion the two repos could not make
|
||||
# about each other before.
|
||||
echo "==> the field names on the wire"
|
||||
cli XRANGE "$STREAM" - + > "$WORK/xrange.txt"
|
||||
for field in farm_id printer_id r2_object_key captured_at print_job_id; do
|
||||
grep -qx "$field" "$WORK/xrange.txt" || bad "field '$field' is not on the wire"
|
||||
done
|
||||
for camel in farmId printerId r2ObjectKey capturedAt printJobId; do
|
||||
grep -qx "$camel" "$WORK/xrange.txt" && bad "camelCase field '$camel' is on the wire" || true
|
||||
done
|
||||
# Exactly one entry carries a job id; the others omit the field entirely
|
||||
# rather than writing "".
|
||||
JOB_FIELDS="$(grep -cx print_job_id "$WORK/xrange.txt" || true)"
|
||||
[ "$JOB_FIELDS" = "1" ] && ok "print_job_id present on one entry, omitted on the others" \
|
||||
|| bad "print_job_id appears $JOB_FIELDS times, expected 1"
|
||||
ok "every field is snake_case, as the schema says"
|
||||
|
||||
# --- 3. an old-format entry ----------------------------------------------
|
||||
|
||||
echo "==> adding one entry in the old camelCase format"
|
||||
# Exactly what the previous producer wrote. The worker must dead-letter it
|
||||
# rather than process it with defaults.
|
||||
cli XADD "$STREAM" '*' \
|
||||
farmId "$FARM" printerId "$PRINTER" printJobId "" \
|
||||
r2ObjectKey "farms/$FARM/frames/legacy.jpg" capturedAt "2026-10-11T12:00:02Z" >/dev/null
|
||||
ok "legacy entry added"
|
||||
|
||||
# --- 4. watch the alert channel ------------------------------------------
|
||||
|
||||
cli SUBSCRIBE "$ALERTS" > "$WORK/alerts.txt" 2>/dev/null &
|
||||
SUB_PID=$!
|
||||
sleep 0.4
|
||||
|
||||
# --- 5. run the real worker ----------------------------------------------
|
||||
|
||||
echo "==> running the worker"
|
||||
CONTINUUM_VALKEY_URL="$VALKEY_URL" \
|
||||
CONTINUUM_FRAME_STREAM="$STREAM" \
|
||||
CONTINUUM_DEAD_LETTER_STREAM="$DEAD" \
|
||||
CONTINUUM_CONSUMER_GROUP="$GROUP" \
|
||||
CONTINUUM_CONSUMER_NAME="e2e-worker" \
|
||||
CONTINUUM_BLOCK_MS=500 \
|
||||
CONTINUUM_CLAIM_IDLE_MS=1000 \
|
||||
CONTINUUM_MAX_DELIVERIES=5 \
|
||||
RUST_LOG=continuum_ai_worker=debug \
|
||||
"$WORKER_BIN" > "$WORK/worker.log" 2>&1 &
|
||||
WORKER_PID=$!
|
||||
|
||||
# Long enough for the claim-idle window (1s) to elapse and a recovery pass to
|
||||
# run after it.
|
||||
for _ in $(seq 1 40); do
|
||||
PENDING="$(cli XPENDING "$STREAM" "$GROUP" | head -1)"
|
||||
DEADLEN="$(cli XLEN "$DEAD")"
|
||||
[ "${PENDING:-1}" = "0" ] && [ "${DEADLEN:-0}" -ge 1 ] && break
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
kill "$WORKER_PID" 2>/dev/null || true
|
||||
sleep 0.5
|
||||
ok "worker ran"
|
||||
|
||||
# --- 6. assertions -------------------------------------------------------
|
||||
|
||||
echo "==> what the worker did"
|
||||
grep -qi "panic" "$WORK/worker.log" && bad "the worker panicked" || ok "no panic"
|
||||
|
||||
PENDING="$(cli XPENDING "$STREAM" "$GROUP" | head -1)"
|
||||
[ "${PENDING:-x}" = "0" ] && ok "nothing left pending: every entry was acknowledged" \
|
||||
|| bad "$PENDING entries are still pending"
|
||||
|
||||
DEADLEN="$(cli XLEN "$DEAD")"
|
||||
[ "$DEADLEN" = "1" ] && ok "exactly one entry was dead-lettered" \
|
||||
|| bad "the dead-letter stream holds $DEADLEN entries, expected 1"
|
||||
|
||||
cli XRANGE "$DEAD" - + > "$WORK/dead.txt"
|
||||
grep -q "dead_reason" "$WORK/dead.txt" || bad "the dead-lettered entry carries no reason"
|
||||
grep -q "missing required field 'farm_id'" "$WORK/dead.txt" \
|
||||
&& ok "the legacy camelCase entry was dead-lettered, naming the field it lacked" \
|
||||
|| bad "wrong dead-letter reason: $(grep -A1 dead_reason "$WORK/dead.txt" | head -4 | tr '\n' ' ')"
|
||||
# The original fields are kept, so it can be inspected and replayed.
|
||||
grep -q "farmId" "$WORK/dead.txt" && ok "the original entry's fields were preserved" \
|
||||
|| bad "the dead-lettered entry did not keep the original fields"
|
||||
|
||||
grep -q "reclaimed a pending frame task" "$WORK/worker.log" \
|
||||
&& ok "the stranded entry was reclaimed from the dead consumer" \
|
||||
|| bad "no evidence the stranded entry was reclaimed"
|
||||
|
||||
# The three well-formed entries (one reclaimed, two read normally) must have
|
||||
# been *parsed*, not merely acknowledged - a parse failure would have
|
||||
# dead-lettered them instead, and an acknowledged-but-unparsed entry is
|
||||
# exactly the old bug. `no detections` is only reachable past the parse.
|
||||
PARSED="$(grep -c "no detections" "$WORK/worker.log" || true)"
|
||||
[ "${PARSED:-0}" -ge 3 ] && ok "all 3 well-formed tasks parsed and scored ($PARSED)" \
|
||||
|| bad "only $PARSED tasks got past parsing, expected 3"
|
||||
|
||||
# The whole point of the inference fix. The stub used to return a fixed
|
||||
# 0.91 "spaghetti" for every frame, including the empty slice it was always
|
||||
# handed, so a running worker was a false-alarm generator.
|
||||
# `redis-cli SUBSCRIBE` prints its own confirmation first - the three lines
|
||||
# "subscribe", the channel name, "1" - so grepping the file for the channel
|
||||
# name matches that and not a message. A delivered message is a line that is
|
||||
# exactly "message".
|
||||
if grep -qx "message" "$WORK/alerts.txt" 2>/dev/null; then
|
||||
bad "an alert was published: $(grep -A2 -x message "$WORK/alerts.txt" | tail -2 | tr '\n' ' ')"
|
||||
else
|
||||
ok "no alert was published - the inference stub abstained instead of inventing one"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$FAILURES" -eq 0 ]; then
|
||||
echo "PASS - the backend's producer and the worker's consumer agree on"
|
||||
echo " stream:camera_frames, against real Valkey."
|
||||
else
|
||||
echo "FAIL - $FAILURES assertion(s) failed." >&2
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user