Phase 7 (partial): CPU thread/profile config + verified multi-worker scaling

Add worker/src/config.rs resolving RENDER_THREADS (auto | manual override)
and RENDER_PROFILE (server | consumer) into a rayon thread-pool size and a
per-xread batch size, unit tested. Restage main.rs's processing loop into
three stages per batch: async fetch, CPU-bound rasterize+mesh parallelized
across the batch on a sized rayon pool, then async store+ack — the
parallelism target is many chunks in flight at once, since a single 16x16
tile is too small for rayon to help within itself (per the pre-existing
doc comment in render/cpu.rs).

Verified locally: 3 worker instances against the same Redis stream split
24 queued dirty-chunk jobs with zero duplicates and zero drops (confirmed
via worker logs and tile_pointers rows), validating the consumer-group
design ahead of a real remote-worker deployment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015tKdPZt78zbPUZMXWzKEKt
This commit is contained in:
2026-08-09 14:07:57 +02:00
parent a134c47152
commit cc3860ce08
4 changed files with 301 additions and 77 deletions
+10 -1
View File
@@ -59,7 +59,16 @@ docker compose up
`api` on :3000, `frontend` on :3001, both behind Caddy on :80. `api` applies its Postgres
migrations on startup (see `api/src/db/migrate.ts`); `worker` consumes the `mcmapper:dirty-chunks`
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely.
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely
scale locally with `docker compose up --scale worker=N`, or run a standalone `worker` container
on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/
Tailscale/LAN — never expose those ports publicly).
`worker`'s `RENDER_THREADS`/`RENDER_PROFILE` (see `worker/.env.example`) size the rayon pool that
renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per
batch: `RENDER_THREADS=auto` uses all available cores, or set a fixed count; `RENDER_PROFILE`
is `server` (default — many small batches, tuned for a many-core box) or `consumer` (fewer,
larger batches, less scheduling overhead on fewer/faster cores).
### Connecting a mod instance