Phase 7 (partial): CPU thread/profile config + verified multi-worker scaling

Add worker/src/config.rs resolving RENDER_THREADS (auto | manual override)
and RENDER_PROFILE (server | consumer) into a rayon thread-pool size and a
per-xread batch size, unit tested. Restage main.rs's processing loop into
three stages per batch: async fetch, CPU-bound rasterize+mesh parallelized
across the batch on a sized rayon pool, then async store+ack — the
parallelism target is many chunks in flight at once, since a single 16x16
tile is too small for rayon to help within itself (per the pre-existing
doc comment in render/cpu.rs).

Verified locally: 3 worker instances against the same Redis stream split
24 queued dirty-chunk jobs with zero duplicates and zero drops (confirmed
via worker logs and tile_pointers rows), validating the consumer-group
design ahead of a real remote-worker deployment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015tKdPZt78zbPUZMXWzKEKt
This commit is contained in:
2026-08-09 14:07:57 +02:00
parent a134c47152
commit cc3860ce08
4 changed files with 301 additions and 77 deletions
+14
View File
@@ -11,3 +11,17 @@ MINIO_SECRET_KEY=changeme-set-in-untracked-env
# cpu | gpu | hybrid — see render::backend. Only `cpu` exists so far (Phase 8 adds gpu/hybrid).
RENDER_BACKEND=cpu
# Thread count for the rayon pool that renders a batch of dirty chunks in parallel (see
# config.rs/main.rs — a single 16x16 tile is too small for rayon to help within itself, so the
# parallelism target is many chunks in flight at once). `auto` (default) uses
# std::thread::available_parallelism(); set a positive integer to override (0/garbage also falls
# back to auto).
RENDER_THREADS=auto
# server | consumer — tunes how many dirty-chunk entries are pulled off the Redis stream per
# xread cycle before being rendered in parallel. `server` (default) batches more aggressively —
# many small batches so every core on a many-core Xeon-style box stays fed; `consumer` uses
# fewer, larger batches, less scheduling overhead per chunk on fewer/faster cores. Unrecognized
# values fall back to `server`, matching this project's own primary deployment target.
RENDER_PROFILE=server