Phase 7 (partial): CPU thread/profile config + verified multi-worker scaling
Add worker/src/config.rs resolving RENDER_THREADS (auto | manual override) and RENDER_PROFILE (server | consumer) into a rayon thread-pool size and a per-xread batch size, unit tested. Restage main.rs's processing loop into three stages per batch: async fetch, CPU-bound rasterize+mesh parallelized across the batch on a sized rayon pool, then async store+ack — the parallelism target is many chunks in flight at once, since a single 16x16 tile is too small for rayon to help within itself (per the pre-existing doc comment in render/cpu.rs). Verified locally: 3 worker instances against the same Redis stream split 24 queued dirty-chunk jobs with zero duplicates and zero drops (confirmed via worker logs and tile_pointers rows), validating the consumer-group design ahead of a real remote-worker deployment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015tKdPZt78zbPUZMXWzKEKt
This commit is contained in:
@@ -59,7 +59,16 @@ docker compose up
|
||||
|
||||
`api` on :3000, `frontend` on :3001, both behind Caddy on :80. `api` applies its Postgres
|
||||
migrations on startup (see `api/src/db/migrate.ts`); `worker` consumes the `mcmapper:dirty-chunks`
|
||||
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely.
|
||||
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely —
|
||||
scale locally with `docker compose up --scale worker=N`, or run a standalone `worker` container
|
||||
on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/
|
||||
Tailscale/LAN — never expose those ports publicly).
|
||||
|
||||
`worker`'s `RENDER_THREADS`/`RENDER_PROFILE` (see `worker/.env.example`) size the rayon pool that
|
||||
renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per
|
||||
batch: `RENDER_THREADS=auto` uses all available cores, or set a fixed count; `RENDER_PROFILE`
|
||||
is `server` (default — many small batches, tuned for a many-core box) or `consumer` (fewer,
|
||||
larger batches, less scheduling overhead on fewer/faster cores).
|
||||
|
||||
### Connecting a mod instance
|
||||
|
||||
|
||||
Reference in New Issue
Block a user