octoturge
cc3860ce08
Add worker/src/config.rs resolving RENDER_THREADS (auto | manual override) and RENDER_PROFILE (server | consumer) into a rayon thread-pool size and a per-xread batch size, unit tested. Restage main.rs's processing loop into three stages per batch: async fetch, CPU-bound rasterize+mesh parallelized across the batch on a sized rayon pool, then async store+ack — the parallelism target is many chunks in flight at once, since a single 16x16 tile is too small for rayon to help within itself (per the pre-existing doc comment in render/cpu.rs). Verified locally: 3 worker instances against the same Redis stream split 24 queued dirty-chunk jobs with zero duplicates and zero drops (confirmed via worker logs and tile_pointers rows), validating the consumer-group design ahead of a real remote-worker deployment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015tKdPZt78zbPUZMXWzKEKt
162 lines
8.8 KiB
Markdown
162 lines
8.8 KiB
Markdown
# MCMapper-Backend
|
|
|
|
Map-render backend for [MCMapper-Mod](https://git.octoturge.com/octoturge/MCMapper-Mod). Does
|
|
all the heavy lifting a thin in-game mod shouldn't: persists world state, renders 2D tiles and
|
|
3D meshes, relays chat, and serves the web map viewer — instead of the MC server itself burning
|
|
CPU/RAM on rendering the way Bluemap/Dynmap do.
|
|
|
|
## Services
|
|
|
|
Three independently-deployable services, each its own docker-compose service:
|
|
|
|
- `api/` — ElysiaJS on Bun. The I/O layer: WS gateway for mod connections, chat relay, chunk
|
|
store, marker/waypoint sharing, admin config, region export, tile/mesh serving.
|
|
- `worker/` — Rust. CPU-bound rendering: tile rasterization and chunk meshing, consumed off a
|
|
Redis dirty-chunk stream. Stateless — scale it with `docker compose up --scale worker=N`, or
|
|
run instances on separate hardware pointed at the same Postgres/Redis/MinIO over a private
|
|
network. Rendering strategy (`cpu`/`gpu`/`hybrid`) is config-selectable behind a
|
|
`RenderBackend` trait; only `cpu` (rayon) exists so far — `gpu`/`hybrid` (wgpu) land in
|
|
Phase 8.
|
|
- `frontend/` — ElysiaJS + Pug + Tailwind 4 + Alpine.js. The public-facing pages (map viewer,
|
|
chat, admin panel). Stateless — no DB access, calls `api` for anything server-rendered; the
|
|
browser's own live map/chat/tile traffic talks to `api` directly, not proxied through here.
|
|
|
|
Plus `postgres` (source chunk data, accounts, chat history, config, render-artifact metadata
|
|
pointers), `redis` (dirty-chunk queue, pub/sub, link-code TTLs), and `caddy` (reverse proxy:
|
|
`/ws` + `/api/*` → `api`, everything else → `frontend`).
|
|
|
|
Postgres/Redis are never exposed publicly — only reachable on the compose network or a private
|
|
network (VPN/Tailscale/LAN) for remote `worker` instances.
|
|
|
|
### Object storage
|
|
|
|
Rendered tile PNGs and mesh binaries live in a `mcmapper-tiles` bucket on an existing shared
|
|
MinIO instance (`devstack-minio` on octo-winsrv) rather than a per-stack `minio` container —
|
|
kept out of Postgres so backups stay free of large binaries, and any `worker` instance (local or
|
|
remote) has a shared place to write output. `api`/`worker` authenticate with a dedicated
|
|
`mcmapper` access key whose policy (`mcmapper-tiles-rw`) only grants
|
|
`GetObject`/`PutObject`/`DeleteObject`/`ListBucket` on that one bucket — it can't see or touch
|
|
anything else on the shared instance. Provisioned via:
|
|
|
|
```
|
|
mc mb myminio/mcmapper-tiles
|
|
mc admin policy create myminio mcmapper-tiles-rw mcmapper-policy.json # see git history for the policy JSON
|
|
mc admin user add myminio mcmapper <generated secret>
|
|
mc admin policy attach myminio mcmapper-tiles-rw --user mcmapper
|
|
```
|
|
|
|
`api/.env.example`/`worker/.env.example` point `MINIO_ENDPOINT`/`MINIO_PORT` at that instance's
|
|
LAN address; swap to the public gateway (`s3.octoturge.com:443`, `MINIO_USE_SSL=true`) if a
|
|
stack isn't on the same LAN. `MINIO_SECRET_KEY` is a real credential and is deliberately **not**
|
|
committed — set it in an untracked `./api/.env` / `./worker/.env` (docker-compose layers those on
|
|
top of the tracked `.env.example`, see `docker-compose.yml`).
|
|
|
|
## Running
|
|
|
|
```
|
|
docker compose up
|
|
```
|
|
|
|
`api` on :3000, `frontend` on :3001, both behind Caddy on :80. `api` applies its Postgres
|
|
migrations on startup (see `api/src/db/migrate.ts`); `worker` consumes the `mcmapper:dirty-chunks`
|
|
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely —
|
|
scale locally with `docker compose up --scale worker=N`, or run a standalone `worker` container
|
|
on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/
|
|
Tailscale/LAN — never expose those ports publicly).
|
|
|
|
`worker`'s `RENDER_THREADS`/`RENDER_PROFILE` (see `worker/.env.example`) size the rayon pool that
|
|
renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per
|
|
batch: `RENDER_THREADS=auto` uses all available cores, or set a fixed count; `RENDER_PROFILE`
|
|
is `server` (default — many small batches, tuned for a many-core box) or `consumer` (fewer,
|
|
larger batches, less scheduling overhead on fewer/faster cores).
|
|
|
|
### Connecting a mod instance
|
|
|
|
Register a server via the admin panel at `/admin` (set `MCMAPPER_ADMIN_TOKEN` first — see
|
|
"Admin panel" below), or seed one row by hand for scripted/headless setup:
|
|
|
|
```
|
|
docker compose run --rm api bun run seed
|
|
```
|
|
|
|
reads `MCMAPPER_SEED_SERVER_NAME`/`MCMAPPER_SEED_SERVER_TOKEN` from `api/.env.example` (edit
|
|
those first, or override with `-e`). Either way, point the mod's `MapperConfig#serverToken` at
|
|
the resulting token. Once the mod connects and sends its initial chunk backfill, tiles appear at
|
|
`GET /api/tiles/:serverId/:dimension/:zoom/:tileX/:tileY.png` (zoom is always `0` for now — see
|
|
`worker/src/render/cpu.rs`) and the frontend's Leaflet viewer picks them up automatically from
|
|
`GET /api/servers`.
|
|
|
|
### Admin panel
|
|
|
|
`/admin` (linked from the map page's header) manages the server registry: register new servers
|
|
(generates their token — never admin-supplied, so it can't collide with or be guessed from
|
|
anything else), and edit `authMode`, `anonymousChatAllowed`, and `waypointFormat` per server.
|
|
It's gated behind a single shared secret, not a per-account role (this is a single-operator
|
|
backend) — set `MCMAPPER_ADMIN_TOKEN` in `api/`'s untracked `.env` (see `api/.env.example`),
|
|
restart `api`, then enter that same value into the panel's unlock prompt. Leaving it unset
|
|
disables every `/api/admin/*` route (401), it does not default to open. The token is remembered
|
|
in the browser's `localStorage` after unlocking, same pattern as the player-facing session token
|
|
(see `api/src/link.ts`'s doc comment).
|
|
|
|
Settings not exposed here yet (which map types render, dimension filtering, player-position
|
|
visibility) don't have underlying features built for them either — no point in a knob nothing
|
|
reads. They'll gain admin UI alongside the feature that needs them.
|
|
|
|
## Running tests
|
|
|
|
`worker`'s tests (`cargo test`, in `worker/`) are pure unit tests (greedy mesher, tile
|
|
rasterizer, block-color palette) and need nothing running. `api`'s and `frontend`'s (`bun test`,
|
|
in each directory) are integration tests against real infra — start it first:
|
|
|
|
```
|
|
docker run -d --name mcmapper-test-pg -e POSTGRES_USER=mcmapper -e POSTGRES_PASSWORD=mcmapper -e POSTGRES_DB=mcmapper -p 15432:5432 postgres:17-alpine
|
|
docker run -d --name mcmapper-test-redis -p 16379:6379 redis:7-alpine
|
|
docker run -d --name mcmapper-test-minio -e MINIO_ROOT_USER=mcmapper -e MINIO_ROOT_PASSWORD=mcmapper-dev-only -p 19000:9000 minio/minio:latest server /data
|
|
cd api && DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper bun run migrate
|
|
```
|
|
|
|
then, from `api/` or `frontend/`:
|
|
|
|
```
|
|
DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper \
|
|
REDIS_URL=redis://localhost:16379 \
|
|
MINIO_ENDPOINT=localhost MINIO_PORT=19000 MINIO_ACCESS_KEY=mcmapper MINIO_SECRET_KEY=mcmapper-dev-only \
|
|
bun test
|
|
```
|
|
|
|
Each test file creates and tears down its own server row (random token per run) so runs never
|
|
collide with each other or with real dev data — see `api/src/test-helpers.ts`.
|
|
|
|
## Running the e2e suite
|
|
|
|
`bun test` above exercises `api` and `frontend` independently — it never proves the browser can
|
|
actually reach both through the same origin the way production's Caddy routing does (`/ws*` and
|
|
`/api/*` -> `api`, everything else -> `frontend`, see `Caddyfile`). `e2e/` is a standing
|
|
Playwright suite that closes that gap: it drives a real Chromium browser against the full stack
|
|
behind a small routing-equivalent proxy (`e2e/proxy.ts` — no `caddy` binary is available in this
|
|
dev environment, so it isn't real Caddy, just the same three routing rules).
|
|
|
|
```
|
|
cd e2e
|
|
bun install
|
|
bunx playwright install chromium # one-time, downloads the browser binary
|
|
bunx playwright test
|
|
```
|
|
|
|
`global-setup.ts` does everything by itself — no manual container/migration steps needed first
|
|
(unlike the `bun test` section above): throwaway Postgres/Redis/MinIO containers
|
|
(`mcmapper-e2e-*`, distinct names/ports from the `bun test` ones so both can run at once),
|
|
migrations, a seeded server + linked account/session + a 5x5-chunk terrain footprint around the
|
|
world origin, then the `api`/`frontend`/proxy processes (the `api` process is started with a
|
|
fixed `MCMAPPER_ADMIN_TOKEN` for `tests/admin.spec.ts` to use — see `config.ts`'s `ADMIN_TOKEN`).
|
|
`global-teardown.ts` kills every spawned process and removes the containers afterward. Covers the
|
|
UI flows most worth a real click-through: the marker click-to-place/edit popup
|
|
(`tests/markers.spec.ts`, including that a marker created while linked shows up in a second
|
|
browser context with the same session — the cross-device sync claim), the region-select drag +
|
|
glTF export (`tests/region-export.spec.ts`, including a real triggered file download), and the
|
|
admin panel's token gate + server register/edit/delete round trip (`tests/admin.spec.ts`).
|
|
|
|
## Attribution
|
|
|
|
See `THIRD_PARTY_NOTICES.md`.
|