MCMapper-Backend/README.md

MCMapper-Backend

Map-render backend for MCMapper-Mod. Does all the heavy lifting a thin in-game mod shouldn't: persists world state, renders 2D tiles and 3D meshes, relays chat, and serves the web map viewer — instead of the MC server itself burning CPU/RAM on rendering the way Bluemap/Dynmap do.

Services

Three independently-deployable services, each its own docker-compose service:

• api/ — ElysiaJS on Bun. The I/O layer: WS gateway for mod connections, chat relay, chunk store, marker/waypoint sharing, admin config, region export, tile/mesh serving.
• worker/ — Rust. Tile rasterization and chunk meshing, consumed off a Redis dirty-chunk stream. Stateless — scale it with docker compose up --scale worker=N, or run instances on separate hardware pointed at the same Postgres/Redis/MinIO over a private network. Rendering strategy is config-selectable (RENDER_BACKEND=cpu/gpu/hybrid, see worker/.env.example) behind a RenderBackend trait (Phase 8): cpu (rayon, default) parallelizes across a whole batch of dirty chunks; gpu additionally offloads tile shading and per-voxel face-visibility extraction to a wgpu compute shader per chunk/section (greedy-mesh merge/compaction stays CPU-only regardless — sequential/branchy, not GPU-parallel-friendly); hybrid offloads only tile shading, keeping meshing on CPU. gpu/hybrid fall back to cpu automatically (logged) if no compatible GPU adapter is found — see worker/src/render/gpu.rs's doc comment for the current known limitation (each tile/section is its own GPU dispatch, so at small batch sizes cpu currently outruns gpu/hybrid — see cargo run --release --example benchmark, in worker/, for real numbers on your own hardware).
• frontend/ — ElysiaJS + Pug + Tailwind 4 + Alpine.js. The public-facing pages (map viewer, chat, admin panel). Stateless — no DB access, calls api for anything server-rendered; the browser's own live map/chat/tile traffic talks to api directly, not proxied through here.

Plus postgres (source chunk data, accounts, chat history, config, render-artifact metadata pointers), redis (dirty-chunk queue, pub/sub, link-code TTLs), and caddy (reverse proxy: /ws + /api/* → api, everything else → frontend).

Postgres/Redis are never exposed publicly — only reachable on the compose network or a private network (VPN/Tailscale/LAN) for remote worker instances.

Object storage

Rendered tile PNGs and mesh binaries live in a mcmapper-tiles bucket on an existing shared MinIO instance (devstack-minio on octo-winsrv) rather than a per-stack minio container — kept out of Postgres so backups stay free of large binaries, and any worker instance (local or remote) has a shared place to write output. api/worker authenticate with a dedicated mcmapper access key whose policy (mcmapper-tiles-rw) only grants GetObject/PutObject/DeleteObject/ListBucket on that one bucket — it can't see or touch anything else on the shared instance. Provisioned via:

mc mb myminio/mcmapper-tiles
mc admin policy create myminio mcmapper-tiles-rw mcmapper-policy.json   # see git history for the policy JSON
mc admin user add myminio mcmapper <generated secret>
mc admin policy attach myminio mcmapper-tiles-rw --user mcmapper

api/.env.example/worker/.env.example point MINIO_ENDPOINT/MINIO_PORT at that instance's LAN address; swap to the public gateway (s3.octoturge.com:443, MINIO_USE_SSL=true) if a stack isn't on the same LAN. MINIO_SECRET_KEY is a real credential and is deliberately not committed — set it in an untracked ./api/.env / ./worker/.env (docker-compose layers those on top of the tracked .env.example, see docker-compose.yml).

Real block textures (Phase 11)

worker/src/palette.rs's hand-picked color table is now a fallback, not the only source of 2D-tile colors. Set ACCEPT_MINECRAFT_EULA=true (worker/.env.example, off by default — mirrors BlueMap's accept-download) and the worker downloads Minecraft's official client jar directly from Mojang's own public launchermeta/piston-meta/piston-data endpoints (same source the real launcher uses — no redistribution, so no licensing issue) on first startup, averages every assets/minecraft/textures/block(s)/*.png into a representative color, and caches the result to disk (TEXTURE_CACHE_DIR, default ./cache) so it isn't re-downloaded every restart. MC_TEXTURE_VERSION (default 1.12.2) picks which version's jar to pull from — this project's priority targets (1.7.10/1.12.2) share the pre-1.13 textures/blocks/ (plural) naming, which worker/src/textures.rs checks alongside the modern textures/block/ path.

A handful of blocks (grass block top, leaves, water, lava) are deliberately excluded from the texture-averaged path (worker/src/block_names.rs's doc comment) and keep their hand-picked color: their real textures are either biome-tinted at runtime (grayscale in the raw file) or animated/transparent frame strips, so averaging the raw asset would produce a wrong color, not just an approximate one.

texturepacks/<name>/ (flat *.png files, mirrors Dynmap) lets an operator override the downloaded defaults — set TEXTURE_PACK=<name> and its colors are layered on top of the vanilla palette at startup. Each server also has an admin-configurable texturePack field (/admin, servers.texture_pack — same shape as waypointFormat) recording which pack an operator wants per server — but render-time application is currently worker-wide only, via TEXTURE_PACK, not yet resolved per-server from that column. True per-server resolution needs the render pipeline to thread a server-scoped palette through main.rs's batch loop instead of one process-wide OnceLock (worker/src/render/mod.rs) — a real architectural change, deferred rather than rushed.

Modded blocks: Forge doesn't split client/server jars, so a modded server's own classpath already has every loaded mod's texture assets, just unused server-side. The forge-1_12_2 mod leaf (Enigmatica 2, this project's primary target) extracts them once at startup (BlockAssetExtractor, best-effort: guesses each block's texture by its registry-name path segment, not a real blockstate/model JSON resolution — that's Phase 12's job) and ships two new WS messages after connecting: block_registry (numeric id -> registry name, needed since a numeric blockId alone is meaningless without the mod list that assigned it) and block_textures (the extracted PNGs, batched). The api stores both (api/src/textures.ts: registry rows in a new block_registry table, texture PNGs in the shared MinIO bucket with pointer rows in block_textures) — but, like the texturepacks/ case above, nothing reads these tables into the render pipeline yet. This is a deliberate two-step boundary: "ingest and store" (done, real, tested) vs. "resolve into a per-server palette at render time" (the same deferred piece as texturepacks/ above, and naturally solved together). forge-1_7_10/neoforge-26_1 don't implement extraction yet either — BackendConnection#sendBlockRegistry/#sendBlockTextures are on the shared interface (so any leaf can adopt them later with no protocol change), but only the 1.12.2 leaf calls them so far, matching this phase's Enigmatica-2-focused verification target.

Running

docker compose up

api on :3000, frontend on :3001, both behind Caddy on :80. api applies its Postgres migrations on startup (see api/src/db/migrate.ts); worker consumes the mcmapper:dirty-chunks Redis stream via a consumer group (mcmapper-workers) so multiple instances split work safely — scale locally with docker compose up --scale worker=N, or run a standalone worker container on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/ Tailscale/LAN — never expose those ports publicly).

worker's RENDER_THREADS/RENDER_PROFILE (see worker/.env.example) size the rayon pool that renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per batch: RENDER_THREADS=auto uses all available cores, or set a fixed count; RENDER_PROFILE is server (default — many small batches, tuned for a many-core box) or consumer (fewer, larger batches, less scheduling overhead on fewer/faster cores).

Connecting a mod instance

Register a server via the admin panel at /admin (set MCMAPPER_ADMIN_TOKEN first — see "Admin panel" below), or seed one row by hand for scripted/headless setup:

docker compose run --rm api bun run seed

reads MCMAPPER_SEED_SERVER_NAME/MCMAPPER_SEED_SERVER_TOKEN from api/.env.example (edit those first, or override with -e). Either way, point the mod's MapperConfig#serverToken at the resulting token. Once the mod connects and sends its initial chunk backfill, tiles appear at GET /api/tiles/:serverId/:dimension/:zoom/:tileX/:tileY.png (zoom is always 0 for now — see worker/src/render/cpu.rs) and the frontend's Leaflet viewer picks them up automatically from GET /api/servers.

Admin panel

/admin (linked from the map page's header) manages the server registry: register new servers (generates their token — never admin-supplied, so it can't collide with or be guessed from anything else), and edit authMode, anonymousChatAllowed, waypointFormat, and playerPositionsVisible per server. It's gated behind a single shared secret, not a per-account role (this is a single-operator backend) — set MCMAPPER_ADMIN_TOKEN in api/'s untracked .env (see api/.env.example), restart api, then enter that same value into the panel's unlock prompt. Leaving it unset disables every /api/admin/* route (401), it does not default to open. The token is remembered in the browser's localStorage after unlocking, same pattern as the player-facing session token (see api/src/link.ts's doc comment).

Settings not exposed here yet (which map types render, dimension filtering) don't have underlying features built for them either — no point in a knob nothing reads. They'll gain admin UI alongside the feature that needs them.

Player positions (Phase 7b)

The mod sends a throttled roster of online players (player_positions over /ws, see api/src/ws-gateway.ts's doc comment) — its own playerTrackingEnabled/ playerPositionIntervalTicks config decides whether/how often it sends this at all. The backend relays it to any browser subscribed on /ws/players/:serverId (api/src/players-gateway.ts), gated per-server on the admin panel's playerPositionsVisible toggle (default on) — an independent, backend-side "should we show it" decision from the mod's own config. Not persisted (no meaningful history for a live position), just the latest roster in Redis (api/src/players.ts) so a browser tab that connects between mod flushes doesn't sit empty. The map renders players as map markers with a show/hide toggle and an online count, right above the region-export panel.

Running tests

worker's tests (cargo test, in worker/) are pure unit tests (greedy mesher, tile rasterizer, block-color palette) and need nothing running — including the gpu/hybrid backend tests (Phase 8), which request a real GPU adapter and skip themselves (rather than failing) if none is found, so cargo test still passes on a machine with no GPU. api's and frontend's (bun test, in each directory) are integration tests against real infra — start it first:

docker run -d --name mcmapper-test-pg -e POSTGRES_USER=mcmapper -e POSTGRES_PASSWORD=mcmapper -e POSTGRES_DB=mcmapper -p 15432:5432 postgres:17-alpine
docker run -d --name mcmapper-test-redis -p 16379:6379 redis:7-alpine
docker run -d --name mcmapper-test-minio -e MINIO_ROOT_USER=mcmapper -e MINIO_ROOT_PASSWORD=mcmapper-dev-only -p 19000:9000 minio/minio:latest server /data
cd api && DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper bun run migrate

then, from api/ or frontend/:

DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper \
REDIS_URL=redis://localhost:16379 \
MINIO_ENDPOINT=localhost MINIO_PORT=19000 MINIO_ACCESS_KEY=mcmapper MINIO_SECRET_KEY=mcmapper-dev-only \
bun test

Each test file creates and tears down its own server row (random token per run) so runs never collide with each other or with real dev data — see api/src/test-helpers.ts.

Running the e2e suite

bun test above exercises api and frontend independently — it never proves the browser can actually reach both through the same origin the way production's Caddy routing does (/ws* and /api/* -> api, everything else -> frontend, see Caddyfile). e2e/ is a standing Playwright suite that closes that gap: it drives a real Chromium browser against the full stack behind a small routing-equivalent proxy (e2e/proxy.ts — no caddy binary is available in this dev environment, so it isn't real Caddy, just the same three routing rules).

cd e2e
bun install
bunx playwright install chromium   # one-time, downloads the browser binary
bunx playwright test

global-setup.ts does everything by itself — no manual container/migration steps needed first (unlike the bun test section above): throwaway Postgres/Redis/MinIO containers (mcmapper-e2e-*, distinct names/ports from the bun test ones so both can run at once), migrations, a seeded server + linked account/session + a 5x5-chunk terrain footprint around the world origin, then the api/frontend/proxy processes (the api process is started with a fixed MCMAPPER_ADMIN_TOKEN for tests/admin.spec.ts to use — see config.ts's ADMIN_TOKEN). global-teardown.ts kills every spawned process and removes the containers afterward. Covers the UI flows most worth a real click-through: the marker click-to-place/edit popup (tests/markers.spec.ts, including that a marker created while linked shows up in a second browser context with the same session — the cross-device sync claim), the region-select drag + glTF export (tests/region-export.spec.ts, including a real triggered file download), the admin panel's token gate + server register/edit/delete round trip (tests/admin.spec.ts), and the player-position relay (tests/players.spec.ts — simulates a mod connection over the real /ws protocol from inside the browser context and confirms a subscribed tab renders the roster, respects the show toggle, and clears markers on an empty roster).

Attribution

See THIRD_PARTY_NOTICES.md.