MCMapper-Backend

Map-render backend for MCMapper-Mod. Does all the heavy lifting a thin in-game mod shouldn't: persists world state, renders 2D tiles and 3D meshes, relays chat, and serves the web map viewer — instead of the MC server itself burning CPU/RAM on rendering the way Bluemap/Dynmap do.

Services

Three independently-deployable services, each its own docker-compose service:

• api/ — ElysiaJS on Bun. The I/O layer: WS gateway for mod connections, chat relay, chunk store, marker/waypoint sharing, admin config, region export, tile/mesh serving.
• worker/ — Rust. Tile rasterization and chunk meshing, consumed off a Redis dirty-chunk stream. Stateless — scale it with docker compose up --scale worker=N, or run instances on separate hardware pointed at the same Postgres/Redis/MinIO over a private network. Rendering strategy is config-selectable (RENDER_BACKEND=cpu/gpu/hybrid, see worker/.env.example) behind a RenderBackend trait (Phase 8): cpu (rayon, default) parallelizes across a whole batch of dirty chunks; gpu additionally offloads tile shading and per-voxel face-visibility extraction to a wgpu compute shader per chunk/section (greedy-mesh merge/compaction stays CPU-only regardless — sequential/branchy, not GPU-parallel-friendly); hybrid offloads only tile shading, keeping meshing on CPU. gpu/hybrid fall back to cpu automatically (logged) if no compatible GPU adapter is found — see worker/src/render/gpu.rs's doc comment for the current known limitation (each tile/section is its own GPU dispatch, so at small batch sizes cpu currently outruns gpu/hybrid — see cargo run --release --example benchmark, in worker/, for real numbers on your own hardware).
• frontend/ — ElysiaJS + Pug + Tailwind 4 + Alpine.js. The public-facing pages (map viewer, chat, admin panel). Stateless — no DB access, calls api for anything server-rendered; the browser's own live map/chat/tile traffic talks to api directly, not proxied through here.

Plus postgres (source chunk data, accounts, chat history, config, render-artifact metadata pointers), redis (dirty-chunk queue, pub/sub, link-code TTLs), and caddy (reverse proxy: /ws + /api/* → api, everything else → frontend).

Postgres/Redis are never exposed publicly — only reachable on the compose network or a private network (VPN/Tailscale/LAN) for remote worker instances.

Object storage

Rendered tile PNGs and mesh binaries live in a mcmapper-tiles bucket on an existing shared MinIO instance (devstack-minio on octo-winsrv) rather than a per-stack minio container — kept out of Postgres so backups stay free of large binaries, and any worker instance (local or remote) has a shared place to write output. api/worker authenticate with a dedicated mcmapper access key whose policy (mcmapper-tiles-rw) only grants GetObject/PutObject/DeleteObject/ListBucket on that one bucket — it can't see or touch anything else on the shared instance. Provisioned via:

mc mb myminio/mcmapper-tiles
mc admin policy create myminio mcmapper-tiles-rw mcmapper-policy.json   # see git history for the policy JSON
mc admin user add myminio mcmapper <generated secret>
mc admin policy attach myminio mcmapper-tiles-rw --user mcmapper

api/.env.example/worker/.env.example point MINIO_ENDPOINT/MINIO_PORT at that instance's LAN address; swap to the public gateway (s3.octoturge.com:443, MINIO_USE_SSL=true) if a stack isn't on the same LAN. MINIO_SECRET_KEY is a real credential and is deliberately not committed — set it in an untracked ./api/.env / ./worker/.env (docker-compose layers those on top of the tracked .env.example, see docker-compose.yml).

Real block textures (Phase 11)

worker/src/palette.rs's hand-picked color table is now a fallback, not the only source of 2D-tile colors. Set ACCEPT_MINECRAFT_EULA=true (worker/.env.example, off by default — mirrors BlueMap's accept-download) and the worker downloads Minecraft's official client jar directly from Mojang's own public launchermeta/piston-meta/piston-data endpoints (same source the real launcher uses — no redistribution, so no licensing issue) on first startup, averages every assets/minecraft/textures/block(s)/*.png into a representative color, and caches the result to disk (TEXTURE_CACHE_DIR, default ./cache) so it isn't re-downloaded every restart. MC_TEXTURE_VERSION (default 1.12.2) picks which version's jar to pull from — this project's priority targets (1.7.10/1.12.2) share the pre-1.13 textures/blocks/ (plural) naming, which worker/src/textures.rs checks alongside the modern textures/block/ path.

A handful of blocks (grass block top, leaves, water, lava) are deliberately excluded from the texture-averaged path (worker/src/block_names.rs's doc comment) and keep their hand-picked color: their real textures are either biome-tinted at runtime (grayscale in the raw file) or animated/transparent frame strips, so averaging the raw asset would produce a wrong color, not just an approximate one.

texturepacks/<name>/ (flat *.png files, mirrors Dynmap) lets an operator override the downloaded defaults — set TEXTURE_PACK=<name> and its colors are layered on top of the vanilla palette at startup. Each server also has an admin-configurable texturePack field (/admin, servers.texture_pack — same shape as waypointFormat) recording which pack an operator wants per server — but render-time application is currently worker-wide only, via TEXTURE_PACK, not yet resolved per-server from that column. True per-server resolution needs the render pipeline to thread a server-scoped palette through main.rs's batch loop instead of one process-wide OnceLock (worker/src/render/mod.rs) — a real architectural change, deferred rather than rushed.

Modded blocks: Forge doesn't split client/server jars, so a modded server's own classpath already has every loaded mod's texture assets, just unused server-side. The forge-1_12_2 mod leaf (Enigmatica 2, this project's primary target) extracts them once at startup (BlockAssetExtractor, best-effort: guesses each block's texture by its registry-name path segment, not a real blockstate/model JSON resolution — that's Phase 13's job, see below) and ships two new WS messages after connecting: block_registry (numeric id -> registry name, needed since a numeric blockId alone is meaningless without the mod list that assigned it) and block_textures (the extracted PNGs, batched). The api stores both (api/src/textures.ts: registry rows in a new block_registry table, texture PNGs in the shared MinIO bucket with pointer rows in block_textures) — but, like the texturepacks/ case above, nothing reads these tables into the render pipeline yet. This is a deliberate two-step boundary: "ingest and store" (done, real, tested) vs. "resolve into a per-server palette at render time" (the same deferred piece as texturepacks/ above, and naturally solved together). forge-1_7_10/neoforge-26_1 don't implement extraction yet either — BackendConnection#sendBlockRegistry/#sendBlockTextures are on the shared interface (so any leaf can adopt them later with no protocol change), but only the 1.12.2 leaf calls them so far, matching this phase's Enigmatica-2-focused verification target.

Texture atlas + UV-mapped 3D meshes (Phase 12)

Phase 11 only fed texture-averaged colors into the 2D tile path — worker/src/mesh.rs's 3D section mesher still called the plain hand-picked palette::color_for, so the 3D viewer's "now accurate" claim in that phase's writeup wasn't actually true yet. Fixed first: mesh.rs now calls color_for_textured too, same as the 2D path.

On top of that, worker/src/atlas.rs packs every block texture the worker already downloads (see Phase 11 above) into a single RGBA PNG atlas (one native 16x16 tile per texture, deterministically laid out in a square-ish grid) plus a texture name -> normalized [u0,v0,u1,v1] rect map, cached to disk like the palette. It does its own jar download rather than sharing Phase 11's — a small, one-time, cached duplicate fetch, accepted to keep the two build paths independent. Unlike the palette's post-hoc texturepacks/ overlay, the atlas always rebuilds (and caches under its own -<pack> suffixed filename) when TEXTURE_PACK is set, since there's no cheap way to patch one tile back out of an already-packed image.

mesh.rs now emits two new per-vertex buffers alongside the existing position/normal/color ones: tile-relative uv (unbounded — a merged quad spanning N blocks has that UV coordinate run 0..N, not 0..1) and atlasRect (the same 4 floats repeated for all 4 vertices of a quad; [0,0,0,0] sentinel when the block has no atlas entry — the frontend falls back to the flat vertex color for that quad). This is a hard break in the section-mesh binary wire format (v2) — safe to do without any migration, since rendered meshes are a fully regenerable cache (MinIO + a Postgres pointer row), not a durable artifact; an old-format blob just gets silently overwritten the next time that section's dirty-chunk job runs.

The atlas PNG + UV-map JSON are uploaded once at worker startup to fixed, version-agnostic MinIO keys (atlas/current.png, atlas/current.json — matches the worker-wide-only scope already established for the palette/texturepack in Phase 11) and served by api at GET /api/atlas.png / GET /api/atlas.json (404 until a worker with ACCEPT_MINECRAFT_EULA=true has built one).

The live Babylon 3D viewer (frontend/src/public/js/mesh.js) uses a custom unlit ShaderMaterial (nearest-neighbor sampling, mipmaps disabled — bilinear/mip blending would bleed a tile's edge pixels into its atlas neighbor) whose fragment shader fract()s the tile-relative UV to repeat a single atlas tile across a merged quad, and falls back to the plain vertex color per-fragment when atlasRect is the [0,0,0,0] sentinel — this per-fragment branch is what makes the live viewer strictly more capable than the exported glTF here (see below), and is what actually makes greedy meshing (which merges many blocks into one quad) compatible with per-block texture tiling at all. Falls back entirely to the pre-Phase-12 flat-color StandardMaterial if no atlas was ever uploaded (fetch 404/error). Not yet empirically verified against a real running worker + browser (no headless-GL environment available in this dev setup) — in particular invertY's row-order assumption against atlas.rs's top-down PNG rows is unconfirmed, worth checking on first live test, same "flagged, not yet live-tested" caveat this project already carries for the Xaero waypoint format (Phase 4).

The client-side region-export mesher (voxel-mesh.js) gained the identical UV/atlasRect output (ported by hand from mesh.rs, same pattern as block-colors.js mirroring palette.rs — see the new block-textures.js mirroring block_names.rs), but gltf-export.js deliberately does not embed the atlas texture into exported glTFs — still vertex-color-only, unchanged from before this phase. Reason: standard glTF materials only support one fixed formula (baseColorTexture * baseColorFactor * COLOR_0, no branching), so the live viewer's per-fragment vertex-color fallback for untextured quads isn't expressible in a way that works in arbitrary external viewers (Blender, generic glTF web viewers) — properly supporting it needs either a reserved always-white atlas tile baked into atlas.rs or splitting merged geometry into per-material primitives, both real scope, deliberately deferred rather than shipping some faces textured and others visibly wrong.

Still not done (see the plan's phase list): real non-cube block/blockentity geometry via blockstate/model JSON parsing (BlockAssetExtractor's texture matching is still a filename convention guess, not a real model resolution) — renumbered to Phase 13 once the atlas/UV scope above turned out to be its own full phase; Thaumcraft remains the named stress test for it.

Real (non-cube) block models (Phase 13)

worker/src/models.rs is a real blockstate/model JSON resolver: given a block's registry name, it picks a variant, walks the model's parent chain (merging textures maps as it goes, child overrides win), and resolves each element's face textures down to the same atlas-lookup key atlas.rs already uses (Phase 12) — so a resolved non-cube model's faces are textured with zero atlas-side changes. ResolvedModel::is_full_cube() tells mesh.rs whether a block still belongs on the existing cube greedy-mesher (untouched — lower risk, and re-rendering a plain cube through the new per-element path would be pure overhead) or needs its own per-element geometry.

Two model sources, deliberately split:

• Vanilla: extracted from the same Mojang client jar Phase 11/12 already download (cached to disk under its own vanilla-<version>-models.json, a third instance of the same accepted "duplicate download, cached after first build" tradeoff as the palette/atlas), worker-wide.
• Modded: shipped by the mod over a new block_models WS message (mirrors Phase 11's block_textures, batched the same way) and stored per-server in a new block_models Postgres table (server_id, kind, name, json, kind distinguishing a blockstate entry from a model entry since both are keyed by resource-location-shaped strings that could otherwise collide) — fetched fresh per chunk-job by the worker, alongside the already-existing block_registry table (Phase 11 wrote it but nothing ever read it back until now). A modded model's parent can point at a vanilla base model (e.g. "minecraft:block/cross") via ModelRegistry::resolve's fallback parameter — common in practice, since plenty of modded blocks just extend a vanilla shape.

Deliberately out of scope, documented in models.rs's doc comments rather than silently dropped:

• No multipart blockstates (fences, walls, redstone wire, glass panes) — there's no way to know a block's neighbor-dependent connection state from this project's raw block-id/meta data model, so a multipart-only blockstate resolves to None and the block falls back to a flat cube, same as pre-Phase-13.
• No property-based variant selection — chunk data here only ever carries a numeric meta (legacy) or a truncated packed state id (modern, per Phase 10), never named property strings, so resolve() always picks a deterministic representative variant (the "" key if present, else alphabetically first) rather than the "correct" one for a given block's actual state.
• No per-face UV rectangle or per-variant rotation — element geometry (from/to) is real, but face texturing reuses the same tile-relative "UV span in block units" scheme the cube mesher already uses, not the model's literal declared UV rect, to avoid needing per-pixel atlas remapping/a more complex shader.
• Weighted multi-model variant lists always take the first entry (weights ignored); meta is never consulted for modded block resolution, since block_registry only carries block_id -> name (no per-state granularity) — a real, pre-existing schema constraint, not new to this phase.

A genuine pre-existing rendering bug got fixed as a side effect: compute_face_masks treated any non-zero block as solid for neighbor face-culling, so a torch (or any non-cube block) sitting next to a solid block incorrectly culled that solid block's adjacent face. Excluding non-cube-resolved voxels from the cube mesher's input array (needed anyway, for Phase 13's own correctness) fixes this for free — see mesh.rs's a_non_cube_neighbor_no_longer_incorrectly_culls_an_adjacent_solid_faces test.

Verified via real cargo build/cargo test --lib (worker, 65/65 passing, up from 52) and a real ./gradlew :forge-1_12_2:compileJava against the actual legacy ForgeGradle toolchain for the mod side. api's new models.test.ts (mirrors textures.test.ts's pattern) has since been run against a real throwaway Postgres — 4/4 passing — once Docker connectivity was fixed (the dev machine's real dockerd runs inside WSL2, not Docker Desktop; Windows' DOCKER_HOST had nothing listening on the other end until a systemd drop-in exposed it on tcp://127.0.0.1:2375). Full api (98/98) and frontend (39/39) suites both pass against real Postgres/Redis/MinIO, closing the same "Docker unavailable" gap Phase 11 had also carried (see "Running tests" below).

Running

docker compose up

api on :3000, frontend on :3001, both behind Caddy on :80. api applies its Postgres migrations on startup (see api/src/db/migrate.ts); worker consumes the mcmapper:dirty-chunks Redis stream via a consumer group (mcmapper-workers) so multiple instances split work safely — scale locally with docker compose up --scale worker=N, or run a standalone worker container on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/ Tailscale/LAN — never expose those ports publicly).

worker's RENDER_THREADS/RENDER_PROFILE (see worker/.env.example) size the rayon pool that renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per batch: RENDER_THREADS=auto uses all available cores, or set a fixed count; RENDER_PROFILE is server (default — many small batches, tuned for a many-core box) or consumer (fewer, larger batches, less scheduling overhead on fewer/faster cores).

Connecting a mod instance

Register a server via the admin panel at /admin (set MCMAPPER_ADMIN_TOKEN first — see "Admin panel" below), or seed one row by hand for scripted/headless setup:

docker compose run --rm api bun run seed

reads MCMAPPER_SEED_SERVER_NAME/MCMAPPER_SEED_SERVER_TOKEN from api/.env.example (edit those first, or override with -e). Either way, point the mod's MapperConfig#serverToken at the resulting token. Once the mod connects and sends its initial chunk backfill, tiles appear at GET /api/tiles/:serverId/:dimension/:zoom/:tileX/:tileY.png (zoom is always 0 for now — see worker/src/render/cpu.rs) and the frontend's Leaflet viewer picks them up automatically from GET /api/servers.

Admin panel

/admin (linked from the map page's header) manages the server registry: register new servers (generates their token — never admin-supplied, so it can't collide with or be guessed from anything else), and edit authMode, anonymousChatAllowed, waypointFormat, and playerPositionsVisible per server. It's gated behind a single shared secret, not a per-account role (this is a single-operator backend) — set MCMAPPER_ADMIN_TOKEN in api/'s untracked .env (see api/.env.example), restart api, then enter that same value into the panel's unlock prompt. Leaving it unset disables every /api/admin/* route (401), it does not default to open. The token is remembered in the browser's localStorage after unlocking, same pattern as the player-facing session token (see api/src/link.ts's doc comment).

Settings not exposed here yet (which map types render, dimension filtering) don't have underlying features built for them either — no point in a knob nothing reads. They'll gain admin UI alongside the feature that needs them.

Player positions (Phase 7b)

The mod sends a throttled roster of online players (player_positions over /ws, see api/src/ws-gateway.ts's doc comment) — its own playerTrackingEnabled/ playerPositionIntervalTicks config decides whether/how often it sends this at all. The backend relays it to any browser subscribed on /ws/players/:serverId (api/src/players-gateway.ts), gated per-server on the admin panel's playerPositionsVisible toggle (default on) — an independent, backend-side "should we show it" decision from the mod's own config. Not persisted (no meaningful history for a live position), just the latest roster in Redis (api/src/players.ts) so a browser tab that connects between mod flushes doesn't sit empty. The map renders players as map markers with a show/hide toggle and an online count, right above the region-export panel.

Running tests

worker's tests (cargo test, in worker/) are pure unit tests (greedy mesher, tile rasterizer, block-color palette) and need nothing running — including the gpu/hybrid backend tests (Phase 8), which request a real GPU adapter and skip themselves (rather than failing) if none is found, so cargo test still passes on a machine with no GPU. api's and frontend's (bun test, in each directory) are integration tests against real infra — start it first:

docker run -d --name mcmapper-test-pg -e POSTGRES_USER=mcmapper -e POSTGRES_PASSWORD=mcmapper -e POSTGRES_DB=mcmapper -p 15432:5432 postgres:17-alpine
docker run -d --name mcmapper-test-redis -p 16379:6379 redis:7-alpine
docker run -d --name mcmapper-test-minio -e MINIO_ROOT_USER=mcmapper -e MINIO_ROOT_PASSWORD=mcmapper-dev-only -p 19000:9000 minio/minio:latest server /data
cd api && DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper bun run migrate

then, from api/ or frontend/:

DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper \
REDIS_URL=redis://localhost:16379 \
MINIO_ENDPOINT=localhost MINIO_PORT=19000 MINIO_ACCESS_KEY=mcmapper MINIO_SECRET_KEY=mcmapper-dev-only \
bun test

Each test file creates and tears down its own server row (random token per run) so runs never collide with each other or with real dev data — see api/src/test-helpers.ts.

Running the e2e suite

bun test above exercises api and frontend independently — it never proves the browser can actually reach both through the same origin the way production's Caddy routing does (/ws* and /api/* -> api, everything else -> frontend, see Caddyfile). e2e/ is a standing Playwright suite that closes that gap: it drives a real Chromium browser against the full stack behind a small routing-equivalent proxy (e2e/proxy.ts — no caddy binary is available in this dev environment, so it isn't real Caddy, just the same three routing rules).

cd e2e
bun install
bunx playwright install chromium   # one-time, downloads the browser binary
bunx playwright test

global-setup.ts does everything by itself — no manual container/migration steps needed first (unlike the bun test section above): throwaway Postgres/Redis/MinIO containers (mcmapper-e2e-*, distinct names/ports from the bun test ones so both can run at once), migrations, a seeded server + linked account/session + a 5x5-chunk terrain footprint around the world origin, then the api/frontend/proxy processes (the api process is started with a fixed MCMAPPER_ADMIN_TOKEN for tests/admin.spec.ts to use — see config.ts's ADMIN_TOKEN). global-teardown.ts kills every spawned process and removes the containers afterward. Covers the UI flows most worth a real click-through: the marker click-to-place/edit popup (tests/markers.spec.ts, including that a marker created while linked shows up in a second browser context with the same session — the cross-device sync claim), the region-select drag + glTF export (tests/region-export.spec.ts, including a real triggered file download), the admin panel's token gate + server register/edit/delete round trip (tests/admin.spec.ts), and the player-position relay (tests/players.spec.ts — simulates a mod connection over the real /ws protocol from inside the browser context and confirms a subscribed tab renders the roster, respects the show toggle, and clears markers on an empty roster).

Attribution

See THIRD_PARTY_NOTICES.md.