MCMapper-Backend/README.md

# MCMapper-Backend

Map-render backend for [MCMapper-Mod](https://git.octoturge.com/octoturge/MCMapper-Mod). Does
all the heavy lifting a thin in-game mod shouldn't: persists world state, renders 2D tiles and
3D meshes, relays chat, and serves the web map viewer — instead of the MC server itself burning
CPU/RAM on rendering the way Bluemap/Dynmap do.

## Services

Three independently-deployable services, each its own docker-compose service:

- `api/` — ElysiaJS on Bun. The I/O layer: WS gateway for mod connections, chat relay, chunk
  store, marker/waypoint sharing, admin config, region export, tile/mesh serving.
- `worker/` — Rust. Tile rasterization and chunk meshing, consumed off a Redis dirty-chunk
  stream. Stateless — scale it with `docker compose up --scale worker=N`, or run instances on
  separate hardware pointed at the same Postgres/Redis/MinIO over a private network. Rendering
  strategy is config-selectable (`RENDER_BACKEND=cpu`/`gpu`/`hybrid`, see `worker/.env.example`)
  behind a `RenderBackend` trait (Phase 8): `cpu` (rayon, default) parallelizes across a whole
  batch of dirty chunks; `gpu` additionally offloads tile shading and per-voxel
  face-visibility extraction to a `wgpu` compute shader per chunk/section (greedy-mesh
  merge/compaction stays CPU-only regardless — sequential/branchy, not GPU-parallel-friendly);
  `hybrid` offloads only tile shading, keeping meshing on CPU. `gpu`/`hybrid` fall back to `cpu`
  automatically (logged) if no compatible GPU adapter is found — see `worker/src/render/gpu.rs`'s
  doc comment for the current known limitation (each tile/section is its own GPU dispatch, so at
  small batch sizes `cpu` currently outruns `gpu`/`hybrid` — see `cargo run --release --example
  benchmark`, in `worker/`, for real numbers on your own hardware).
- `frontend/` — ElysiaJS + Pug + Tailwind 4 + Alpine.js. The public-facing pages (map viewer,
  chat, admin panel). Stateless — no DB access, calls `api` for anything server-rendered; the
  browser's own live map/chat/tile traffic talks to `api` directly, not proxied through here.

Plus `postgres` (source chunk data, accounts, chat history, config, render-artifact metadata
pointers), `redis` (dirty-chunk queue, pub/sub, link-code TTLs), and `caddy` (reverse proxy:
`/ws` + `/api/*` → `api`, everything else → `frontend`).

Postgres/Redis are never exposed publicly — only reachable on the compose network or a private
network (VPN/Tailscale/LAN) for remote `worker` instances.

### Object storage

Rendered tile PNGs and mesh binaries live in a `mcmapper-tiles` bucket on an existing shared
MinIO instance (`devstack-minio` on octo-winsrv) rather than a per-stack `minio` container —
kept out of Postgres so backups stay free of large binaries, and any `worker` instance (local or
remote) has a shared place to write output. `api`/`worker` authenticate with a dedicated
`mcmapper` access key whose policy (`mcmapper-tiles-rw`) only grants
`GetObject`/`PutObject`/`DeleteObject`/`ListBucket` on that one bucket — it can't see or touch
anything else on the shared instance. Provisioned via:

```
mc mb myminio/mcmapper-tiles
mc admin policy create myminio mcmapper-tiles-rw mcmapper-policy.json   # see git history for the policy JSON
mc admin user add myminio mcmapper <generated secret>
mc admin policy attach myminio mcmapper-tiles-rw --user mcmapper
```

`api/.env.example`/`worker/.env.example` point `MINIO_ENDPOINT`/`MINIO_PORT` at that instance's
LAN address; swap to the public gateway (`s3.octoturge.com:443`, `MINIO_USE_SSL=true`) if a
stack isn't on the same LAN. `MINIO_SECRET_KEY` is a real credential and is deliberately **not**
committed — set it in an untracked `./api/.env` / `./worker/.env` (docker-compose layers those on
top of the tracked `.env.example`, see `docker-compose.yml`).

### Real block textures (Phase 11)

`worker/src/palette.rs`'s hand-picked color table is now a *fallback*, not the only source of
2D-tile colors. Set `ACCEPT_MINECRAFT_EULA=true` (`worker/.env.example`, off by default — mirrors
BlueMap's `accept-download`) and the worker downloads Minecraft's official client jar directly
from Mojang's own public `launchermeta`/`piston-meta`/`piston-data` endpoints (same source the
real launcher uses — no redistribution, so no licensing issue) on first startup, averages every
`assets/minecraft/textures/block(s)/*.png` into a representative color, and caches the result to
disk (`TEXTURE_CACHE_DIR`, default `./cache`) so it isn't re-downloaded every restart.
`MC_TEXTURE_VERSION` (default `1.12.2`) picks which version's jar to pull from — this project's
priority targets (1.7.10/1.12.2) share the pre-1.13 `textures/blocks/` (plural) naming, which
`worker/src/textures.rs` checks alongside the modern `textures/block/` path.

A handful of blocks (grass block top, leaves, water, lava) are deliberately *excluded* from the
texture-averaged path (`worker/src/block_names.rs`'s doc comment) and keep their hand-picked
color: their real textures are either biome-tinted at runtime (grayscale in the raw file) or
animated/transparent frame strips, so averaging the raw asset would produce a wrong color, not
just an approximate one.

**`texturepacks/<name>/`** (flat `*.png` files, mirrors Dynmap) lets an operator override the
downloaded defaults — set `TEXTURE_PACK=<name>` and its colors are layered on top of the vanilla
palette at startup. Each server also has an admin-configurable `texturePack` field (`/admin`,
`servers.texture_pack` — same shape as `waypointFormat`) recording *which* pack an operator wants
per server — **but render-time application is currently worker-wide only, via `TEXTURE_PACK`, not
yet resolved per-server from that column.** True per-server resolution needs the render pipeline
to thread a server-scoped palette through `main.rs`'s batch loop instead of one process-wide
`OnceLock` (`worker/src/render/mod.rs`) — a real architectural change, deferred rather than rushed.

**Modded blocks**: Forge doesn't split client/server jars, so a modded server's own classpath
already has every loaded mod's texture assets, just unused server-side. The `forge-1_12_2` mod
leaf (Enigmatica 2, this project's primary target) extracts them once at startup
(`BlockAssetExtractor`, best-effort: guesses each block's texture by its registry-name path
segment, not a real blockstate/model JSON resolution — that's Phase 13's job, see below) and ships two new
WS messages after connecting: `block_registry` (numeric id -> registry name, needed since a
numeric `blockId` alone is meaningless without the mod list that assigned it) and `block_textures`
(the extracted PNGs, batched). The api stores both (`api/src/textures.ts`: registry rows in a new
`block_registry` table, texture PNGs in the shared MinIO bucket with pointer rows in
`block_textures`) — **but, like the `texturepacks/` case above, nothing reads these tables into
the render pipeline yet.** This is a deliberate two-step boundary: "ingest and store" (done, real,
tested) vs. "resolve into a per-server palette at render time" (the same deferred piece as
`texturepacks/` above, and naturally solved together). `forge-1_7_10`/`neoforge-26_1` don't
implement extraction yet either — `BackendConnection#sendBlockRegistry`/`#sendBlockTextures` are
on the shared interface (so any leaf can adopt them later with no protocol change), but only the
1.12.2 leaf calls them so far, matching this phase's Enigmatica-2-focused verification target.

### Texture atlas + UV-mapped 3D meshes (Phase 12)

Phase 11 only fed texture-averaged colors into the 2D tile path — `worker/src/mesh.rs`'s 3D
section mesher still called the plain hand-picked `palette::color_for`, so the 3D viewer's "now
accurate" claim in that phase's writeup wasn't actually true yet. Fixed first: `mesh.rs` now calls
`color_for_textured` too, same as the 2D path.

On top of that, `worker/src/atlas.rs` packs every block texture the worker already downloads (see
Phase 11 above) into a single RGBA PNG atlas (one native `16x16` tile per texture, deterministically
laid out in a square-ish grid) plus a `texture name -> normalized [u0,v0,u1,v1]` rect map, cached
to disk like the palette. It does its own jar download rather than sharing Phase 11's — a small,
one-time, cached duplicate fetch, accepted to keep the two build paths independent. Unlike the
palette's post-hoc `texturepacks/` overlay, the atlas always rebuilds (and caches under its own
`-<pack>` suffixed filename) when `TEXTURE_PACK` is set, since there's no cheap way to patch one
tile back out of an already-packed image.

`mesh.rs` now emits two new per-vertex buffers alongside the existing position/normal/color ones:
tile-relative `uv` (unbounded — a merged quad spanning N blocks has that UV coordinate run 0..N,
not 0..1) and `atlasRect` (the same 4 floats repeated for all 4 vertices of a quad; `[0,0,0,0]`
sentinel when the block has no atlas entry — the frontend falls back to the flat vertex color for
that quad). This is a hard break in the section-mesh binary wire format (v2) — safe to do without
any migration, since rendered meshes are a fully regenerable cache (MinIO + a Postgres pointer
row), not a durable artifact; an old-format blob just gets silently overwritten the next time that
section's dirty-chunk job runs.

The atlas PNG + UV-map JSON are uploaded once at worker startup to fixed, version-agnostic MinIO
keys (`atlas/current.png`, `atlas/current.json` — matches the worker-wide-only scope already
established for the palette/texturepack in Phase 11) and served by `api` at `GET /api/atlas.png`
/ `GET /api/atlas.json` (404 until a worker with `ACCEPT_MINECRAFT_EULA=true` has built one).

The live Babylon 3D viewer (`frontend/src/public/js/mesh.js`) uses a custom unlit `ShaderMaterial`
(nearest-neighbor sampling, mipmaps disabled — bilinear/mip blending would bleed a tile's edge
pixels into its atlas neighbor) whose fragment shader `fract()`s the tile-relative UV to repeat a
single atlas tile across a merged quad, and falls back to the plain vertex color per-fragment when
`atlasRect` is the `[0,0,0,0]` sentinel — this per-fragment branch is what makes the live viewer
strictly more capable than the exported glTF here (see below), and is what actually makes greedy
meshing (which merges many blocks into one quad) compatible with per-block texture tiling at all.
Falls back entirely to the pre-Phase-12 flat-color `StandardMaterial` if no atlas was ever
uploaded (fetch 404/error). **Not yet empirically verified against a real running worker + browser**
(no headless-GL environment available in this dev setup) — in particular `invertY`'s row-order
assumption against `atlas.rs`'s top-down PNG rows is unconfirmed, worth checking on first live
test, same "flagged, not yet live-tested" caveat this project already carries for the Xaero
waypoint format (Phase 4).

The client-side region-export mesher (`voxel-mesh.js`) gained the identical UV/atlasRect output
(ported by hand from `mesh.rs`, same pattern as `block-colors.js` mirroring `palette.rs` — see the
new `block-textures.js` mirroring `block_names.rs`), but **`gltf-export.js` deliberately does not
embed the atlas texture into exported glTFs** — still vertex-color-only, unchanged from before this
phase. Reason: standard glTF materials only support one fixed formula (`baseColorTexture *
baseColorFactor * COLOR_0`, no branching), so the live viewer's per-fragment vertex-color fallback
for untextured quads isn't expressible in a way that works in arbitrary external viewers (Blender,
generic glTF web viewers) — properly supporting it needs either a reserved always-white atlas tile
baked into `atlas.rs` or splitting merged geometry into per-material primitives, both real scope,
deliberately deferred rather than shipping some faces textured and others visibly wrong.

**Still not done** (see the plan's phase list): real non-cube block/blockentity geometry via
blockstate/model JSON parsing (`BlockAssetExtractor`'s texture matching is still a filename
convention guess, not a real model resolution) — renumbered to **Phase 13** once the atlas/UV
scope above turned out to be its own full phase; Thaumcraft remains the named stress test for it.

### Real (non-cube) block models (Phase 13)

`worker/src/models.rs` is a real blockstate/model JSON resolver: given a block's registry name, it
picks a variant, walks the model's `parent` chain (merging `textures` maps as it goes, child
overrides win), and resolves each element's face textures down to the same atlas-lookup key
`atlas.rs` already uses (Phase 12) — so a resolved non-cube model's faces are textured with zero
atlas-side changes. `ResolvedModel::is_full_cube()` tells `mesh.rs` whether a block still belongs
on the existing cube greedy-mesher (untouched — lower risk, and re-rendering a plain cube through
the new per-element path would be pure overhead) or needs its own per-element geometry.

Two model sources, deliberately split:
- **Vanilla**: extracted from the same Mojang client jar Phase 11/12 already download (cached to
  disk under its own `vanilla-<version>-models.json`, a third instance of the same accepted
  "duplicate download, cached after first build" tradeoff as the palette/atlas), worker-wide.
- **Modded**: shipped by the mod over a new `block_models` WS message (mirrors Phase 11's
  `block_textures`, batched the same way) and stored per-server in a new `block_models` Postgres
  table (`server_id, kind, name, json`, `kind` distinguishing a blockstate entry from a model entry
  since both are keyed by resource-location-shaped strings that could otherwise collide) — fetched
  fresh per chunk-job by the worker, alongside the already-existing `block_registry` table (Phase
  11 wrote it but nothing ever read it back until now). A modded model's `parent` can point at a
  vanilla base model (e.g. `"minecraft:block/cross"`) via `ModelRegistry::resolve`'s `fallback`
  parameter — common in practice, since plenty of modded blocks just extend a vanilla shape.

**Deliberately out of scope**, documented in `models.rs`'s doc comments rather than silently
dropped:
- **No `multipart` blockstates** (fences, walls, redstone wire, glass panes) — there's no way to
  know a block's neighbor-dependent connection state from this project's raw block-id/meta data
  model, so a multipart-only blockstate resolves to `None` and the block falls back to a flat cube,
  same as pre-Phase-13.
- **No property-based variant selection** — chunk data here only ever carries a numeric `meta`
  (legacy) or a truncated packed state id (modern, per Phase 10), never named property strings, so
  `resolve()` always picks a deterministic representative variant (the `""` key if present, else
  alphabetically first) rather than the "correct" one for a given block's actual state.
- **No per-face UV rectangle or per-variant rotation** — element geometry (`from`/`to`) is real,
  but face texturing reuses the same tile-relative "UV span in block units" scheme the cube mesher
  already uses, not the model's literal declared UV rect, to avoid needing per-pixel atlas
  remapping/a more complex shader.
- Weighted multi-model variant lists always take the first entry (weights ignored); `meta` is never
  consulted for modded block resolution, since `block_registry` only carries `block_id -> name`
  (no per-state granularity) — a real, pre-existing schema constraint, not new to this phase.

A genuine pre-existing rendering bug got fixed as a side effect: `compute_face_masks` treated *any*
non-zero block as solid for neighbor face-culling, so a torch (or any non-cube block) sitting next
to a solid block incorrectly culled that solid block's adjacent face. Excluding non-cube-resolved
voxels from the cube mesher's input array (needed anyway, for Phase 13's own correctness) fixes
this for free — see `mesh.rs`'s `a_non_cube_neighbor_no_longer_incorrectly_culls_an_adjacent_solid_faces`
test.

Verified via real `cargo build`/`cargo test --lib` (worker, 65/65 passing, up from 52) and a real
`./gradlew :forge-1_12_2:compileJava` against the actual legacy ForgeGradle toolchain for the mod
side. `api`'s new `models.test.ts` (mirrors `textures.test.ts`'s pattern) has since been run against
a real throwaway Postgres — 4/4 passing — once Docker connectivity was fixed (the dev machine's real
`dockerd` runs inside WSL2, not Docker Desktop; Windows' `DOCKER_HOST` had nothing listening on the
other end until a systemd drop-in exposed it on `tcp://127.0.0.1:2375`). Full `api` (98/98) and
`frontend` (39/39) suites both pass against real Postgres/Redis/MinIO, closing the same "Docker
unavailable" gap Phase 11 had also carried (see "Running tests" below).

## Running

```
docker compose up
```

`api` on :3000, `frontend` on :3001, both behind Caddy on :80. `api` applies its Postgres
migrations on startup (see `api/src/db/migrate.ts`); `worker` consumes the `mcmapper:dirty-chunks`
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely —
scale locally with `docker compose up --scale worker=N`, or run a standalone `worker` container
on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/
Tailscale/LAN — never expose those ports publicly).

`worker`'s `RENDER_THREADS`/`RENDER_PROFILE` (see `worker/.env.example`) size the rayon pool that
renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per
batch: `RENDER_THREADS=auto` uses all available cores, or set a fixed count; `RENDER_PROFILE`
is `server` (default — many small batches, tuned for a many-core box) or `consumer` (fewer,
larger batches, less scheduling overhead on fewer/faster cores).

### Connecting a mod instance

Register a server via the admin panel at `/admin` (set `MCMAPPER_ADMIN_TOKEN` first — see
"Admin panel" below), or seed one row by hand for scripted/headless setup:

```
docker compose run --rm api bun run seed
```

reads `MCMAPPER_SEED_SERVER_NAME`/`MCMAPPER_SEED_SERVER_TOKEN` from `api/.env.example` (edit
those first, or override with `-e`). Either way, point the mod's `MapperConfig#serverToken` at
the resulting token. Once the mod connects and sends its initial chunk backfill, tiles appear at
`GET /api/tiles/:serverId/:dimension/:zoom/:tileX/:tileY.png` (zoom is always `0` for now — see
`worker/src/render/cpu.rs`) and the frontend's Leaflet viewer picks them up automatically from
`GET /api/servers`.

### Admin panel

`/admin` (linked from the map page's header) manages the server registry: register new servers
(generates their token — never admin-supplied, so it can't collide with or be guessed from
anything else), and edit `authMode`, `anonymousChatAllowed`, `waypointFormat`, and
`playerPositionsVisible` per server. It's gated behind a single shared secret, not a per-account
role (this is a single-operator backend) — set `MCMAPPER_ADMIN_TOKEN` in `api/`'s untracked
`.env` (see `api/.env.example`), restart `api`, then enter that same value into the panel's
unlock prompt. Leaving it unset disables every `/api/admin/*` route (401), it does not default to
open. The token is remembered in the browser's `localStorage` after unlocking, same pattern as
the player-facing session token (see `api/src/link.ts`'s doc comment).

Settings not exposed here yet (which map types render, dimension filtering) don't have underlying
features built for them either — no point in a knob nothing reads. They'll gain admin UI
alongside the feature that needs them.

### Player positions (Phase 7b)

The mod sends a throttled roster of online players (`player_positions` over `/ws`, see
`api/src/ws-gateway.ts`'s doc comment) — its own `playerTrackingEnabled`/
`playerPositionIntervalTicks` config decides whether/how often it sends this at all. The backend
relays it to any browser subscribed on `/ws/players/:serverId` (`api/src/players-gateway.ts`),
gated per-server on the admin panel's `playerPositionsVisible` toggle (default on) — an
independent, backend-side "should we show it" decision from the mod's own config. Not persisted
(no meaningful history for a live position), just the latest roster in Redis (`api/src/players.ts`)
so a browser tab that connects between mod flushes doesn't sit empty. The map renders players as
map markers with a `show`/hide toggle and an online count, right above the region-export panel.

## Running tests

`worker`'s tests (`cargo test`, in `worker/`) are pure unit tests (greedy mesher, tile
rasterizer, block-color palette) and need nothing running — including the `gpu`/`hybrid` backend
tests (Phase 8), which request a real GPU adapter and skip themselves (rather than failing) if
none is found, so `cargo test` still passes on a machine with no GPU. `api`'s and `frontend`'s (`bun test`,
in each directory) are integration tests against real infra — start it first:

```
docker run -d --name mcmapper-test-pg -e POSTGRES_USER=mcmapper -e POSTGRES_PASSWORD=mcmapper -e POSTGRES_DB=mcmapper -p 15432:5432 postgres:17-alpine
docker run -d --name mcmapper-test-redis -p 16379:6379 redis:7-alpine
docker run -d --name mcmapper-test-minio -e MINIO_ROOT_USER=mcmapper -e MINIO_ROOT_PASSWORD=mcmapper-dev-only -p 19000:9000 minio/minio:latest server /data
cd api && DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper bun run migrate
```

then, from `api/` or `frontend/`:

```
DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper \
REDIS_URL=redis://localhost:16379 \
MINIO_ENDPOINT=localhost MINIO_PORT=19000 MINIO_ACCESS_KEY=mcmapper MINIO_SECRET_KEY=mcmapper-dev-only \
bun test
```

Each test file creates and tears down its own server row (random token per run) so runs never
collide with each other or with real dev data — see `api/src/test-helpers.ts`.

## Running the e2e suite

`bun test` above exercises `api` and `frontend` independently — it never proves the browser can
actually reach both through the same origin the way production's Caddy routing does (`/ws*` and
`/api/*` -> `api`, everything else -> `frontend`, see `Caddyfile`). `e2e/` is a standing
Playwright suite that closes that gap: it drives a real Chromium browser against the full stack
behind a small routing-equivalent proxy (`e2e/proxy.ts` — no `caddy` binary is available in this
dev environment, so it isn't real Caddy, just the same three routing rules).

```
cd e2e
bun install
bunx playwright install chromium   # one-time, downloads the browser binary
bunx playwright test
```

`global-setup.ts` does everything by itself — no manual container/migration steps needed first
(unlike the `bun test` section above): throwaway Postgres/Redis/MinIO containers
(`mcmapper-e2e-*`, distinct names/ports from the `bun test` ones so both can run at once),
migrations, a seeded server + linked account/session + a 5x5-chunk terrain footprint around the
world origin, then the `api`/`frontend`/proxy processes (the `api` process is started with a
fixed `MCMAPPER_ADMIN_TOKEN` for `tests/admin.spec.ts` to use — see `config.ts`'s `ADMIN_TOKEN`).
`global-teardown.ts` kills every spawned process and removes the containers afterward. Covers the
UI flows most worth a real click-through: the marker click-to-place/edit popup
(`tests/markers.spec.ts`, including that a marker created while linked shows up in a second
browser context with the same session — the cross-device sync claim), the region-select drag +
glTF export (`tests/region-export.spec.ts`, including a real triggered file download), the
admin panel's token gate + server register/edit/delete round trip (`tests/admin.spec.ts`), and
the player-position relay (`tests/players.spec.ts` — simulates a mod connection over the real
`/ws` protocol from inside the browser context and confirms a subscribed tab renders the roster,
respects the `show` toggle, and clears markers on an empty roster).

## Attribution

See `THIRD_PARTY_NOTICES.md`.