MCMapper-Backend/README.md

# MCMapper-Backend

Map-render backend for [MCMapper-Mod](https://git.octoturge.com/octoturge/MCMapper-Mod). Does
all the heavy lifting a thin in-game mod shouldn't: persists world state, renders 2D tiles and
3D meshes, relays chat, and serves the web map viewer — instead of the MC server itself burning
CPU/RAM on rendering the way Bluemap/Dynmap do.

## Services

Three independently-deployable services, each its own docker-compose service:

- `api/` — ElysiaJS on Bun. The I/O layer: WS gateway for mod connections, chat relay, chunk
  store, marker/waypoint sharing, admin config, region export, tile/mesh serving.
- `worker/` — Rust. Tile rasterization and chunk meshing, consumed off a Redis dirty-chunk
  stream. Stateless — scale it with `docker compose up --scale worker=N`, or run instances on
  separate hardware pointed at the same Postgres/Redis/MinIO over a private network. Rendering
  strategy is config-selectable (`RENDER_BACKEND=cpu`/`gpu`/`hybrid`, see `worker/.env.example`)
  behind a `RenderBackend` trait (Phase 8): `cpu` (rayon, default) parallelizes across a whole
  batch of dirty chunks; `gpu` additionally offloads tile shading and per-voxel
  face-visibility extraction to a `wgpu` compute shader per chunk/section (greedy-mesh
  merge/compaction stays CPU-only regardless — sequential/branchy, not GPU-parallel-friendly);
  `hybrid` offloads only tile shading, keeping meshing on CPU. `gpu`/`hybrid` fall back to `cpu`
  automatically (logged) if no compatible GPU adapter is found — see `worker/src/render/gpu.rs`'s
  doc comment for the current known limitation (each tile/section is its own GPU dispatch, so at
  small batch sizes `cpu` currently outruns `gpu`/`hybrid` — see `cargo run --release --example
  benchmark`, in `worker/`, for real numbers on your own hardware).
- `frontend/` — ElysiaJS + Pug + Tailwind 4 + Alpine.js. The public-facing pages (map viewer,
  chat, admin panel). Stateless — no DB access, calls `api` for anything server-rendered; the
  browser's own live map/chat/tile traffic talks to `api` directly, not proxied through here.

Plus `postgres` (source chunk data, accounts, chat history, config, render-artifact metadata
pointers), `redis` (dirty-chunk queue, pub/sub, link-code TTLs), and `caddy` (reverse proxy:
`/ws` + `/api/*` → `api`, everything else → `frontend`).

Postgres/Redis are never exposed publicly — only reachable on the compose network or a private
network (VPN/Tailscale/LAN) for remote `worker` instances.

### Object storage

Rendered tile PNGs and mesh binaries live in a `mcmapper-tiles` bucket on an existing shared
MinIO instance (`devstack-minio` on octo-winsrv) rather than a per-stack `minio` container —
kept out of Postgres so backups stay free of large binaries, and any `worker` instance (local or
remote) has a shared place to write output. `api`/`worker` authenticate with a dedicated
`mcmapper` access key whose policy (`mcmapper-tiles-rw`) only grants
`GetObject`/`PutObject`/`DeleteObject`/`ListBucket` on that one bucket — it can't see or touch
anything else on the shared instance. Provisioned via:

```
mc mb myminio/mcmapper-tiles
mc admin policy create myminio mcmapper-tiles-rw mcmapper-policy.json   # see git history for the policy JSON
mc admin user add myminio mcmapper <generated secret>
mc admin policy attach myminio mcmapper-tiles-rw --user mcmapper
```

`api/.env.example`/`worker/.env.example` point `MINIO_ENDPOINT`/`MINIO_PORT` at that instance's
LAN address; swap to the public gateway (`s3.octoturge.com:443`, `MINIO_USE_SSL=true`) if a
stack isn't on the same LAN. `MINIO_SECRET_KEY` is a real credential and is deliberately **not**
committed — set it in an untracked `./api/.env` / `./worker/.env` (docker-compose layers those on
top of the tracked `.env.example`, see `docker-compose.yml`).

### Real block textures (Phase 11)

`worker/src/palette.rs`'s hand-picked color table is now a *fallback*, not the only source of
2D-tile colors. Set `ACCEPT_MINECRAFT_EULA=true` (`worker/.env.example`, off by default — mirrors
BlueMap's `accept-download`) and the worker downloads Minecraft's official client jar directly
from Mojang's own public `launchermeta`/`piston-meta`/`piston-data` endpoints (same source the
real launcher uses — no redistribution, so no licensing issue) on first startup, averages every
`assets/minecraft/textures/block(s)/*.png` into a representative color, and caches the result to
disk (`TEXTURE_CACHE_DIR`, default `./cache`) so it isn't re-downloaded every restart.
`MC_TEXTURE_VERSION` (default `1.12.2`) picks which version's jar to pull from — this project's
priority targets (1.7.10/1.12.2) share the pre-1.13 `textures/blocks/` (plural) naming, which
`worker/src/textures.rs` checks alongside the modern `textures/block/` path.

A handful of blocks (grass block top, leaves, water, lava) are deliberately *excluded* from the
texture-averaged path (`worker/src/block_names.rs`'s doc comment) and keep their hand-picked
color: their real textures are either biome-tinted at runtime (grayscale in the raw file) or
animated/transparent frame strips, so averaging the raw asset would produce a wrong color, not
just an approximate one.

**`texturepacks/<name>/`** (flat `*.png` files, mirrors Dynmap) lets an operator override the
downloaded defaults — set `TEXTURE_PACK=<name>` and its colors are layered on top of the vanilla
palette at startup. Each server also has an admin-configurable `texturePack` field (`/admin`,
`servers.texture_pack` — same shape as `waypointFormat`) recording *which* pack an operator wants
per server — **but render-time application is currently worker-wide only, via `TEXTURE_PACK`, not
yet resolved per-server from that column.** True per-server resolution needs the render pipeline
to thread a server-scoped palette through `main.rs`'s batch loop instead of one process-wide
`OnceLock` (`worker/src/render/mod.rs`) — a real architectural change, deferred rather than rushed.

**Modded blocks**: Forge doesn't split client/server jars, so a modded server's own classpath
already has every loaded mod's texture assets, just unused server-side. The `forge-1_12_2` mod
leaf (Enigmatica 2, this project's primary target) extracts them once at startup
(`BlockAssetExtractor`, best-effort: guesses each block's texture by its registry-name path
segment, not a real blockstate/model JSON resolution — that's Phase 12's job) and ships two new
WS messages after connecting: `block_registry` (numeric id -> registry name, needed since a
numeric `blockId` alone is meaningless without the mod list that assigned it) and `block_textures`
(the extracted PNGs, batched). The api stores both (`api/src/textures.ts`: registry rows in a new
`block_registry` table, texture PNGs in the shared MinIO bucket with pointer rows in
`block_textures`) — **but, like the `texturepacks/` case above, nothing reads these tables into
the render pipeline yet.** This is a deliberate two-step boundary: "ingest and store" (done, real,
tested) vs. "resolve into a per-server palette at render time" (the same deferred piece as
`texturepacks/` above, and naturally solved together). `forge-1_7_10`/`neoforge-26_1` don't
implement extraction yet either — `BackendConnection#sendBlockRegistry`/`#sendBlockTextures` are
on the shared interface (so any leaf can adopt them later with no protocol change), but only the
1.12.2 leaf calls them so far, matching this phase's Enigmatica-2-focused verification target.

## Running

```
docker compose up
```

`api` on :3000, `frontend` on :3001, both behind Caddy on :80. `api` applies its Postgres
migrations on startup (see `api/src/db/migrate.ts`); `worker` consumes the `mcmapper:dirty-chunks`
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely —
scale locally with `docker compose up --scale worker=N`, or run a standalone `worker` container
on separate hardware pointed at the same Postgres/Redis/MinIO over a private network (VPN/
Tailscale/LAN — never expose those ports publicly).

`worker`'s `RENDER_THREADS`/`RENDER_PROFILE` (see `worker/.env.example`) size the rayon pool that
renders a batch of dirty chunks in parallel and how many chunks are pulled off the stream per
batch: `RENDER_THREADS=auto` uses all available cores, or set a fixed count; `RENDER_PROFILE`
is `server` (default — many small batches, tuned for a many-core box) or `consumer` (fewer,
larger batches, less scheduling overhead on fewer/faster cores).

### Connecting a mod instance

Register a server via the admin panel at `/admin` (set `MCMAPPER_ADMIN_TOKEN` first — see
"Admin panel" below), or seed one row by hand for scripted/headless setup:

```
docker compose run --rm api bun run seed
```

reads `MCMAPPER_SEED_SERVER_NAME`/`MCMAPPER_SEED_SERVER_TOKEN` from `api/.env.example` (edit
those first, or override with `-e`). Either way, point the mod's `MapperConfig#serverToken` at
the resulting token. Once the mod connects and sends its initial chunk backfill, tiles appear at
`GET /api/tiles/:serverId/:dimension/:zoom/:tileX/:tileY.png` (zoom is always `0` for now — see
`worker/src/render/cpu.rs`) and the frontend's Leaflet viewer picks them up automatically from
`GET /api/servers`.

### Admin panel

`/admin` (linked from the map page's header) manages the server registry: register new servers
(generates their token — never admin-supplied, so it can't collide with or be guessed from
anything else), and edit `authMode`, `anonymousChatAllowed`, `waypointFormat`, and
`playerPositionsVisible` per server. It's gated behind a single shared secret, not a per-account
role (this is a single-operator backend) — set `MCMAPPER_ADMIN_TOKEN` in `api/`'s untracked
`.env` (see `api/.env.example`), restart `api`, then enter that same value into the panel's
unlock prompt. Leaving it unset disables every `/api/admin/*` route (401), it does not default to
open. The token is remembered in the browser's `localStorage` after unlocking, same pattern as
the player-facing session token (see `api/src/link.ts`'s doc comment).

Settings not exposed here yet (which map types render, dimension filtering) don't have underlying
features built for them either — no point in a knob nothing reads. They'll gain admin UI
alongside the feature that needs them.

### Player positions (Phase 7b)

The mod sends a throttled roster of online players (`player_positions` over `/ws`, see
`api/src/ws-gateway.ts`'s doc comment) — its own `playerTrackingEnabled`/
`playerPositionIntervalTicks` config decides whether/how often it sends this at all. The backend
relays it to any browser subscribed on `/ws/players/:serverId` (`api/src/players-gateway.ts`),
gated per-server on the admin panel's `playerPositionsVisible` toggle (default on) — an
independent, backend-side "should we show it" decision from the mod's own config. Not persisted
(no meaningful history for a live position), just the latest roster in Redis (`api/src/players.ts`)
so a browser tab that connects between mod flushes doesn't sit empty. The map renders players as
map markers with a `show`/hide toggle and an online count, right above the region-export panel.

## Running tests

`worker`'s tests (`cargo test`, in `worker/`) are pure unit tests (greedy mesher, tile
rasterizer, block-color palette) and need nothing running — including the `gpu`/`hybrid` backend
tests (Phase 8), which request a real GPU adapter and skip themselves (rather than failing) if
none is found, so `cargo test` still passes on a machine with no GPU. `api`'s and `frontend`'s (`bun test`,
in each directory) are integration tests against real infra — start it first:

```
docker run -d --name mcmapper-test-pg -e POSTGRES_USER=mcmapper -e POSTGRES_PASSWORD=mcmapper -e POSTGRES_DB=mcmapper -p 15432:5432 postgres:17-alpine
docker run -d --name mcmapper-test-redis -p 16379:6379 redis:7-alpine
docker run -d --name mcmapper-test-minio -e MINIO_ROOT_USER=mcmapper -e MINIO_ROOT_PASSWORD=mcmapper-dev-only -p 19000:9000 minio/minio:latest server /data
cd api && DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper bun run migrate
```

then, from `api/` or `frontend/`:

```
DATABASE_URL=postgres://mcmapper:mcmapper@localhost:15432/mcmapper \
REDIS_URL=redis://localhost:16379 \
MINIO_ENDPOINT=localhost MINIO_PORT=19000 MINIO_ACCESS_KEY=mcmapper MINIO_SECRET_KEY=mcmapper-dev-only \
bun test
```

Each test file creates and tears down its own server row (random token per run) so runs never
collide with each other or with real dev data — see `api/src/test-helpers.ts`.

## Running the e2e suite

`bun test` above exercises `api` and `frontend` independently — it never proves the browser can
actually reach both through the same origin the way production's Caddy routing does (`/ws*` and
`/api/*` -> `api`, everything else -> `frontend`, see `Caddyfile`). `e2e/` is a standing
Playwright suite that closes that gap: it drives a real Chromium browser against the full stack
behind a small routing-equivalent proxy (`e2e/proxy.ts` — no `caddy` binary is available in this
dev environment, so it isn't real Caddy, just the same three routing rules).

```
cd e2e
bun install
bunx playwright install chromium   # one-time, downloads the browser binary
bunx playwright test
```

`global-setup.ts` does everything by itself — no manual container/migration steps needed first
(unlike the `bun test` section above): throwaway Postgres/Redis/MinIO containers
(`mcmapper-e2e-*`, distinct names/ports from the `bun test` ones so both can run at once),
migrations, a seeded server + linked account/session + a 5x5-chunk terrain footprint around the
world origin, then the `api`/`frontend`/proxy processes (the `api` process is started with a
fixed `MCMAPPER_ADMIN_TOKEN` for `tests/admin.spec.ts` to use — see `config.ts`'s `ADMIN_TOKEN`).
`global-teardown.ts` kills every spawned process and removes the containers afterward. Covers the
UI flows most worth a real click-through: the marker click-to-place/edit popup
(`tests/markers.spec.ts`, including that a marker created while linked shows up in a second
browser context with the same session — the cross-device sync claim), the region-select drag +
glTF export (`tests/region-export.spec.ts`, including a real triggered file download), the
admin panel's token gate + server register/edit/delete round trip (`tests/admin.spec.ts`), and
the player-position relay (`tests/players.spec.ts` — simulates a mod connection over the real
`/ws` protocol from inside the browser context and confirms a subscribed tab renders the roster,
respects the `show` toggle, and clears markers on an empty roster).

## Attribution

See `THIRD_PARTY_NOTICES.md`.