Phase 1: chunk store, tile rendering pipeline, and Leaflet viewer
api: WS gateway with token auth (Postgres-backed servers table, seeded via `bun run seed`), column-granularity chunk store (hand-written SQL migrations, no drizzle-kit CLI — its config loader needs esbuild, which doesn't install cleanly here), dirty-chunk Redis stream producer with per-flush dedup, and a tile-serving route. worker: consumes the dirty-chunk stream via a proper consumer group, rasterizes each chunk's columns into a single-resolution top-down PNG (static pre-Flattening block-id palette), uploads to MinIO, and upserts the tile pointer. frontend: barebones Leaflet 2D viewer (CRS.Simple, one native zoom level) wired to /api/servers and /api/tiles. Object storage: tiles live in a dedicated `mcmapper-tiles` bucket on the existing shared MinIO instance (devstack-minio on octo-winsrv) instead of a per-stack container, via a scoped access key limited to that one bucket — see README's "Object storage" section. MINIO_SECRET_KEY is real and is deliberately not committed; docker-compose layers an untracked .env over .env.example for it. Full pipeline verified end-to-end against live containers: WS auth -> Postgres upsert -> deduped Redis dirty-chunk event -> worker rasterize -> MinIO upload -> tile fetch through the api route, including from the actual mod-side WS client (see MCMapper-Mod's matching commit).
This commit is contained in:
@@ -22,13 +22,34 @@ Three independently-deployable services, each its own docker-compose service:
|
||||
browser's own live map/chat/tile traffic talks to `api` directly, not proxied through here.
|
||||
|
||||
Plus `postgres` (source chunk data, accounts, chat history, config, render-artifact metadata
|
||||
pointers), `redis` (dirty-chunk queue, pub/sub, link-code TTLs), `minio` (S3-compatible object
|
||||
storage for the actual rendered tile PNGs and mesh binaries — kept out of Postgres so backups
|
||||
stay free of large binaries and remote `worker` instances have a shared place to write output),
|
||||
and `caddy` (reverse proxy: `/ws` + `/api/*` → `api`, everything else → `frontend`).
|
||||
pointers), `redis` (dirty-chunk queue, pub/sub, link-code TTLs), and `caddy` (reverse proxy:
|
||||
`/ws` + `/api/*` → `api`, everything else → `frontend`).
|
||||
|
||||
Postgres/Redis/MinIO are never exposed publicly — only reachable on the compose network or a
|
||||
private network (VPN/Tailscale/LAN) for remote `worker` instances.
|
||||
Postgres/Redis are never exposed publicly — only reachable on the compose network or a private
|
||||
network (VPN/Tailscale/LAN) for remote `worker` instances.
|
||||
|
||||
### Object storage
|
||||
|
||||
Rendered tile PNGs and mesh binaries live in a `mcmapper-tiles` bucket on an existing shared
|
||||
MinIO instance (`devstack-minio` on octo-winsrv) rather than a per-stack `minio` container —
|
||||
kept out of Postgres so backups stay free of large binaries, and any `worker` instance (local or
|
||||
remote) has a shared place to write output. `api`/`worker` authenticate with a dedicated
|
||||
`mcmapper` access key whose policy (`mcmapper-tiles-rw`) only grants
|
||||
`GetObject`/`PutObject`/`DeleteObject`/`ListBucket` on that one bucket — it can't see or touch
|
||||
anything else on the shared instance. Provisioned via:
|
||||
|
||||
```
|
||||
mc mb myminio/mcmapper-tiles
|
||||
mc admin policy create myminio mcmapper-tiles-rw mcmapper-policy.json # see git history for the policy JSON
|
||||
mc admin user add myminio mcmapper <generated secret>
|
||||
mc admin policy attach myminio mcmapper-tiles-rw --user mcmapper
|
||||
```
|
||||
|
||||
`api/.env.example`/`worker/.env.example` point `MINIO_ENDPOINT`/`MINIO_PORT` at that instance's
|
||||
LAN address; swap to the public gateway (`s3.octoturge.com:443`, `MINIO_USE_SSL=true`) if a
|
||||
stack isn't on the same LAN. `MINIO_SECRET_KEY` is a real credential and is deliberately **not**
|
||||
committed — set it in an untracked `./api/.env` / `./worker/.env` (docker-compose layers those on
|
||||
top of the tracked `.env.example`, see `docker-compose.yml`).
|
||||
|
||||
## Running
|
||||
|
||||
@@ -36,8 +57,24 @@ private network (VPN/Tailscale/LAN) for remote `worker` instances.
|
||||
docker compose up
|
||||
```
|
||||
|
||||
`api` on :3000, `frontend` on :3001, both behind Caddy on :80. First boot pulls/builds all
|
||||
images; `worker` starts polling an empty Redis stream until Phase 1 gives it something to do.
|
||||
`api` on :3000, `frontend` on :3001, both behind Caddy on :80. `api` applies its Postgres
|
||||
migrations on startup (see `api/src/db/migrate.ts`); `worker` consumes the `mcmapper:dirty-chunks`
|
||||
Redis stream via a consumer group (`mcmapper-workers`) so multiple instances split work safely.
|
||||
|
||||
### Phase 1: connecting a mod instance
|
||||
|
||||
There's no admin registration API yet (Phase 6) — seed one server row by hand, then point the
|
||||
mod's `MapperConfig` at the same token:
|
||||
|
||||
```
|
||||
docker compose run --rm api bun run seed
|
||||
```
|
||||
|
||||
reads `MCMAPPER_SEED_SERVER_NAME`/`MCMAPPER_SEED_SERVER_TOKEN` from `api/.env.example` (edit
|
||||
those first, or override with `-e`). Once the mod connects and sends its initial chunk backfill,
|
||||
tiles appear at `GET /api/tiles/:serverId/:dimension/:zoom/:tileX/:tileY.png` (zoom is always `0`
|
||||
for now — see `worker/src/render/cpu.rs`) and the frontend's Leaflet viewer picks them up
|
||||
automatically from `GET /api/servers`.
|
||||
|
||||
## Attribution
|
||||
|
||||
|
||||
Reference in New Issue
Block a user