obmp-docker/docs/PORTABILITY-FINDINGS.md

# Portability findings — what breaks on a greenfield deploy

Every artifact hit while standing this stack up on a fresh host (WSL2
`NWE-LT02`, Ubuntu 24.04, native docker-ce 29.x — 2026-07), ordered roughly as
encountered. Each one is something a clean deploy has to survive; most are now
handled automatically by `deploy.sh`/`setup.sh`. Kept as a checklist for the
next new host and as the rationale for the deploy tooling.

## 1. Bind-mounted volumes need pre-existing source dirs
`docker-compose.yml` binds `${OBMP_DATA_ROOT}/postgres/{data,ts}` with
`type:none,o:bind`. Docker will not create these; if absent, Postgres fails
with `no such file or directory`. `setup.sh` creates them — the error only
appears when setup.sh wasn't run on the target. deploy.sh double-checks the
dirs exist after setup.sh.

## 2. Stale HOST_IP survives a copied .env
A `.env` copied from another host carries that host's `HOST_IP`; setup.sh
validation only checks non-empty/not-"changeme", so the stale IP passes and
gets rendered into `gobgpd.conf` and the Grafana root URL. **This is the core
reason deploy.sh exists** — it reconciles HOST_IP (and warns when the .env
value isn't a local address) *before* setup.sh renders anything.

## 3. A populated .env is not a deployed host
The same copied .env had late-stage artifacts (Authelia secrets) while
early-stage host state (Postgres dirs) was missing — proof that setup.sh
completed on a *different* host. Run setup.sh per target, always.

## 4. WSL: HOST_IP is the NAT address; routers can't reach it
`hostname -I` in WSL returns the VM's NAT address — it changes on
`wsl --shutdown` and physical routers cannot reach it. Routers must target the
**Windows LAN IP**, forwarded into WSL via `netsh portproxy`
(`scripts/wsl-portproxy.ps1`). deploy.sh separates "router-facing IP:port"
from HOST_IP and auto-detects the Windows LAN IP via powershell.exe interop.

## 5. Cold-start contention masquerades as component bugs
Bringing the whole stack up at once crash-looped Kafka on a preflight
"writable" check that was actually resource contention, not permissions.
deploy.sh stages the bring-up: infra (zookeeper/kafka) -> psql -> core ->
feeders.

## 6. Size for BMP burst, not steady state
Generic docs say the collector is light — true, and misleading: the store
saturates under burst (full table x monitored sessions, Postgres write
amplification). See DEPLOYMENT-TYPES.md for the full model.

## 7. Ports 5000 and 3000 are crowded
The collector's 5000 and Grafana's 3000 are commonly already occupied.
deploy.sh runs a port-collision preflight before touching anything. The
router-facing BMP port default moved 5000 -> **1790** (the IANA BMP port);
the collector still listens on 5000 inside the container.

## 8. Docker Desktop breaks host bind mounts on WSL (the big one)
Symptom: Postgres mount fails `no such file or directory` on a path `ls`
shows exists. Cause: Docker Desktop runs the daemon in its own WSL distro
(`docker-desktop` VM), so host bind mounts resolve against *that* filesystem,
not the distro you ran compose from. No mkdir fixes it. **Fix: native
docker-in-WSL** — disable Desktop's WSL integration, install docker-ce,
`systemctl enable --now docker`, and confirm
`docker info | grep 'Operating System'` does not say Docker Desktop.
deploy.sh detects Docker Desktop and warns/refuses.

### 8b. Desktop residue splits the mount namespace
After installing native docker, the CLI couldn't reach `/run/docker.sock`
even though dockerd held it — a stale mount-namespace split left by Desktop.
`wsl --shutdown` and reopening the distro cleared it.

## 9. Grafana customizations must live in the repo, not grafana.db
Dashboards hand-built in the UI exist only in that host's `grafana.db` and
vanish on a greenfield deploy. All dashboards are now committed as JSON under
`obmp-grafana/dashboards/` and setup.sh syncs both provisioning YAML *and*
dashboard JSONs to the data root. If you edit a dashboard in the UI, export
and commit the JSON.

## 10. Blanket `chmod -R 777` on the data root breaks Postgres re-deploys
setup.sh's lab-permissive chmod over an *existing* data tree made
`psql_server.key` world-accessible; Postgres refuses to boot
(`FATAL: private key file ... has group or world access`) and the whole core
cascades down. Fresh installs never hit it (the key is generated after the
chmod) — it is strictly a re-deploy bug. setup.sh now skips `postgres/`
(the image entrypoint owns those perms) and re-tightens the key/cert to 0600
if a previous run already clobbered them.

## 11. docker-ce 29 rejects Docker API < 1.40 — old pinned clients break
telegraf 1.28's `[[inputs.docker]]` hardcodes Docker API version 1.24, which
docker-ce >= 29 refuses (`client version 1.24 is too old`). Every
per-container stack metric silently never reached InfluxDB; the stack
dashboards sat empty while `disk`/`postgresql_*` inputs worked fine.
`DOCKER_API_VERSION` env is ignored (version pinned in code) — the fix is
telegraf >= 1.30, which negotiates the API version (telegraf/Dockerfile now
pins 1.33). Watch for the same failure in any other tooling that talks to the
daemon with an old vendored client.

## 12. Router-side automation targeted the old lab's values
`cml/proxmox_bmp_config.py` sent routers at `HOST_IP` port `5000`, and
`cml/xrd-node-definition.yaml` baked in `10.40.40.202:5000` — both correct on
the old native-Linux dev lab, wrong on a WSL host (finding 4: routers must
target the router-facing address). Resolved 2026-07: the router-facing port
is **1790** everywhere; compose publishes `${ROUTER_FACING_PORT:-1790}:5000`,
setup.sh renders the same port into gobgp's BMP export (gobgp is
host-networked and must hit the *published* port), and the cml tooling reads
`ROUTER_FACING_IP`/`ROUTER_FACING_PORT` from `.env`. Remaining hands-on step:
re-apply the config to routers still pointed at 5000. See
router-bmp-config.md.