9e67ef10ba
Repo root keeps only README.md; DOCS.md and DB_SCHEMA.md move to docs/ with their cross-references updated. The chat-session handoff packets land in docs/handoff/ as point-in-time records with a README marking the repo docs as the live source of truth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
75 lines
4.3 KiB
Markdown
75 lines
4.3 KiB
Markdown
# Deployment findings — what broke on greenfield and why
|
||
|
||
Ordered roughly as encountered. Each is a real portability artifact that a
|
||
clean deploy has to survive. These are captured in full in the bundle's
|
||
`PORTABILITY-FINDINGS.md`; this is the digest for you.
|
||
|
||
## 1. Bind-mounted volumes need pre-existing source dirs
|
||
`docker-compose.yml` binds `${OBMP_DATA_ROOT}/postgres/{data,ts}` with
|
||
`type:none,o:bind`. Docker won't create these; if absent, Postgres fails with
|
||
`no such file or directory`. `setup.sh` DOES create them (its data-tree loop) —
|
||
the error only appears when setup.sh wasn't run on the target.
|
||
|
||
## 2. Stale HOST_IP survives a copied .env, and setup.sh doesn't catch it
|
||
`.env` carried `HOST_IP=10.13.21.65` from another host — not an address on the
|
||
new box. setup.sh validation only checks non-empty/not-"changeme", so a stale
|
||
IP passes and gets rendered into `gobgpd.conf` + Grafana URL. deploy.sh fixes
|
||
HOST_IP before setup.sh runs. **This is the core reason deploy.sh exists.**
|
||
|
||
## 3. .env travels, host state doesn't
|
||
Authelia secrets were populated in the copied `.env` (late setup.sh step) but
|
||
the Postgres dirs (early step) were missing — proving setup.sh completed on a
|
||
*different* host and the .env was copied over without re-running setup.sh here.
|
||
Lesson: a populated .env is not a deployed host; run setup.sh per target.
|
||
|
||
## 4. WSL: HOST_IP is the NAT addr; not router-reachable
|
||
`hostname -I` on WSL returns the VM's NAT address — changes on `wsl --shutdown`,
|
||
not reachable by physical routers. Routers must target the **Windows LAN IP**,
|
||
forwarded via `netsh portproxy` into WSL. deploy.sh now auto-detects the Windows
|
||
LAN IP via `powershell.exe` interop (read-only, no admin needed) and separates
|
||
"router-facing IP" from "HOST_IP".
|
||
|
||
## 5. Cold-start contention looked like a Kafka permission bug
|
||
Bringing the whole stack up at once crash-looped Kafka on a preflight
|
||
"writable" check that was actually resource contention, not permissions.
|
||
deploy.sh stages the bring-up (infra → core → feeders).
|
||
|
||
## 6. Sizing: dimension for BMP burst, not steady state
|
||
Generic docs say the collector is light. True for steady state, misleading for
|
||
sizing: the store (Postgres) saturates under burst. Sam's lab is GoBGP pulling
|
||
full v4/v6 (~1.18M paths) reflected through RRs to clients — load is
|
||
full-table × monitored-sessions, and Postgres write amplification is the wall.
|
||
Numbers in DEPLOYMENT-TYPES.md are engineering estimates to calibrate against
|
||
Sam's watermarking, explicitly not measured specs.
|
||
|
||
## 7. Port 5000/3000 collisions
|
||
5000 (OpenBMP collector) and 3000 (Grafana) are commonly occupied. deploy.sh
|
||
now runs a port-collision preflight before bring-up. Also: router-facing BMP
|
||
port default moved 5000 → **1790** (IANA standard) to avoid the crowded 5000;
|
||
collector still listens 5000 internally, external maps down.
|
||
|
||
## 8. **Docker Desktop breaks host bind mounts on WSL** (the big one)
|
||
Symptom: Postgres mount fails `no such file or directory` on a path that `ls`
|
||
shows exists. Cause: Docker Desktop runs the daemon in its OWN wsl distro
|
||
(`docker-desktop` VM); host bind mounts resolve against THAT filesystem, not
|
||
the working distro. Two different filesystems. No `mkdir` in the working distro
|
||
fixes it. **Fix: switch to native docker-in-WSL** (disable Desktop WSL
|
||
integration, install docker-ce, `systemctl enable --now docker`). Confirm with
|
||
`docker info | grep 'Operating System'` — must NOT say Docker Desktop. This also
|
||
makes the WSL box behave like the native-Linux prod host. deploy.sh now detects
|
||
Docker Desktop and warns/refuses. Resolving this is what finally let the stack
|
||
pull images and progress.
|
||
|
||
## 8b. Native docker socket namespace snag (post-switch)
|
||
After installing native docker, the CLI couldn't reach the socket
|
||
(`/run/docker.sock` listened-on by dockerd but not visible to the shell — mount
|
||
namespace split left by Desktop residue). Fixed by `wsl --shutdown` and
|
||
reopening, which cleared the stale namespaces. After restart `docker info`
|
||
correctly reported Ubuntu.
|
||
|
||
## 9. OPEN: Grafana customizations missing / branch uncertainty
|
||
Grafana came up without expected customizations. Provisioning is supposed to be
|
||
repo-committed (`obmp-grafana/provisioning/`). Sam then questioned whether the
|
||
correct branch was even checked out. **This is the handoff question for the
|
||
VSCode instance** — see 00-HANDOFF-BRIEF.md.
|