75 lines
4.3 KiB
Markdown
75 lines
4.3 KiB
Markdown
|
# Deployment findings — what broke on greenfield and why
|
|||
|
|
|
|||
|
|
Ordered roughly as encountered. Each is a real portability artifact that a
|
|||
|
|
clean deploy has to survive. These are captured in full in the bundle's
|
|||
|
|
`PORTABILITY-FINDINGS.md`; this is the digest for you.
|
|||
|
|
|
|||
|
|
## 1. Bind-mounted volumes need pre-existing source dirs
|
|||
|
|
`docker-compose.yml` binds `${OBMP_DATA_ROOT}/postgres/{data,ts}` with
|
|||
|
|
`type:none,o:bind`. Docker won't create these; if absent, Postgres fails with
|
|||
|
|
`no such file or directory`. `setup.sh` DOES create them (its data-tree loop) —
|
|||
|
|
the error only appears when setup.sh wasn't run on the target.
|
|||
|
|
|
|||
|
|
## 2. Stale HOST_IP survives a copied .env, and setup.sh doesn't catch it
|
|||
|
|
`.env` carried `HOST_IP=10.13.21.65` from another host — not an address on the
|
|||
|
|
new box. setup.sh validation only checks non-empty/not-"changeme", so a stale
|
|||
|
|
IP passes and gets rendered into `gobgpd.conf` + Grafana URL. deploy.sh fixes
|
|||
|
|
HOST_IP before setup.sh runs. **This is the core reason deploy.sh exists.**
|
|||
|
|
|
|||
|
|
## 3. .env travels, host state doesn't
|
|||
|
|
Authelia secrets were populated in the copied `.env` (late setup.sh step) but
|
|||
|
|
the Postgres dirs (early step) were missing — proving setup.sh completed on a
|
|||
|
|
*different* host and the .env was copied over without re-running setup.sh here.
|
|||
|
|
Lesson: a populated .env is not a deployed host; run setup.sh per target.
|
|||
|
|
|
|||
|
|
## 4. WSL: HOST_IP is the NAT addr; not router-reachable
|
|||
|
|
`hostname -I` on WSL returns the VM's NAT address — changes on `wsl --shutdown`,
|
|||
|
|
not reachable by physical routers. Routers must target the **Windows LAN IP**,
|
|||
|
|
forwarded via `netsh portproxy` into WSL. deploy.sh now auto-detects the Windows
|
|||
|
|
LAN IP via `powershell.exe` interop (read-only, no admin needed) and separates
|
|||
|
|
"router-facing IP" from "HOST_IP".
|
|||
|
|
|
|||
|
|
## 5. Cold-start contention looked like a Kafka permission bug
|
|||
|
|
Bringing the whole stack up at once crash-looped Kafka on a preflight
|
|||
|
|
"writable" check that was actually resource contention, not permissions.
|
|||
|
|
deploy.sh stages the bring-up (infra → core → feeders).
|
|||
|
|
|
|||
|
|
## 6. Sizing: dimension for BMP burst, not steady state
|
|||
|
|
Generic docs say the collector is light. True for steady state, misleading for
|
|||
|
|
sizing: the store (Postgres) saturates under burst. Sam's lab is GoBGP pulling
|
|||
|
|
full v4/v6 (~1.18M paths) reflected through RRs to clients — load is
|
|||
|
|
full-table × monitored-sessions, and Postgres write amplification is the wall.
|
|||
|
|
Numbers in DEPLOYMENT-TYPES.md are engineering estimates to calibrate against
|
|||
|
|
Sam's watermarking, explicitly not measured specs.
|
|||
|
|
|
|||
|
|
## 7. Port 5000/3000 collisions
|
|||
|
|
5000 (OpenBMP collector) and 3000 (Grafana) are commonly occupied. deploy.sh
|
|||
|
|
now runs a port-collision preflight before bring-up. Also: router-facing BMP
|
|||
|
|
port default moved 5000 → **1790** (IANA standard) to avoid the crowded 5000;
|
|||
|
|
collector still listens 5000 internally, external maps down.
|
|||
|
|
|
|||
|
|
## 8. **Docker Desktop breaks host bind mounts on WSL** (the big one)
|
|||
|
|
Symptom: Postgres mount fails `no such file or directory` on a path that `ls`
|
|||
|
|
shows exists. Cause: Docker Desktop runs the daemon in its OWN wsl distro
|
|||
|
|
(`docker-desktop` VM); host bind mounts resolve against THAT filesystem, not
|
|||
|
|
the working distro. Two different filesystems. No `mkdir` in the working distro
|
|||
|
|
fixes it. **Fix: switch to native docker-in-WSL** (disable Desktop WSL
|
|||
|
|
integration, install docker-ce, `systemctl enable --now docker`). Confirm with
|
|||
|
|
`docker info | grep 'Operating System'` — must NOT say Docker Desktop. This also
|
|||
|
|
makes the WSL box behave like the native-Linux prod host. deploy.sh now detects
|
|||
|
|
Docker Desktop and warns/refuses. Resolving this is what finally let the stack
|
|||
|
|
pull images and progress.
|
|||
|
|
|
|||
|
|
## 8b. Native docker socket namespace snag (post-switch)
|
|||
|
|
After installing native docker, the CLI couldn't reach the socket
|
|||
|
|
(`/run/docker.sock` listened-on by dockerd but not visible to the shell — mount
|
|||
|
|
namespace split left by Desktop residue). Fixed by `wsl --shutdown` and
|
|||
|
|
reopening, which cleared the stale namespaces. After restart `docker info`
|
|||
|
|
correctly reported Ubuntu.
|
|||
|
|
|
|||
|
|
## 9. OPEN: Grafana customizations missing / branch uncertainty
|
|||
|
|
Grafana came up without expected customizations. Provisioning is supposed to be
|
|||
|
|
repo-committed (`obmp-grafana/provisioning/`). Sam then questioned whether the
|
|||
|
|
correct branch was even checked out. **This is the handoff question for the
|
|||
|
|
VSCode instance** — see 00-HANDOFF-BRIEF.md.
|