diff --git a/README.md b/README.md index 45cbab1..bd77960 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ Deployment types (`--scope full-stack|remote|central-store`) are described in > docker compose up -d # BMP collector core > docker compose --profile test --profile auth up -d # full stack > ``` -> See [DOCS.md](DOCS.md) section 4 for details and the manual alternative below. +> See [docs/DOCS.md](docs/DOCS.md) section 4 for details and the manual alternative below. ### Install Docker Compose You will need docker-compose. You can install that via [Docker Compose](https://docs.docker.com/compose/install/) diff --git a/DB_SCHEMA.md b/docs/DB_SCHEMA.md similarity index 100% rename from DB_SCHEMA.md rename to docs/DB_SCHEMA.md diff --git a/DOCS.md b/docs/DOCS.md similarity index 99% rename from DOCS.md rename to docs/DOCS.md index 3e73d40..d2b4229 100644 --- a/DOCS.md +++ b/docs/DOCS.md @@ -584,7 +584,7 @@ Four advanced dashboards that go beyond basic BMP monitoring, unlocking TE/SR da ### Database Schema Reference -A standalone database schema reference is also available at `DB_SCHEMA.md` in the repo root. It documents all 33 tables, 11 views, TE/SR columns, enum types, and common query patterns. +A standalone database schema reference is also available at `docs/DB_SCHEMA.md`. It documents all 33 tables, 11 views, TE/SR columns, enum types, and common query patterns. --- diff --git a/docs/handoff/00-HANDOFF-BRIEF.md b/docs/handoff/00-HANDOFF-BRIEF.md new file mode 100644 index 0000000..38d15b4 --- /dev/null +++ b/docs/handoff/00-HANDOFF-BRIEF.md @@ -0,0 +1,95 @@ +# Handoff brief — OpenBMP greenfield deployment work + +**From:** Claude (chat instance) working with Sam on deployment/portability. +**To:** Claude (VSCode instance) working directly in the `obmp-docker` Gitea repo. +**Date:** 2026-07-20. + +We don't share memory. This packet is the bridge. Sam is the channel between us. + +--- + +## One-paragraph summary + +Sam is standing the `obmp-docker` stack up on a **new WSL2 host** (`NWE-LT02`, +Ubuntu 24.04), separate from the dev lab, to shake out greenfield-portability +problems before prod. Over this session we built a `deploy.sh` wrapper around +the repo's existing `setup.sh` that reconciles host-specific `.env` values +*before* provisioning, plus supporting docs. We hit and fixed a long chain of +deployment artifacts (details in `02-FINDINGS.md`). The stack is now **finally +pulling images and coming up on a native docker daemon** after the big +blocker — Docker Desktop's cross-VM bind-mount failure — was resolved by +switching to native docker-in-WSL. **Current open thread: Grafana came up +without its expected customizations, and we're not sure the repo checkout is on +the right branch.** That's where you come in — you can see the repo directly. + +--- + +## What this session produced (lives in Sam's chat outputs, not the repo yet) + +A `deploy.sh` + docs bundle intended to drop into the repo root. Sam has these +files; they are NOT yet committed to the repo (that's a decision for you + Sam): + +- `deploy.sh` — interactive greenfield deploy wrapper (see `01-DEPLOY-SCRIPT.md`) +- `DEPLOYMENT-TYPES.md` — the three deployment types, locked +- `PORTABILITY-FINDINGS.md` — everything that broke and why +- `router-bmp-config.md` — IOS-XR BMP config reference +- `README.md` — bundle overview + +If you want to integrate these into the repo, coordinate branch/placement with +Sam. They were written against the repo's current `setup.sh`, `.env.example`, +and `docker-compose.yml` as pasted into chat. + +--- + +## THE OPEN QUESTION FOR YOU (highest priority) + +Grafana is up but **its customizations are missing**. Sam expected them to be +"part of the repo." Then Sam raised the real doubt: **"did I even pull the +correct branch?"** + +You can answer this directly — you have the repo. Please determine: + +1. **What branch/commit is the working checkout on?** + `git branch --show-current && git log --oneline -5 && git status` + +2. **Is the Grafana provisioning committed, and on which branch?** + The stack provisions Grafana from `obmp-grafana/provisioning/` (setup.sh + copies it to `${OBMP_DATA_ROOT}/grafana/provisioning/`, bind-mounted into + the container). Check whether dashboards/datasources actually exist in the + tree, and on which branch: + ``` + git ls-tree -r --name-only HEAD -- obmp-grafana/provisioning/ + git branch -a + git ls-tree -r --name-only origin/main -- obmp-grafana/provisioning/ + # and any feature branch you've been working on + ``` + +3. **Were the Grafana customizations ever committed as code**, or did they only + ever exist as UI state in a `grafana.db` (i.e. hand-built, never exported to + provisioning)? This is the crux: "supposed to be in the repo" may have been + intent, not reality. You'd know — did you commit dashboard JSON? + +The fork: +- **On wrong branch** → check out the right one, redeploy; provisioning returns. +- **Right branch, provisioning present but not loading** → permissions/timing; + `docker compose restart grafana` after confirming files are in + `/var/openbmp/grafana/provisioning/`. +- **Provisioning never committed** → the customizations were UI-only; they need + exporting to the repo now so future greenfield deploys reproduce them. This is + itself a portability finding. + +--- + +## Immediate deployment state (as of handoff) + +- Host: WSL2, Ubuntu 24.04.4, `NWE-LT02`, 22 vCPU / ~15 GB RAM. +- Daemon: **native docker-ce 29.6.2** (NOT Docker Desktop — that was the bug). + `docker info` confirms `Operating System: Ubuntu 24.04.4 LTS`. +- `OBMP_DATA_ROOT=/var/openbmp` (native ext4, `/dev/sdg`). +- Deploy command in use: `./deploy.sh --wsl --scope full-stack --auth local` +- Stack was pulling images and progressing past the volume-mount failure that + blocked every earlier attempt. +- Grafana reachable at `http://:3000/grafana/`, login `admin`/`openbmp`. + +See `02-FINDINGS.md` for the full list of what broke and how it was fixed, and +`03-NEXT-STEPS.md` for what remains. diff --git a/docs/handoff/01-DEPLOY-SCRIPT.md b/docs/handoff/01-DEPLOY-SCRIPT.md new file mode 100644 index 0000000..a9dc235 --- /dev/null +++ b/docs/handoff/01-DEPLOY-SCRIPT.md @@ -0,0 +1,55 @@ +# deploy.sh — what it is and how it behaves + +An interactive wrapper around the repo's `setup.sh`. Its job: reconcile +**host-specific** values in `.env` for THIS host *before* `setup.sh` renders +config from them, then bring the stack up in stages. It exists because copying +a populated `.env` between hosts silently propagates stale values (see +`02-FINDINGS.md`, finding 2/3). + +## Flow (what happens on a run) + +1. Detect host type (WSL vs native Linux); let user override. +2. Guard: refuse/warn if the active daemon is **Docker Desktop** (bind mounts + break under it — see findings). +3. Resolve deployment **type** (menu with descriptions + resource estimates). +4. Resolve `HOST_IP` (internal bind), **router-facing IP+port** (separate — what + `bmp server` targets), auth mode, and for `remote` type the central Kafka. +5. On WSL: auto-detect the **Windows LAN IP** via `powershell.exe` interop for + the router-facing default (with graceful fallback to a placeholder). +6. **Port-collision preflight** — checks host ports the type will bind + (ss/netstat//proc), warns/refuses on conflict. +7. Show a plan; one confirmation. +8. Write reconciled values to `.env`. +9. Clear stale volume handles → run `setup.sh` → verify bind dirs exist → + clear handles again → **staged bring-up** → verify. +10. Print post-deploy notes (WSL: netsh portproxy commands; native: firewall + + router `bmp server` snippet). + +## Deployment types (also in DEPLOYMENT-TYPES.md) + +- `full-stack` — collector + Kafka + Postgres + Grafana + feeders, one host. +- `remote` — collector + local Kafka only, forwards to a central store's Kafka. + Needs `--central-kafka HOST:PORT`. No local store. +- `central-store` — store + Grafana + feeders, NO local collector; ingest via + remotes. + +Old scope names (`core`/`feeders`/`full`/`standalone`) are accepted as aliases. + +## Flags + +`--wsl|--prod`, `--host-ip`, `--router-ip`, `--router-port` (default **1790**, +the IANA BMP port — collector still listens 5000 internally, mapped), +`--auth local|authelia`, `--scope full-stack|remote|central-store`, +`--central-kafka HOST:PORT`, `--reset`, `--yes`. + +## Notes for repo integration + +- It calls `./setup.sh` and checks its exit status. It does NOT reimplement + setup.sh's provisioning — it drives it with correct inputs. +- It assumes `.env.example`, `setup.sh`, `docker-compose.yml` are in the same + dir. If `setup.sh` key names change, update deploy.sh's `get_env`/`set_env` + references. +- Router port default was changed to 1790; the lab routers are currently + configured to send BMP to **5000**, so until they're reconfigured, deploy + with `--router-port 5000` OR update the routers. This is an open decision + (see 03-NEXT-STEPS.md). diff --git a/docs/handoff/02-FINDINGS.md b/docs/handoff/02-FINDINGS.md new file mode 100644 index 0000000..7977b72 --- /dev/null +++ b/docs/handoff/02-FINDINGS.md @@ -0,0 +1,74 @@ +# Deployment findings — what broke on greenfield and why + +Ordered roughly as encountered. Each is a real portability artifact that a +clean deploy has to survive. These are captured in full in the bundle's +`PORTABILITY-FINDINGS.md`; this is the digest for you. + +## 1. Bind-mounted volumes need pre-existing source dirs +`docker-compose.yml` binds `${OBMP_DATA_ROOT}/postgres/{data,ts}` with +`type:none,o:bind`. Docker won't create these; if absent, Postgres fails with +`no such file or directory`. `setup.sh` DOES create them (its data-tree loop) — +the error only appears when setup.sh wasn't run on the target. + +## 2. Stale HOST_IP survives a copied .env, and setup.sh doesn't catch it +`.env` carried `HOST_IP=10.13.21.65` from another host — not an address on the +new box. setup.sh validation only checks non-empty/not-"changeme", so a stale +IP passes and gets rendered into `gobgpd.conf` + Grafana URL. deploy.sh fixes +HOST_IP before setup.sh runs. **This is the core reason deploy.sh exists.** + +## 3. .env travels, host state doesn't +Authelia secrets were populated in the copied `.env` (late setup.sh step) but +the Postgres dirs (early step) were missing — proving setup.sh completed on a +*different* host and the .env was copied over without re-running setup.sh here. +Lesson: a populated .env is not a deployed host; run setup.sh per target. + +## 4. WSL: HOST_IP is the NAT addr; not router-reachable +`hostname -I` on WSL returns the VM's NAT address — changes on `wsl --shutdown`, +not reachable by physical routers. Routers must target the **Windows LAN IP**, +forwarded via `netsh portproxy` into WSL. deploy.sh now auto-detects the Windows +LAN IP via `powershell.exe` interop (read-only, no admin needed) and separates +"router-facing IP" from "HOST_IP". + +## 5. Cold-start contention looked like a Kafka permission bug +Bringing the whole stack up at once crash-looped Kafka on a preflight +"writable" check that was actually resource contention, not permissions. +deploy.sh stages the bring-up (infra → core → feeders). + +## 6. Sizing: dimension for BMP burst, not steady state +Generic docs say the collector is light. True for steady state, misleading for +sizing: the store (Postgres) saturates under burst. Sam's lab is GoBGP pulling +full v4/v6 (~1.18M paths) reflected through RRs to clients — load is +full-table × monitored-sessions, and Postgres write amplification is the wall. +Numbers in DEPLOYMENT-TYPES.md are engineering estimates to calibrate against +Sam's watermarking, explicitly not measured specs. + +## 7. Port 5000/3000 collisions +5000 (OpenBMP collector) and 3000 (Grafana) are commonly occupied. deploy.sh +now runs a port-collision preflight before bring-up. Also: router-facing BMP +port default moved 5000 → **1790** (IANA standard) to avoid the crowded 5000; +collector still listens 5000 internally, external maps down. + +## 8. **Docker Desktop breaks host bind mounts on WSL** (the big one) +Symptom: Postgres mount fails `no such file or directory` on a path that `ls` +shows exists. Cause: Docker Desktop runs the daemon in its OWN wsl distro +(`docker-desktop` VM); host bind mounts resolve against THAT filesystem, not +the working distro. Two different filesystems. No `mkdir` in the working distro +fixes it. **Fix: switch to native docker-in-WSL** (disable Desktop WSL +integration, install docker-ce, `systemctl enable --now docker`). Confirm with +`docker info | grep 'Operating System'` — must NOT say Docker Desktop. This also +makes the WSL box behave like the native-Linux prod host. deploy.sh now detects +Docker Desktop and warns/refuses. Resolving this is what finally let the stack +pull images and progress. + +## 8b. Native docker socket namespace snag (post-switch) +After installing native docker, the CLI couldn't reach the socket +(`/run/docker.sock` listened-on by dockerd but not visible to the shell — mount +namespace split left by Desktop residue). Fixed by `wsl --shutdown` and +reopening, which cleared the stale namespaces. After restart `docker info` +correctly reported Ubuntu. + +## 9. OPEN: Grafana customizations missing / branch uncertainty +Grafana came up without expected customizations. Provisioning is supposed to be +repo-committed (`obmp-grafana/provisioning/`). Sam then questioned whether the +correct branch was even checked out. **This is the handoff question for the +VSCode instance** — see 00-HANDOFF-BRIEF.md. diff --git a/docs/handoff/03-NEXT-STEPS.md b/docs/handoff/03-NEXT-STEPS.md new file mode 100644 index 0000000..af78d4c --- /dev/null +++ b/docs/handoff/03-NEXT-STEPS.md @@ -0,0 +1,66 @@ +# Next steps & open decisions + +## Immediate (blocking the current deploy) + +1. **Resolve the branch/provisioning question** (VSCode instance — you own this): + - Confirm which branch the checkout is on and whether it's the intended one. + - Confirm whether `obmp-grafana/provisioning/` (dashboards + datasources) is + committed, and on which branch. + - If provisioning exists but Grafana didn't load it: verify files reached + `/var/openbmp/grafana/provisioning/`, check ownership (Grafana UID 472), + `docker compose restart grafana`, check logs for provisioning errors. + - If provisioning was never committed (UI-only): export current/expected + dashboards to JSON and commit them so greenfield deploys reproduce them. + This closes a real portability gap. + +2. **Pin the deploy to a known branch/commit.** A greenfield deploy is only + reproducible if the checkout is pinned. Recommend documenting "clone, check + out at , run deploy.sh" — otherwise "clone and run" silently + depends on whatever branch was checked out. Consider adding a branch/commit + check to deploy.sh or the README. + +## Deployment decisions still open + +3. **Router BMP port: 5000 vs 1790.** deploy.sh now defaults router-facing to + 1790 (IANA standard, avoids crowded 5000). BUT the lab routers are currently + configured to send to **5000**. Decision: either + - reconfigure the routers to 1790 (Sam leaned this way — cleaner, avoids + common-port collisions), which needs the IOS-XR `bmp server` port change + on each device and a compose mapping `1790:5000` on native hosts; or + - keep 5000 and deploy with `--router-port 5000`. + If moving to 1790: router-bmp-config.md needs the port updated, and the + native-host compose `ports:` for the collector needs `1790:5000`. + +4. **Sizing profiles.** DEPLOYMENT-TYPES.md has three types with ballpark + resources (estimates, not measured). Sam is watermarking a prod-realistic + node to replace estimates with real numbers. When those land, update the + figures. There was an idea to add a `--sizing lab|prod-lite|prod` selector + that writes `*_MEM_LIMIT` blocks into `.env`; deferred until watermarks exist. + +5. **Doc name sync.** deploy.sh was renamed to the three deployment types after + the docs were written; a couple of README/doc tables may still reference the + old scope names (feeders/full/standalone). Cosmetic — aliases keep it + working — but worth a sync pass. + +## Integration decisions + +6. **Where do the bundle files live?** deploy.sh + the 4 docs currently exist + only in Sam's chat outputs. Decide whether they get committed to the repo + (and to which branch), and whether deploy.sh supersedes or complements any + existing deploy tooling in the repo you may know about that I don't. + +7. **Reconcile with whatever you (VSCode) have been building.** I have no + visibility into your repo work — the EVPN consumer, gobgp config, any CI, or + other tooling. If there's overlap or conflict between deploy.sh and existing + repo scripts, that's a reconciliation for you + Sam. Flag anything I've + duplicated or contradicted. + +## Things that are DONE (don't redo) + +- Native docker-in-WSL is installed and working; Docker Desktop is the wrong + substrate (finding 8). Don't re-recommend Desktop. +- The `.env` HOST_IP/auth reconciliation, staged bring-up, port preflight, + Windows-LAN-IP detection, and Docker Desktop guard are all built and tested + in deploy.sh. +- The bind-dir + stale-volume handling in deploy.sh is hardened (down+rm before + AND after setup.sh, plus a dir-existence check). diff --git a/docs/handoff/README.md b/docs/handoff/README.md new file mode 100644 index 0000000..08f36e2 --- /dev/null +++ b/docs/handoff/README.md @@ -0,0 +1,10 @@ +# Handoff packets + +Point-in-time bridge documents written by one Claude session for another +during the 2026-07 greenfield portability push (chat instance -> repo/VSCode +instance, with Sam as the channel). Kept for the record; **the live versions +of their content are the repo docs** — findings in +[../PORTABILITY-FINDINGS.md](../PORTABILITY-FINDINGS.md), deployment types in +[../DEPLOYMENT-TYPES.md](../DEPLOYMENT-TYPES.md), router config in +[../router-bmp-config.md](../router-bmp-config.md). Where these packets and +the repo disagree, the repo is right.