docs: audit fixes -- retire stale claims, add navigation, correct init_db story

From a two-agent audit (docs accuracy + greenfield deploy path): - init_db docs were describing upstream behavior: this repo's psql-app auto-creates the schema on first run and drops config/do_not_init_db to skip later (psql-app/scripts/run:74). README/DOCS/backup-restore now describe the marker semantics; the restore flow creates the marker BEFORE first start instead of 'not creating init_db' - DOCS.md contained heavy retired-lab drift: banner declares it a legacy walkthrough with illustrative values; fixed its self-contradictions -- external port 1790, folder OBMP-Reference, datasource 'PostgreSQL', OpenConfig gNMI paths (matching telegraf.conf), EXABGP_PEERS, TRAFFIC_GEN_PORT - deploy.sh --help now shows the current scope names (old ones remain accepted aliases) - navigation: new docs/README.md index with operator and network-engineer tracks (links the previously orphaned backup-restore, security-hardening, ROADMAP, DB_SCHEMA); README gains a Start-here router - intra-docs prose paths no longer carry the docs/ prefix (they resolve from within docs/); RR-CLIENTS -> RR-CLIENT matches the blueprint; ROADMAP A6 marked done - scripts/deploy-lib.sh added: shared log/ask/confirm/get_env/set_env helpers for the deploy tooling consolidation (wiring lands next) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 17:05:47 -07:00 · 2026-07-20 17:05:47 -07:00 · da975d7375
commit da975d7375
parent 65fd88a0a2
10 changed files with 180 additions and 38 deletions
--- a/README.md
+++ b/README.md
@ -1,6 +1,14 @@
 # OpenBMP docker files
 Docker files for OpenBMP.

+> **Start here:**
+> - **Deploying the stack?** Jump to [Greenfield deploy](#greenfield-deploy-recommended-deploysh)
+>   below, then see the [docs index](docs/README.md) for deployment types,
+>   portability findings, sizing, and backup.
+> - **Configuring routers to feed it?** Go straight to
+>   [docs/router-integration.md](docs/router-integration.md) and the
+>   copy-paste fragments in [router-blueprints/](router-blueprints/).
+
 ## (Prerequisite) Platform Docker Install

 > Ignore this step if you already have a current docker install
@ -104,8 +112,10 @@ sudo chmod -R 7777 $OBMP_DATA_ROOT
 > refuse to start. Skip `postgres/` (setup.sh does this for you — see
 > [docs/PORTABILITY-FINDINGS.md](docs/PORTABILITY-FINDINGS.md) finding 10).

-> In order to init the DB tables, you must create the file ```${OBMP_DATA_ROOT}/config/init_db```.  This should
-> only be done once or whenever you want to completely wipe out the DB and start over. 
+> DB tables are created **automatically** by psql-app on its first run (it
+> drops a `config/do_not_init_db` marker afterward so restarts skip the
+> migration). No `init_db` trigger file is needed — that was upstream
+> behavior this repo's psql-app replaces.

 Change ```OBMP_DATA_ROOT=<path>``` to where you created the directories above.  The default is ```/var/openbmp```

--- a/deploy.sh
+++ b/deploy.sh
@ -19,8 +19,8 @@
 #   ./deploy.sh --host-ip 10.0.0.50   # preset HOST_IP (internal stack IP)
 #   ./deploy.sh --router-ip 10.0.0.9 --router-port 5000   # BMP-facing target
 #   ./deploy.sh --auth local|authelia
-#   ./deploy.sh --scope core|feeders|full|standalone
-#   ./deploy.sh --central-kafka 10.0.0.50:9092   # for --scope standalone
+#   ./deploy.sh --scope full-stack|remote|central-store
+#   ./deploy.sh --central-kafka 10.0.0.50:9092   # for --scope remote
 #   ./deploy.sh --reset               # wipe data tree first (guarded on prod)
 #   ./deploy.sh --prod --yes          # unattended (needs enough presets)
 #   REPO_DIR=/opt/obmp-docker ./deploy.sh
@ -103,7 +103,7 @@
 #   Mitigations: NVMe with real IOPS; BMP-monitor pre-policy on the RRs only
 #   (not every client session) to avoid ingesting N reflected copies of the
 #   same table; stagger connects (initial-refresh delay/spread); raise
-#   KAFKA/PSQL_APP mem; or split ingest with --scope standalone collectors.
+#   KAFKA/PSQL_APP mem; or split ingest with --scope remote collectors.
 # ---------------------------------------------------------------------------
 #
 # HOST TYPE affects default HOST_IP source, default auth/scope, reset guarding,
--- a/docs/DEPLOYMENT-TYPES.md
+++ b/docs/DEPLOYMENT-TYPES.md
@ -64,4 +64,4 @@ Storage scales with prefixes: ~1 GB per peer with a full internet table

 > All figures are engineering estimates to calibrate against your own
 > watermarking, not measured specs. Replace them with real numbers once a
-> prod-realistic node has been load-tested (see docs/production-sizing.md).
+> prod-realistic node has been load-tested (see production-sizing.md).
--- a/docs/DOCS.md
+++ b/docs/DOCS.md
@ -1,5 +1,14 @@
 # OpenBMP + ExaBGP Route Injector — Full Documentation

+> **LEGACY LAB WALKTHROUGH.** This guide was written against the original
+> single-host dev lab and keeps its literal values (e.g. host `10.40.40.202`,
+> NETCONF user `webui`) as *illustrations* — substitute your own `.env`
+> values (`HOST_IP`, `ROUTER_FACING_IP`/`PORT`, `IOSXR_NETCONF_*`). For
+> current deployments start instead at:
+> **[deploy.sh + README quickstart](../README.md)** (operators) and
+> **[router-integration.md](router-integration.md)** (network engineers).
+> Where this file disagrees with those, they win.
+
 ## Table of Contents

 1. [What Is This Project?](#1-what-is-this-project)
@ -48,7 +57,7 @@ This is a **BGP Monitoring Platform (BMP) lab stack** deployed via Docker Compos

 ```
 IOS-XR Routers (9x, AS 65020)
-  BMP telemetry on TCP 5000
+  BMP telemetry on TCP 1790 (maps to collector :5000 in-container)
         |
         v
  obmp-collector (openbmp/collector:2.2.3)
@ -101,7 +110,7 @@ Traffic Generator (Phase 4):
 |-----------|-------|---------|------|
 | obmp-zookeeper | confluentinc/cp-zookeeper:7.1.1 | 2181 (internal) | Kafka coordination |
 | obmp-kafka | confluentinc/cp-kafka:7.1.1 | 9092 | Message broker |
-| obmp-collector | openbmp/collector:2.2.3 | 5000 | BMP receiver |
+| obmp-collector | openbmp/collector:2.2.3 | 1790 -> 5000 | BMP receiver (external ${ROUTER_FACING_PORT:-1790} maps to in-container 5000) |
 | obmp-psql-app | openbmp/psql-app:2.2.2 | 9005 | Kafka→PostgreSQL consumer |
 | obmp-psql | openbmp/postgres:2.2.1 | 5432 | TimescaleDB storage |
 | obmp-grafana | grafana/grafana:9.1.7 | 3000 | Visualization |
@ -119,7 +128,7 @@ Traffic Generator (Phase 4):

 - Docker Engine (20.10+) and Docker Compose v2
 - Host IP `10.40.40.202` reachable from the CML management network
- CML routers with BMP configured pointing to the collector's router-facing address (see docs/router-bmp-config.md)
+- CML routers with BMP configured pointing to the collector's router-facing address (see router-bmp-config.md)
 - CML CORE routers configured with ExaBGP as eBGP neighbor (see Section 5)
 - `OBMP_DATA_ROOT` directory created (default: `/var/openbmp`)

@ -181,15 +190,15 @@ mkdir -p ${OBMP_DATA_ROOT}/grafana/dashboards
 sudo chmod -R 777 $OBMP_DATA_ROOT
 ```

-### 4.3 Initialise the database (first run only)
+### 4.3 Database initialisation (automatic)

-Create the init trigger file — this causes psql-app to create all tables on startup:
+psql-app creates the full schema **automatically on its first run** and then
+drops a marker file `${OBMP_DATA_ROOT}/config/do_not_init_db` so later
+restarts skip it. No manual trigger is needed.

-```bash
-touch ${OBMP_DATA_ROOT}/config/init_db
-```
-
-> **Warning:** Do not create this file on subsequent runs unless you want to wipe and recreate the entire database.
+> To force a schema re-init against an empty database, delete the marker:
+> `rm ${OBMP_DATA_ROOT}/config/do_not_init_db` and restart `obmp-psql-app`.
+> Never do this against a database whose data you want to keep.

 ### 4.4 Copy Grafana provisioning files

@ -554,7 +563,7 @@ Default credentials: `admin` / `openbmp` (anonymous access also enabled)

 > History dashboards require `ip_rib_log` and `stats_chg_*` table data. Run `inject.py churn` to populate these.

-### OBMP-Learning Dashboards (folder: `OBMP-Learning`)
+### OBMP-Learning Dashboards (folder: `OBMP-Reference`)

 Six learning-focused dashboards in a separate folder, designed to teach BGP concepts using live lab data.

@ -569,7 +578,7 @@ Six learning-focused dashboards in a separate folder, designed to teach BGP conc

 > **RPKI note:** The `rpki_validator` table is populated by a cron job in `psql-app` every 2 hours. Dashboard `obmp-learn-04` will show zero counts until the cron runs — check `ENABLE_RPKI=1` in `docker-compose.yml`.

-### Advanced Analytics Dashboards (folder: `OBMP-Learning`)
+### Advanced Analytics Dashboards (folder: `OBMP-Reference`)

 Four advanced dashboards that go beyond basic BMP monitoring, unlocking TE/SR data and providing heuristic analysis.

@ -584,7 +593,7 @@ Four advanced dashboards that go beyond basic BMP monitoring, unlocking TE/SR da

 ### Database Schema Reference

-A standalone database schema reference is also available at `docs/DB_SCHEMA.md`. It documents all 33 tables, 11 views, TE/SR columns, enum types, and common query patterns.
+A standalone database schema reference is also available at `DB_SCHEMA.md`. It documents all 33 tables, 11 views, TE/SR columns, enum types, and common query patterns.

 ---

@ -653,7 +662,7 @@ Should show topics like `openbmp.parsed.unicast_prefix`, `openbmp.parsed.peer`,

 ### 9.6 Grafana datasource

-Open `http://10.40.40.202:3000` → Configuration → Data Sources → OpenBMP → Test.
+Open `http://${HOST_IP}:3000` → Configuration → Data Sources → PostgreSQL → Test.
 Should return "Database Connection OK".

 ### 9.7 BMP collector receiving data
@ -822,7 +831,7 @@ Verify: `show bgp 1.1.1.0/24` — should show `Status: s (active), bestpath`.

 ### Grafana shows no data

-1. Check datasource: Configuration → Data Sources → OpenBMP → Test
+1. Check datasource: Configuration → Data Sources → PostgreSQL → Test
 2. Verify psql-app is writing: `docker compose -p obmp logs psql-app`
 3. Check the database directly (see database queries above)
 4. History dashboards need route churn — run `python3 inject.py churn`
@ -881,9 +890,7 @@ Adjust in `docker-compose.yml` under the `psql-app` service environment block.
 |----------|---------|-------------|
 | `EXABGP_LOCAL_IP` | `10.40.40.202` | Host IP ExaBGP binds to and uses as router-id |
 | `EXABGP_LOCAL_AS` | `65100` | ExaBGP's AS number |
-| `EXABGP_PEER_AS` | `65020` | AS of the IOS-XR lab |
-| `EXABGP_PEER_1` | `10.100.0.100` | First CORE router to peer with |
-| `EXABGP_PEER_2` | `10.100.0.200` | Second CORE router to peer with |
+| `EXABGP_PEERS` | (semicolon list) | Peers as `ip:peer_as:description;...` — replaces the retired `EXABGP_PEER_AS`/`EXABGP_PEER_1`/`EXABGP_PEER_2` |
 | `EXABGP_API_PORT` | `5050` | Flask API port |

 ### psql-app container (key variables)
@ -955,12 +962,13 @@ Expected: gRPC listening on port 57400.

 ### Telemetry Data Collected

-Telegraf subscribes to two IOS-XR YANG paths at 10-second intervals:
+Telegraf subscribes to two OpenConfig paths (see telegraf/telegraf.conf,
+the source of truth):

-| Subscription | YANG Path | Data |
-|-------------|-----------|------|
-| interface_counters | `Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-counters` | bytes/packets in/out, errors, drops, CRC |
-| interface_rates | `Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/data-rate` | bits/sec in/out, packet rate |
+| Subscription | OpenConfig path | Interval | Data |
+|-------------|-----------------|----------|------|
+| interface_counters | `openconfig-interfaces:/interfaces/interface/state/counters` | 10s | bytes/packets in/out, errors, drops |
+| interface_state | `openconfig-interfaces:/interfaces/interface/state` | 30s | oper status, MTU, speed |

 ### InfluxDB Access

@ -1110,6 +1118,6 @@ The **Combined BMP + Telemetry** dashboard shows both control-plane (BMP BGP upd

 | Variable | Default | Description |
 |----------|---------|-------------|
-| `TRAFFIC_GEN_API_PORT` | `5051` | Flask API listen port |
+| `TRAFFIC_GEN_PORT` | `5051` | Flask API listen port |
 | `TRAFFIC_GEN_MODE` | `sender` | Operating mode: `sender` or `responder` |
 | `INFLUXDB_TOKEN` | `openbmp-telemetry-token` | InfluxDB auth token (Telegraf) |
--- a/docs/PORTABILITY-FINDINGS.md
+++ b/docs/PORTABILITY-FINDINGS.md
@ -41,7 +41,7 @@ feeders.
 ## 6. Size for BMP burst, not steady state
 Generic docs say the collector is light — true, and misleading: the store
 saturates under burst (full table x monitored sessions, Postgres write
-amplification). See docs/DEPLOYMENT-TYPES.md for the full model.
+amplification). See DEPLOYMENT-TYPES.md for the full model.

 ## 7. Ports 5000 and 3000 are crowded
 The collector's 5000 and Grafana's 3000 are commonly already occupied.
@ -100,4 +100,4 @@ setup.sh renders the same port into gobgp's BMP export (gobgp is
 host-networked and must hit the *published* port), and the cml tooling reads
 `ROUTER_FACING_IP`/`ROUTER_FACING_PORT` from `.env`. Remaining hands-on step:
 re-apply the config to routers still pointed at 5000. See
-docs/router-bmp-config.md.
+router-bmp-config.md.
--- a/docs/README.md
+++ b/docs/README.md
@ -0,0 +1,32 @@
+# Documentation index
+
+Two tracks depending on who you are. Docs higher in each list are the entry
+points; the rest are references you'll be routed to when needed.
+
+## Operators — deploying and running the stack
+
+| Doc | What it covers |
+|---|---|
+| [../README.md](../README.md) | Quickstart: clone, pin the checkout, run `deploy.sh` |
+| [DEPLOYMENT-TYPES.md](DEPLOYMENT-TYPES.md) | The three `--scope` types (full-stack / remote / central-store) + burst sizing model |
+| [PORTABILITY-FINDINGS.md](PORTABILITY-FINDINGS.md) | Every greenfield trap and how the tooling handles it — read before deploying to a new host |
+| [production-sizing.md](production-sizing.md) | Memory limits and host sizing for production |
+| [backup-restore.md](backup-restore.md) | Postgres backup/restore (`scripts/pg-backup.sh`) |
+| [security-hardening.md](security-hardening.md) | Hardening checklist for exposed deployments |
+| [DB_SCHEMA.md](DB_SCHEMA.md) | Full database schema reference (33 tables, views, query patterns) |
+| [ROADMAP.md](ROADMAP.md) | Longer-term work items |
+
+## Network engineers — feeding routers into the stack
+
+| Doc | What it covers |
+|---|---|
+| [router-integration.md](router-integration.md) | **Start here.** The four ingest paths (BMP, BGP-LS, gNMI, NETCONF), bring-up order, verification |
+| [router-bmp-config.md](router-bmp-config.md) | BMP deep-dive: address/port selection (WSL vs native), activation scope, troubleshooting |
+| [../router-blueprints/](../router-blueprints/) | Copy-paste IOS-XR fragments with `<PLACEHOLDER>` substitutions |
+
+## Legacy
+
+| Doc | Status |
+|---|---|
+| [DOCS.md](DOCS.md) | Original single-host dev-lab walkthrough. Values are illustrative; where it disagrees with the docs above, they win. |
+| [handoff/](handoff/) | Point-in-time session handoff packets, kept for the record. |
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@ -96,9 +96,10 @@ Replace hardcoded IPs in `docker-compose.yml` (Kafka listener, ExaBGP env vars).

 Replace hardcoded gNMI addresses in `telegraf/telegraf.conf` with env var substitution. Pass `GNMI_TARGETS` from docker-compose.yml.

-### A6. Fix InfluxDB datasource URL
+### A6. Fix InfluxDB datasource URL — DONE

-`obmp-grafana/provisioning/datasources/influxdb-ds.yml`: replace `http://10.40.40.202:8086` with `http://obmp-influxdb:8086`.
+Completed: `obmp-grafana/provisioning/datasources/influxdb-ds.yml` now reads
+`http://obmp-influxdb:8086`.

 ---

--- a/docs/backup-restore.md
+++ b/docs/backup-restore.md
@ -115,8 +115,9 @@ EOSQL
 ```

 > Restoring into a **brand-new container**? Bring `obmp-psql` up first and let
-> it initialize, but **do not** create the `config/init_db` trigger file —
-> the schema comes from the dump, not from psql-app's first-run migration.
+> it initialize, and create the skip-marker BEFORE psql-app's first start —
+> `touch ${OBMP_DATA_ROOT}/config/do_not_init_db` — so psql-app does not run
+> its first-run schema migration; the schema comes from the dump instead.

 ### 3. Restore the dump

--- a/docs/router-bmp-config.md
+++ b/docs/router-bmp-config.md
@ -44,7 +44,7 @@ bmp server 1 initial-refresh delay 60 spread 30

 The `initial-refresh delay/spread` staggering matters at scale: it spreads
 the full-RIB dumps out when many routers (re)connect at once, which is the
-burst that sizes the whole store tier (docs/DEPLOYMENT-TYPES.md).
+burst that sizes the whole store tier (DEPLOYMENT-TYPES.md).

 ## Activating monitoring on BGP sessions

@ -52,7 +52,7 @@ BMP only reports sessions that are `bmp-activate`d:

 ```
 router bgp <ASN>
- neighbor-group RR-CLIENTS
+ neighbor-group RR-CLIENT
  bmp-activate server 1
 !
 !
--- a/scripts/deploy-lib.sh
+++ b/scripts/deploy-lib.sh
@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+#
+# deploy-lib.sh - shared helpers for deploy.sh and setup.sh.
+#
+# Both scripts source this file so output styling and .env handling stay
+# identical and are defined exactly once. Safe to source multiple times.
+#
+# Provides:
+#   log / ok / warn / die / hr        - consistent colored output
+#   banner "title"                    - boxed section banner
+#   step N M "title"                  - numbered decision/stage header
+#   ask VAR "prompt" "default"        - editable-default prompt (honors ARG_YES)
+#   confirm "prompt"                  - y/N confirm (honors ARG_YES)
+#   get_env KEY / set_env KEY VAL     - .env read/write without sourcing it
+#   is_ipv4 ADDR                      - dotted-quad shape check
+#
+# Conventions: callers may set ARG_YES=1 for unattended runs and TAG to
+# change the log prefix (defaults to "deploy").
+
+TAG="${TAG:-deploy}"
+ARG_YES="${ARG_YES:-0}"
+
+log()  { printf '\033[1;36m[%s]\033[0m %s\n' "$TAG" "$*"; }
+ok()   { printf '\033[1;32m[%s] \xe2\x9c\x94\033[0m %s\n' "$TAG" "$*"; }
+warn() { printf '\033[1;33m[%s][warn]\033[0m %s\n' "$TAG" "$*" >&2; }
+die()  { printf '\033[1;31m[%s][err]\033[0m %s\n' "$TAG" "$*" >&2; exit 1; }
+hr()   { printf '\033[2m%s\033[0m\n' "----------------------------------------------------------------"; }
+
+# Boxed banner, e.g.:
+#   +--------------------------------------------------------------+
+#   |  OpenBMP stack deploy                                        |
+#   +--------------------------------------------------------------+
+banner() {
+    local title="$1" width=62
+    printf '\033[1;36m+%s+\n' "$(printf -- '-%.0s' $(seq 1 $width))"
+    printf '|  %-*s|\n' "$((width - 2))" "$title"
+    printf '+%s+\033[0m\n' "$(printf -- '-%.0s' $(seq 1 $width))"
+}
+
+# Numbered header for decisions and stages: step 2 5 "Deployment type"
+step() {
+    local n="$1" total="$2" title="$3"
+    printf '\n\033[1;36m[%s] Step %s/%s\033[0m \033[1m%s\033[0m\n' "$TAG" "$n" "$total" "$title"
+}
+
+# Prompt with an editable default. Honors ARG_YES (accepts default silently).
+ask() {
+    local __var="$1" __prompt="$2" __default="${3:-}" __reply=""
+    if [ "$ARG_YES" -eq 1 ]; then
+        printf -v "$__var" '%s' "$__default"; return
+    fi
+    if [ -n "$__default" ]; then
+        read -r -p "$__prompt [$__default]: " __reply
+        __reply="${__reply:-$__default}"
+    else
+        read -r -p "$__prompt: " __reply
+    fi
+    printf -v "$__var" '%s' "$__reply"
+}
+
+confirm() {
+    local __prompt="$1" __reply=""
+    [ "$ARG_YES" -eq 1 ] && return 0
+    read -r -p "$__prompt [y/N]: " __reply
+    [ "$__reply" = "y" ] || [ "$__reply" = "Y" ]
+}
+
+# Read a single KEY=value from .env without sourcing it -- .env contains keys
+# with hyphens (PROX-CML_*) that a shell `source` would choke on.
+get_env() { grep -E "^$1=" .env | head -1 | cut -d= -f2- || true; }
+
+# Set KEY=value in .env: replace the line if present, else append.
+set_env() {
+    local key="$1" val="$2"
+    if grep -qE "^${key}=" .env; then
+        # `|` delimiter -- values are hex/IPs, no `|`.
+        sed -i "s|^${key}=.*|${key}=${val}|" .env
+    else
+        printf '%s=%s\n' "$key" "$val" >> .env
+    fi
+}
+
+# Dotted-quad shape check (rejects placeholders like <WINDOWS_LAN_IP>).
+is_ipv4() {
+    local ip="$1" o
+    [[ "$ip" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]] || return 1
+    IFS=. read -r -a o <<<"$ip"
+    for x in "${o[@]}"; do [ "$x" -le 255 ] || return 1; done
+    return 0
+}