obmp-portability-test/docs/redesign-real-routers.md

Redesign: Real-Router BMP Ingest Test (v2)

Where we landed after the first run

The original two-VM design (docs/design.md) proved deploy.sh runs to completion on a fresh Ubuntu host and that a synthetic GoBGP peer can open a BMP session to the collector across a real network boundary. That work found and fixed three real obmp-docker bugs (bb71c9f, 745f066, 6df67a7 on deploy-refinement) plus one design issue on the bmpgen VM (d0b82c4 here). Portability of the deploy is now demonstrated.

Then the intent shifted: use the test collector to receive BMP from the real lab routers, running alongside the production collector via IOS-XR's second-server slot (bmp server 2). The router-side extension was written (scripts/router_bmp.py, terraform/router_bmp.tf) but is dormant (enable_router_bmp=false). The 4 GB store VM OOM'd under the existing internal ingest (251 K routes from Bromirski via --profile test) before we ever turned the extension on.

That's the pivot: v1 was "prove deploy works". v2 is "receive real router BMP as a canary collector". Different workload, different sizing, different scope.

Current state

• Both v1 VMs (9001 store, 9002 bmpgen) destroyed via PVE API after terraform destroy hung on the OOM-killed guest agent.
• Terraform state cleaned; only the cached Ubuntu image resource remains (kept for a fast next apply).
• Router extension code committed but disabled by default.
• Nothing on prox940 or the lab routers has been touched.

Proposed v2 topology

Single VM, not two. Bmpgen has no role once real routers are the BMP source — it was a stand-in for exactly that.

   +---------------------------------------------------+
   |             store VM (v2)                         |
   |   ubuntu 24.04 cloud + deploy.sh                  |
   |                                                   |
   |   scope: full-stack (no --profile test)           |
   |   collector on ROUTER_FACING_IP : 1790            |
   +---------------------------------------------------+
              ^         ^         ^
              |         |         |
         bmp server 2 from every lab router
         (all 18: CML + PROX; RR-scope on BMP-MONITORED
         neighbor-group; mirrors production activation)

Sizing (needs your call)

The single biggest miss in v1 was memory. Real load drivers:

• collector: light, but the burst on PEER_UP full-RIB dumps is where memory spikes.
• kafka: 4 GB heap plus page cache; 8 GB minimum for the ingest spool under a full-table burst.
• psql: shared_buffers = 6 GB in the shipped image. Under real ingest, work_mem + autovacuum churn adds several GB more.
• psql-app: 3 GB JVM heap per current defaults.
• grafana + monitors + rib-poller: ~1 GB combined.

Production (fleet-feed-stress-test in your memory notes) is on 62 GB after the RAM bump. For the duplicate stream, we're not asking for production sizing — the test collector doesn't need to sustain the same peak indefinitely, just enough to receive, ingest, and let you run diff queries. My best guess:

• Minimum viable: 16 GB / 4 vCPU / 64 GB disk. Handles all 18 RR-scope BMP sessions if none are pushing full internet tables at the same instant. Likely to swap under a coordinated PEER UP storm.
• Comfortable: 32 GB / 8 vCPU / 128 GB disk. Matches the production sizing for the store tier per docs/DEPLOYMENT-TYPES.md ("full-stack: prod-realistic 16 vCPU / 48-64 GB / NVMe >=250 GB").
• Overkill for a canary: 64 GB. Only if you plan to leave it up long-term ingesting real routes for extended comparison studies.

Recommendation: 32 GB / 8 vCPU / 128 GB. Matches your "canary that sees what prod sees" intent without dominating prox940 (which has 754 GB and 96 cores, so 32 GB is 4%).

Compose scope

Skip --profile test. The internal gobgp there was what pulled the 251 K routes from Bromirski that filled up v1's 4 GB. For a real-router test, we want the collector receiving only real router BMP — no synthetic feed muddying the diffs.

Current deploy.sh only offers scope full-stack (implies --profile test), remote, or central-store. Fresh portability find: full-stack conflates "the full stack" with "the lab feeders", which is fine for a demo but not for a canary collector.

Two options:

1. Extend deploy.sh --scope full-stack --no-feeders as a new option (adds a real portability enhancement to obmp-docker; landing on deploy-refinement).
2. Sidestep in cloud-init: run deploy.sh --scope full-stack --yes, then explicitly docker compose --profile test down after to remove the synthetic feeders.

Option 1 is cleaner and lands a real feature. Option 2 is a workaround we could remove once option 1 exists.

Router extension (unchanged in intent)

scripts/router_bmp.py + terraform/router_bmp.tf are already written and correct for v2. Two-step safety still holds: first apply with enable_router_bmp=true runs a dry-run (prints per-router pending diffs, does NOT commit); set confirm_router_push=true and re-apply to actually push. Destroy provisioner removes it cleanly, on_failure = continue so a single unreachable router doesn't block VM teardown.

Scope: bmp-activate server 2 on BMP-MONITORED neighbor-group per router — matches production activation pattern (enables diff-based validation between the two collectors).

Router creds: read via ROUTER_USER/ROUTER_PASS env for CML; hardcoded admin/cisco for PROX (matches the inventory pattern in obmp-docker/obmp-rib-poller/poller.py).

Assertion set (needs updating)

scripts/run-test.sh's six assertions were written for the v1 synthetic-feed topology. For v2:

#	Old (v1)	New (v2)
1	All obmp containers healthy	Same
2	Collector accepted a BMP session from bmpgen	Collector accepted BMP sessions from >= N of the 18 routers
3	routers table has >= 1 row	routers table contains the expected router names
4	bgp_peers has >= 1 UP session	bgp_peers count matches production within +/- 10%
5	ip_rib has >= 1 row	ip_rib count matches production within +/- 5% (canary!)
6	Grafana /api/health	Same

The "matches production" assertions are what make this a canary collector — we're not just checking that data lands, we're checking that it lands in about the same shape as the production feed. Needs the production DB to be queryable from the same host that runs the assertions.

What I did NOT do while you were asleep

• Did not enable enable_router_bmp or touch any lab router.
• Did not create new VMs.
• Did not change compose scope / deploy.sh flags in obmp-docker.
• Did not push v2 sizing changes to terraform.tfvars.

What I DID do

• Force-destroyed VMs 9001 + 9002 via PVE API (terraform destroy hung on the dead guest agent).
• Cleaned terraform state to match reality.
• Committed the dormant router extension code + the earlier bug-fix commit set from the v1 run.
• Wrote this doc.

Decisions you'll want to make when you're back

1. Sizing: 16 GB / 32 GB / other for the v2 store VM?
2. Compose scope: add --no-feeders to deploy.sh (option 1 above), or workaround in cloud-init (option 2)?
3. Ready to enable router extension? Once v2 store VM is up and healthy, first apply is a safe dry-run.
4. Assertion updates: leave the v1 assertions in place for now, or evolve to the "matches production" set right away? (The latter requires network reach + creds for the production Postgres.)

Any of these can wait until we've stood up the v2 VM and confirmed it's ingesting fine.