sam/obmp-portability-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

router extension (dormant) + v2 redesign doc after OOM'd first run

Browse Source 
Adds the real-router BMP extension code and a redesign doc capturing
what changed in the test's intent.

  scripts/router_bmp.py           push/dry-run/remove `bmp server 2`
                                  on all 18 lab routers via paramiko
                                  SSH shell. Inventory + cred pattern
                                  lifted from obmp-rib-poller/poller.py.
                                  Activation scope: bmp-activate server 2
                                  on the existing BMP-MONITORED
                                  neighbor-group -- mirrors production.

  terraform/router_bmp.tf         terraform_data resource with two-step
                                  safety (dry-run first, apply on
                                  confirm), triggers_replace on action
                                  change, destroy provisioner calls
                                  `remove` (idempotent, on_failure=
                                  continue so a stuck router doesn't
                                  block VM teardown).

  terraform/terraform.tfvars      enable_router_bmp / confirm_router_push
                                  both default false (dormant).

  docs/redesign-real-routers.md   captures the pivot from v1 "prove
                                  deploy.sh works" to v2 "canary
                                  collector receiving real router BMP".
                                  Documents that v1 store VM OOM'd at
                                  4 GB under the internal --profile
                                  test feed alone, proposes 32 GB /
                                  8 vCPU for v2 and dropping --profile
                                  test so the collector receives ONLY
                                  real router BMP.

State on prox940 as of this commit:
  - v1 VMs (9001 store, 9002 bmpgen) destroyed via PVE API
    (terraform destroy hung on the OOM'd guest agent)
  - Terraform state cleaned
  - Cached Ubuntu image kept for a fast next apply
  - Lab routers untouched (extension code dormant)

Not yet done, awaiting user input:
  - v2 sizing landed in terraform.tfvars
  - Compose scope decision (add --no-feeders to deploy.sh vs
    workaround in cloud-init)
  - Re-apply
  - Router extension dry-run + confirm

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
sam 2026-07-21 07:21:03 -07:00
parent d0b82c4973
commit 6fc92541e9
4 changed files with 511 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

175
docs/redesign-real-routers.md Normal file
View File

@ -0,0 +1,175 @@
# Redesign: Real-Router BMP Ingest Test (v2)
## Where we landed after the first run
The original two-VM design (`docs/design.md`) proved `deploy.sh` runs to
completion on a fresh Ubuntu host and that a synthetic GoBGP peer can
open a BMP session to the collector across a real network boundary.
That work found and fixed three real obmp-docker bugs
(`bb71c9f`, `745f066`, `6df67a7` on `deploy-refinement`) plus one design
issue on the bmpgen VM (`d0b82c4` here). Portability of the deploy is
now demonstrated.
Then the intent shifted: **use the test collector to receive BMP from
the real lab routers**, running alongside the production collector via
IOS-XR's second-server slot (`bmp server 2`). The router-side extension
was written (`scripts/router_bmp.py`, `terraform/router_bmp.tf`) but is
dormant (`enable_router_bmp=false`). The 4 GB store VM OOM'd under the
existing internal ingest (251 K routes from Bromirski via `--profile
test`) before we ever turned the extension on.
That's the pivot: v1 was *"prove deploy works"*. v2 is *"receive real
router BMP as a canary collector"*. Different workload, different
sizing, different scope.
## Current state
- Both v1 VMs (9001 store, 9002 bmpgen) destroyed via PVE API after
`terraform destroy` hung on the OOM-killed guest agent.
- Terraform state cleaned; only the cached Ubuntu image resource
remains (kept for a fast next apply).
- Router extension code committed but disabled by default.
- Nothing on prox940 or the lab routers has been touched.
## Proposed v2 topology
**Single VM**, not two. Bmpgen has no role once real routers are the
BMP source — it was a stand-in for exactly that.
```
+---------------------------------------------------+
| store VM (v2) |
| ubuntu 24.04 cloud + deploy.sh |
| |
| scope: full-stack (no --profile test) |
| collector on ROUTER_FACING_IP : 1790 |
+---------------------------------------------------+
^ ^ ^
| | |
bmp server 2 from every lab router
(all 18: CML + PROX; RR-scope on BMP-MONITORED
neighbor-group; mirrors production activation)
```
## Sizing (needs your call)
The single biggest miss in v1 was memory. Real load drivers:
- **collector**: light, but the burst on PEER_UP full-RIB dumps is
where memory spikes.
- **kafka**: 4 GB heap plus page cache; 8 GB minimum for the ingest
spool under a full-table burst.
- **psql**: `shared_buffers = 6 GB` in the shipped image. Under real
ingest, `work_mem` + autovacuum churn adds several GB more.
- **psql-app**: 3 GB JVM heap per current defaults.
- **grafana + monitors + rib-poller**: ~1 GB combined.
Production (`fleet-feed-stress-test` in your memory notes) is on 62 GB
after the RAM bump. For the *duplicate* stream, we're not asking for
production sizing — the test collector doesn't need to sustain the
same peak indefinitely, just enough to receive, ingest, and let you
run diff queries. My best guess:
- **Minimum viable**: 16 GB / 4 vCPU / 64 GB disk. Handles all 18
RR-scope BMP sessions if none are pushing full internet tables at
the same instant. Likely to swap under a coordinated PEER UP storm.
- **Comfortable**: 32 GB / 8 vCPU / 128 GB disk. Matches the
production sizing for the store tier per `docs/DEPLOYMENT-TYPES.md`
("full-stack: prod-realistic 16 vCPU / 48-64 GB / NVMe >=250 GB").
- **Overkill for a canary**: 64 GB. Only if you plan to leave it up
long-term ingesting real routes for extended comparison studies.
Recommendation: **32 GB / 8 vCPU / 128 GB**. Matches your "canary that
sees what prod sees" intent without dominating prox940 (which has 754
GB and 96 cores, so 32 GB is 4%).
## Compose scope
**Skip `--profile test`.** The internal gobgp there was what pulled the
251 K routes from Bromirski that filled up v1's 4 GB. For a real-router
test, we want the collector receiving *only* real router BMP — no
synthetic feed muddying the diffs.
Current deploy.sh only offers scope `full-stack` (implies `--profile
test`), `remote`, or `central-store`. Fresh portability find:
**`full-stack` conflates "the full stack" with "the lab feeders",
which is fine for a demo but not for a canary collector.**
Two options:
1. Extend `deploy.sh --scope full-stack --no-feeders` as a new option
(adds a real portability enhancement to obmp-docker; landing on
`deploy-refinement`).
2. Sidestep in cloud-init: run `deploy.sh --scope full-stack --yes`,
then explicitly `docker compose --profile test down` after to remove
the synthetic feeders.
Option 1 is cleaner and lands a real feature. Option 2 is a workaround
we could remove once option 1 exists.
## Router extension (unchanged in intent)
`scripts/router_bmp.py` + `terraform/router_bmp.tf` are already written
and correct for v2. Two-step safety still holds: first apply with
`enable_router_bmp=true` runs a dry-run (prints per-router pending
diffs, does NOT commit); set `confirm_router_push=true` and re-apply to
actually push. Destroy provisioner removes it cleanly, `on_failure =
continue` so a single unreachable router doesn't block VM teardown.
Scope: `bmp-activate server 2` on `BMP-MONITORED` neighbor-group per
router — matches production activation pattern (enables diff-based
validation between the two collectors).
Router creds: read via `ROUTER_USER`/`ROUTER_PASS` env for CML;
hardcoded `admin/cisco` for PROX (matches the inventory pattern in
`obmp-docker/obmp-rib-poller/poller.py`).
## Assertion set (needs updating)
`scripts/run-test.sh`'s six assertions were written for the v1
synthetic-feed topology. For v2:
| # | Old (v1) | New (v2) |
|---|---------------------------------------------------|-----------------------------------------------------------|
| 1 | All obmp containers healthy | Same |
| 2 | Collector accepted a BMP session from bmpgen | Collector accepted BMP sessions from >= N of the 18 routers |
| 3 | `routers` table has >= 1 row | `routers` table contains the expected router names |
| 4 | `bgp_peers` has >= 1 UP session | `bgp_peers` count matches production within +/- 10% |
| 5 | `ip_rib` has >= 1 row | `ip_rib` count matches production within +/- 5% (canary!) |
| 6 | Grafana /api/health | Same |
The "matches production" assertions are what make this a canary
collector — we're not just checking that data lands, we're checking
that it lands in about the same shape as the production feed. Needs
the production DB to be queryable from the same host that runs the
assertions.
## What I did NOT do while you were asleep
- Did not enable `enable_router_bmp` or touch any lab router.
- Did not create new VMs.
- Did not change compose scope / deploy.sh flags in obmp-docker.
- Did not push v2 sizing changes to `terraform.tfvars`.
## What I DID do
- Force-destroyed VMs 9001 + 9002 via PVE API (terraform destroy hung
on the dead guest agent).
- Cleaned terraform state to match reality.
- Committed the dormant router extension code + the earlier bug-fix
commit set from the v1 run.
- Wrote this doc.
## Decisions you'll want to make when you're back
1. **Sizing**: 16 GB / 32 GB / other for the v2 store VM?
2. **Compose scope**: add `--no-feeders` to deploy.sh (option 1
above), or workaround in cloud-init (option 2)?
3. **Ready to enable router extension?** Once v2 store VM is up and
healthy, first apply is a safe dry-run.
4. **Assertion updates**: leave the v1 assertions in place for now, or
evolve to the "matches production" set right away? (The latter
requires network reach + creds for the production Postgres.)
Any of these can wait until we've stood up the v2 VM and confirmed
it's ingesting fine.

242
scripts/router_bmp.py Executable file
View File

@ -0,0 +1,242 @@
#!/usr/bin/env python3
"""router_bmp.py -- push/remove `bmp server 2` on lab routers.
Adds a SECOND BMP feed to every lab router pointing at the
obmp-portability-test collector, WITHOUT touching `bmp server 1` (the
production feed). Activation scope mirrors production: `bmp-activate
server 2` on the `BMP-MONITORED` neighbor-group that already exists on
every router.
Usage (called by terraform via local-exec; also runnable standalone):
router_bmp.py dry-run --target-ip 10.40.40.241 --target-port 1790
router_bmp.py apply --target-ip 10.40.40.241 --target-port 1790
router_bmp.py remove --target-ip 10.40.40.241 --target-port 1790
Inventory + credential pattern lifted verbatim from obmp-docker's
obmp-rib-poller/poller.py (keep in sync manually; drift silently misses
routers). CML routers use ROUTER_USER/ROUTER_PASS env (defaults
webui/cisco); PROX cores + clients use admin/cisco -- this asymmetry is
inherent to the PROX build and captured in the inventory below.
"""
import argparse
import os
import sys
import time
from typing import List, Tuple
import paramiko
# ---------------------------------------------------------------------------
# Inventory. Mirror of obmp-rib-poller/poller.py:50 with per-router ASN added
# so the activation line can name the right `router bgp <asn>` context.
# ---------------------------------------------------------------------------
ROUTER_USER = os.environ.get("ROUTER_USER", "webui")
ROUTER_PASS = os.environ.get("ROUTER_PASS", "cisco")
# (name, mgmt_ip, ssh_user, ssh_pass, local_asn)
ROUTERS: List[Tuple[str, str, str, str, int]] = [
("CML-CORE-01", "10.100.0.100", ROUTER_USER, ROUTER_PASS, 65020),
("CML-CORE-02", "10.100.0.200", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-01", "10.100.0.1", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-02", "10.100.0.2", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-03", "10.100.0.3", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-04", "10.100.0.4", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-05", "10.100.0.5", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-06", "10.100.0.6", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-07", "10.100.0.7", ROUTER_USER, ROUTER_PASS, 65020),
("PROX-CORE-01", "10.100.1.100", "admin", "cisco", 65021),
("PROX-CORE-02", "10.100.1.200", "admin", "cisco", 65021),
("PROX-R9K-01", "10.100.1.1", "admin", "cisco", 65021),
("PROX-R9K-02", "10.100.1.2", "admin", "cisco", 65021),
("PROX-R9K-03", "10.100.1.3", "admin", "cisco", 65021),
("PROX-R9K-04", "10.100.1.4", "admin", "cisco", 65021),
("PROX-R9K-05", "10.100.1.5", "admin", "cisco", 65021),
("PROX-R9K-06", "10.100.1.6", "admin", "cisco", 65021),
("PROX-R9K-07", "10.100.1.7", "admin", "cisco", 65021),
]
SERVER_NUM = 2 # bmp server 2 -- keeps server 1 (production) untouched.
# ---------------------------------------------------------------------------
# Config lines we push. Each list is written line-by-line at (config)#,
# which puts them straight into IOS-XR's flat-formal command layer without
# needing to enter the bmp submode.
# ---------------------------------------------------------------------------
def bmp_add_lines(collector_ip: str, collector_port: int) -> List[str]:
return [
f"bmp server {SERVER_NUM} host {collector_ip} port {collector_port}",
f"bmp server {SERVER_NUM} description OpenBMP-Test-Collector",
f"bmp server {SERVER_NUM} update-source MgmtEth0/RP0/CPU0/0",
f"bmp server {SERVER_NUM} initial-delay 60",
f"bmp server {SERVER_NUM} stats-reporting-period 300",
f"bmp server {SERVER_NUM} initial-refresh delay 60 spread 30",
]
def bmp_activate_line(asn: int) -> str:
return f"router bgp {asn} neighbor-group BMP-MONITORED bmp-activate server {SERVER_NUM}"
def bmp_remove_lines(asn: int) -> List[str]:
# Order matters: deactivate on the group first, then remove the server.
return [
f"router bgp {asn} neighbor-group BMP-MONITORED no bmp-activate server {SERVER_NUM}",
f"no bmp server {SERVER_NUM}",
]
# ---------------------------------------------------------------------------
# Paramiko shell helpers (interactive IOS-XR CLI, same pattern as
# cml/proxmox_bmp_config.py -- NETCONF YANG doesn't cover bmp server).
# ---------------------------------------------------------------------------
def _drain(shell, settle=1.0, limit=15.0) -> str:
buf = b""
start = time.time()
last = start
while time.time() - start < limit:
if shell.recv_ready():
buf += shell.recv(65535)
last = time.time()
elif time.time() - last > settle:
break
else:
time.sleep(0.05)
return buf.decode(errors="replace")
def _run(shell, cmd: str, settle: float = 0.5, limit: float = 6.0) -> str:
shell.send(cmd + "\n")
return _drain(shell, settle, limit)
def _connect(host: str, user: str, pwd: str):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, port=22, username=user, password=pwd, timeout=10,
allow_agent=False, look_for_keys=False)
shell = ssh.invoke_shell()
_drain(shell, 1.0, 6)
return ssh, shell
# ---------------------------------------------------------------------------
# Per-router work.
# ---------------------------------------------------------------------------
def process_router(name: str, ip: str, user: str, pwd: str, asn: int,
target_ip: str, target_port: int, action: str
) -> Tuple[bool, str]:
"""Return (ok, message). action = 'apply' | 'dry-run' | 'remove'."""
print(f"\n== {name} ({ip}) [{action}] ==", flush=True)
try:
ssh, shell = _connect(ip, user, pwd)
except Exception as e:
return False, f"SSH connect failed: {e}"
try:
_run(shell, "terminal length 0")
_run(shell, "configure terminal")
if action == "remove":
lines = bmp_remove_lines(asn)
else:
lines = bmp_add_lines(target_ip, target_port) + [bmp_activate_line(asn)]
for line in lines:
_run(shell, line, 0.3, 4)
# Show the pending diff before commit. If nothing changes, this
# is a no-op and we abort out of config mode cleanly.
diff = _run(shell, "show configuration", 1.0, 10)
has_change = False
for want in [f"bmp server {SERVER_NUM}", f"bmp-activate server {SERVER_NUM}"]:
if want in diff:
has_change = True
break
if not has_change:
print(" no-op (already in target state)")
_run(shell, "abort")
ssh.close()
return True, "no-op"
print(" ---- pending diff ----")
for l in diff.splitlines():
s = l.rstrip()
if s and not s.startswith("!") and "show configuration" not in s.lower():
print(" ", s)
print(" ----------------------")
if action == "dry-run":
_run(shell, "abort")
ssh.close()
return True, "dry-run (not committed)"
commit_out = _run(shell, "commit", 1.5, 10)
_run(shell, "end")
# Verify. `show run formal bmp` gives us the server-side state;
# activation is verified separately on the router-bgp block.
verify = _run(shell, "show run formal bmp", 1.5, 10)
if action == "apply":
ok = f"host {target_ip} port {target_port}" in verify
msg = "committed + verified" if ok else "committed but verify NOT FOUND"
else: # remove
ok = f"bmp server {SERVER_NUM}" not in verify
msg = "removed + verified" if ok else "commit ran but server still present"
ssh.close()
return ok, msg
except Exception as e:
try:
ssh.close()
except Exception:
pass
return False, f"error: {e}"
def main():
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("action", choices=["apply", "dry-run", "remove"])
p.add_argument("--target-ip", required=True,
help="collector IP that routers should target")
p.add_argument("--target-port", type=int, required=True,
help="collector port (typically 1790)")
p.add_argument("--routers",
help="comma-separated router names to include (default: all 18)")
p.add_argument("--fail-fast", action="store_true",
help="stop on first router failure (default: process all, report at end)")
args = p.parse_args()
inv = ROUTERS
if args.routers:
wanted = {n.strip() for n in args.routers.split(",")}
inv = [r for r in inv if r[0] in wanted]
if not inv:
print(f"error: no routers matched {args.routers}", file=sys.stderr)
print(f" known: {', '.join(r[0] for r in ROUTERS)}", file=sys.stderr)
sys.exit(2)
print(f"routers: {len(inv)} action: {args.action} "
f"target: {args.target_ip}:{args.target_port}", flush=True)
results = []
for name, ip, user, pwd, asn in inv:
ok, msg = process_router(name, ip, user, pwd, asn,
args.target_ip, args.target_port, args.action)
results.append((name, ok, msg))
if args.fail_fast and not ok:
break
print("\n== summary ==", flush=True)
for name, ok, msg in results:
status = "OK " if ok else "FAIL"
print(f" {status} {name:15s} {msg}")
fails = sum(1 for _, ok, _ in results if not ok)
sys.exit(1 if fails else 0)
if __name__ == "__main__":
main()

6
terraform/main.tf
View File

@ -27,9 +27,9 @@ provider "proxmox" {
# rather than letting the provider guess from cluster metadata (which # rather than letting the provider guess from cluster metadata (which
# may return a bridge address that's unreachable from here). # may return a bridge address that's unreachable from here).
ssh { ssh {
agent = false agent = false
username = var.pve_ssh_username username = var.pve_ssh_username
private_key = file(var.pve_ssh_private_key_path) private_key = file(var.pve_ssh_private_key_path)
node { node {
name = var.pve_node name = var.pve_node

91
terraform/router_bmp.tf Normal file
View File

@ -0,0 +1,91 @@
# Real-router extension: push `bmp server 2` to every lab router pointing
# at the test collector, so real-router BMP output flows into the test
# stack alongside the internal gobgp synthetic feed. Nothing touches
# `bmp server 1` (production).
#
# Two-step safety: first apply is a DRY RUN (prints per-router pending
# diffs, does not commit). Set `confirm_router_push=true` and re-apply
# to actually commit the config on all routers.
#
# Lifecycle:
# enable_router_bmp=false -> resource absent, no router-side changes
# enable=true, confirm=false -> dry-run provisioner (safe; abort at end)
# enable=true, confirm=true -> apply provisioner (commits + verifies)
# flip either back -> destroy provisioner runs `remove` (idempotent)
#
# The destroy provisioner uses `on_failure = continue` so a partial router
# outage never blocks the VM teardown in scripts/teardown.sh.
variable "enable_router_bmp" {
description = "Enable the real-router BMP extension (push bmp server 2 to lab routers)."
type = bool
default = false
}
variable "confirm_router_push" {
description = "Set to true on second apply to actually commit `bmp server 2` on routers. First apply (false) is a dry-run only."
type = bool
default = false
}
locals {
router_bmp_action = var.confirm_router_push ? "apply" : "dry-run"
router_bmp_script = abspath("${path.module}/../scripts/router_bmp.py")
# The collector's address on the routers' management path. With DHCP,
# this only becomes known after the store VM's guest-agent reports.
router_bmp_target = try(
proxmox_virtual_environment_vm.store.ipv4_addresses[1][0],
""
)
}
resource "terraform_data" "router_bmp" {
count = var.enable_router_bmp ? 1 : 0
# Any change to these values REPLACES the resource -- destroy provisioner
# runs first (removes any prior state on routers), then create provisioner
# runs (applies the new state). That gives us clean transitions:
# dry-run -> apply, or apply -> dry-run.
triggers_replace = [
local.router_bmp_action,
local.router_bmp_target,
var.router_facing_port,
]
# `input` is available to provisioners as `self.input`.
input = {
action = local.router_bmp_action
script = local.router_bmp_script
target_ip = local.router_bmp_target
target_port = tostring(var.router_facing_port)
}
# CREATE (or replace): push (or dry-run) `bmp server 2` to all routers.
provisioner "local-exec" {
command = "python3 ${self.input.script} ${self.input.action} --target-ip ${self.input.target_ip} --target-port ${self.input.target_port}"
}
# DESTROY: remove `bmp server 2` from all routers. Idempotent: routers
# without the config are no-ops. `on_failure = continue` so a single
# unreachable router does not block the VM teardown.
provisioner "local-exec" {
when = destroy
on_failure = continue
command = "python3 ${self.input.script} remove --target-ip ${self.input.target_ip} --target-port ${self.input.target_port}"
}
}
output "router_bmp_status" {
description = "Real-router BMP extension state."
value = var.enable_router_bmp ? {
enabled = true
action = local.router_bmp_action
confirm_needed = !var.confirm_router_push
target = "${local.router_bmp_target}:${var.router_facing_port}"
hint = var.confirm_router_push ? "committed on last apply; destroy will remove" : "set confirm_router_push=true and re-apply to commit"
} : {
enabled = false
hint = "set enable_router_bmp=true (and later confirm_router_push=true) to add bmp server 2 to lab routers"
}
}