diff --git a/docs/redesign-real-routers.md b/docs/redesign-real-routers.md new file mode 100644 index 0000000..6d42099 --- /dev/null +++ b/docs/redesign-real-routers.md @@ -0,0 +1,175 @@ +# Redesign: Real-Router BMP Ingest Test (v2) + +## Where we landed after the first run + +The original two-VM design (`docs/design.md`) proved `deploy.sh` runs to +completion on a fresh Ubuntu host and that a synthetic GoBGP peer can +open a BMP session to the collector across a real network boundary. +That work found and fixed three real obmp-docker bugs +(`bb71c9f`, `745f066`, `6df67a7` on `deploy-refinement`) plus one design +issue on the bmpgen VM (`d0b82c4` here). Portability of the deploy is +now demonstrated. + +Then the intent shifted: **use the test collector to receive BMP from +the real lab routers**, running alongside the production collector via +IOS-XR's second-server slot (`bmp server 2`). The router-side extension +was written (`scripts/router_bmp.py`, `terraform/router_bmp.tf`) but is +dormant (`enable_router_bmp=false`). The 4 GB store VM OOM'd under the +existing internal ingest (251 K routes from Bromirski via `--profile +test`) before we ever turned the extension on. + +That's the pivot: v1 was *"prove deploy works"*. v2 is *"receive real +router BMP as a canary collector"*. Different workload, different +sizing, different scope. + +## Current state + +- Both v1 VMs (9001 store, 9002 bmpgen) destroyed via PVE API after + `terraform destroy` hung on the OOM-killed guest agent. +- Terraform state cleaned; only the cached Ubuntu image resource + remains (kept for a fast next apply). +- Router extension code committed but disabled by default. +- Nothing on prox940 or the lab routers has been touched. + +## Proposed v2 topology + +**Single VM**, not two. Bmpgen has no role once real routers are the +BMP source — it was a stand-in for exactly that. + +``` + +---------------------------------------------------+ + | store VM (v2) | + | ubuntu 24.04 cloud + deploy.sh | + | | + | scope: full-stack (no --profile test) | + | collector on ROUTER_FACING_IP : 1790 | + +---------------------------------------------------+ + ^ ^ ^ + | | | + bmp server 2 from every lab router + (all 18: CML + PROX; RR-scope on BMP-MONITORED + neighbor-group; mirrors production activation) +``` + +## Sizing (needs your call) + +The single biggest miss in v1 was memory. Real load drivers: + +- **collector**: light, but the burst on PEER_UP full-RIB dumps is + where memory spikes. +- **kafka**: 4 GB heap plus page cache; 8 GB minimum for the ingest + spool under a full-table burst. +- **psql**: `shared_buffers = 6 GB` in the shipped image. Under real + ingest, `work_mem` + autovacuum churn adds several GB more. +- **psql-app**: 3 GB JVM heap per current defaults. +- **grafana + monitors + rib-poller**: ~1 GB combined. + +Production (`fleet-feed-stress-test` in your memory notes) is on 62 GB +after the RAM bump. For the *duplicate* stream, we're not asking for +production sizing — the test collector doesn't need to sustain the +same peak indefinitely, just enough to receive, ingest, and let you +run diff queries. My best guess: + +- **Minimum viable**: 16 GB / 4 vCPU / 64 GB disk. Handles all 18 + RR-scope BMP sessions if none are pushing full internet tables at + the same instant. Likely to swap under a coordinated PEER UP storm. +- **Comfortable**: 32 GB / 8 vCPU / 128 GB disk. Matches the + production sizing for the store tier per `docs/DEPLOYMENT-TYPES.md` + ("full-stack: prod-realistic 16 vCPU / 48-64 GB / NVMe >=250 GB"). +- **Overkill for a canary**: 64 GB. Only if you plan to leave it up + long-term ingesting real routes for extended comparison studies. + +Recommendation: **32 GB / 8 vCPU / 128 GB**. Matches your "canary that +sees what prod sees" intent without dominating prox940 (which has 754 +GB and 96 cores, so 32 GB is 4%). + +## Compose scope + +**Skip `--profile test`.** The internal gobgp there was what pulled the +251 K routes from Bromirski that filled up v1's 4 GB. For a real-router +test, we want the collector receiving *only* real router BMP — no +synthetic feed muddying the diffs. + +Current deploy.sh only offers scope `full-stack` (implies `--profile +test`), `remote`, or `central-store`. Fresh portability find: +**`full-stack` conflates "the full stack" with "the lab feeders", +which is fine for a demo but not for a canary collector.** + +Two options: + +1. Extend `deploy.sh --scope full-stack --no-feeders` as a new option + (adds a real portability enhancement to obmp-docker; landing on + `deploy-refinement`). +2. Sidestep in cloud-init: run `deploy.sh --scope full-stack --yes`, + then explicitly `docker compose --profile test down` after to remove + the synthetic feeders. + +Option 1 is cleaner and lands a real feature. Option 2 is a workaround +we could remove once option 1 exists. + +## Router extension (unchanged in intent) + +`scripts/router_bmp.py` + `terraform/router_bmp.tf` are already written +and correct for v2. Two-step safety still holds: first apply with +`enable_router_bmp=true` runs a dry-run (prints per-router pending +diffs, does NOT commit); set `confirm_router_push=true` and re-apply to +actually push. Destroy provisioner removes it cleanly, `on_failure = +continue` so a single unreachable router doesn't block VM teardown. + +Scope: `bmp-activate server 2` on `BMP-MONITORED` neighbor-group per +router — matches production activation pattern (enables diff-based +validation between the two collectors). + +Router creds: read via `ROUTER_USER`/`ROUTER_PASS` env for CML; +hardcoded `admin/cisco` for PROX (matches the inventory pattern in +`obmp-docker/obmp-rib-poller/poller.py`). + +## Assertion set (needs updating) + +`scripts/run-test.sh`'s six assertions were written for the v1 +synthetic-feed topology. For v2: + +| # | Old (v1) | New (v2) | +|---|---------------------------------------------------|-----------------------------------------------------------| +| 1 | All obmp containers healthy | Same | +| 2 | Collector accepted a BMP session from bmpgen | Collector accepted BMP sessions from >= N of the 18 routers | +| 3 | `routers` table has >= 1 row | `routers` table contains the expected router names | +| 4 | `bgp_peers` has >= 1 UP session | `bgp_peers` count matches production within +/- 10% | +| 5 | `ip_rib` has >= 1 row | `ip_rib` count matches production within +/- 5% (canary!) | +| 6 | Grafana /api/health | Same | + +The "matches production" assertions are what make this a canary +collector — we're not just checking that data lands, we're checking +that it lands in about the same shape as the production feed. Needs +the production DB to be queryable from the same host that runs the +assertions. + +## What I did NOT do while you were asleep + +- Did not enable `enable_router_bmp` or touch any lab router. +- Did not create new VMs. +- Did not change compose scope / deploy.sh flags in obmp-docker. +- Did not push v2 sizing changes to `terraform.tfvars`. + +## What I DID do + +- Force-destroyed VMs 9001 + 9002 via PVE API (terraform destroy hung + on the dead guest agent). +- Cleaned terraform state to match reality. +- Committed the dormant router extension code + the earlier bug-fix + commit set from the v1 run. +- Wrote this doc. + +## Decisions you'll want to make when you're back + +1. **Sizing**: 16 GB / 32 GB / other for the v2 store VM? +2. **Compose scope**: add `--no-feeders` to deploy.sh (option 1 + above), or workaround in cloud-init (option 2)? +3. **Ready to enable router extension?** Once v2 store VM is up and + healthy, first apply is a safe dry-run. +4. **Assertion updates**: leave the v1 assertions in place for now, or + evolve to the "matches production" set right away? (The latter + requires network reach + creds for the production Postgres.) + +Any of these can wait until we've stood up the v2 VM and confirmed +it's ingesting fine. diff --git a/scripts/router_bmp.py b/scripts/router_bmp.py new file mode 100755 index 0000000..46ee949 --- /dev/null +++ b/scripts/router_bmp.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""router_bmp.py -- push/remove `bmp server 2` on lab routers. + +Adds a SECOND BMP feed to every lab router pointing at the +obmp-portability-test collector, WITHOUT touching `bmp server 1` (the +production feed). Activation scope mirrors production: `bmp-activate +server 2` on the `BMP-MONITORED` neighbor-group that already exists on +every router. + +Usage (called by terraform via local-exec; also runnable standalone): + router_bmp.py dry-run --target-ip 10.40.40.241 --target-port 1790 + router_bmp.py apply --target-ip 10.40.40.241 --target-port 1790 + router_bmp.py remove --target-ip 10.40.40.241 --target-port 1790 + +Inventory + credential pattern lifted verbatim from obmp-docker's +obmp-rib-poller/poller.py (keep in sync manually; drift silently misses +routers). CML routers use ROUTER_USER/ROUTER_PASS env (defaults +webui/cisco); PROX cores + clients use admin/cisco -- this asymmetry is +inherent to the PROX build and captured in the inventory below. +""" + +import argparse +import os +import sys +import time +from typing import List, Tuple + +import paramiko + +# --------------------------------------------------------------------------- +# Inventory. Mirror of obmp-rib-poller/poller.py:50 with per-router ASN added +# so the activation line can name the right `router bgp ` context. +# --------------------------------------------------------------------------- +ROUTER_USER = os.environ.get("ROUTER_USER", "webui") +ROUTER_PASS = os.environ.get("ROUTER_PASS", "cisco") + +# (name, mgmt_ip, ssh_user, ssh_pass, local_asn) +ROUTERS: List[Tuple[str, str, str, str, int]] = [ + ("CML-CORE-01", "10.100.0.100", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-CORE-02", "10.100.0.200", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-01", "10.100.0.1", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-02", "10.100.0.2", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-03", "10.100.0.3", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-04", "10.100.0.4", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-05", "10.100.0.5", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-06", "10.100.0.6", ROUTER_USER, ROUTER_PASS, 65020), + ("CML-R9K-07", "10.100.0.7", ROUTER_USER, ROUTER_PASS, 65020), + ("PROX-CORE-01", "10.100.1.100", "admin", "cisco", 65021), + ("PROX-CORE-02", "10.100.1.200", "admin", "cisco", 65021), + ("PROX-R9K-01", "10.100.1.1", "admin", "cisco", 65021), + ("PROX-R9K-02", "10.100.1.2", "admin", "cisco", 65021), + ("PROX-R9K-03", "10.100.1.3", "admin", "cisco", 65021), + ("PROX-R9K-04", "10.100.1.4", "admin", "cisco", 65021), + ("PROX-R9K-05", "10.100.1.5", "admin", "cisco", 65021), + ("PROX-R9K-06", "10.100.1.6", "admin", "cisco", 65021), + ("PROX-R9K-07", "10.100.1.7", "admin", "cisco", 65021), +] + +SERVER_NUM = 2 # bmp server 2 -- keeps server 1 (production) untouched. + + +# --------------------------------------------------------------------------- +# Config lines we push. Each list is written line-by-line at (config)#, +# which puts them straight into IOS-XR's flat-formal command layer without +# needing to enter the bmp submode. +# --------------------------------------------------------------------------- +def bmp_add_lines(collector_ip: str, collector_port: int) -> List[str]: + return [ + f"bmp server {SERVER_NUM} host {collector_ip} port {collector_port}", + f"bmp server {SERVER_NUM} description OpenBMP-Test-Collector", + f"bmp server {SERVER_NUM} update-source MgmtEth0/RP0/CPU0/0", + f"bmp server {SERVER_NUM} initial-delay 60", + f"bmp server {SERVER_NUM} stats-reporting-period 300", + f"bmp server {SERVER_NUM} initial-refresh delay 60 spread 30", + ] + + +def bmp_activate_line(asn: int) -> str: + return f"router bgp {asn} neighbor-group BMP-MONITORED bmp-activate server {SERVER_NUM}" + + +def bmp_remove_lines(asn: int) -> List[str]: + # Order matters: deactivate on the group first, then remove the server. + return [ + f"router bgp {asn} neighbor-group BMP-MONITORED no bmp-activate server {SERVER_NUM}", + f"no bmp server {SERVER_NUM}", + ] + + +# --------------------------------------------------------------------------- +# Paramiko shell helpers (interactive IOS-XR CLI, same pattern as +# cml/proxmox_bmp_config.py -- NETCONF YANG doesn't cover bmp server). +# --------------------------------------------------------------------------- +def _drain(shell, settle=1.0, limit=15.0) -> str: + buf = b"" + start = time.time() + last = start + while time.time() - start < limit: + if shell.recv_ready(): + buf += shell.recv(65535) + last = time.time() + elif time.time() - last > settle: + break + else: + time.sleep(0.05) + return buf.decode(errors="replace") + + +def _run(shell, cmd: str, settle: float = 0.5, limit: float = 6.0) -> str: + shell.send(cmd + "\n") + return _drain(shell, settle, limit) + + +def _connect(host: str, user: str, pwd: str): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(host, port=22, username=user, password=pwd, timeout=10, + allow_agent=False, look_for_keys=False) + shell = ssh.invoke_shell() + _drain(shell, 1.0, 6) + return ssh, shell + + +# --------------------------------------------------------------------------- +# Per-router work. +# --------------------------------------------------------------------------- +def process_router(name: str, ip: str, user: str, pwd: str, asn: int, + target_ip: str, target_port: int, action: str + ) -> Tuple[bool, str]: + """Return (ok, message). action = 'apply' | 'dry-run' | 'remove'.""" + print(f"\n== {name} ({ip}) [{action}] ==", flush=True) + try: + ssh, shell = _connect(ip, user, pwd) + except Exception as e: + return False, f"SSH connect failed: {e}" + + try: + _run(shell, "terminal length 0") + _run(shell, "configure terminal") + + if action == "remove": + lines = bmp_remove_lines(asn) + else: + lines = bmp_add_lines(target_ip, target_port) + [bmp_activate_line(asn)] + + for line in lines: + _run(shell, line, 0.3, 4) + + # Show the pending diff before commit. If nothing changes, this + # is a no-op and we abort out of config mode cleanly. + diff = _run(shell, "show configuration", 1.0, 10) + has_change = False + for want in [f"bmp server {SERVER_NUM}", f"bmp-activate server {SERVER_NUM}"]: + if want in diff: + has_change = True + break + if not has_change: + print(" no-op (already in target state)") + _run(shell, "abort") + ssh.close() + return True, "no-op" + + print(" ---- pending diff ----") + for l in diff.splitlines(): + s = l.rstrip() + if s and not s.startswith("!") and "show configuration" not in s.lower(): + print(" ", s) + print(" ----------------------") + + if action == "dry-run": + _run(shell, "abort") + ssh.close() + return True, "dry-run (not committed)" + + commit_out = _run(shell, "commit", 1.5, 10) + _run(shell, "end") + + # Verify. `show run formal bmp` gives us the server-side state; + # activation is verified separately on the router-bgp block. + verify = _run(shell, "show run formal bmp", 1.5, 10) + if action == "apply": + ok = f"host {target_ip} port {target_port}" in verify + msg = "committed + verified" if ok else "committed but verify NOT FOUND" + else: # remove + ok = f"bmp server {SERVER_NUM}" not in verify + msg = "removed + verified" if ok else "commit ran but server still present" + + ssh.close() + return ok, msg + except Exception as e: + try: + ssh.close() + except Exception: + pass + return False, f"error: {e}" + + +def main(): + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("action", choices=["apply", "dry-run", "remove"]) + p.add_argument("--target-ip", required=True, + help="collector IP that routers should target") + p.add_argument("--target-port", type=int, required=True, + help="collector port (typically 1790)") + p.add_argument("--routers", + help="comma-separated router names to include (default: all 18)") + p.add_argument("--fail-fast", action="store_true", + help="stop on first router failure (default: process all, report at end)") + args = p.parse_args() + + inv = ROUTERS + if args.routers: + wanted = {n.strip() for n in args.routers.split(",")} + inv = [r for r in inv if r[0] in wanted] + if not inv: + print(f"error: no routers matched {args.routers}", file=sys.stderr) + print(f" known: {', '.join(r[0] for r in ROUTERS)}", file=sys.stderr) + sys.exit(2) + + print(f"routers: {len(inv)} action: {args.action} " + f"target: {args.target_ip}:{args.target_port}", flush=True) + + results = [] + for name, ip, user, pwd, asn in inv: + ok, msg = process_router(name, ip, user, pwd, asn, + args.target_ip, args.target_port, args.action) + results.append((name, ok, msg)) + if args.fail_fast and not ok: + break + + print("\n== summary ==", flush=True) + for name, ok, msg in results: + status = "OK " if ok else "FAIL" + print(f" {status} {name:15s} {msg}") + + fails = sum(1 for _, ok, _ in results if not ok) + sys.exit(1 if fails else 0) + + +if __name__ == "__main__": + main() diff --git a/terraform/main.tf b/terraform/main.tf index 4cf13da..a9e4a08 100644 --- a/terraform/main.tf +++ b/terraform/main.tf @@ -27,9 +27,9 @@ provider "proxmox" { # rather than letting the provider guess from cluster metadata (which # may return a bridge address that's unreachable from here). ssh { - agent = false - username = var.pve_ssh_username - private_key = file(var.pve_ssh_private_key_path) + agent = false + username = var.pve_ssh_username + private_key = file(var.pve_ssh_private_key_path) node { name = var.pve_node diff --git a/terraform/router_bmp.tf b/terraform/router_bmp.tf new file mode 100644 index 0000000..69f8940 --- /dev/null +++ b/terraform/router_bmp.tf @@ -0,0 +1,91 @@ +# Real-router extension: push `bmp server 2` to every lab router pointing +# at the test collector, so real-router BMP output flows into the test +# stack alongside the internal gobgp synthetic feed. Nothing touches +# `bmp server 1` (production). +# +# Two-step safety: first apply is a DRY RUN (prints per-router pending +# diffs, does not commit). Set `confirm_router_push=true` and re-apply +# to actually commit the config on all routers. +# +# Lifecycle: +# enable_router_bmp=false -> resource absent, no router-side changes +# enable=true, confirm=false -> dry-run provisioner (safe; abort at end) +# enable=true, confirm=true -> apply provisioner (commits + verifies) +# flip either back -> destroy provisioner runs `remove` (idempotent) +# +# The destroy provisioner uses `on_failure = continue` so a partial router +# outage never blocks the VM teardown in scripts/teardown.sh. + +variable "enable_router_bmp" { + description = "Enable the real-router BMP extension (push bmp server 2 to lab routers)." + type = bool + default = false +} + +variable "confirm_router_push" { + description = "Set to true on second apply to actually commit `bmp server 2` on routers. First apply (false) is a dry-run only." + type = bool + default = false +} + +locals { + router_bmp_action = var.confirm_router_push ? "apply" : "dry-run" + router_bmp_script = abspath("${path.module}/../scripts/router_bmp.py") + + # The collector's address on the routers' management path. With DHCP, + # this only becomes known after the store VM's guest-agent reports. + router_bmp_target = try( + proxmox_virtual_environment_vm.store.ipv4_addresses[1][0], + "" + ) +} + +resource "terraform_data" "router_bmp" { + count = var.enable_router_bmp ? 1 : 0 + + # Any change to these values REPLACES the resource -- destroy provisioner + # runs first (removes any prior state on routers), then create provisioner + # runs (applies the new state). That gives us clean transitions: + # dry-run -> apply, or apply -> dry-run. + triggers_replace = [ + local.router_bmp_action, + local.router_bmp_target, + var.router_facing_port, + ] + + # `input` is available to provisioners as `self.input`. + input = { + action = local.router_bmp_action + script = local.router_bmp_script + target_ip = local.router_bmp_target + target_port = tostring(var.router_facing_port) + } + + # CREATE (or replace): push (or dry-run) `bmp server 2` to all routers. + provisioner "local-exec" { + command = "python3 ${self.input.script} ${self.input.action} --target-ip ${self.input.target_ip} --target-port ${self.input.target_port}" + } + + # DESTROY: remove `bmp server 2` from all routers. Idempotent: routers + # without the config are no-ops. `on_failure = continue` so a single + # unreachable router does not block the VM teardown. + provisioner "local-exec" { + when = destroy + on_failure = continue + command = "python3 ${self.input.script} remove --target-ip ${self.input.target_ip} --target-port ${self.input.target_port}" + } +} + +output "router_bmp_status" { + description = "Real-router BMP extension state." + value = var.enable_router_bmp ? { + enabled = true + action = local.router_bmp_action + confirm_needed = !var.confirm_router_push + target = "${local.router_bmp_target}:${var.router_facing_port}" + hint = var.confirm_router_push ? "committed on last apply; destroy will remove" : "set confirm_router_push=true and re-apply to commit" + } : { + enabled = false + hint = "set enable_router_bmp=true (and later confirm_router_push=true) to add bmp server 2 to lab routers" + } +}