sam 6fc92541e9 router extension (dormant) + v2 redesign doc after OOM'd first run
Adds the real-router BMP extension code and a redesign doc capturing
what changed in the test's intent.

  scripts/router_bmp.py           push/dry-run/remove `bmp server 2`
                                  on all 18 lab routers via paramiko
                                  SSH shell. Inventory + cred pattern
                                  lifted from obmp-rib-poller/poller.py.
                                  Activation scope: bmp-activate server 2
                                  on the existing BMP-MONITORED
                                  neighbor-group -- mirrors production.

  terraform/router_bmp.tf         terraform_data resource with two-step
                                  safety (dry-run first, apply on
                                  confirm), triggers_replace on action
                                  change, destroy provisioner calls
                                  `remove` (idempotent, on_failure=
                                  continue so a stuck router doesn't
                                  block VM teardown).

  terraform/terraform.tfvars      enable_router_bmp / confirm_router_push
                                  both default false (dormant).

  docs/redesign-real-routers.md   captures the pivot from v1 "prove
                                  deploy.sh works" to v2 "canary
                                  collector receiving real router BMP".
                                  Documents that v1 store VM OOM'd at
                                  4 GB under the internal --profile
                                  test feed alone, proposes 32 GB /
                                  8 vCPU for v2 and dropping --profile
                                  test so the collector receives ONLY
                                  real router BMP.

State on prox940 as of this commit:
  - v1 VMs (9001 store, 9002 bmpgen) destroyed via PVE API
    (terraform destroy hung on the OOM'd guest agent)
  - Terraform state cleaned
  - Cached Ubuntu image kept for a fast next apply
  - Lab routers untouched (extension code dormant)

Not yet done, awaiting user input:
  - v2 sizing landed in terraform.tfvars
  - Compose scope decision (add --no-feeders to deploy.sh vs
    workaround in cloud-init)
  - Re-apply
  - Router extension dry-run + confirm

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-21 07:21:03 -07:00

243 lines
9.7 KiB
Python
Executable File

#!/usr/bin/env python3
"""router_bmp.py -- push/remove `bmp server 2` on lab routers.
Adds a SECOND BMP feed to every lab router pointing at the
obmp-portability-test collector, WITHOUT touching `bmp server 1` (the
production feed). Activation scope mirrors production: `bmp-activate
server 2` on the `BMP-MONITORED` neighbor-group that already exists on
every router.
Usage (called by terraform via local-exec; also runnable standalone):
router_bmp.py dry-run --target-ip 10.40.40.241 --target-port 1790
router_bmp.py apply --target-ip 10.40.40.241 --target-port 1790
router_bmp.py remove --target-ip 10.40.40.241 --target-port 1790
Inventory + credential pattern lifted verbatim from obmp-docker's
obmp-rib-poller/poller.py (keep in sync manually; drift silently misses
routers). CML routers use ROUTER_USER/ROUTER_PASS env (defaults
webui/cisco); PROX cores + clients use admin/cisco -- this asymmetry is
inherent to the PROX build and captured in the inventory below.
"""
import argparse
import os
import sys
import time
from typing import List, Tuple
import paramiko
# ---------------------------------------------------------------------------
# Inventory. Mirror of obmp-rib-poller/poller.py:50 with per-router ASN added
# so the activation line can name the right `router bgp <asn>` context.
# ---------------------------------------------------------------------------
ROUTER_USER = os.environ.get("ROUTER_USER", "webui")
ROUTER_PASS = os.environ.get("ROUTER_PASS", "cisco")
# (name, mgmt_ip, ssh_user, ssh_pass, local_asn)
ROUTERS: List[Tuple[str, str, str, str, int]] = [
("CML-CORE-01", "10.100.0.100", ROUTER_USER, ROUTER_PASS, 65020),
("CML-CORE-02", "10.100.0.200", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-01", "10.100.0.1", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-02", "10.100.0.2", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-03", "10.100.0.3", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-04", "10.100.0.4", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-05", "10.100.0.5", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-06", "10.100.0.6", ROUTER_USER, ROUTER_PASS, 65020),
("CML-R9K-07", "10.100.0.7", ROUTER_USER, ROUTER_PASS, 65020),
("PROX-CORE-01", "10.100.1.100", "admin", "cisco", 65021),
("PROX-CORE-02", "10.100.1.200", "admin", "cisco", 65021),
("PROX-R9K-01", "10.100.1.1", "admin", "cisco", 65021),
("PROX-R9K-02", "10.100.1.2", "admin", "cisco", 65021),
("PROX-R9K-03", "10.100.1.3", "admin", "cisco", 65021),
("PROX-R9K-04", "10.100.1.4", "admin", "cisco", 65021),
("PROX-R9K-05", "10.100.1.5", "admin", "cisco", 65021),
("PROX-R9K-06", "10.100.1.6", "admin", "cisco", 65021),
("PROX-R9K-07", "10.100.1.7", "admin", "cisco", 65021),
]
SERVER_NUM = 2 # bmp server 2 -- keeps server 1 (production) untouched.
# ---------------------------------------------------------------------------
# Config lines we push. Each list is written line-by-line at (config)#,
# which puts them straight into IOS-XR's flat-formal command layer without
# needing to enter the bmp submode.
# ---------------------------------------------------------------------------
def bmp_add_lines(collector_ip: str, collector_port: int) -> List[str]:
return [
f"bmp server {SERVER_NUM} host {collector_ip} port {collector_port}",
f"bmp server {SERVER_NUM} description OpenBMP-Test-Collector",
f"bmp server {SERVER_NUM} update-source MgmtEth0/RP0/CPU0/0",
f"bmp server {SERVER_NUM} initial-delay 60",
f"bmp server {SERVER_NUM} stats-reporting-period 300",
f"bmp server {SERVER_NUM} initial-refresh delay 60 spread 30",
]
def bmp_activate_line(asn: int) -> str:
return f"router bgp {asn} neighbor-group BMP-MONITORED bmp-activate server {SERVER_NUM}"
def bmp_remove_lines(asn: int) -> List[str]:
# Order matters: deactivate on the group first, then remove the server.
return [
f"router bgp {asn} neighbor-group BMP-MONITORED no bmp-activate server {SERVER_NUM}",
f"no bmp server {SERVER_NUM}",
]
# ---------------------------------------------------------------------------
# Paramiko shell helpers (interactive IOS-XR CLI, same pattern as
# cml/proxmox_bmp_config.py -- NETCONF YANG doesn't cover bmp server).
# ---------------------------------------------------------------------------
def _drain(shell, settle=1.0, limit=15.0) -> str:
buf = b""
start = time.time()
last = start
while time.time() - start < limit:
if shell.recv_ready():
buf += shell.recv(65535)
last = time.time()
elif time.time() - last > settle:
break
else:
time.sleep(0.05)
return buf.decode(errors="replace")
def _run(shell, cmd: str, settle: float = 0.5, limit: float = 6.0) -> str:
shell.send(cmd + "\n")
return _drain(shell, settle, limit)
def _connect(host: str, user: str, pwd: str):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, port=22, username=user, password=pwd, timeout=10,
allow_agent=False, look_for_keys=False)
shell = ssh.invoke_shell()
_drain(shell, 1.0, 6)
return ssh, shell
# ---------------------------------------------------------------------------
# Per-router work.
# ---------------------------------------------------------------------------
def process_router(name: str, ip: str, user: str, pwd: str, asn: int,
target_ip: str, target_port: int, action: str
) -> Tuple[bool, str]:
"""Return (ok, message). action = 'apply' | 'dry-run' | 'remove'."""
print(f"\n== {name} ({ip}) [{action}] ==", flush=True)
try:
ssh, shell = _connect(ip, user, pwd)
except Exception as e:
return False, f"SSH connect failed: {e}"
try:
_run(shell, "terminal length 0")
_run(shell, "configure terminal")
if action == "remove":
lines = bmp_remove_lines(asn)
else:
lines = bmp_add_lines(target_ip, target_port) + [bmp_activate_line(asn)]
for line in lines:
_run(shell, line, 0.3, 4)
# Show the pending diff before commit. If nothing changes, this
# is a no-op and we abort out of config mode cleanly.
diff = _run(shell, "show configuration", 1.0, 10)
has_change = False
for want in [f"bmp server {SERVER_NUM}", f"bmp-activate server {SERVER_NUM}"]:
if want in diff:
has_change = True
break
if not has_change:
print(" no-op (already in target state)")
_run(shell, "abort")
ssh.close()
return True, "no-op"
print(" ---- pending diff ----")
for l in diff.splitlines():
s = l.rstrip()
if s and not s.startswith("!") and "show configuration" not in s.lower():
print(" ", s)
print(" ----------------------")
if action == "dry-run":
_run(shell, "abort")
ssh.close()
return True, "dry-run (not committed)"
commit_out = _run(shell, "commit", 1.5, 10)
_run(shell, "end")
# Verify. `show run formal bmp` gives us the server-side state;
# activation is verified separately on the router-bgp block.
verify = _run(shell, "show run formal bmp", 1.5, 10)
if action == "apply":
ok = f"host {target_ip} port {target_port}" in verify
msg = "committed + verified" if ok else "committed but verify NOT FOUND"
else: # remove
ok = f"bmp server {SERVER_NUM}" not in verify
msg = "removed + verified" if ok else "commit ran but server still present"
ssh.close()
return ok, msg
except Exception as e:
try:
ssh.close()
except Exception:
pass
return False, f"error: {e}"
def main():
p = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("action", choices=["apply", "dry-run", "remove"])
p.add_argument("--target-ip", required=True,
help="collector IP that routers should target")
p.add_argument("--target-port", type=int, required=True,
help="collector port (typically 1790)")
p.add_argument("--routers",
help="comma-separated router names to include (default: all 18)")
p.add_argument("--fail-fast", action="store_true",
help="stop on first router failure (default: process all, report at end)")
args = p.parse_args()
inv = ROUTERS
if args.routers:
wanted = {n.strip() for n in args.routers.split(",")}
inv = [r for r in inv if r[0] in wanted]
if not inv:
print(f"error: no routers matched {args.routers}", file=sys.stderr)
print(f" known: {', '.join(r[0] for r in ROUTERS)}", file=sys.stderr)
sys.exit(2)
print(f"routers: {len(inv)} action: {args.action} "
f"target: {args.target_ip}:{args.target_port}", flush=True)
results = []
for name, ip, user, pwd, asn in inv:
ok, msg = process_router(name, ip, user, pwd, asn,
args.target_ip, args.target_port, args.action)
results.append((name, ok, msg))
if args.fail_fast and not ok:
break
print("\n== summary ==", flush=True)
for name, ok, msg in results:
status = "OK " if ok else "FAIL"
print(f" {status} {name:15s} {msg}")
fails = sum(1 for _, ok, _ in results if not ok)
sys.exit(1 if fails else 0)
if __name__ == "__main__":
main()