Docker Compose's .env parser treats an unquoted mid-line comma as a
variable-name boundary. The current value:
GNMI_ADDRESSES="10.100.0.100:57400", "10.100.0.200:57400"
fails compose pull / build / up with:
failed to read .env: line 56: unexpected character "," in variable name
The double-quoted-list form is what telegraf.conf needs when it
substitutes `addresses = [ ${GNMI_ADDRESSES} ]`, but the raw comma is
what breaks compose. Wrap the whole value in SINGLE quotes: compose
treats it as a literal string (single-quoted values are not interpolated
and preserve special characters), and telegraf still gets the
double-quoted list body verbatim on substitution.
Also fix the docker-compose.yml default fallback -- same comma issue,
drop the second address from the default (users can extend via .env).
Found by the obmp-portability-test two-VM run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follow-up to bb71c9f. `sudo -n -v` also fails on NOPASSWD users because
`sudo -v` demands a credential *timestamp* and NOPASSWD never produces
one (verified on Ubuntu 24.04 sudo 1.9.x: `sudo -n true` returns 0 while
`sudo -n -v` returns 1 with "a password is required").
Replace the probe with `sudo -n true`. When it fails, additionally
require a TTY for the interactive fallback -- previously we would fall
through to `sudo -v` on a TTY-less run and die with a confusing
"authentication failed" instead of a clear message about the missing
NOPASSWD entry.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`sudo -v` always attempts interactive credential validation; it does not
consult NOPASSWD entries. On any user configured with `NOPASSWD:ALL`
(cloud-init's default ubuntu, most CI runners), `sudo -v` still demands
a TTY password prompt and dies with "sudo authentication failed" when
there is no TTY.
Found by the obmp-portability-test two-VM run: deploy.sh failed at line
433 on a cloud-init-provisioned Ubuntu 24.04 VM despite the ubuntu user
having passwordless sudo (`sudo -n whoami` returned root correctly).
Fix: try `sudo -n -v` first (succeeds silently on NOPASSWD users, fails
non-interactively on others), and only fall through to the interactive
`sudo -v` prompt when the non-interactive probe fails. Humans still get
the same visible up-front prompt; automation now succeeds without one.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the one-shot goal from the greenfield audit: tear down fully
and/or deploy cleanly on a pristine WSL instance with nothing but the
interactive script.
- './deploy.sh teardown': leave-it-down removal across ALL compose
profiles (core/test/auth/evpn-test/scale-out -- the old reset missed the
last two), networks, named volumes, rendered gobgpd.confs. Data wipe is
opt-in (--wipe-data, guarded on native hosts) and preserves backups/
unless --purge-backups; --purge-images reclaims images. Prints the
Windows-side cleanup (wsl-portproxy.ps1 -Remove, new switch).
- fresh-host preflight: verifies the docker DAEMON (not just the client),
offers to install docker-ce from the official repo (--install-prereqs
for unattended), and handles the classic pristine-WSL trap: systemd off
-> writes /etc/wsl.conf [boot] systemd=true and stops with exact
restart instructions instead of dying cryptically.
- sudo authenticates ONCE up front (no more password prompt buried
mid-provision); --reset now covers all profiles and preserves backups/
- guided flow: banner, Step N/6 headers, context lines explaining
HOST_IP-vs-router-facing before the address prompts, router-facing IP
validated as a dotted quad (a failed Windows-LAN autodetect can no
longer write a <WINDOWS_LAN_IP> placeholder into .env), stage
completions print green checkmarks, loud abort note on the plan
confirm, post-deploy Verify + teardown hints.
- shared helpers moved to scripts/deploy-lib.sh; setup.sh sources it too
(duplicate get_env/set_env/ask/confirm definitions deleted), keeping
setup.sh as a standalone provisioning primitive.
Verified live on this host: teardown left zero obmp containers and
removed rendered configs; unattended redeploy brought the stack back.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From a two-agent audit (docs accuracy + greenfield deploy path):
- init_db docs were describing upstream behavior: this repo's psql-app
auto-creates the schema on first run and drops config/do_not_init_db to
skip later (psql-app/scripts/run:74). README/DOCS/backup-restore now
describe the marker semantics; the restore flow creates the marker
BEFORE first start instead of 'not creating init_db'
- DOCS.md contained heavy retired-lab drift: banner declares it a legacy
walkthrough with illustrative values; fixed its self-contradictions --
external port 1790, folder OBMP-Reference, datasource 'PostgreSQL',
OpenConfig gNMI paths (matching telegraf.conf), EXABGP_PEERS,
TRAFFIC_GEN_PORT
- deploy.sh --help now shows the current scope names (old ones remain
accepted aliases)
- navigation: new docs/README.md index with operator and network-engineer
tracks (links the previously orphaned backup-restore, security-hardening,
ROADMAP, DB_SCHEMA); README gains a Start-here router
- intra-docs prose paths no longer carry the docs/ prefix (they resolve
from within docs/); RR-CLIENTS -> RR-CLIENT matches the blueprint;
ROADMAP A6 marked done
- scripts/deploy-lib.sh added: shared log/ask/confirm/get_env/set_env
helpers for the deploy tooling consolidation (wiring lands next)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings from verifying the router-blueprints drop against the
deployed stack:
- router-blueprints/README.md linked docs/router-config-guide.md, which
does not exist -- the guide is docs/router-integration.md
- router-integration.md still advised --router-port 5000 for lab routers;
1790 is the standard now, the note points at re-applying via
cml/proxmox_bmp_config.py instead
- verify sections referenced Grafana paths that don't exist in the
provisioned UI: 'Base-1001 -> Peers Table' -> Inventory / Peer Detail,
and folder 'OBMP-Learning' -> OBMP-Reference
- role-route-reflector.cfg: loud caveat block on next-hop-self -- it only
rewrites eBGP-learned/local routes (what this lab wants); reflected
iBGP routes need 'ibgp policy out enforce-modifications', so on a pure
RR the line is a silent no-op
- cml/xrd-node-definition.yaml BMP timers synced to the blueprint values
(initial-delay 60, refresh 60/30, drop flapping-delay) per the
keep-hand-config-and-automation-in-sync rule
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decision landed: routers target 1790 (IANA BMP port); the collector keeps
listening on 5000 in-container. One .env variable now drives every hop:
- compose publishes ${ROUTER_FACING_PORT:-1790}:5000 (was hardcoded 5000:5000)
- gobgpd.conf.tmpl (+ evpn) gains __BMP_PORT__; setup.sh renders it from
ROUTER_FACING_PORT -- gobgp is host-networked, so it must chase the
published port or the full-table feeder silently breaks
- deploy.sh + scripts/wsl-portproxy.ps1: the WSL side now publishes the
router-facing port itself, so the portproxy forwards the same port
end-to-end instead of down-mapping to 5000
- cml/proxmox_bmp_config.py reads ROUTER_FACING_IP/PORT from .env (HOST_IP
fallback for pre-key .envs); xrd-node-definition.yaml baked port -> 1790
- .env.example documents ROUTER_FACING_IP/PORT; docs updated, finding 12
closed on the repo side (routers still on 5000 need a config re-apply)
Verified live: collector republished 0.0.0.0:1790->5000, gobgp BMP session
Up to HOST_IP:1790 with the RIB dump flowing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stack side is portable; the router side had no reference material -- a
new operator standing up OBMP has no guide for what to configure on their
devices to feed it. Adds two artifacts, both vendor-neutral in intent,
IOS-XR-specific in syntax:
docs/router-integration.md
Multi-path tour of the four ingest paths (BMP, BGP-LS, gNMI,
NETCONF) with a bring-up checklist and cross-references. The BMP
section is deliberately thin -- it hands off to docs/router-bmp-config.md
for the address-selection, port-choice, and RR-scope activation
detail already covered there.
router-blueprints/iosxr/*.cfg
Copy-paste config fragments, one per ingest path plus a
role-route-reflector overlay. All values are <PLACEHOLDER>
substitutions -- no hardcoded IPs, no lab-gear names. Fragments
align to the ROUTER_FACING_IP / ROUTER_FACING_PORT convention
introduced in docs/router-bmp-config.md.
router-blueprints/README.md
Substitution legend + layout, cross-links to PORTABILITY-FINDINGS.md
finding 4 (why routers can't target HOST_IP on WSL).
Also appends the multi-path guide link to the Greenfield deploy section
in README.md so router-side integration is discoverable from the top.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Repo root keeps only README.md; DOCS.md and DB_SCHEMA.md move to docs/ with
their cross-references updated. The chat-session handoff packets land in
docs/handoff/ as point-in-time records with a README marking the repo docs
as the live source of truth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deploy.sh referenced DEPLOYMENT-TYPES.md and router-bmp-config.md, which were
never committed (they lived only in a chat session's outputs). Land them under
docs/ along with PORTABILITY-FINDINGS.md covering all 12 greenfield findings,
including the two from today's deploy (chmod-777-vs-psql, telegraf docker API)
and the open router-side gap (cml tooling still targets HOST_IP:5000).
- scripts/wsl-portproxy.ps1: idempotent elevated-PS helper for the Windows
portproxy + firewall rules; auto-detects the current WSL IP. Replaces the
copy-paste netsh block as the primary path (raw commands kept as fallback).
- deploy.sh: plan now prints the branch @ commit (+dirty marker) so every
deploy records exactly what it ran from; doc references point at docs/.
- README: greenfield quickstart with pinned-checkout guidance; warning on the
manual chmod path that breaks existing Postgres trees (finding 10).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
telegraf 1.28's [[inputs.docker]] hardcodes Docker API version 1.24, which
docker-ce >= 29 refuses (min supported API 1.40), so every per-container
stack metric (cpu/mem/net/blkio/health) silently never reached InfluxDB.
DOCKER_API_VERSION is ignored (version pinned in code); newer telegraf
negotiates the API version with the daemon. Verified docker_container_*
measurements land in the telemetry bucket after the bump.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Blanket 777 over an existing PGDATA makes psql_server.key world-accessible;
postgres refuses to boot (FATAL: private key file has group or world access)
and the whole core cascades down. Fresh installs were unaffected since the
key is generated after the chmod. Skip postgres/ (the image entrypoint owns
its perms) and re-tighten the key/cert if a prior run already clobbered them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh remote hosts came up with an empty Grafana: the provisioning YAML
pointed Grafana at /var/lib/grafana/dashboards/... (bind-mounted from
${OBMP_DATA_ROOT}/grafana/dashboards) but nothing on the host ever populated
that path. Grafana loaded the providers, found no JSON files, and showed
the built-in Home dashboard only.
setup.sh already copied obmp-grafana/provisioning/. -> the equivalent host
path; add the same copy for obmp-grafana/dashboards/. so a fresh host gets
all 42 committed dashboards on first run. Grafana's file provider re-scans
every 30s, so they appear without a container restart.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bootstrap previously hard-required OBMP_DOMAIN and OBMP_COOKIE_DOMAIN
even when a user just wanted a local lab deployment with Grafana's built-in
login -- those vars only feed Authelia's session-cookie domain and the
public URL it lives behind. On a fresh host with no FQDN this made
./setup.sh impossible to pass without inventing dummy values.
New OBMP_AUTH_MODE=local|authelia in .env (default local) gates the FQDN
validation, Authelia secret generation, Authelia config rendering, and the
auth-profile image pull/build. setup.sh also writes GF_SERVER_ROOT_URL into
.env -- http://HOST_IP:3000/grafana/ for local, https://OBMP_DOMAIN/grafana/
for authelia -- and docker-compose.yml now reads ${GF_SERVER_ROOT_URL}
instead of hardcoding the apodacalab.com fallback.
Back-compat: an existing .env with no OBMP_AUTH_MODE but a real OBMP_DOMAIN
or an existing AUTHELIA_SESSION_SECRET is inferred as 'authelia' and the
mode is persisted -- a re-run on a live Authelia host won't silently flip
it to local and break the next docker compose up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two recurring layout issues across dashboards I built this session:
1) Right-placed legend tables ate 30% of each panel width.
2) Default h:9 panels left ~50% of the viewport empty on a 1080p
display (total dashboard height ~18 grid rows vs ~30 available).
Stack Resources (Telemetry-3001/stack_resources.json):
* 3 timeseries: legend placement right -> bottom, calcs [max] -> [last,max],
added sortBy: Max desc so top consumers float to the top of the legend.
* Bumped all 4 panels h: 9 -> 14 (dashboard total 18 -> 28 rows).
Kafka Ingestion Lag and Live BGP Churn (Telemetry-3001/*):
* Bumped timeseries panels h: 9 -> 12; second-row y: 13 -> 16.
Dashboard total 22 -> 28 rows.
Policy Diff (obmp/History-1002/policy_diff.json):
* Bumped bottom-row panels h: 8 -> 11. Total 24 -> 27 rows.
Untouched (already adequate, scrollable by design, or built earlier):
evpn_rib (30 rows), global_table (38), router_diff (52), and the
Maps-1006 dashboards (already h:22-28 single panels).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes hardcoded 10.40.40.202 references so a fresh clone + .env-only
edit can stand the stack up on a new compute node.
* docker-compose.yml: rib-poller PG_DSN now uses ${HOST_IP:-...}.
* obmp-rib-poller/poller.py: default PG_DSN host falls back to
${HOST_IP} env (compose passes it; manual runs honour $HOST_IP too).
* cml/gobgp_peering_config.py: GOBGP_IP read from $HOST_IP or the
HOST_IP= line in repo-root .env, with a small _env_default helper.
* cml/proxmox_bmp_config.py: COLLECTOR_HOST resolved the same way.
For gobgp/gobgpd.conf and gobgp-evpn/gobgpd.conf -- jauderho/gobgp is
distroless (no shell), so we can't sed-substitute at container start.
Pattern instead:
* gobgpd.conf is now gobgpd.conf.tmpl with __HOST_IP__ placeholders
(committed). The rendered gobgpd.conf is gitignored.
* setup.sh renders the .tmpl(s) to .conf using $HOST_IP from .env.
* compose `command` stays the simple `gobgpd -f /config/gobgpd.conf`.
After cloning on a new host: cp .env.example .env -> edit HOST_IP ->
./setup.sh -> docker compose up -d. Verified locally by force-recreating
gobgp; all 6 sessions (4 cores + 2 Bromirski) re-established in <60s.
Known portability gaps still to address (separate work):
* Hardcoded lab-router inventories in cml/*.py and
obmp-rib-poller/poller.py.
* The /etc/cron.d/openbmp */5 -> */15 edit inside obmp-psql-app is
not persistent (regenerated by config_cron on every container start).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Derived from the 2026-05-19 ingestion stress-test session. psql-app's
unicast_prefix drain rate caps at a few-hundred msg/s under continuous
Postgres maintenance (autovacuum on ip_rib + update_global_ip_rib() /
update_chg_stats() / update_peer_rib_counts() crons) competing for
ip_rib disk I/O.
ALTER TABLE ip_rib SET autovacuum_vacuum_scale_factor=0.02 -- run more
often on smaller chunks. cost_limit kept at its OpenBMP-default 3000 so
each run finishes fast; the consumer runs flat out between bursts
instead of being throttled continuously.
DROP INDEX for four unused/redundant indexes (every INSERT updates every
index; these all had 0 scans in ~2h of heavy activity):
- ip_rib_hash_id_idx (907 MB)
- ip_rib_base_attr_hash_id_idx (558 MB)
- ip_rib_prefix_idx (1538 MB, GiST)
- ip_rib_origin_as_idx (364 MB)
9 -> 5 indexes; ~3.4 GB freed (6,715 MB -> 3,348 MB). Reduces index
write-amplification per UPSERT by ~45% and shortens autovacuum on
ip_rib by ~the same.
Measurement note: across-cycle 25-min runs were inconclusive on the
sustained-rate effect (inflow was near-zero by then -- gobgp stopped --
so the consumer was largely idle). The real test is re-enabling the
fleet-wide feed with the consumer-replica + 62 GiB RAM and seeing
whether unicast_prefix keeps up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
psql-app-consumer: profile-gated (scale-out) horizontal scale-out for the
Kafka->Postgres ingestion path. Shares the primary's /config read-only so
it reuses obmp-psql.yml, whose fixed group.id makes Kafka rebalance
partitions across the primary and every replica. Its command runs ONLY
the consumer jar -- no cron, RPKI/IRR/DBIP or initdb -- so it does not
duplicate the primary's DB-maintenance jobs (config_cron wires those up
unconditionally in /usr/sbin/run). Each replica brings its own consumer
and writer threads.
Measured: one consumer-only replica took the post-storm backlog drain
from a cold-start ~3.7k msg/s to ~48k msg/s; group membership 8->16. With
2 consumers feeding it, Postgres becomes the next bottleneck (~500% CPU)
-- DB write capacity is the ceiling beyond ~2-3 consumers.
docker compose --profile scale-out up -d --scale psql-app-consumer=2
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On a cold boot all containers start together; psql-app finishes its
RPKI/IRR/DBIP setup and opens its single Postgres connection while the
DB is still initialising -> "the database system is starting up" ->
ConsumerApp.main throws and the consumer dies. The container does NOT
exit (the wrapper keeps cron/rsyslog alive), so restart: unless-stopped
never fires and the consumer stays dead silently.
Add depends_on psql: condition: service_healthy (plus kafka) so Compose
holds psql-app until Postgres passes its pg_isready healthcheck.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
obmp-churn-monitor: a decoupled fast-path BGP churn consumer. Reads
openbmp.parsed.unicast_prefix with its own Kafka consumer group and only
counts announcements/withdrawals per (router,peer) into churn_metrics
(010_churn_metrics.sql) -- no relational RIB write. Storm-tested: it
stayed real-time (tracked 1k->85k msg/s) while the psql-app bulk
pipeline lag grew 3.8M->5.6M. Live BGP Churn dashboard reads it.
tools/churn_storm.py: programmatic churn-storm generator (flaps GoBGP's
eBGP sessions to the lab cores) for load testing.
Stress-test finding: fleet-wide full table from 18 routers exceeds this
31 GiB host. The bottleneck is RAM, not CPU -- at 16 cores the host
still hit load 33 because it was swap-thrashing (swap 2/2 full, <1.5 GiB
free). Lag ran away 3.8M->20M+. Recourse: more host RAM for bulk
throughput; the fast-path consumer for visibility regardless.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Policy Diff (roadmap E2 follow-up): obmp-rib-poller pulls per-router
post-policy accepted/advertised prefix counts and route-policy bindings
over CLI+NETCONF (BMP on XRv9000 24.3.1 carries only pre-policy
Adj-RIB-In). New tables in 008_obmp_policy_diff.sql; Policy Diff
dashboard joins them against BMP ip_rib for received-vs-kept-vs-rejected.
GoBGP fleet-wide feed: GoBGP re-advertises the full Bromirski table to
both labs' core routers (CML AS65020, PROX AS65021) over eBGP; as route
reflectors the cores propagate it to every R9K client, so all 18 lab
routers carry and BMP-export a full table -- an intentional stress test
of the ingestion/storage path. cml/gobgp_peering_config.py applies and
rolls back the core-side config; gobgp/README.md documents the rollback.
Kafka lag monitoring: kafka-lag-monitor samples consumer-group lag every
30s into TimescaleDB (009_kafka_lag.sql); Kafka Ingestion Lag dashboard
gives visibility into the pipeline under churn load.
Peer Detail dashboard: the Peer selector is now router-qualified
(router -> peer) so it is unambiguous in an iBGP route-reflector mesh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
evpn_rib table, gobgp-evpn injector, obmp-evpn-consumer and the EVPN
RIB dashboard are built and verified for type-2/type-3. type-5 and
real (non-synthetic) EVPN remain limited by collector 2.2.3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Visualises BGP EVPN routes from evpn_rib: route counts and EVIs,
route-type breakdown, a per-EVI summary, and detail tables for type-2
MAC/IP advertisements (MAC, host IP, VNI/label, route-targets, ESI)
and type-3 inclusive-multicast routes. Scoped by an RD/EVI variable.
Lives in the OBMP-L3VPN folder.
Completes roadmap E5's lab-testable scope: evpn_rib table, gobgp-evpn
injector, obmp-evpn-consumer, and this dashboard — verified end to
end with synthetic type-2/type-3 routes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A standalone Python Kafka consumer that subscribes to the
openbmp.parsed.evpn topic (which the stock psql-app ignores) and
writes BGP EVPN routes into evpn_rib. Field positions are pinned to
the verified collector 2.2.3 / v1.7 message layout; route_type is
derived from which fields populate. Profile-gated ('evpn-test')
alongside the gobgp-evpn injector.
Verified end to end: 5 injected type-2/type-3 routes land in evpn_rib
with correct RD, ethernet-tag, MAC, IP, label and route-target.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verified against the live collector: EVPN type-2 (MAC/IP) and type-3
(inclusive multicast) parse cleanly onto openbmp.parsed.evpn, but
type-5 (IP-prefix) is mis-decoded — the IP prefix corrupts the RD
field. inject-evpn.sh now injects only type-2/3; the type-5
limitation is documented in the injector README and roadmap E5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A profile-gated GoBGP instance (Compose profile 'evpn-test', not part
of the normal stack) that originates synthetic BGP EVPN routes and
BMP-exports its local RIB to the collector. Verified end to end: the
injected type-2/3/5 routes are parsed by the collector and land on
the openbmp.parsed.evpn Kafka topic, ready for the EVPN consumer.
inject-evpn.sh pushes type-2 (MAC/IP), type-3 (inclusive multicast)
and type-5 (IP-prefix) routes. Start with:
docker compose --profile evpn-test up -d gobgp-evpn
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- postgres/scripts/007_obmp_evpn.sql: the evpn_rib landing table
(roadmap E5 step 1), applied to the live DB. Mirrors l3vpn_rib;
a dedicated consumer will populate it.
- production-sizing.md: corrected retention figures to the actual
policy values, added a measured-data section (one full feed ≈
+5 GB current state; DB now ~30 GB), and a horizontal-scaling
section — the bottleneck is the psql-app consumer + disk IOPS, so
scale psql-app as a Kafka consumer group (cap = partition count),
treat multi-collector as HA/locality not throughput.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The OpenBMP collector already decodes EVPN and emits openbmp.parsed.evpn;
the gap is solely the psql-app (no subscription/handler) and the missing
schema table. L2VPN-VPLS is unsupported entirely. Records the two
implementation paths: fork the Java psql-app, or run GoBMP as a second
EVPN-capable collector with a thin Postgres consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surfaces the new Telegraf disk/DB metrics plus GoBGP feed health:
openbmp database size (current + trend), largest tables, host
filesystem usage % and free space, GoBGP feed route count, and the
state of the IPv4/IPv6 BGP sessions to AS57355. Lives in the
OBMP-Telemetry folder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Telegraf now collects host filesystem usage ([[inputs.disk]], via a
read-only /hostfs mount) and PostgreSQL database + per-table sizes
([[inputs.postgresql_extensible]]) into InfluxDB. Surfaces RIB growth
and disk pressure — relevant now that the full-table GoBGP feed has
pushed the openbmp DB to ~30 GB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The IPv6 eBGP session never established because the Docker bridge
has no IPv6. Switch the gobgp container to network_mode: host so it
uses the host's real dual-stack connectivity — both sessions to
AS57355 now source from the host's public v4/v6 addresses.
Host mode binds the host's port namespace, so disable GoBGP's
inbound BGP listener (port = -1) — we only originate outbound
sessions, and a non-root container cannot bind privileged port 179.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This OpenBMP deployment has no EVPN/L2VPN schema; supporting it
requires collector + psql-app + schema changes upstream, not a
Grafana dashboard. Captured as E5 with a research-spike first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both L3VPN dashboards had a static custom 'rd' variable holding only
the '-' (all) sentinel — you could not actually filter by a VRF.
Convert 'rd' to a query variable that discovers route distinguishers
from l3vpn_rib. Degrades cleanly on the (currently empty) lab table:
the query always returns '-', so behaviour is unchanged until real
L3VPN data exists, then RDs auto-populate. Existing panel SQL
('$rd' = '-' OR rd = '$rd') is untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Explores the real DFZ table received from the AS57355 route server
via the GoBGP feed (the '$feed' / GoBGP BMP peer): IPv4/IPv6 prefix
counts, distinct origin ASes, prefix-length distribution, top origin
ASes by prefix count, and an overlap-based prefix lookup. Serves as
the comparison baseline for the Router Diff dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GoBGP's BMP config requires a literal IP — 'obmp-collector' failed
to parse and the container crash-looped. Point BMP export at the
docker host IP (10.40.40.202) where the collector publishes port
5000; stable across container recreation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generalizes the RR-specific RR Loc-RIB Diff into a comparison of up
to four selectable routers. router1/router2 are required; router3/
router4 default to a '-- none --' sentinel and drop cleanly out of
every query (no empty-IN, no dangling predicates).
Panels: per-router prefix counts, divergent-prefix count, a presence
matrix (row per prefix, column per router, cell = best-path next-hop),
a divergence detail table classifying missing / next-hop / AS-path
disagreement, and a per-prefix all-paths drill-down. Once the GoBGP
global feed (E1) is up, GLOBAL-FEED is selectable as any of the four
for lab-vs-Internet diffing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New gobgp service: GoBGP peers eBGP-multihop with the AS57355 lab
route server (Bromirski) for the full real IPv4 + IPv6 Internet table
and BMP-exports it to the OpenBMP collector, landing in ip_rib as a
monitored peer.
Config follows the route server's published peering spec: local AS
65001, no password, keepalive 3600 / hold-time 7200, IPv4 feed on the
v4 session and IPv6 feed on the v6 session. gobgp/mrt-refresh.sh is a
cron-safe fallback that injects RouteViews MRT RIB dumps when the live
session is down. The live BGP session is not started here — bringing
gobgp up establishes the external session and loads ~1M routes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan for a local full-Internet routing table, a generalized N-way
router diff, and VRF/RD scoping:
- E1: GoBGP container peering AS57355 (Bromirski lab route server)
for a live full v4/v6 table, MRT RIB dumps as a 2-hourly fallback,
BMP-exported into ip_rib as a GLOBAL-FEED peer.
- E2: generic up-to-4-router diff dashboard (presence matrix),
generalized from the RR-specific rr_locrib_diff.
- E3: global table exploration dashboard.
- E4: VRF/RD scoping across unicast + L3VPN dashboards (built to
schema; not lab-verifiable with CML IOS-XR).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The AS map previously exploded ~4.4M base_attrs AS_PATH rows live,
three times per load (one per panel), ~1.8s each — slow enough that
navigating away cancelled the queries mid-flight.
Add mv_as_adjacency: undirected consecutive-AS pairs with occurrence
counts over the full RIB (17k rows), refreshed hourly by pg_cron via
REFRESH ... CONCURRENTLY. The dashboard panels now read the view in
~1ms. Min-occurrence options rescaled for full-RIB counts
(2000/5000/10000/50000, default 2000 -> ~63-node graph).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The node graph rendered blank because the two CML/PROX labs formed
two disconnected components (iBGP-only meshes within each lab), and
Grafana's nodeGraph layout renders nothing for a disconnected graph.
Match BGP sessions to monitored routers by peer IP as well as peer
BGP-ID, so the real cross-lab eBGP sessions become graph edges. The
graph is now one connected component (30 iBGP + 4 eBGP edges) and
lays out. The companion external-neighbours table uses the same
peer-IP check so those sessions are no longer double-listed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The node graph rendered empty because the edges target returned a
string mainstat ('iBGP'/'eBGP'). Grafana's nodeGraph treats edge
mainStat as numeric for layout/labelling; a string value silently
breaks the layout so no nodes are drawn (the working LS map and the
original ls_topo both cast edge mainstat to an integer).
Edge mainstat is now COUNT(DISTINCT feed)::int (BMP peer-feed count
for the router pair); the iBGP/eBGP label moves to secondarystat and
detail__session_type, which accept strings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch the route-reflector membership test to
= ANY(ARRAY[${rr_loopbacks:singlequote}]::text[]) — the singlequote
format is the one already proven to interpolate correctly in this
Grafana instance (rr_locrib_diff uses it), and the ARRAY[...]::text[]
wrapper stays valid (empty array) when the variable resolves empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The protocol variable has includeAll enabled, so Grafana auto-quotes
its value ('IS-IS_L2'); the SQL then wrapped it again, producing
''IS-IS_L2'' and a syntax error that blanked the node graph. Replace
the quoted equality filter with IN ($protocol) — Grafana already
emits a quoted CSV — and make the variable multi-select so "All"
expands cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the rr_loopbacks variable resolved empty, the IN
(${rr_loopbacks:singlequote}) clause expanded to IN (), a SQL
syntax error that blanked the topology panel. Switch to
= ANY(string_to_array('${rr_loopbacks:csv}', ',')), which yields
a no-match (not a syntax error) on an empty variable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Create a new OBMP-Maps Grafana folder (folderUid 1006) with four
data-visualization dashboards built on nodeGraph and geomap panels:
- BGP Peer Map: routers as nodes, BGP sessions as edges; iBGP/eBGP
edge typing and operator-editable rr_loopbacks variable to denote
route reflectors; companion table for sessions to non-monitored
neighbours.
- IGP / Link-State Topology Map: reworked from LinkState-1004 and
moved here (uid preserved); scoped by peer feed / protocol / AS so
the 489-node BGP-LS topology stays readable; SR-capability rings.
- AS Relationship Map: AS adjacency graph from consecutive AS_PATH
pairs over a 200k-route sample; min-occurrence and focus-AS
variables; nodes enriched from info_asn whois.
- Geographic Prefix Map: geomap of RIB prefixes and origin ASes by
IP geolocation, with a note that lab 10.x loopbacks do not
geolocate; bounded geo_ip join via a sample-size variable.
Also add a data link on the Looking Glass ASN Info panel's origin_as
column that jumps to the ASN View dashboard scoped to that AS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
asn_num was a fixed custom variable; converting it to a textbox lets an
operator look up any origin AS and see all of its RIB prefixes, upstreams,
and downstreams.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RCA: the exabgp container was OOM-killed — its 512m mem_limit was far too
small for the full-table feature (900K route objects in memory). Raises the
limit to a parameterized 6g default (EXABGP_MEM_LIMIT).
Adds Docker healthchecks to 14 services (port/HTTP probes) so unhealthy
containers are visible. Adds a Telegraf docker input that collects per-
container CPU/memory/IO into InfluxDB, plus a "Stack Resources" dashboard —
so resource pressure is caught before it causes an OOM crash. telegraf runs
with an overridden entrypoint so it keeps root and can read the docker socket.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The IOS-XR routers negotiate IPv6 unicast capability, but the generated
exabgp.conf declared only ipv4 unicast — producing repeated "route family
(ipv6/unicast) is not configured" errors that crashed ExaBGP. Declaring
ipv6 unicast on the neighbor matches the routers' capabilities and stops
the crash-restart cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a panel that flags the next-hop-self-on-an-RR anti-pattern: reflected
routes (those carrying ORIGINATOR_ID) whose NEXT_HOP is an RR loopback while
the route was originated by a different router — meaning the RR rewrote
next-hop to itself and has been pulled into the forwarding path. RR-originated
routes and legitimately-imported eBGP routes (originator == next-hop) are
excluded. An editable rr_loopbacks template variable keeps it environment-
agnostic — useful for validating RR behavior during an IOS-XR to Junos
migration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
