netbox-diode-project/collectors/README.md

NetBox Diode Collectors

A suite of data collectors that discover infrastructure from various sources and ingest it into NetBox through the Diode reconciliation pipeline (or, for cables, directly via the NetBox REST API).

Architecture

┌─────────────────┐     ┌──────────────┐     ┌──────────────┐     ┌────────┐
│  Data Sources    │     │  Collectors   │     │   Diode      │     │ NetBox │
│                  │     │              │     │  Pipeline    │     │        │
│  Proxmox VE     │────▶│  proxmox_    │────▶│              │     │        │
│  Proxmox PBS    │────▶│  pbs_        │────▶│  ingester    │────▶│  DCIM  │
│  VMware vCenter │────▶│  vmware_     │────▶│  reconciler  │     │  IPAM  │
│  Docker         │────▶│  docker_     │────▶│              │     │  Virt  │
│  Cisco CML      │────▶│  cml_        │────▶│              │     │        │
│  Zabbix         │────▶│  zabbix_     │────▶│              │     │        │
│  Observium      │────▶│  observium_  │────▶│              │     │        │
│  UniFi          │────▶│  unifi_      │────▶│              │     │        │
│  Network (SSH)  │────▶│  network_    │────▶│              │     │        │
│                  │     └──────────────┘     └──────────────┘     │        │
│                  │     ┌──────────────┐                          │        │
│  LLDP Neighbors │────▶│  cable_      │─────────────────────────▶│ Cables │
│  (via SSH)      │     │  exporter    │  (REST API direct)       │        │
└─────────────────┘     └──────────────┘                          └────────┘

All collectors except cable_exporter.py use the Diode SDK to push entities through the Diode ingestion pipeline (gRPC → ingester → reconciler → NetBox). The cable exporter uses the NetBox REST API directly because the Diode reconciler does not yet support cable entities (diode#191).

Quick Start

# Activate the project virtual environment
source .venv/bin/activate

# Configure credentials
cp .env.example .env   # Edit with your credentials
vi .env

# Test a collector (dry-run — no changes made)
python collectors/proxmox_collector.py --dry-run

# Run for real
python collectors/proxmox_collector.py

All collectors support --dry-run, --log-level, and --env-file flags.

---

Collectors Overview

Collector	Source	NetBox Entities	Auth Method
proxmox_collector	Proxmox VE API	Devices, Clusters, VMs, LXC, Interfaces, IPs, Disks	API token
pbs_collector	PBS API	Devices, Interfaces, IPs, Services (datastores)	API token
vmware_collector	vSphere API	Devices, Clusters, VMs, Interfaces, IPs, Disks	Username/password
docker_collector	Docker API	Clusters, VMs (containers), VMInterfaces, IPs	Docker socket/TCP
cml_collector	CML REST API	Devices, Interfaces, IPs, Cables, Configs	Username/password
network_collector	SSH (NAPALM/Netmiko)	Devices, Interfaces, IPs, VLANs, VRFs, Prefixes, Cables, Configs, Inventory	SSH credentials
unifi_collector	UniFi Controller API	Devices, Interfaces (ports + radios), VLANs, WLANs, Cables, Prefixes	Username/password
zabbix_collector	Zabbix API	Devices, Interfaces, IPs, Custom Fields	Username/password or API token
observium_collector	Observium REST API	Devices, Interfaces, IPs	Username/password
cable_exporter	SSH (LLDP) + NetBox API	Cables	SSH + NetBox API token

---

Proxmox VE Collector

File: proxmox_collector.py (843 lines)

Discovers Proxmox VE clusters including hypervisor nodes, QEMU VMs, LXC containers, network interfaces, IP addresses, and virtual disks. Supports multiple standalone PVE hosts via numbered environment variables.

Entity Mapping

Proxmox Object	NetBox Entity	Notes
PVE Cluster	Cluster (type: Proxmox VE)	One cluster per PVE host
PVE Node	Device	Model: Proxmox VE Node
Node NIC	Interface	Type detected from name (bond, bridge, veth, etc.)
Node IP	IPAddress	From network config
QEMU VM	VirtualMachine	Status mapped from PVE state
LXC Container	VirtualMachine	Status mapped from PVE state
VM/LXC NIC	VMInterface	MAC and VLAN extracted from config
VM/LXC Disk	InventoryItem	Size parsed from config string
Guest Agent IP	IPAddress	Requires qemu-guest-agent running
LXC static IP	IPAddress	From LXC network config

Key Functions

Function	Description
get_pve_hosts()	Builds host list from numbered env vars (PVE_HOST, PVE_HOST_2, etc.)
connect_pve(config)	Creates ProxmoxAPI connection with token auth
collect_node_info(prox, node)	Gets node CPU, memory, status
collect_qemu_vms(prox, node)	Lists all QEMU VMs on a node
collect_vm_config(prox, node, vmid)	Gets full VM config (NICs, disks, boot order)
collect_vm_guest_agent_ips(prox, node, vmid)	Gets IPs from QEMU guest agent
collect_lxc_containers(prox, node)	Lists all LXC containers on a node
parse_pve_net_config(net_str)	Parses PVE NIC config: virtio=AA:BB:CC:DD:EE:FF,bridge=vmbr0,tag=100
parse_lxc_net_config(net_str)	Parses LXC NIC config: name=eth0,bridge=vmbr0,ip=10.0.0.5/24
parse_disk_size(size_str)	Converts 32G, 500M, 1T to bytes
build_vm_entity(...)	Builds VirtualMachine entity with platform, cluster, resources
build_vm_disk_entities(...)	Builds InventoryItem entities for each disk

Usage

python collectors/proxmox_collector.py --dry-run
python collectors/proxmox_collector.py --log-level DEBUG

Environment Variables

PVE_HOST=192.168.1.190         # First PVE host
PVE_USER=root@pam              # API user
PVE_TOKEN_NAME=diode           # API token name
PVE_TOKEN_VALUE=<token>        # API token value (required)
PVE_VERIFY_SSL=false           # SSL verification
PVE_PORT=8006                  # API port

# Additional hosts use numbered suffixes
PVE_HOST_2=10.40.40.107
PVE_TOKEN_VALUE_2=<token>
# ... up to PVE_HOST_N

---

Proxmox Backup Server Collector

File: pbs_collector.py (464 lines)

Discovers PBS hosts as devices with their network interfaces, IP addresses, and datastores (modeled as Services in NetBox).

Entity Mapping

PBS Object	NetBox Entity	Notes
PBS Host	Device	Model: Proxmox Backup Server
Network Interface	Interface	Type detected from name
Interface IP	IPAddress	With prefix length
Datastore	Service (custom)	Size/usage in description

Key Functions

Function	Description
get_pbs_hosts()	Builds host list from numbered env vars
connect_pbs(config)	Connects to PBS API with token auth
collect_datastores(pbs)	Lists all datastores with sizes
collect_datastore_usage(pbs, store)	Gets datastore disk usage
collect_datastore_snapshots(pbs, store)	Lists backup snapshots per datastore
build_datastore_entities(...)	Creates Service entities for datastores

Usage

python collectors/pbs_collector.py --dry-run

Environment Variables

PBS_HOST_1=10.40.40.150        # First PBS host
PBS_USER_1=diode@pbs
PBS_TOKEN_NAME_1=diode
PBS_TOKEN_VALUE_1=<token>      # Required
PBS_PORT_1=8007                # Default: 8007
# ... up to PBS_HOST_N

---

VMware vSphere Collector

File: vmware_collector.py (544 lines)

Discovers ESXi hosts and VMs from vCenter or standalone ESXi via the vSphere API (pyVmomi).

Entity Mapping

vSphere Object	NetBox Entity	Notes
Cluster	Cluster (type: VMware vSphere)	—
ESXi Host	Device	Manufacturer from vendor string
Host vNIC	Interface	Speed-based type detection
VM	VirtualMachine	Power state mapped to status
VM vNIC	VMInterface	MAC address from config
VM Disk	InventoryItem	Size from disk backing
VM Guest IP	IPAddress	From VMware Tools guest info

Key Functions

Function	Description
connect_vsphere(cfg)	Connects to vCenter/ESXi via pyVmomi
get_all_objects(si, obj_type, folder)	Retrieves all managed objects of a type
build_cluster_entities(si, site)	Creates Cluster entities from vSphere clusters
build_host_entities(si, site)	Creates Device entities for ESXi hosts + interfaces
build_vm_entities(si, site, host_map)	Creates VM + interface + disk + IP entities
_mask_to_prefix(mask)	Converts subnet mask string to prefix length

Usage

python collectors/vmware_collector.py --dry-run

Environment Variables

VCENTER_HOST=vcenter.local     # vCenter or ESXi IP/hostname
VCENTER_USER=administrator@vsphere.local
VCENTER_PASSWORD=<password>
VCENTER_PORT=443
VCENTER_VERIFY_SSL=false
VCENTER_SITE=main              # NetBox site name

---

Docker Collector

File: docker_collector.py (358 lines)

Discovers Docker containers as VirtualMachines grouped into Clusters (one per Docker host). Supports local Docker socket and remote TCP connections.

Entity Mapping

Docker Object	NetBox Entity	Notes
Docker Host	Cluster (type: Docker)	Host name from docker info
Container	VirtualMachine	Status mapped (running→active, exited→offline)
Container Network	VMInterface	MAC address, one per Docker network
Container IPv4	IPAddress	With prefix length from Docker
Container IPv6	IPAddress	Global IPv6 if assigned

Custom Fields

• docker_container_id — Container short ID
• docker_compose_project — Compose project name (from labels)

Key Functions

Function	Description
connect_docker(host, tls)	Connects to Docker (local socket or remote TCP)
get_host_info(client)	Gets Docker host system info
get_containers(client, all)	Lists containers (running or all)
build_cluster_entity(host, site)	Creates Cluster entity for Docker host
build_container_entities(container, host, site)	Creates VM + interfaces + IPs per container

Usage

# Local Docker (no config needed)
python collectors/docker_collector.py --dry-run

# Include stopped containers
python collectors/docker_collector.py --dry-run --all

Environment Variables

DOCKER_HOSTS=tcp://10.0.0.5:2375,tcp://10.0.0.6:2375   # Comma-separated (default: local socket)
DOCKER_SITE=main
DOCKER_TLS_VERIFY=false

---

Cisco CML Collector

File: cml_collector.py (456 lines)

Syncs Cisco Modeling Labs topology into NetBox: lab nodes become devices, interfaces are mapped with types inferred from names, links become cables, and L3 addresses and device configs are captured.

Entity Mapping

CML Object	NetBox Entity	Notes
Lab Node	Device	Model/role/platform from node definition
Node Interface	Interface	Type inferred from name (Gi→1000base-t, etc.)
Node L3 Address	IPAddress	From CML's discovered IPv4/IPv6
Lab Link	Cable	Between interface endpoints
Node Config	DeviceConfig	Startup configuration text

Node Definition Mappings

The collector maps CML node definitions to NetBox device attributes:

Node Definition	Platform	Role	Manufacturer
iosv	Cisco IOS	Router	Cisco
iosvl2	Cisco IOS	Switch	Cisco
iosxrv / iosxrv9000	Cisco IOS-XR	Router	Cisco
csr1000v / cat8000v	Cisco IOS-XE	Router	Cisco
cat9000v	Cisco IOS-XE	Switch	Cisco
nxosv / nxosv9000	Cisco NX-OS	Switch	Cisco
asav	Cisco ASA	Firewall	Cisco
server / ubuntu / alpine	Linux / Ubuntu / Alpine	Server	Generic

Key Functions

Function	Description
build_node_entity(node, site)	Creates Device entity from CML node with mapped type/role/platform
build_interface_entities(node, site)	Creates Interface entities, type inferred from name regex
build_ip_entities(node, site)	Creates IPAddress entities from CML L3 discovery
build_cable_entities(lab, site, node_map)	Creates Cable entities from CML links
build_config_entity(node, site)	Captures node startup config as DeviceConfig

Usage

python collectors/cml_collector.py --dry-run
python collectors/cml_collector.py                     # All labs
CML_LAB="my-lab" python collectors/cml_collector.py    # Specific lab

Environment Variables

CML_HOST=10.40.40.50           # CML controller address
CML_USER=admin
CML_PASSWORD=<password>
CML_LAB=                       # Optional: specific lab name or ID
CML_VERIFY_SSL=false
CML_SITE=CML                   # NetBox site for CML devices

---

Network Collector

File: network_collector.py (1930 lines)

The most comprehensive collector — discovers Cisco and Brocade network devices via SSH using NAPALM (with Netmiko fallback for unsupported platforms). Collects device facts, interfaces with speeds and types, IP addresses, LLDP neighbors, VLANs, VRFs, prefixes, device configs, and hardware inventory. Optionally uses pyATS/Genie for CDP and IGP neighbor data, and pushes BGP sessions to the netbox-bgp plugin API.

Supported Platforms

Platform	Driver	Connection Method	Notes
Cisco IOS	ios	NAPALM	Full feature support
Cisco IOS-XE	ios	NAPALM	Same as IOS
Cisco IOS-XR	iosxr	Netmiko (cisco_xr)	Custom parsers for facts, interfaces, IPs, LLDP
Cisco NX-OS	nxos / nxos_ssh	NAPALM	—
Arista EOS	eos	NAPALM	—
Juniper Junos	junos	NAPALM	—
Brocade FastIron (ICX)	ruckus_fastiron	Netmiko	Custom parsers
Brocade VDX/NOS	nos / brocade_nos	Netmiko (extreme_nos)	Custom parsers

Entity Mapping

Network Object	NetBox Entity	Notes
Device	Device	With model, serial, platform, role
Physical Interface	Interface	Type from name or speed
Loopback/VLAN/Tunnel	Interface (virtual)	—
Port-channel/LAG	Interface (lag)	—
Bundle-Ether	Interface (lag)	IOS-XR LAG interfaces
Interface IP	IPAddress	With prefix length
LLDP Neighbor	Cable	Deduplicated bidirectional
VLAN	VLAN + VLANGroup	Per-device VLAN group
VRF	VRF	From NAPALM get_network_instances
IP Subnet	Prefix	Computed from IP + mask
Running Config	DeviceConfig	Full running-config text
Hardware Module	InventoryItem	From NAPALM environment data
BGP Neighbor	Plugin API push	To netbox-bgp (if installed)

Interface Type Detection

Interface types are determined by a two-pass system:

1. Name-based (NAME_TO_TYPE): Regex patterns match interface names

   • GigabitEthernet / Gi → 1000base-t
   • TenGigabitEthernet / Te → 10gbase-x-sfpp
   • FortyGig / Fo → 40gbase-x-qsfpp
   • HundredGig / Hu → 100gbase-x-qsfp28
   • Loopback / Vlan / Tunnel → virtual
   • Port-channel / Bundle-Ether → lag
   • e 1/1/1 (Brocade) → 1000base-t

2. Speed-based (SPEED_TO_TYPE): Falls back to reported link speed

   • 10 Mbps → 10base-t, 100 → 100base-tx, 1000 → 1000base-t, etc.

IOS-XR Support (Netmiko)

IOS-XR devices use Netmiko instead of NAPALM due to better reliability. Custom parsers handle IOS-XR-specific output formats:

Function	Description
_netmiko_parse_facts_iosxr(conn, facts)	Parses show version, show inventory, show running hostname — strips RP/0/RP0/CPU0: prefix from hostname, extracts model (cleaned), serial (prefers "Rack 0" chassis)
_netmiko_parse_interfaces_iosxr(conn, driver)	Parses show interfaces for description, MAC, MTU, bandwidth (Kbit→Mbps conversion)
_netmiko_parse_interfaces_ip_iosxr(conn, driver, ifaces)	Parses show ipv4 interface brief + show running-config interface for IP addresses with correct prefix lengths; also collects IPv6

Brocade Support (Netmiko)

Function	Description
_netmiko_parse_facts(conn, driver)	Parses show version for model/serial, dispatches to IOS-XR if needed
_netmiko_parse_interfaces(conn, driver)	Parses show interface brief / show interface for port status
_netmiko_parse_interfaces_ip(conn, driver, ifaces)	Parses show ip interface / show running-config for IPs
_netmiko_parse_lldp(conn, driver)	Parses show lldp neighbors detail for LLDP neighbor data

Other Key Functions

Function	Description
load_inventory(path)	Loads inventory.yaml with defaults, per-device overrides
merge_device_config(entry, defaults)	Merges per-device config with inventory defaults
normalize_interface_name(name)	Expands abbreviations: Gi0/1 → GigabitEthernet0/1
map_interface_type(name, speed)	Returns NetBox interface type slug
connect_device(host, driver, ...)	Creates NAPALM connection
connect_netmiko(host, driver, ...)	Creates Netmiko connection
collect_napalm_data(dev)	Collects all data via NAPALM getters
collect_netmiko_data(conn, driver)	Collects all data via Netmiko parsers
collect_pyats_data(...)	Optional pyATS/Genie for CDP, OSPF, IS-IS
build_device_entity(...)	Creates Device entity with full attributes
build_interface_entities(...)	Creates Interface entities with types
build_ip_entities(...)	Creates IPAddress + Prefix entities
build_vlan_entities(...)	Creates VLAN + VLANGroup entities
build_cable_entities(...)	Creates Cable entities from LLDP (bidirectional dedup)
build_config_entity(...)	Creates DeviceConfig entity
build_inventory_entities(...)	Creates InventoryItem entities from hardware modules
push_bgp_sessions(...)	Pushes BGP data to netbox-bgp plugin REST API

Usage

python collectors/network_collector.py -i collectors/inventory.yaml --dry-run
python collectors/network_collector.py -i collectors/inventory.yaml
python collectors/network_collector.py -i collectors/inventory.yaml --no-pyats --log-level DEBUG

Inventory File Format

Credentials are stored in collectors/inventory.yaml (gitignored), not in .env:

defaults:
  driver: ios
  username: admin
  password: secret
  secret: enable_secret    # Enable password (optional)
  timeout: 60

devices:
  - host: 10.0.0.1
    driver: ios
    # Uses defaults for username/password

  - host: 10.0.0.2
    driver: iosxr
    username: cisco        # Override default credentials

  - host: 10.0.0.3
    driver: ruckus_fastiron
    username: admin
    password: brocade_pass

---

UniFi Collector

File: unifi_collector.py (754 lines)

Discovers Ubiquiti UniFi infrastructure including UDM/UDM-SE gateways, managed switches, access points, switch ports with PoE and SFP detection, WiFi radios, VLANs, WLANs, and LLDP-based cabling.

Entity Mapping

UniFi Object	NetBox Entity	Notes
UDM/Switch/AP	Device	Model from model field, serial from serial
Switch Port	Interface	Speed/PoE/SFP detection
WiFi Radio	Interface (wireless)	Band-specific type (802.11n/ac/ax)
Network/VLAN	VLAN + Prefix	From UniFi network config
WLAN (SSID)	WirelessLAN	Auth type from security settings
LLDP Neighbor	Cable	Deduplicated bidirectional

Key Functions

Function	Description
connect_unifi(cfg)	Connects to UniFi Controller API (UDM or legacy)
_build_device_entities(dev, site)	Creates Device entity with model/serial/firmware
_build_port_entities(dev, site)	Creates port Interface entities with speed/PoE/SFP detection
_build_radio_entities(dev, site)	Creates WiFi radio Interface entities with band/channel
_build_cable_entities(devices, site)	Creates Cable entities from LLDP uplink data
_build_network_entities(networks, site)	Creates VLAN + Prefix from UniFi networks
_build_wlan_entities(wlans, site)	Creates WirelessLAN entities with auth type

Usage

python collectors/unifi_collector.py --dry-run

Environment Variables

UNIFI_HOST=192.168.1.1         # UDM-SE or Controller IP
UNIFI_USER=admin
UNIFI_PASSWORD=<password>
UNIFI_SITE=default             # UniFi site name
UNIFI_VERIFY_SSL=false
UNIFI_IS_UDM=true              # true for UDM/UDM-SE, false for legacy controller
UNIFI_NETBOX_SITE=main         # NetBox site name

---

Zabbix Collector

File: zabbix_collector.py (372 lines)

Imports device inventory from Zabbix monitoring into NetBox. Automatically infers platform and device role from Zabbix host groups, templates, and inventory fields. Adds a zabbix_host_id custom field for cross-referencing.

Entity Mapping

Zabbix Object	NetBox Entity	Notes
Host	Device	Status: monitored→active, unmonitored→offline
Host Interface	Interface	Named by type: mgmt0, snmp0, ipmi0
Interface IP	IPAddress	/32 prefix (Zabbix doesn't provide prefix length)

Intelligent Mapping

• Platform detection (guess_platform): Checks inventory os_full/os_short fields and template names against keyword list (linux, windows, cisco, juniper, vmware, etc.)
• Role detection (guess_role): Checks host group names for keywords (router, switch, firewall, hypervisor, server)
• Custom fields: zabbix_host_id stored for cross-referencing back to Zabbix

Key Functions

Function	Description
connect_zabbix(cfg)	Connects to Zabbix API (user/pass or API token)
collect_hosts(zapi)	Fetches all hosts with interfaces, inventory, groups, templates
guess_platform(host_data)	Infers platform from inventory OS and template names
guess_role(host_data, default)	Infers device role from host group names
build_host_entities(host_data, cfg)	Creates Device + Interface + IP entities

Usage

python collectors/zabbix_collector.py --dry-run

Environment Variables

ZABBIX_URL=http://10.40.40.20/api_jsonrpc.php
ZABBIX_USER=Admin
ZABBIX_PASSWORD=<password>
# OR use API token (Zabbix 5.4+):
ZABBIX_API_TOKEN=<token>
ZABBIX_SITE=main
ZABBIX_DEFAULT_ROLE=Server

---

Observium Collector

File: observium_collector.py (378 lines)

Imports device, port, and IP data from the Observium REST API. Requires Observium Professional or Enterprise edition (Community Edition has no REST API).

Entity Mapping

Observium Object	NetBox Entity	Notes
Device	Device	Platform/role inferred from OS type
Port	Interface	Type from SNMP ifType
Port IP	IPAddress	With prefix length

Key Functions

Function	Description
api_get(base_url, endpoint, auth, params)	Makes authenticated GET request to Observium API
collect_all_entities(cfg)	Fetches devices → ports → IPs, builds all entities

Usage

python collectors/observium_collector.py --dry-run

Environment Variables

OBSERVIUM_URL=http://10.40.40.30/api/v0
OBSERVIUM_USER=admin
OBSERVIUM_PASSWORD=<password>
OBSERVIUM_SITE=main
OBSERVIUM_DEFAULT_ROLE=Network Device

---

LLDP Cable Exporter

File: cable_exporter.py (576 lines)

A standalone tool that creates network cables in NetBox by collecting LLDP neighbor data from devices via SSH, validating both endpoints against the NetBox inventory, and creating cables via the NetBox REST API. This bypasses the Diode pipeline because cable entities are not yet supported by the Diode reconciler.

Why REST API Instead of Diode?

The Diode reconciler returns "entity is nil" errors for cable entities (diode#191). The cable exporter uses the NetBox REST API directly, which provides:

• Endpoint validation — both devices and interfaces must exist in NetBox
• Duplicate detection — checks existing cables before creating
• Precise matching — uses interface IDs, not names
• Per-cable error handling — one failure doesn't stop the batch

How It Works

1. Build Cache     Query NetBox API for devices, interfaces, existing cables
                   at the target site → in-memory lookup tables

2. Collect LLDP    SSH to each device in inventory.yaml
                   NAPALM get_lldp_neighbors_detail() or Netmiko fallback

3. Match & Validate  For each LLDP pair:
                     - Normalize interface names
                     - Look up both devices by hostname in NetBox
                     - Look up both interfaces by (device_id, name)
                     - Skip LAG/virtual endpoints (NetBox rejects them)
                     - Skip if cable already exists
                     - Deduplicate bidirectional links (A→B = B→A)

4. Create Cables   POST /api/dcim/cables/ for each validated pair
                   Also generates CSV audit trail

Interface Name Matching

LLDP reports interface names that may differ from what's stored in NetBox. The exporter uses fuzzy matching:

LLDP Reports	NetBox Has	Match Strategy
GigabitEthernet0/0/0/1	GigabitEthernet0/0/0/1	Exact match
FortyGigabitEthernet1/0/1	FortyGigabitEthernet 1/0/1	Space variation
TenGigabitEthernet1/0/34:1	Te 1/0/34:1	Abbreviated form
Gi0/1	GigabitEthernet0/1	Long form expansion

Supported abbreviation mappings: Gi↔GigabitEthernet, Te↔TenGigabitEthernet, Fa↔FastEthernet, Fo↔FortyGigabitEthernet, Hu↔HundredGigabitEthernet, BE↔Bundle-Ether, Eth↔Ethernet, Mg↔MgmtEth.

Key Classes and Functions

Component	Description
NetBoxClient	REST API wrapper with pagination, auth, GET/POST methods
NetBoxClient.get_devices(site)	Fetches all devices at a site
NetBoxClient.get_interfaces(device_id)	Fetches interfaces for a device
NetBoxClient.get_cables()	Fetches existing cables for duplicate detection
NetBoxClient.ensure_tag(name, slug)	Creates tag if it doesn't exist (idempotent)
NetBoxClient.create_cable(a_id, b_id, tag_id)	POSTs a cable with terminations
build_netbox_cache(nb, site)	Builds device/interface/cable lookup dicts
collect_lldp_from_device(cfg)	SSH to one device, returns LLDP data
collect_all_lldp(inventory)	Collects LLDP from all inventory devices
lookup_interface(cache, dev_id, name)	Fuzzy interface name matching
build_cable_list(lldp, cache, nb)	Validates and deduplicates LLDP → cable list
write_csv(cables, path, site)	Writes CSV audit trail
create_cables_via_api(nb, cables, tag_id)	Creates cables via REST API

Usage

# Dry run — show what cables would be created
python collectors/cable_exporter.py -i collectors/inventory.yaml --dry-run

# Create cables in NetBox
python collectors/cable_exporter.py -i collectors/inventory.yaml --env-file ../.env

# CSV only (no API calls)
python collectors/cable_exporter.py -i collectors/inventory.yaml --csv-only

# Target a specific site
python collectors/cable_exporter.py -i collectors/inventory.yaml --site CML

Environment Variables

NETBOX_API_URL=http://172.19.77.160:8000
NETBOX_API_TOKEN=nbt_<key>.<token>

Output

NetBox inventory: 45 devices, 1200 interfaces loaded
LLDP collection: 18/19 devices connected, 62 neighbor pairs found
Validation: 48 cables matched (14 skipped: 6 device not found, 5 interface not found, 3 LAG endpoint)
Existing cables: 0 (0 duplicates skipped)
Created: 48 cables
CSV written to: cables_export.csv

---

Common Patterns

Diode Ingestion Flow

All Diode-based collectors follow the same pattern:

# 1. Build Entity objects
entities = []
entities.append(Entity(device=Device(name="router1", ...)))
entities.append(Entity(interface=Interface(device=dev_ref, name="Gi0/1", ...)))
entities.append(Entity(ip_address=IPAddress(address="10.0.0.1/24", ...)))

# 2. Ingest via Diode SDK
with DiodeClient(target=target, client_id=id, client_secret=secret, ...) as client:
    resp = client.ingest(entities=entities)

Shared Environment Variables (Diode)

All Diode-based collectors use:

DIODE_TARGET=grpc://localhost:8080/diode    # Diode ingester endpoint
DIODE_CLIENT_ID=diode-ingester              # Or INGESTER_CLIENT_ID
DIODE_CLIENT_SECRET=<secret>                # Or INGESTER_CLIENT_SECRET (required)

Dry Run Mode

Every collector supports --dry-run which prints entities to stdout without making any changes. Always test with --dry-run first.

Logging

All collectors support --log-level (DEBUG, INFO, WARNING, ERROR). Use DEBUG for troubleshooting connection issues.

---

Dependencies

Python Packages

netboxlabs-diode-sdk          # All Diode-based collectors
napalm                         # network_collector, cable_exporter
netmiko                        # network_collector, cable_exporter (Netmiko fallback)
pyyaml                         # network_collector, cable_exporter (inventory.yaml)
requests                       # cable_exporter, network_collector (BGP plugin)
python-dotenv                  # cable_exporter
proxmoxer                      # proxmox_collector, pbs_collector
pyvmomi                        # vmware_collector
docker                         # docker_collector
pyzabbix                       # zabbix_collector
virl2_client                   # cml_collector
pyats + genie                  # network_collector (optional — CDP, OSPF, IS-IS)

Install

pip install -r requirements.txt

---

File Reference

File	Lines	Description
network_collector.py	1930	Multi-vendor network device collector (NAPALM/Netmiko)
proxmox_collector.py	843	Proxmox VE hypervisor + VM/LXC collector
unifi_collector.py	754	UniFi controller device/port/radio collector
cable_exporter.py	576	LLDP cable creator via NetBox REST API
vmware_collector.py	544	VMware vSphere host/VM collector
pbs_collector.py	464	Proxmox Backup Server collector
cml_collector.py	456	Cisco CML lab topology collector
observium_collector.py	378	Observium NMS device/port collector
zabbix_collector.py	372	Zabbix monitoring device collector
docker_collector.py	358	Docker container collector
ENV_REFERENCE.md	245	Environment variable reference and setup guide