# NetBox Diode Collectors

A suite of data collectors that discover infrastructure from various sources and ingest it into NetBox through the [Diode](https://github.com/netboxlabs/diode) reconciliation pipeline (or, for cables, directly via the NetBox REST API).

## Architecture

```
┌─────────────────┐     ┌──────────────┐     ┌──────────────┐     ┌────────┐
│  Data Sources    │     │  Collectors   │     │   Diode      │     │ NetBox │
│                  │     │              │     │  Pipeline    │     │        │
│  Proxmox VE     │────▶│  proxmox_    │────▶│              │     │        │
│  Proxmox PBS    │────▶│  pbs_        │────▶│  ingester    │────▶│  DCIM  │
│  VMware vCenter │────▶│  vmware_     │────▶│  reconciler  │     │  IPAM  │
│  Docker         │────▶│  docker_     │────▶│              │     │  Virt  │
│  Cisco CML      │────▶│  cml_        │────▶│              │     │        │
│  Zabbix         │────▶│  zabbix_     │────▶│              │     │        │
│  Observium      │────▶│  observium_  │────▶│              │     │        │
│  UniFi          │────▶│  unifi_      │────▶│              │     │        │
│  Network (SSH)  │────▶│  network_    │────▶│              │     │        │
│                  │     └──────────────┘     └──────────────┘     │        │
│                  │     ┌──────────────┐                          │        │
│  LLDP Neighbors │────▶│  cable_      │─────────────────────────▶│ Cables │
│  (via SSH)      │     │  exporter    │  (REST API direct)       │        │
└─────────────────┘     └──────────────┘                          └────────┘
```

All collectors except `cable_exporter.py` use the **Diode SDK** to push entities through the Diode ingestion pipeline (gRPC → ingester → reconciler → NetBox). The cable exporter uses the **NetBox REST API** directly because the Diode reconciler does not yet support cable entities ([diode#191](https://github.com/netboxlabs/diode/issues/191)).

## Quick Start

```bash
# Activate the project virtual environment
source .venv/bin/activate

# Configure credentials
cp .env.example .env   # Edit with your credentials
vi .env

# Test a collector (dry-run — no changes made)
python collectors/proxmox_collector.py --dry-run

# Run for real
python collectors/proxmox_collector.py
```

All collectors support `--dry-run`, `--log-level`, and `--env-file` flags.

---

## Collectors Overview

| Collector | Source | NetBox Entities | Auth Method |
|-----------|--------|----------------|-------------|
| [proxmox_collector](#proxmox-ve-collector) | Proxmox VE API | Devices, Clusters, VMs, LXC, Interfaces, IPs, Disks | API token |
| [pbs_collector](#proxmox-backup-server-collector) | PBS API | Devices, Interfaces, IPs, Services (datastores) | API token |
| [vmware_collector](#vmware-vsphere-collector) | vSphere API | Devices, Clusters, VMs, Interfaces, IPs, Disks | Username/password |
| [docker_collector](#docker-collector) | Docker API | Clusters, VMs (containers), VMInterfaces, IPs | Docker socket/TCP |
| [cml_collector](#cisco-cml-collector) | CML REST API | Devices, Interfaces, IPs, Cables, Configs | Username/password |
| [network_collector](#network-collector) | SSH (NAPALM/Netmiko) | Devices, Interfaces, IPs, VLANs, VRFs, Prefixes, Cables, Configs, Inventory | SSH credentials |
| [unifi_collector](#unifi-collector) | UniFi Controller API | Devices, Interfaces (ports + radios), VLANs, WLANs, Cables, Prefixes | Username/password |
| [zabbix_collector](#zabbix-collector) | Zabbix API | Devices, Interfaces, IPs, Custom Fields | Username/password or API token |
| [observium_collector](#observium-collector) | Observium REST API | Devices, Interfaces, IPs | Username/password |
| [cable_exporter](#lldp-cable-exporter) | SSH (LLDP) + NetBox API | Cables | SSH + NetBox API token |

---

## Proxmox VE Collector

**File:** `proxmox_collector.py` (843 lines)

Discovers Proxmox VE clusters including hypervisor nodes, QEMU VMs, LXC containers, network interfaces, IP addresses, and virtual disks. Supports multiple standalone PVE hosts via numbered environment variables.

### Entity Mapping

| Proxmox Object | NetBox Entity | Notes |
|----------------|---------------|-------|
| PVE Cluster | Cluster (type: Proxmox VE) | One cluster per PVE host |
| PVE Node | Device | Model: Proxmox VE Node |
| Node NIC | Interface | Type detected from name (bond, bridge, veth, etc.) |
| Node IP | IPAddress | From network config |
| QEMU VM | VirtualMachine | Status mapped from PVE state |
| LXC Container | VirtualMachine | Status mapped from PVE state |
| VM/LXC NIC | VMInterface | MAC and VLAN extracted from config |
| VM/LXC Disk | InventoryItem | Size parsed from config string |
| Guest Agent IP | IPAddress | Requires qemu-guest-agent running |
| LXC static IP | IPAddress | From LXC network config |

### Key Functions

| Function | Description |
|----------|-------------|
| `get_pve_hosts()` | Builds host list from numbered env vars (PVE_HOST, PVE_HOST_2, etc.) |
| `connect_pve(config)` | Creates ProxmoxAPI connection with token auth |
| `collect_node_info(prox, node)` | Gets node CPU, memory, status |
| `collect_qemu_vms(prox, node)` | Lists all QEMU VMs on a node |
| `collect_vm_config(prox, node, vmid)` | Gets full VM config (NICs, disks, boot order) |
| `collect_vm_guest_agent_ips(prox, node, vmid)` | Gets IPs from QEMU guest agent |
| `collect_lxc_containers(prox, node)` | Lists all LXC containers on a node |
| `parse_pve_net_config(net_str)` | Parses PVE NIC config: `virtio=AA:BB:CC:DD:EE:FF,bridge=vmbr0,tag=100` |
| `parse_lxc_net_config(net_str)` | Parses LXC NIC config: `name=eth0,bridge=vmbr0,ip=10.0.0.5/24` |
| `parse_disk_size(size_str)` | Converts `32G`, `500M`, `1T` to bytes |
| `build_vm_entity(...)` | Builds VirtualMachine entity with platform, cluster, resources |
| `build_vm_disk_entities(...)` | Builds InventoryItem entities for each disk |

### Usage

```bash
python collectors/proxmox_collector.py --dry-run
python collectors/proxmox_collector.py --log-level DEBUG
```

### Environment Variables

```bash
PVE_HOST=192.168.1.190         # First PVE host
PVE_USER=root@pam              # API user
PVE_TOKEN_NAME=diode           # API token name
PVE_TOKEN_VALUE=<token>        # API token value (required)
PVE_VERIFY_SSL=false           # SSL verification
PVE_PORT=8006                  # API port

# Additional hosts use numbered suffixes
PVE_HOST_2=10.40.40.107
PVE_TOKEN_VALUE_2=<token>
# ... up to PVE_HOST_N
```

---

## Proxmox Backup Server Collector

**File:** `pbs_collector.py` (464 lines)

Discovers PBS hosts as devices with their network interfaces, IP addresses, and datastores (modeled as Services in NetBox).

### Entity Mapping

| PBS Object | NetBox Entity | Notes |
|------------|---------------|-------|
| PBS Host | Device | Model: Proxmox Backup Server |
| Network Interface | Interface | Type detected from name |
| Interface IP | IPAddress | With prefix length |
| Datastore | Service (custom) | Size/usage in description |

### Key Functions

| Function | Description |
|----------|-------------|
| `get_pbs_hosts()` | Builds host list from numbered env vars |
| `connect_pbs(config)` | Connects to PBS API with token auth |
| `collect_datastores(pbs)` | Lists all datastores with sizes |
| `collect_datastore_usage(pbs, store)` | Gets datastore disk usage |
| `collect_datastore_snapshots(pbs, store)` | Lists backup snapshots per datastore |
| `build_datastore_entities(...)` | Creates Service entities for datastores |

### Usage

```bash
python collectors/pbs_collector.py --dry-run
```

### Environment Variables

```bash
PBS_HOST_1=10.40.40.150        # First PBS host
PBS_USER_1=diode@pbs
PBS_TOKEN_NAME_1=diode
PBS_TOKEN_VALUE_1=<token>      # Required
PBS_PORT_1=8007                # Default: 8007
# ... up to PBS_HOST_N
```

---

## VMware vSphere Collector

**File:** `vmware_collector.py` (544 lines)

Discovers ESXi hosts and VMs from vCenter or standalone ESXi via the vSphere API (pyVmomi).

### Entity Mapping

| vSphere Object | NetBox Entity | Notes |
|----------------|---------------|-------|
| Cluster | Cluster (type: VMware vSphere) | — |
| ESXi Host | Device | Manufacturer from vendor string |
| Host vNIC | Interface | Speed-based type detection |
| VM | VirtualMachine | Power state mapped to status |
| VM vNIC | VMInterface | MAC address from config |
| VM Disk | InventoryItem | Size from disk backing |
| VM Guest IP | IPAddress | From VMware Tools guest info |

### Key Functions

| Function | Description |
|----------|-------------|
| `connect_vsphere(cfg)` | Connects to vCenter/ESXi via pyVmomi |
| `get_all_objects(si, obj_type, folder)` | Retrieves all managed objects of a type |
| `build_cluster_entities(si, site)` | Creates Cluster entities from vSphere clusters |
| `build_host_entities(si, site)` | Creates Device entities for ESXi hosts + interfaces |
| `build_vm_entities(si, site, host_map)` | Creates VM + interface + disk + IP entities |
| `_mask_to_prefix(mask)` | Converts subnet mask string to prefix length |

### Usage

```bash
python collectors/vmware_collector.py --dry-run
```

### Environment Variables

```bash
VCENTER_HOST=vcenter.local     # vCenter or ESXi IP/hostname
VCENTER_USER=administrator@vsphere.local
VCENTER_PASSWORD=<password>
VCENTER_PORT=443
VCENTER_VERIFY_SSL=false
VCENTER_SITE=main              # NetBox site name
```

---

## Docker Collector

**File:** `docker_collector.py` (358 lines)

Discovers Docker containers as VirtualMachines grouped into Clusters (one per Docker host). Supports local Docker socket and remote TCP connections.

### Entity Mapping

| Docker Object | NetBox Entity | Notes |
|---------------|---------------|-------|
| Docker Host | Cluster (type: Docker) | Host name from `docker info` |
| Container | VirtualMachine | Status mapped (running→active, exited→offline) |
| Container Network | VMInterface | MAC address, one per Docker network |
| Container IPv4 | IPAddress | With prefix length from Docker |
| Container IPv6 | IPAddress | Global IPv6 if assigned |

### Custom Fields

- `docker_container_id` — Container short ID
- `docker_compose_project` — Compose project name (from labels)

### Key Functions

| Function | Description |
|----------|-------------|
| `connect_docker(host, tls)` | Connects to Docker (local socket or remote TCP) |
| `get_host_info(client)` | Gets Docker host system info |
| `get_containers(client, all)` | Lists containers (running or all) |
| `build_cluster_entity(host, site)` | Creates Cluster entity for Docker host |
| `build_container_entities(container, host, site)` | Creates VM + interfaces + IPs per container |

### Usage

```bash
# Local Docker (no config needed)
python collectors/docker_collector.py --dry-run

# Include stopped containers
python collectors/docker_collector.py --dry-run --all
```

### Environment Variables

```bash
DOCKER_HOSTS=tcp://10.0.0.5:2375,tcp://10.0.0.6:2375   # Comma-separated (default: local socket)
DOCKER_SITE=main
DOCKER_TLS_VERIFY=false
```

---

## Cisco CML Collector

**File:** `cml_collector.py` (456 lines)

Syncs Cisco Modeling Labs topology into NetBox: lab nodes become devices, interfaces are mapped with types inferred from names, links become cables, and L3 addresses and device configs are captured.

### Entity Mapping

| CML Object | NetBox Entity | Notes |
|------------|---------------|-------|
| Lab Node | Device | Model/role/platform from node definition |
| Node Interface | Interface | Type inferred from name (Gi→1000base-t, etc.) |
| Node L3 Address | IPAddress | From CML's discovered IPv4/IPv6 |
| Lab Link | Cable | Between interface endpoints |
| Node Config | DeviceConfig | Startup configuration text |

### Node Definition Mappings

The collector maps CML node definitions to NetBox device attributes:

| Node Definition | Platform | Role | Manufacturer |
|----------------|----------|------|--------------|
| `iosv` | Cisco IOS | Router | Cisco |
| `iosvl2` | Cisco IOS | Switch | Cisco |
| `iosxrv` / `iosxrv9000` | Cisco IOS-XR | Router | Cisco |
| `csr1000v` / `cat8000v` | Cisco IOS-XE | Router | Cisco |
| `cat9000v` | Cisco IOS-XE | Switch | Cisco |
| `nxosv` / `nxosv9000` | Cisco NX-OS | Switch | Cisco |
| `asav` | Cisco ASA | Firewall | Cisco |
| `server` / `ubuntu` / `alpine` | Linux / Ubuntu / Alpine | Server | Generic |

### Key Functions

| Function | Description |
|----------|-------------|
| `build_node_entity(node, site)` | Creates Device entity from CML node with mapped type/role/platform |
| `build_interface_entities(node, site)` | Creates Interface entities, type inferred from name regex |
| `build_ip_entities(node, site)` | Creates IPAddress entities from CML L3 discovery |
| `build_cable_entities(lab, site, node_map)` | Creates Cable entities from CML links |
| `build_config_entity(node, site)` | Captures node startup config as DeviceConfig |

### Usage

```bash
python collectors/cml_collector.py --dry-run
python collectors/cml_collector.py                     # All labs
CML_LAB="my-lab" python collectors/cml_collector.py    # Specific lab
```

### Environment Variables

```bash
CML_HOST=10.40.40.50           # CML controller address
CML_USER=admin
CML_PASSWORD=<password>
CML_LAB=                       # Optional: specific lab name or ID
CML_VERIFY_SSL=false
CML_SITE=CML                   # NetBox site for CML devices
```

---

## Network Collector

**File:** `network_collector.py` (1930 lines)

The most comprehensive collector — discovers Cisco and Brocade network devices via SSH using NAPALM (with Netmiko fallback for unsupported platforms). Collects device facts, interfaces with speeds and types, IP addresses, LLDP neighbors, VLANs, VRFs, prefixes, device configs, and hardware inventory. Optionally uses pyATS/Genie for CDP and IGP neighbor data, and pushes BGP sessions to the netbox-bgp plugin API.

### Supported Platforms

| Platform | Driver | Connection Method | Notes |
|----------|--------|-------------------|-------|
| Cisco IOS | `ios` | NAPALM | Full feature support |
| Cisco IOS-XE | `ios` | NAPALM | Same as IOS |
| Cisco IOS-XR | `iosxr` | Netmiko (`cisco_xr`) | Custom parsers for facts, interfaces, IPs, LLDP |
| Cisco NX-OS | `nxos` / `nxos_ssh` | NAPALM | — |
| Arista EOS | `eos` | NAPALM | — |
| Juniper Junos | `junos` | NAPALM | — |
| Brocade FastIron (ICX) | `ruckus_fastiron` | Netmiko | Custom parsers |
| Brocade VDX/NOS | `nos` / `brocade_nos` | Netmiko (`extreme_nos`) | Custom parsers |

### Entity Mapping

| Network Object | NetBox Entity | Notes |
|----------------|---------------|-------|
| Device | Device | With model, serial, platform, role |
| Physical Interface | Interface | Type from name or speed |
| Loopback/VLAN/Tunnel | Interface (virtual) | — |
| Port-channel/LAG | Interface (lag) | — |
| Bundle-Ether | Interface (lag) | IOS-XR LAG interfaces |
| Interface IP | IPAddress | With prefix length |
| LLDP Neighbor | Cable | Deduplicated bidirectional |
| VLAN | VLAN + VLANGroup | Per-device VLAN group |
| VRF | VRF | From NAPALM get_network_instances |
| IP Subnet | Prefix | Computed from IP + mask |
| Running Config | DeviceConfig | Full running-config text |
| Hardware Module | InventoryItem | From NAPALM environment data |
| BGP Neighbor | Plugin API push | To netbox-bgp (if installed) |

### Interface Type Detection

Interface types are determined by a two-pass system:

1. **Name-based** (`NAME_TO_TYPE`): Regex patterns match interface names
   - `GigabitEthernet` / `Gi` → `1000base-t`
   - `TenGigabitEthernet` / `Te` → `10gbase-x-sfpp`
   - `FortyGig` / `Fo` → `40gbase-x-qsfpp`
   - `HundredGig` / `Hu` → `100gbase-x-qsfp28`
   - `Loopback` / `Vlan` / `Tunnel` → `virtual`
   - `Port-channel` / `Bundle-Ether` → `lag`
   - `e 1/1/1` (Brocade) → `1000base-t`

2. **Speed-based** (`SPEED_TO_TYPE`): Falls back to reported link speed
   - 10 Mbps → `10base-t`, 100 → `100base-tx`, 1000 → `1000base-t`, etc.

### IOS-XR Support (Netmiko)

IOS-XR devices use Netmiko instead of NAPALM due to better reliability. Custom parsers handle IOS-XR-specific output formats:

| Function | Description |
|----------|-------------|
| `_netmiko_parse_facts_iosxr(conn, facts)` | Parses `show version`, `show inventory`, `show running hostname` — strips `RP/0/RP0/CPU0:` prefix from hostname, extracts model (cleaned), serial (prefers "Rack 0" chassis) |
| `_netmiko_parse_interfaces_iosxr(conn, driver)` | Parses `show interfaces` for description, MAC, MTU, bandwidth (Kbit→Mbps conversion) |
| `_netmiko_parse_interfaces_ip_iosxr(conn, driver, ifaces)` | Parses `show ipv4 interface brief` + `show running-config interface` for IP addresses with correct prefix lengths; also collects IPv6 |

### Brocade Support (Netmiko)

| Function | Description |
|----------|-------------|
| `_netmiko_parse_facts(conn, driver)` | Parses `show version` for model/serial, dispatches to IOS-XR if needed |
| `_netmiko_parse_interfaces(conn, driver)` | Parses `show interface brief` / `show interface` for port status |
| `_netmiko_parse_interfaces_ip(conn, driver, ifaces)` | Parses `show ip interface` / `show running-config` for IPs |
| `_netmiko_parse_lldp(conn, driver)` | Parses `show lldp neighbors detail` for LLDP neighbor data |

### Other Key Functions

| Function | Description |
|----------|-------------|
| `load_inventory(path)` | Loads `inventory.yaml` with defaults, per-device overrides |
| `merge_device_config(entry, defaults)` | Merges per-device config with inventory defaults |
| `normalize_interface_name(name)` | Expands abbreviations: `Gi0/1` → `GigabitEthernet0/1` |
| `map_interface_type(name, speed)` | Returns NetBox interface type slug |
| `connect_device(host, driver, ...)` | Creates NAPALM connection |
| `connect_netmiko(host, driver, ...)` | Creates Netmiko connection |
| `collect_napalm_data(dev)` | Collects all data via NAPALM getters |
| `collect_netmiko_data(conn, driver)` | Collects all data via Netmiko parsers |
| `collect_pyats_data(...)` | Optional pyATS/Genie for CDP, OSPF, IS-IS |
| `build_device_entity(...)` | Creates Device entity with full attributes |
| `build_interface_entities(...)` | Creates Interface entities with types |
| `build_ip_entities(...)` | Creates IPAddress + Prefix entities |
| `build_vlan_entities(...)` | Creates VLAN + VLANGroup entities |
| `build_cable_entities(...)` | Creates Cable entities from LLDP (bidirectional dedup) |
| `build_config_entity(...)` | Creates DeviceConfig entity |
| `build_inventory_entities(...)` | Creates InventoryItem entities from hardware modules |
| `push_bgp_sessions(...)` | Pushes BGP data to netbox-bgp plugin REST API |

### Usage

```bash
python collectors/network_collector.py -i collectors/inventory.yaml --dry-run
python collectors/network_collector.py -i collectors/inventory.yaml
python collectors/network_collector.py -i collectors/inventory.yaml --no-pyats --log-level DEBUG
```

### Inventory File Format

Credentials are stored in `collectors/inventory.yaml` (gitignored), not in `.env`:

```yaml
defaults:
  driver: ios
  username: admin
  password: secret
  secret: enable_secret    # Enable password (optional)
  timeout: 60

devices:
  - host: 10.0.0.1
    driver: ios
    # Uses defaults for username/password

  - host: 10.0.0.2
    driver: iosxr
    username: cisco        # Override default credentials

  - host: 10.0.0.3
    driver: ruckus_fastiron
    username: admin
    password: brocade_pass
```

---

## UniFi Collector

**File:** `unifi_collector.py` (754 lines)

Discovers Ubiquiti UniFi infrastructure including UDM/UDM-SE gateways, managed switches, access points, switch ports with PoE and SFP detection, WiFi radios, VLANs, WLANs, and LLDP-based cabling.

### Entity Mapping

| UniFi Object | NetBox Entity | Notes |
|-------------|---------------|-------|
| UDM/Switch/AP | Device | Model from `model` field, serial from `serial` |
| Switch Port | Interface | Speed/PoE/SFP detection |
| WiFi Radio | Interface (wireless) | Band-specific type (802.11n/ac/ax) |
| Network/VLAN | VLAN + Prefix | From UniFi network config |
| WLAN (SSID) | WirelessLAN | Auth type from security settings |
| LLDP Neighbor | Cable | Deduplicated bidirectional |

### Key Functions

| Function | Description |
|----------|-------------|
| `connect_unifi(cfg)` | Connects to UniFi Controller API (UDM or legacy) |
| `_build_device_entities(dev, site)` | Creates Device entity with model/serial/firmware |
| `_build_port_entities(dev, site)` | Creates port Interface entities with speed/PoE/SFP detection |
| `_build_radio_entities(dev, site)` | Creates WiFi radio Interface entities with band/channel |
| `_build_cable_entities(devices, site)` | Creates Cable entities from LLDP uplink data |
| `_build_network_entities(networks, site)` | Creates VLAN + Prefix from UniFi networks |
| `_build_wlan_entities(wlans, site)` | Creates WirelessLAN entities with auth type |

### Usage

```bash
python collectors/unifi_collector.py --dry-run
```

### Environment Variables

```bash
UNIFI_HOST=192.168.1.1         # UDM-SE or Controller IP
UNIFI_USER=admin
UNIFI_PASSWORD=<password>
UNIFI_SITE=default             # UniFi site name
UNIFI_VERIFY_SSL=false
UNIFI_IS_UDM=true              # true for UDM/UDM-SE, false for legacy controller
UNIFI_NETBOX_SITE=main         # NetBox site name
```

---

## Zabbix Collector

**File:** `zabbix_collector.py` (372 lines)

Imports device inventory from Zabbix monitoring into NetBox. Automatically infers platform and device role from Zabbix host groups, templates, and inventory fields. Adds a `zabbix_host_id` custom field for cross-referencing.

### Entity Mapping

| Zabbix Object | NetBox Entity | Notes |
|---------------|---------------|-------|
| Host | Device | Status: monitored→active, unmonitored→offline |
| Host Interface | Interface | Named by type: mgmt0, snmp0, ipmi0 |
| Interface IP | IPAddress | /32 prefix (Zabbix doesn't provide prefix length) |

### Intelligent Mapping

- **Platform detection** (`guess_platform`): Checks inventory `os_full`/`os_short` fields and template names against keyword list (linux, windows, cisco, juniper, vmware, etc.)
- **Role detection** (`guess_role`): Checks host group names for keywords (router, switch, firewall, hypervisor, server)
- **Custom fields**: `zabbix_host_id` stored for cross-referencing back to Zabbix

### Key Functions

| Function | Description |
|----------|-------------|
| `connect_zabbix(cfg)` | Connects to Zabbix API (user/pass or API token) |
| `collect_hosts(zapi)` | Fetches all hosts with interfaces, inventory, groups, templates |
| `guess_platform(host_data)` | Infers platform from inventory OS and template names |
| `guess_role(host_data, default)` | Infers device role from host group names |
| `build_host_entities(host_data, cfg)` | Creates Device + Interface + IP entities |

### Usage

```bash
python collectors/zabbix_collector.py --dry-run
```

### Environment Variables

```bash
ZABBIX_URL=http://10.40.40.20/api_jsonrpc.php
ZABBIX_USER=Admin
ZABBIX_PASSWORD=<password>
# OR use API token (Zabbix 5.4+):
ZABBIX_API_TOKEN=<token>
ZABBIX_SITE=main
ZABBIX_DEFAULT_ROLE=Server
```

---

## Observium Collector

**File:** `observium_collector.py` (378 lines)

Imports device, port, and IP data from the Observium REST API. Requires Observium Professional or Enterprise edition (Community Edition has no REST API).

### Entity Mapping

| Observium Object | NetBox Entity | Notes |
|-----------------|---------------|-------|
| Device | Device | Platform/role inferred from OS type |
| Port | Interface | Type from SNMP ifType |
| Port IP | IPAddress | With prefix length |

### Key Functions

| Function | Description |
|----------|-------------|
| `api_get(base_url, endpoint, auth, params)` | Makes authenticated GET request to Observium API |
| `collect_all_entities(cfg)` | Fetches devices → ports → IPs, builds all entities |

### Usage

```bash
python collectors/observium_collector.py --dry-run
```

### Environment Variables

```bash
OBSERVIUM_URL=http://10.40.40.30/api/v0
OBSERVIUM_USER=admin
OBSERVIUM_PASSWORD=<password>
OBSERVIUM_SITE=main
OBSERVIUM_DEFAULT_ROLE=Network Device
```

---

## LLDP Cable Exporter

**File:** `cable_exporter.py` (576 lines)

A standalone tool that creates network cables in NetBox by collecting LLDP neighbor data from devices via SSH, validating both endpoints against the NetBox inventory, and creating cables via the NetBox REST API. This bypasses the Diode pipeline because cable entities are not yet supported by the Diode reconciler.

### Why REST API Instead of Diode?

The Diode reconciler returns "entity is nil" errors for cable entities ([diode#191](https://github.com/netboxlabs/diode/issues/191)). The cable exporter uses the NetBox REST API directly, which provides:

- **Endpoint validation** — both devices and interfaces must exist in NetBox
- **Duplicate detection** — checks existing cables before creating
- **Precise matching** — uses interface IDs, not names
- **Per-cable error handling** — one failure doesn't stop the batch

### How It Works

```
1. Build Cache     Query NetBox API for devices, interfaces, existing cables
                   at the target site → in-memory lookup tables

2. Collect LLDP    SSH to each device in inventory.yaml
                   NAPALM get_lldp_neighbors_detail() or Netmiko fallback

3. Match & Validate  For each LLDP pair:
                     - Normalize interface names
                     - Look up both devices by hostname in NetBox
                     - Look up both interfaces by (device_id, name)
                     - Skip LAG/virtual endpoints (NetBox rejects them)
                     - Skip if cable already exists
                     - Deduplicate bidirectional links (A→B = B→A)

4. Create Cables   POST /api/dcim/cables/ for each validated pair
                   Also generates CSV audit trail
```

### Interface Name Matching

LLDP reports interface names that may differ from what's stored in NetBox. The exporter uses fuzzy matching:

| LLDP Reports | NetBox Has | Match Strategy |
|-------------|-----------|----------------|
| `GigabitEthernet0/0/0/1` | `GigabitEthernet0/0/0/1` | Exact match |
| `FortyGigabitEthernet1/0/1` | `FortyGigabitEthernet 1/0/1` | Space variation |
| `TenGigabitEthernet1/0/34:1` | `Te 1/0/34:1` | Abbreviated form |
| `Gi0/1` | `GigabitEthernet0/1` | Long form expansion |

Supported abbreviation mappings: `Gi`↔`GigabitEthernet`, `Te`↔`TenGigabitEthernet`, `Fa`↔`FastEthernet`, `Fo`↔`FortyGigabitEthernet`, `Hu`↔`HundredGigabitEthernet`, `BE`↔`Bundle-Ether`, `Eth`↔`Ethernet`, `Mg`↔`MgmtEth`.

### Key Classes and Functions

| Component | Description |
|-----------|-------------|
| `NetBoxClient` | REST API wrapper with pagination, auth, GET/POST methods |
| `NetBoxClient.get_devices(site)` | Fetches all devices at a site |
| `NetBoxClient.get_interfaces(device_id)` | Fetches interfaces for a device |
| `NetBoxClient.get_cables()` | Fetches existing cables for duplicate detection |
| `NetBoxClient.ensure_tag(name, slug)` | Creates tag if it doesn't exist (idempotent) |
| `NetBoxClient.create_cable(a_id, b_id, tag_id)` | POSTs a cable with terminations |
| `build_netbox_cache(nb, site)` | Builds device/interface/cable lookup dicts |
| `collect_lldp_from_device(cfg)` | SSH to one device, returns LLDP data |
| `collect_all_lldp(inventory)` | Collects LLDP from all inventory devices |
| `lookup_interface(cache, dev_id, name)` | Fuzzy interface name matching |
| `build_cable_list(lldp, cache, nb)` | Validates and deduplicates LLDP → cable list |
| `write_csv(cables, path, site)` | Writes CSV audit trail |
| `create_cables_via_api(nb, cables, tag_id)` | Creates cables via REST API |

### Usage

```bash
# Dry run — show what cables would be created
python collectors/cable_exporter.py -i collectors/inventory.yaml --dry-run

# Create cables in NetBox
python collectors/cable_exporter.py -i collectors/inventory.yaml --env-file ../.env

# CSV only (no API calls)
python collectors/cable_exporter.py -i collectors/inventory.yaml --csv-only

# Target a specific site
python collectors/cable_exporter.py -i collectors/inventory.yaml --site CML
```

### Environment Variables

```bash
NETBOX_API_URL=http://172.19.77.160:8000
NETBOX_API_TOKEN=nbt_<key>.<token>
```

### Output

```
NetBox inventory: 45 devices, 1200 interfaces loaded
LLDP collection: 18/19 devices connected, 62 neighbor pairs found
Validation: 48 cables matched (14 skipped: 6 device not found, 5 interface not found, 3 LAG endpoint)
Existing cables: 0 (0 duplicates skipped)
Created: 48 cables
CSV written to: cables_export.csv
```

---

## Common Patterns

### Diode Ingestion Flow

All Diode-based collectors follow the same pattern:

```python
# 1. Build Entity objects
entities = []
entities.append(Entity(device=Device(name="router1", ...)))
entities.append(Entity(interface=Interface(device=dev_ref, name="Gi0/1", ...)))
entities.append(Entity(ip_address=IPAddress(address="10.0.0.1/24", ...)))

# 2. Ingest via Diode SDK
with DiodeClient(target=target, client_id=id, client_secret=secret, ...) as client:
    resp = client.ingest(entities=entities)
```

### Shared Environment Variables (Diode)

All Diode-based collectors use:

```bash
DIODE_TARGET=grpc://localhost:8080/diode    # Diode ingester endpoint
DIODE_CLIENT_ID=diode-ingester              # Or INGESTER_CLIENT_ID
DIODE_CLIENT_SECRET=<secret>                # Or INGESTER_CLIENT_SECRET (required)
```

### Dry Run Mode

Every collector supports `--dry-run` which prints entities to stdout without making any changes. Always test with `--dry-run` first.

### Logging

All collectors support `--log-level` (DEBUG, INFO, WARNING, ERROR). Use `DEBUG` for troubleshooting connection issues.

---

## Dependencies

### Python Packages

```
netboxlabs-diode-sdk          # All Diode-based collectors
napalm                         # network_collector, cable_exporter
netmiko                        # network_collector, cable_exporter (Netmiko fallback)
pyyaml                         # network_collector, cable_exporter (inventory.yaml)
requests                       # cable_exporter, network_collector (BGP plugin)
python-dotenv                  # cable_exporter
proxmoxer                      # proxmox_collector, pbs_collector
pyvmomi                        # vmware_collector
docker                         # docker_collector
pyzabbix                       # zabbix_collector
virl2_client                   # cml_collector
pyats + genie                  # network_collector (optional — CDP, OSPF, IS-IS)
```

### Install

```bash
pip install -r requirements.txt
```

---

## File Reference

| File | Lines | Description |
|------|-------|-------------|
| `network_collector.py` | 1930 | Multi-vendor network device collector (NAPALM/Netmiko) |
| `proxmox_collector.py` | 843 | Proxmox VE hypervisor + VM/LXC collector |
| `unifi_collector.py` | 754 | UniFi controller device/port/radio collector |
| `cable_exporter.py` | 576 | LLDP cable creator via NetBox REST API |
| `vmware_collector.py` | 544 | VMware vSphere host/VM collector |
| `pbs_collector.py` | 464 | Proxmox Backup Server collector |
| `cml_collector.py` | 456 | Cisco CML lab topology collector |
| `observium_collector.py` | 378 | Observium NMS device/port collector |
| `zabbix_collector.py` | 372 | Zabbix monitoring device collector |
| `docker_collector.py` | 358 | Docker container collector |
| `ENV_REFERENCE.md` | 245 | Environment variable reference and setup guide |