Bug detection across the observation surface

l2trace was built to answer “where is MAC X at time T?” The byproduct of collecting enough data to answer that question turned out to be enough data to answer a different question: “is anything in this fabric currently inconsistent with itself?” The eight detectors documented below are the result. Each one is a few-line SQL query that runs against the bitemporal observation tables and surfaces rows that violate a structural L2 or L3 invariant.

The matrix

The detectors are organized along two axes: which observation surface they query, and what kind of violation they look for.

Detector	Surface	Mode	CLI
audit_adjacencies	adjacency	single-source, bidirectional check	audit-adjacencies
disagreements	mac_observation	cross-source spatial	(TUI / SQL view)
detect_mac_flaps	mac_observation	single-source temporal	audit-flaps
detect_multi_vlan_macs	mac_observation	cross-VLAN spatial	audit-multi-vlan
detect_stp_cam_inconsistency	stp_state × mac_observation	cross-layer (control × data)	audit-stp-cam
detect_stp_root_disagreement	stp_state.root_id	single-source, cross-device	audit-stp-roots
detect_arp_ip_collision	arp_observation	cross-device, L3	audit-arp-collision
detect_port_state_cam_inconsistency	port_state × mac_observation	cross-layer (PHY × data)	audit-port-state-cam

Together they touch every observation table the schema exposes — mac_observation, adjacency, stp_state, arp_observation, port_state — across single-source, cross-source, temporal, cross-VLAN, cross-table, and cross-layer modes.

The shared invariant: upper_inf(recorded_during)

Every detector filters on currently-believed rows (upper_inf(recorded_during)). That predicate is doing more work than it looks like. F31’s TCN-driven belief revision closes the recorded_during.upper of every open mac_observation on a device when the bridge reports a spanning-tree topology change. Without the filter, every TCN would spuriously fire detectors that look at retracted rows (“here’s a flap! we saw the MAC on N ports inside the window!”). With it, the detectors and F31’s retraction compose cleanly by construction: retracted state is invisible to detection.

The same predicate also makes the detectors safe under multi-source ingest. A row whose recorded_during is closed represents a former belief — typically because a higher-priority source’s observation superseded it. Detection only sees the current, resolved view.

The detectors, by category

audit_adjacencies — bidirectional LLDP

Every healthy LLDP adjacency is reported from both ends: switch A says “I see B on Eth1,” and B says “I see A on Eth1.” Asymmetry means one side isn’t observing the other — one-way cable (single fiber strand broken), LLDP-TX disabled by config on one side, vendor LLDP-MIB bug, or simply collector observability gap on one of the two sides.

Literature: Lopes 2015 §3 (Reachability Consistency), Anteater 2011 §5.1 (asymmetric route inconsistencies).

disagreements view — cross-source CAM mismatch

l2trace ingests from multiple collectors (gNMI, SNMP, SSH, …) into the same bitemporal log. When two sources see the same (mac, device, vlan) triple but report different port_ids, the disagreement view surfaces it. Real cause: stale SNMP poll racing streaming gNMI, or cache-coherence breaks across collectors.

Literature: Lowekamp 2001 §4 cross-validation.

detect_mac_flaps — single-source temporal

A host MAC oscillating between ports faster than the switch’s aging timer is the signature of a switching loop, a misconfigured server bond, or active-active VRRP gone split-brain. The sliding-window detector counts distinct ports per (mac, device, vlan) over a configurable interval.

Defaults match a conservative production rate: 300 seconds (one Cisco aging cycle) and 3+ distinct ports. Tighter for forensics, looser for low-churn environments.

Literature: Breitbart 2004 §V — switching-loop signature.

detect_multi_vlan_macs — cross-VLAN

A host MAC observed in multiple VLANs simultaneously indicates 802.1Q double-tagging (a VLAN-hopping attack vector), accidental L2 segment merging across VLAN-trunked switches, or a misconfigured trunk-port allowed-VLAN list. The output’s device_count field is the operator’s triage hint: 1 device + N vlans is usually a dot1q sub-interface (legitimate); N devices is almost always a bug.

detect_stp_cam_inconsistency — cross-layer (control × data)

802.1D §8.6 says ports in STP BLOCKING state cannot learn MACs — FDB learning is disabled on blocked ports. So the intersection of “currently blocking” and “currently has CAM entry” must be empty in a healthy fabric. When non-empty: transient STP flap during detection, a vendor bug where learning isn’t gated by STP state, or stale CAM the compactor hasn’t aged.

detect_stp_root_disagreement — STP root-bridge election

802.1D root-bridge election converges every device on a connected segment to the same root_id per VLAN. Multiple distinct root_ids mean the segment is partitioned (each half elects its own root), BPDUs are filtered somewhere (ACL or QoS misconfig), or PVST/RSTP/MSTP modes disagree across vendor peers.

The detector skips rows with NULL root_id deliberately — many collectors leave it unset, and NULL means “we don’t know” rather than “we disagree.” Treating NULL as a value would invent false positives.

detect_arp_ip_collision — L3 cross-table

The first detector to cross the L2/L3 boundary. An IP inside one VRF should map to exactly one MAC at any moment. Two routers ARP-resolving the same IP to different MACs is the classic IP-conflict signature: DHCP lease race, static-route typo, L2 partition with duplicate hosts on each half, or active ARP-spoofing (man-in-the-middle).

The output’s device_count triages: 1 router + 2 macs is likely a single-router multi-source race; 2+ routers + 2+ macs is almost always a real network conflict.

detect_port_state_cam_inconsistency — cross-layer (PHY × data)

A port marked admin='down' or oper IN ('down', 'lower-layer-down') drops frames at the PHY layer — no frames means no MAC learning. So an open CAM entry on a down port is impossible in a healthy fabric. Real cause: stale CAM the compactor hasn’t aged, vendor FDB-learning not gated by port state, or a port-state collector lagging behind the CAM collector.

oper='testing' is deliberately NOT treated as down — some vendors keep learning behavior intact during diagnostics, so flagging would be noisy.

F19: per-category coverage measurement

How do we know each detector actually catches what it claims to? The F19 simulation harness in tests/sim/bugs/ runs each detector against a parameterized random topology that has been deliberately mutated to embed one of the eight bug categories. The injector returns ground truth; the runner drives every detector; the test asserts that the correct detector fires.

The result is a per-category coverage table:

category	trials	detected	rate
lldp_asymmetry	3	3	100%
cam_disagreement	3	3	100%
mac_flap_historical	3	3	100%
multi_vlan_mac	3	3	100%
stp_cam_inconsistency	3	3	100%
stp_root_disagreement	3	3	100%
arp_ip_collision	3	3	100%
port_state_cam	3	3	100%

The CI-scale numbers run on every commit. The paper-scale sweep (higher trial count, more parameter cells) lives in tests/sim/__main__.py.

Inspired by Anteater 2011 §5.2’s coverage methodology — they measured 86% against 78 real Quagga bugs. Public L2-bug corpora at that scale don’t exist for the bitemporal-CAM domain, so the corpus here is synthetic but the rigor is the same shape.

See also

• Audit LLDP adjacencies — operator playbook for the bidirectional check
• Spot disagreements — cross-source CAM mismatch walkthrough
• Bitemporality — why recorded_during matters and how F31 retraction composes through every detector
• CLI reference — full flag lists for every audit command