CLI commands

l2trace ships one binary, l2trace, with several subcommands. The Makefile wraps the most common ones behind targets so you don’t usually invoke it directly — but here’s the full surface.

All commands run inside the reconciler container by default:

docker compose run --rm --no-deps reconciler l2trace <subcommand> ...

l2trace migrate

Apply alembic migrations to head.

docker compose run --rm migrate alembic upgrade head
# or, equivalently:
make migrate

The Make target uses the migrate compose service which is configured for one-shot alembic runs. Direct invocation of l2trace migrate works too but goes through the longer-running reconciler image.

l2trace reconcile

Start the reconciler and the compactor and the collector orchestrator with a shared in-memory LiveSet. Long-running. The default make up starts this as a daemon in the reconciler container.

l2trace reconcile

Co-running is not optional: the compactor must invalidate LiveSet entries when it closes valid_during ranges. Splitting them across two processes would silently corrupt the bitemporal log. See the compactor invariant for why.

The orchestrator picks up rows from the device_collector table on boot and on a ~30s reconfig pass — so anything added/disabled via l2trace device … takes effect without a process restart.

l2trace device …

Register and manage per-device collector configurations. The orchestrator inside l2trace reconcile reads this table and (re)spawns gNMI/SNMP/SSH workers to match. Five subcommands:

device add

l2trace device add --hostname HOST --mgmt-ip IP --source SRC \
  [--community COMMUNITY | --community-uri URI] [--snmp-port 161] \
  [--username USER | --username-uri URI] \
  [--password PW | --password-uri URI] \
  [--enable-secret PW | --enable-secret-uri URI] [--ssh-port 22] \
  [--poll-interval 60] [--vendor VENDOR]

Flag	Default	Description
--hostname	(required)	Switch hostname — unique per install
--mgmt-ip	(required)	Management IP the collector connects to
--source	snmp	One of gnmi, snmp, ssh, netconf
--community / --community-uri	—	SNMPv2c community as cleartext, or as a secrets URI (env://VAR, file:///path#/ptr). Required for --source=snmp
--snmp-port	161	UDP port for SNMP collection
--username / --username-uri	—	SSH username (cleartext or URI). Required for --source=ssh
--password / --password-uri	—	SSH password (cleartext or URI). One of --password* OR --ssh-key-file is required for --source=ssh
--ssh-key-file	—	Absolute path to private key file (alternative to password). Typically a Docker secret mount like /run/secrets/ssh/sw-edge-7-key
--ssh-passphrase / --ssh-passphrase-uri	—	Passphrase for an encrypted private key (cleartext or URI)
--enable-secret / --enable-secret-uri	—	Optional privileged-mode password for IOS-XE
--ssh-port	22	TCP port for SSH collection
--poll-interval	60.0	Seconds between polls (FDB + LLDP + chassis-id)
--vendor	—	Free-text vendor hint. Required for SSH — picks the napalm driver: cisco-ios-xe / cisco-ios / cisco-nxos

URI-form credential flags are preferred for production. The URI form stores a reference to the secret (env var name, JSON pointer into a mounted file, etc.) instead of the value itself — so pg_dump of the database doesn’t expose credentials. See Manage collector credentials for the full URI scheme catalog. Passing both forms for the same field (e.g. --password and --password-uri) is a user error.

Idempotent. Re-running with the same --hostname updates fields in place; the orchestrator picks up the change on its next reconfig pass. You can register the same device twice with different --source values to get redundant collection (e.g. gNMI primary + SNMP backup, or SNMP primary + SSH fallback).

For SSH specifically — see How to add an SSH-only switch for the operator walkthrough including a working IOS-XE example.

device list

l2trace device list

Renders a Rich table showing every registered device, its enabled sources, last successful poll, and the most recent collector error. Quick health check before opening a screenshot session.

device disable / device enable

l2trace device disable --hostname HOST --source SRC
l2trace device enable  --hostname HOST --source SRC

Toggles the enabled flag on the (device, source) collector row. Keeps history intact — re-enabling is one command. The orchestrator gracefully stops the worker on the next reconfig pass.

device remove

l2trace device remove --hostname HOST --yes

DELETEs the device + cascade-deletes its entire observation history. Destructive. The --yes flag is required to skip the confirmation prompt. Most operators want device disable instead — that stops collection but preserves the bitemporal log for forensic queries.

l2trace trace

Run a one-shot L2 traceroute from the CLI.

l2trace trace --src AA:BB:CC:11:22:33 --dst AA:BB:CC:44:55:66 --vlan 10 \
  [--as-of TIMESTAMP|now] [--audit-at TIMESTAMP] [--max-hops 32]

Flag	Default	Description
--src	(required)	Source MAC, any common format
--dst	(required)	Destination MAC
--vlan	(required)	VLAN ID, 1..4094
--as-of	now	Wire-time anchor: “what was the path at T?”
--audit-at	current belief	Belief-time anchor: “what did we believe at T?”
--max-hops	32	Loop guard — abort after N hops

The Make wrapper is:

make trace SRC=... DST=... VLAN=... [AS_OF=...]

(AUDIT_AT is not in the Make wrapper — pass it via the docker compose invocation directly.)

l2trace trace-diff

Run the L2 traceroute query at two as_of timestamps and surface a structured per-hop diff.

l2trace trace-diff --src MAC --dst MAC --vlan N \
  --t1 ISO-8601 --t2 ISO-8601 [--audit-at ISO-8601] [--max-hops 32]

Flag	Default	Description
--src	(required)	Source MAC, any common format
--dst	(required)	Destination MAC
--vlan	(required)	VLAN ID, 1..4094
--t1	(required)	First as-of (the “before” snapshot)
--t2	(required)	Second as-of (the “after” snapshot)
--audit-at	current belief	Belief-time; applied to BOTH snapshots
--max-hops	32	Loop guard, same as trace

Status glyphs per hop: ✓ unchanged, Δ changed, + added, - removed. Summary line reports the total change count and any termination change (e.g. reached → flood).

See How to diff a traceroute between two times for operator workflow + sample output.

l2trace mlag …

Declare MLAG / vPC / MC-LAG peer groups so the traceroute walker collapses paired switches into one logical hop and the adjacency audit doesn’t drown in peer-link rows. See Configure MLAG groups for the workflow and How MLAG-collapsed traceroute works for the design.

Subcommand	Purpose
mlag create --hosts A,B[,C...]	Assign all listed hosts to one MLAG group. Auto-picks the group ID. Idempotent.
mlag list	Rich table of every declared group + its hostnames
mlag dissolve --hosts A,B	Reset mlag_group_id = NULL on the named hosts

Conflict semantics: trying to merge hosts that already belong to different groups errors loudly (would otherwise create quad-groups operators rarely actually want).

Takes effect immediately — no reconciler restart needed; the next trace query and AUDIT refresh use the new grouping.

l2trace audit-adjacencies

Audit every open LLDP adjacency for one-way / asymmetric observations. For each A → B row, checks whether B → A exists from the same source and from any source.

l2trace audit-adjacencies [--source SRC] [--show-healthy] [--include-peer-links]
# or
make audit-adjacencies

Flag	Default	Effect
--source	—	Limit audit to one telemetry source (gnmi, snmp, …)
--show-healthy	off	Include bidirectionally-confirmed rows in output
--include-peer-links	off	Include MLAG peer-link adjacencies in the audit. By default they’re hidden — peer-links are bidirectional by design and would always be ✓ healthy, cluttering the view. See Configure MLAG groups.

Output is a Rich table with one row per open adjacency:

Symbol	Meaning
✗ no reverse	No peer observation back from any source — likely one-way cable, LLDP disabled on peer, or port down
△ source asym	Link is bidirectional, but the same source only sees one direction (telemetry asymmetry, not a cable problem)
✓	Both directions confirmed by the same source

A green “all open adjacencies are bidirectional” line means nothing to investigate. Empty table with --show-healthy off (the default) is also the all-clear signal. See Audit LLDP adjacencies for the full operator playbook.

l2trace audit-flaps

Sliding-window count of distinct ports per (mac, device, vlan). A MAC oscillating between ports faster than the switch’s aging timer is the signature of a switching loop, a misconfigured server bond, or active-active VRRP gone split-brain.

l2trace audit-flaps [--window 300] [--min-moves 3]

Flag	Default	Effect
--window	300	Sliding-window size in seconds. 300s matches one Cisco aging cycle.
--min-moves	3	Minimum distinct ports inside the window to flag (≥ 2)

Defaults track a conservative production rate. Tighter (--window 60 --min-moves 2) for forensic searches; looser for very-low-churn environments.

l2trace audit-multi-vlan

Surface MACs currently observed in 2+ VLANs simultaneously. Catches 802.1Q double-tagging, accidental L2 segment merging, or misconfigured trunk-port allowed-VLAN lists.

l2trace audit-multi-vlan [--min-vlans 2]

The output’s device_count column is the triage hint: 1 device + N vlans is usually a dot1q sub-interface (legitimate); N devices is almost always a bug.

l2trace audit-stp-cam

Surface (mac, port, vlan) triples where the port is STP-blocking but the CAM still has the MAC open. Violates 802.1D §8.6 (blocked ports drop frames at L2 and don’t learn MACs).

l2trace audit-stp-cam

No parameters — the violation is binary. Causes when non-empty: STP flap during detection (transient), vendor bug, or stale CAM the compactor hasn’t aged out.

l2trace audit-stp-roots

Surface VLANs where devices report 2+ different STP root bridges. 802.1D root-bridge election converges every device on a connected segment to the same root_id per VLAN. Multiple distinct root_ids mean partition, BPDU filtering, or PVST/RSTP/MSTP-mode disagreement across vendor peers.

l2trace audit-stp-roots

Only counts rows with non-NULL root_id — NULL means “we don’t know,” not “we disagree.” Treating NULL as a value would invent false positives.

l2trace audit-arp-collision

Cross-table L2/L3 check: surface IPs claimed by 2+ different MACs in the same VRF. Classic IP-conflict signature: DHCP race, static-route typo, L2 partition with duplicate hosts on each half, or active ARP-spoofing (man-in-the-middle).

l2trace audit-arp-collision

device_count distinguishes a single-router multi-source race (1 witness) from a real network conflict (≥ 2 independent witnesses).

l2trace audit-port-state-cam

Cross-layer PHY/data check: surface (mac, port, vlan) triples where the port is admin='down' or oper IN ('down', 'lower-layer-down') but CAM still has the MAC open. Down ports drop frames at the PHY layer; no frames means no MAC learning.

l2trace audit-port-state-cam

oper='testing' is NOT treated as down — some vendors keep learning behavior intact during diagnostics.

See Bug detection across the observation surface for the methodology, the cross-detector invariants, and the F19 coverage table that measures each detector against synthetic injected bugs.

l2trace tui

Launch the Textual TUI operator console.

l2trace tui
# or
make tui

The TUI runs inside the tui compose service (which has TTY allocated via --service-ports). It needs the postgres and nats services running.

Four modes accessible from the HOME screen:

Binding	Mode	What it does
Ctrl+T	TRACE	L2 traceroute form (src/dst MAC + VLAN)
Ctrl+H	HISTORY	MAC bitemporal timeline
Ctrl+O	OPS	Live FDB tree + disagreements + quarantine tail
Ctrl+U	AUDIT	Bidirectional LLDP audit (same query as audit-adjacencies)

F1 opens a per-screen help overlay. The ambient as-of timestamp picker at the top of every screen re-fires whichever query is on display, so “what did the fabric look like an hour ago?” is one input change away. See the TUI tour.

l2trace oui-refresh

Fetch IEEE’s three OUI registries (MA-L / MA-M / MA-S) and UPSERT them into oui_vendor. Safe to run on a live system — UPSERT, no TRUNCATE.

l2trace oui-refresh
# or
make oui-refresh

Pulls ~5 MB across three HTTPS requests to standards-oui.ieee.org. ~52k rows land per refresh. Recommended cadence: weekly.

l2trace collect (one-shot)

Run a one-off poll cycle for a single (device, source) outside the orchestrator — useful for debugging a new device config without waiting for the next reconfig pass.

l2trace collect --device-id ID --source gnmi

Most operators want l2trace reconcile (which is what make up starts) — that runs the orchestrator that owns this lifecycle. Use collect only when you need a single foreground cycle whose stderr you can read directly.

Configuration via .env

Most flags also have an env-var form. The reconciler reads them on start via Pydantic settings. See .env.example for the full list.

Common ones:

Variable	Default	Effect
DATABASE_URL	(in compose)	asyncpg URL — set by compose to the in-network postgres
NATS_URL	(in compose)	NATS URL — set by compose to the in-network nats
GNMI_AGING_THRESHOLD_SECONDS	300	Compactor closes gNMI rows after this much silence
SNMP_POLL_INTERVAL_SECONDS	60	Compactor uses 2 × this as the SNMP aging threshold
COMPACTOR_INTERVAL_SECONDS	30	How often the compactor pass runs
TUI_THEME	(empty / default)	One of hacker, amber, or empty for terminal default

See also

• Data model reference — what the tables actually look like
• Event envelope reference — the canonical event schema collectors emit