Diff a traceroute between two times

When you need this

“We rebooted sw-core-1 at 14:00 yesterday. Did any traffic flows end up rerouted afterwards?”
“Customer says streaming broke at 09:35. Was the L2 path between their two hosts the same at 09:00 as at 09:40?”
“Post-mortem for the incident on Tuesday — show me which hops in the production fabric had ports flip between yesterday and today.”

l2trace trace-diff runs the same recursive-CTE traceroute at two different as_of timestamps and produces a structured per-hop diff. This is the operator interaction the bitemporal log was built for — without valid_during, “what was the path at 14:00 yesterday?” isn’t even a question you can ask.

The command

l2trace trace-diff \
  --src 00:03:93:44:20:82 \
  --dst 00:50:56:78:9b:34 \
  --vlan 10 \
  --t1 2026-05-11T22:10:00Z \
  --t2 2026-05-11T22:20:00Z

Required flags: --src, --dst, --vlan, --t1, --t2. The two timestamps are ISO-8601 — same shape as l2trace trace --as-of.

Reading the output

Real capture from the seeded make seed-realistic demo data, running the diff inside the recent-observation window:

           trace-diff 00:03:93:44:20:82 → 00:50:56:78:9b:34 vlan=10
            2026-05-11T22:10:00+00:00 vs 2026-05-11T22:20:00+00:00
┏━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ step ┃ status ┃ before                   ┃ after                    ┃ note ┃
┡━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━┩
│    0 │ ✓      │ sw-access-1: Eth3 → Eth8 │ sw-access-1: Eth3 → Eth8 │      │
│    1 │ ✓      │ sw-core-1: Eth1 → Eth2   │ sw-core-1: Eth1 → Eth2   │      │
│    2 │ ✓      │ sw-access-2: Eth8 → Eth4 │ sw-access-2: Eth8 → Eth4 │      │
└──────┴────────┴──────────────────────────┴──────────────────────────┴──────┘
✓ trace unchanged between 2026-05-11T22:10:00+00:00 and
2026-05-11T22:20:00+00:00 (termination: reached)

This is the common operator outcome: the path is stable, every hop is unchanged, and the summary line confirms it. Most diffs across short windows in a healthy fabric look exactly like this.

When the diff is non-trivial

The status column uses four glyphs:

Glyph	Status	Meaning
✓	`unchanged`	Same device + same out-port at this step. Path is stable here.
Δ	`changed`	Both traces have a hop at this step but they differ. The `note` column names what changed (out-port, in-port, device, or telemetry source).
+	`added`	Only the AFTER trace has a hop here — the path grew longer at T2 (extra hop).
-	`removed`	Only the BEFORE trace had a hop here — the path shortened at T2.

The summary line at the bottom reports N change(s) · X modified · Y added · Z removed. If termination changed (e.g. T1 reached but T2 floods), that’s surfaced separately as termination changed: reached → flood.

Example: a host moved between ports

A constructed example of what an out-port change looks like in practice — this is the kind of output operators see when, say, a laptop disconnected from Eth1 and re-attached on Eth3:

┏━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ step ┃ status ┃ before                   ┃ after                    ┃ note                      ┃
┡━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│    0 │ ✓      │ sw-access-1: Eth1 → Eth8 │ sw-access-1: Eth3 → Eth8 │                           │
│    1 │ ✓      │ sw-core-1: Eth1 → Eth2   │ sw-core-1: Eth1 → Eth2   │                           │
│    2 │ Δ      │ sw-access-2: Eth8 → Eth1 │ sw-access-2: Eth8 → Eth4 │ out-port changed: Eth1 → Eth4 │
└──────┴────────┴──────────────────────────┴──────────────────────────┴───────────────────────────┘
1 change(s) · 1 modified · 0 added · 0 removed

Step 2 (sw-access-2) flagged — the dst MAC is now learned on Eth4 instead of Eth1, telling the operator the host moved on the egress switch. Step 0 looks ✓ because the in-port + out-port pair at sw-access-1 are still the same labels (Eth3 → Eth8); the renderer groups by device+ports for the “unchanged” check, and a hop where the out-port is the same trunk gets ✓ even if the in-port differs slightly.

The integration test tests/test_trace_diff_db.py walks through an explicit move scenario that produces a real ✓/Δ/+/- mixture against the testcontainers DB — useful as a guaranteed-working example to point at.

When to use this vs HISTORY vs OPS

Three l2trace surfaces look adjacent but answer different questions — worth keeping straight to avoid foot-guns:

Tool	Asks	Returns
`trace-diff`	”Did the path between MAC A and MAC B change between T1 and T2?”	Per-flow per-hop diff: ✓/Δ/+/- per step
HISTORY screen	”Where has MAC X been seen, ever?”	Per-MAC timeline of port assignments across all switches and sources
OPS screen	”What’s the current state of the fabric right now?”	Live FDB tree, disagreements pane, quarantine tail

A common mistake: reaching for trace-diff when you actually want HISTORY (“where has my host been today?”) — HISTORY is the single-MAC walk. The other direction is rarer: HISTORY won’t tell you how a flow’s path through the fabric changed, only where the endpoint MAC sat at each instant. Use trace-diff for path-shape questions; use HISTORY for endpoint-location questions.

Step-aligned, not LCS

The diff compares position i in BEFORE to position i in AFTER — NOT a longest-common-subsequence alignment. This matches operator intuition for L2 paths: when a path changes, it usually changes AT a specific step (an FDB entry got updated on one switch) rather than shifting all subsequent steps by ±1 (which would require a fundamental topology event).

The minority case — actual path-length changes — shows up as + or - entries at the tail end of the longer trace. If you find yourself wishing for LCS-style alignment, file an issue with the specific scenario; in practice we haven’t needed it.

Same audit-time on both sides

trace-diff takes one --audit-at belief-time, applied to both snapshots. The interpretation: “compare two valid-time states through the same belief lens.”

There’s a different question — “did our beliefs about T change between two retrieval times?” — that would vary BOTH axes. We haven’t built a helper for it because it’s rarer in practice (operators usually care about valid-time changes); when it matters, two separate l2trace trace --audit-at ... invocations + manual comparison covers it.

How it works under the hood

The implementation is two traceroute() calls plus a pure step-aligned diff:

async def trace_diff(session, src_mac, dst_mac, vlan, t1, t2, ...):
    before = await traceroute(..., as_of=t1, ...)
    after  = await traceroute(..., as_of=t2, ...)
    return diff_traces(before, after)

The bitemporal CTE was the hard work — trace-diff is conceptually shape on top of it. See src/l2trace/db/queries.py::diff_traces for the per-hop alignment logic; the precedence rule for the note column is device → out-port → in-port → telemetry-source, headline change wins.