CLI commands

l2trace ships one binary, l2trace, with several subcommands. The Makefile wraps the most common ones behind targets so you don’t usually invoke it directly — but here’s the full surface.

All commands run inside the reconciler container by default:

docker compose run --rm --no-deps reconciler l2trace <subcommand> ...

l2trace migrate

Apply alembic migrations to head.

docker compose run --rm migrate alembic upgrade head
# or, equivalently:
make migrate

The Make target uses the migrate compose service which is configured for one-shot alembic runs. Direct invocation of l2trace migrate works too but goes through the longer-running reconciler image.

l2trace reconcile

Start the reconciler and the compactor and the collector orchestrator with a shared in-memory LiveSet. Long-running. The default make up starts this as a daemon in the reconciler container.

l2trace reconcile

Co-running is not optional: the compactor must invalidate LiveSet entries when it closes valid_during ranges. Splitting them across two processes would silently corrupt the bitemporal log. See the compactor invariant for why.

The orchestrator picks up rows from the device_collector table on boot and on a ~30s reconfig pass — so anything added/disabled via l2trace device … takes effect without a process restart.

Flag	Default	Description
--migrate	off	Run alembic migrations to head before starting, after the singleton lock is held so a crash-looping replacement can’t race them. For supervised deploys (Swarm) that have no separate one-shot migrate step.

The singleton advisory lock makes this a cluster-wide singleton: a second instance (partition reschedule, double-start) exits non-zero (111 lock held, 112 DB unreachable) rather than corrupting the bitemporal log — the supervisor’s restart policy turns that into “wait for the holder to die.”

Opt-in co-running services fold into the same event loop when their env flag is set: SYSLOG_ENABLED (syslog listener), RADIUS_ENABLED (802.1X server), NAUTOBOT_ENABLED (inventory sync), AUTHZ_SYNC_ENABLED (802.1X authorization-intent sync). PTR enrichment runs unless PTR_ENRICHMENT_ENABLED is turned off.

l2trace orchestrate

Run only the collector orchestrator — no reconciler, compactor, or syslog. The entry point for a remote poller host that can reach devices the central stack can’t (e.g. an SNMP-allowlisted box).

l2trace orchestrate

Spawns one collector per enabled device_collector row whose source is in ORCHESTRATOR_SOURCES, and emits events to the central NATS the reconcile process consumes. Point DATABASE_URL (collector config + port ensure) and NATS_URL (event emit) at the central stack. The reconciler/compactor stay on the central host under reconcile.

No flags — everything is driven by env (ORCHESTRATOR_SOURCES, DATABASE_URL, NATS_URL). The poller-side firewall that restricts the exposed Postgres/NATS ports to the poller’s source IP is a deploy-time concern, not a CLI flag.

l2trace device …

Register and manage per-device collector configurations. The orchestrator inside l2trace reconcile reads this table and (re)spawns gNMI/SNMP/SSH workers to match. Seven subcommands: add, list, disable / enable, remove, plus assign-profile / clear-profile (which bind a device to a polling-cadence profile — see l2trace profile … below).

device add

l2trace device add --hostname HOST --mgmt-ip IP --source SRC \
  [--community COMMUNITY | --community-uri URI] [--snmp-port 161] \
  [--username USER | --username-uri URI] \
  [--password PW | --password-uri URI] \
  [--enable-secret PW | --enable-secret-uri URI] [--ssh-port 22] \
  [--poll-interval 60] [--vendor VENDOR]

Flag	Default	Description
--hostname	(required)	Switch hostname — unique per install
--mgmt-ip	(required)	Management IP the collector connects to
--source	snmp	One of gnmi, snmp, ssh, netconf
--community / --community-uri	—	SNMPv2c community as cleartext, or as a secrets URI (env://VAR, file:///path#/ptr). Required for --source=snmp
--snmp-port	161	UDP port for SNMP collection
--username / --username-uri	—	SSH username (cleartext or URI). Required for --source=ssh
--password / --password-uri	—	SSH password (cleartext or URI). One of --password* OR --ssh-key-file is required for --source=ssh
--ssh-key-file	—	Absolute path to private key file (alternative to password). Typically a Docker secret mount like /run/secrets/ssh/sw-edge-7-key
--ssh-passphrase / --ssh-passphrase-uri	—	Passphrase for an encrypted private key (cleartext or URI)
--enable-secret / --enable-secret-uri	—	Optional privileged-mode password for IOS-XE
--ssh-port	22	TCP port for SSH collection
--poll-interval	60.0	Seconds between polls (FDB + LLDP + chassis-id)
--vendor	—	Free-text vendor hint. Required for SSH — picks the napalm driver: cisco-ios-xe / cisco-ios / cisco-nxos

URI-form credential flags are preferred for production. The URI form stores a reference to the secret (env var name, JSON pointer into a mounted file, etc.) instead of the value itself — so pg_dump of the database doesn’t expose credentials. See Manage collector credentials for the full URI scheme catalog. Passing both forms for the same field (e.g. --password and --password-uri) is a user error.

Idempotent. Re-running with the same --hostname updates fields in place; the orchestrator picks up the change on its next reconfig pass. You can register the same device twice with different --source values to get redundant collection (e.g. gNMI primary + SNMP backup, or SNMP primary + SSH fallback).

For SSH specifically — see How to add an SSH-only switch for the operator walkthrough including a working IOS-XE example.

device list

l2trace device list

Renders a Rich table showing every registered device, its enabled sources, last successful poll, and the most recent collector error. Quick health check before opening a screenshot session.

device disable / device enable

l2trace device disable --hostname HOST --source SRC
l2trace device enable  --hostname HOST --source SRC

Toggles the enabled flag on the (device, source) collector row. Keeps history intact — re-enabling is one command. The orchestrator gracefully stops the worker on the next reconfig pass.

device remove

l2trace device remove --hostname HOST --yes

DELETEs the device + cascade-deletes its entire observation history. Destructive. The --yes flag is required to skip the confirmation prompt. Most operators want device disable instead — that stops collection but preserves the bitemporal log for forensic queries.

device assign-profile / device clear-profile

l2trace device assign-profile --hostname HOST --profile NAME
l2trace device clear-profile  --hostname HOST

Bind a device to a named collection profile (the l2trace profile … sub-app below) or remove that binding. clear-profile falls the device back to the global default interval. Both flags are required for assign-profile; clear-profile takes only --hostname.

l2trace profile …

Named per-device polling-cadence profiles. A profile carries a default poll interval plus optional per-source (snmp / gnmi / ssh) overrides. The orchestrator resolves the effective interval per (device, source) as: per-row extras → per-source profile override → profile default → built-in default. Three subcommands:

profile create

l2trace profile create --name NAME --default-interval SECONDS \
  [--description TEXT] [--snmp-interval N] [--gnmi-interval N] [--ssh-interval N]

Flag	Default	Description
--name	(required)	Profile identifier — unique
--default-interval	(required)	Default poll interval in seconds
--description	—	Free-text description
--snmp-interval	—	SNMP-specific override (seconds)
--gnmi-interval	—	gNMI-specific override (seconds)
--ssh-interval	—	SSH-specific override (seconds)

Idempotent on --name — re-running with the same name updates the intervals and description in place.

profile list

l2trace profile list

Rich table of every profile with its default and per-source intervals.

profile delete

l2trace profile delete --name NAME

Deletes a profile. Devices currently assigned to it have their profile set to NULL — they fall back to the global default interval.

l2trace trace

Run a one-shot L2 traceroute from the CLI.

l2trace trace --src AA:BB:CC:11:22:33 --dst AA:BB:CC:44:55:66 --vlan 10 \
  [--as-of TIMESTAMP|now] [--audit-at TIMESTAMP] [--max-hops 32]

Flag	Default	Description
--src	(required)	Source MAC, any common format
--dst	(required)	Destination MAC
--vlan	(required)	VLAN ID, 1..4094
--as-of	now	Wire-time anchor: “what was the path at T?”
--audit-at	current belief	Belief-time anchor: “what did we believe at T?”
--max-hops	32	Loop guard — abort after N hops

The Make wrapper is:

make trace SRC=... DST=... VLAN=... [AS_OF=...]

(AUDIT_AT is not in the Make wrapper — pass it via the docker compose invocation directly.)

l2trace trace-diff

Run the L2 traceroute query at two as_of timestamps and surface a structured per-hop diff.

l2trace trace-diff --src MAC --dst MAC --vlan N \
  --t1 ISO-8601 --t2 ISO-8601 [--audit-at ISO-8601] [--max-hops 32]

Flag	Default	Description
--src	(required)	Source MAC, any common format
--dst	(required)	Destination MAC
--vlan	(required)	VLAN ID, 1..4094
--t1	(required)	First as-of (the “before” snapshot)
--t2	(required)	Second as-of (the “after” snapshot)
--audit-at	current belief	Belief-time; applied to BOTH snapshots
--max-hops	32	Loop guard, same as trace

Status glyphs per hop: ✓ unchanged, Δ changed, + added, - removed. Summary line reports the total change count and any termination change (e.g. reached → flood).

See How to diff a traceroute between two times for operator workflow + sample output.

l2trace mlag …

Declare MLAG / vPC / MC-LAG peer groups so the traceroute walker collapses paired switches into one logical hop and the adjacency audit doesn’t drown in peer-link rows. See Configure MLAG groups for the workflow and How MLAG-collapsed traceroute works for the design.

Subcommand	Purpose
mlag create --hosts A,B[,C...]	Assign all listed hosts to one MLAG group. Auto-picks the group ID. Idempotent.
mlag list	Rich table of every declared group + its hostnames
mlag dissolve --hosts A,B	Reset mlag_group_id = NULL on the named hosts

Conflict semantics: trying to merge hosts that already belong to different groups errors loudly (would otherwise create quad-groups operators rarely actually want).

Takes effect immediately — no reconciler restart needed; the next trace query and AUDIT refresh use the new grouping.

l2trace audit-adjacencies

Audit every open LLDP adjacency for one-way / asymmetric observations. For each A → B row, checks whether B → A exists from the same source and from any source.

l2trace audit-adjacencies [--source SRC] [--show-healthy] [--include-peer-links]
# or
make audit-adjacencies

Flag	Default	Effect
--source	—	Limit audit to one telemetry source (gnmi, snmp, …)
--show-healthy	off	Include bidirectionally-confirmed rows in output
--include-peer-links	off	Include MLAG peer-link adjacencies in the audit. By default they’re hidden — peer-links are bidirectional by design and would always be ✓ healthy, cluttering the view. See Configure MLAG groups.

Output is a Rich table with one row per open adjacency:

Symbol	Meaning
✗ no reverse	No peer observation back from any source — likely one-way cable, LLDP disabled on peer, or port down
△ source asym	Link is bidirectional, but the same source only sees one direction (telemetry asymmetry, not a cable problem)
✓	Both directions confirmed by the same source

A green “all open adjacencies are bidirectional” line means nothing to investigate. Empty table with --show-healthy off (the default) is also the all-clear signal. See Audit LLDP adjacencies for the full operator playbook.

l2trace audit-flaps

Sliding-window count of distinct ports per (mac, device, vlan). A MAC oscillating between ports faster than the switch’s aging timer is the signature of a switching loop, a misconfigured server bond, or active-active VRRP gone split-brain.

l2trace audit-flaps [--window 300] [--min-moves 3]

Flag	Default	Effect
--window	300	Sliding-window size in seconds. 300s matches one Cisco aging cycle.
--min-moves	3	Minimum distinct ports inside the window to flag (≥ 2)

Defaults track a conservative production rate. Tighter (--window 60 --min-moves 2) for forensic searches; looser for very-low-churn environments.

l2trace audit-multi-vlan

Surface MACs currently observed in 2+ VLANs simultaneously. Catches 802.1Q double-tagging, accidental L2 segment merging, or misconfigured trunk-port allowed-VLAN lists.

l2trace audit-multi-vlan [--min-vlans 2]

The output’s device_count column is the triage hint: 1 device + N vlans is usually a dot1q sub-interface (legitimate); N devices is almost always a bug.

l2trace audit-stp-cam

Surface (mac, port, vlan) triples where the port is STP-blocking but the CAM still has the MAC open. Violates 802.1D §8.6 (blocked ports drop frames at L2 and don’t learn MACs).

l2trace audit-stp-cam

No parameters — the violation is binary. Causes when non-empty: STP flap during detection (transient), vendor bug, or stale CAM the compactor hasn’t aged out.

l2trace audit-stp-roots

Surface VLANs where devices report 2+ different STP root bridges. 802.1D root-bridge election converges every device on a connected segment to the same root_id per VLAN. Multiple distinct root_ids mean partition, BPDU filtering, or PVST/RSTP/MSTP-mode disagreement across vendor peers.

l2trace audit-stp-roots

Only counts rows with non-NULL root_id — NULL means “we don’t know,” not “we disagree.” Treating NULL as a value would invent false positives.

l2trace audit-arp-collision

Cross-table L2/L3 check: surface IPs claimed by 2+ different MACs in the same VRF. Classic IP-conflict signature: DHCP race, static-route typo, L2 partition with duplicate hosts on each half, or active ARP-spoofing (man-in-the-middle).

l2trace audit-arp-collision

device_count distinguishes a single-router multi-source race (1 witness) from a real network conflict (≥ 2 independent witnesses).

l2trace audit-port-state-cam

Cross-layer PHY/data check: surface (mac, port, vlan) triples where the port is admin='down' or oper IN ('down', 'lower-layer-down') but CAM still has the MAC open. Down ports drop frames at the PHY layer; no frames means no MAC learning.

l2trace audit-port-state-cam

oper='testing' is NOT treated as down — some vendors keep learning behavior intact during diagnostics.

See Bug detection across the observation surface for the methodology, the cross-detector invariants, and the F19 coverage table that measures each detector against synthetic injected bugs.

l2trace web

Serve the web UI backend (FastAPI). A read-only JSON API over the same bitemporal engine the TUI uses, plus the built frontend.

l2trace web [--host 0.0.0.0] [--port 8080]

Flag	Default	Description
--host	0.0.0.0	Bind address
--port	8080	Bind port

Optional OIDC SSO (Authentik) via OIDC_ENABLED; otherwise front it with caddy for TLS. The app trusts X-Forwarded-Proto (proxy_headers, forwarded_allow_ips=*) so it sees https behind the proxy — correct scheme for OIDC redirects and Secure session cookies.

l2trace tui

Launch the Textual TUI operator console.

l2trace tui
# or
make tui

The TUI runs inside the tui compose service (which has TTY allocated via --service-ports). It needs the postgres and nats services running.

Modes accessible from the HOME screen:

Binding	Mode	What it does
Ctrl+T	TRACE	L2 traceroute form (src/dst MAC + VLAN)
Ctrl+H	HISTORY	MAC bitemporal timeline
Ctrl+O	OPS	Live FDB tree + disagreements + quarantine tail
Ctrl+U	AUDIT	Bidirectional LLDP audit (same query as audit-adjacencies)
Ctrl+G	CHANGES	Moved / new / removed / IP-rebinding review over a window
Ctrl+D	DIFF	Same traceroute at two times, aligned hop-by-hop
Ctrl+W	ROGUE-SW	Unmanaged / rogue switch detector (current-belief only)

F1 opens a per-screen help overlay. The ambient as-of timestamp picker at the top of every screen re-fires whichever query is on display, so “what did the fabric look like an hour ago?” is one input change away. See the TUI tour.

l2trace oui-refresh

Fetch IEEE’s three OUI registries (MA-L / MA-M / MA-S) and UPSERT them into oui_vendor. Safe to run on a live system — UPSERT, no TRUNCATE.

l2trace oui-refresh
# or
make oui-refresh

Pulls ~5 MB across three HTTPS requests to standards-oui.ieee.org. ~52k rows land per refresh. Recommended cadence: weekly.

l2trace collect (one-shot)

Run a one-off poll cycle for a single (device, source) outside the orchestrator — useful for debugging a new device config without waiting for the next reconfig pass.

l2trace collect --device-id ID --source gnmi

Most operators want l2trace reconcile (which is what make up starts) — that runs the orchestrator that owns this lifecycle. Use collect only when you need a single foreground cycle whose stderr you can read directly.

l2trace infer-adjacency

Derive switch adjacencies from CAM-table intersections (Lowekamp 2001 §5) — a backstop for gear that strips LLDP/CDP. Reads the bitemporal mac_observation table at --as-of, applies the Simple Connection Theorem to every cross-device port pair not already covered by LLDP, and prints (or writes) the port-pairs it believes are directly linked.

l2trace infer-adjacency [--as-of now] [--vlan N] \
  [--min-observations 3] [--min-coverage 1.0] [--write]

Flag	Default	Description
--as-of	now	Valid-time point (ISO-8601 or now)
--vlan	—	Restrict to one VLAN; omit to run across the whole universe
--min-observations	3	Minimum MACs per side (Lowekamp Lemma 5.2: 3 is sufficient)
--min-coverage	1.0	Joint host-universe coverage threshold; 1.0 = strict Lowekamp
--write	off	Persist inferences as adjacency rows (source=reconciler, bidirectional). Default is dry-run

Output respects existing LLDP adjacencies and won’t re-suggest known links. Without --write it prints a confidence-scored table and exits; --write inserts bidirectional rows so the traceroute CTE can walk the inferred link either way.

l2trace seed-cam

Pipelined ICMP CAM-table seeder (Breitbart 2004 §VI.B). Walks the listed subnets so each ICMP reply refreshes the switch’s CAM entry for the responding host’s MAC — the next SNMP poll then catches everyone before aging fires (Cisco default ~5 minutes).

l2trace seed-cam --subnets CIDR[,CIDR...] \
  [--burst 16] [--rate 160] [--timeout 1.0] [--duration 0]

Flag	Default	Description
--subnets	(required)	Comma-separated CIDRs to seed, e.g. 10.0.0.0/24,10.0.1.0/24
--burst	16	ICMPs per burst (Breitbart: 16)
--rate	160.0	Target IPs/sec across the sweep (Breitbart: 160)
--timeout	1.0	Per-ping timeout in seconds
--duration	0.0	Run for this many seconds, repeating the sweep. 0 = one sweep then exit

The seeder does NOT write to the bitemporal log — it only provokes traffic. Visibility into who responded lands in the regular SNMP poll cycle, processed by the reconciler as usual.

l2trace radius

Run only the RADIUS/802.1X server — no reconciler, compactor, or orchestrator. The standalone entry point for a dedicated radius service, so switch authentication availability is independent of the reconciler advisory lock.

l2trace radius

Binds UDP RADIUS_AUTH_PORT, resolves each NAS by source IP + shared secret, and emits an AUTH_EVENT per Access-Request to the central NATS the reconciler consumes. Point DATABASE_URL (decision reads) and NATS_URL (emit) at the central stack. Register switches as device_collector rows with source=radius before starting.

Monitor mode (RADIUS_MONITOR_MODE, default on) makes zero network changes: every reply is Access-Accept with no VLAN, and the decision the server would have made is recorded as a discrepancy flag on each auth_event. No flags — everything is env-driven.

l2trace radius-mock-switch

A software NAS for smoke-testing l2trace radius without hardware. Sends MAB Access-Requests (or a PEAP exchange), then optionally listens for CoA/Disconnect and ACKs them.

l2trace radius-mock-switch --secret SECRET --mac AA:BB:CC:11:22:33 \
  [--server 127.0.0.1] [--auth-port 1812] [--nas-port GigabitEthernet1/0/1] \
  [--coa-port 3799] [--listen/--no-listen] [--duration 0] \
  [--peap --username USER --password PW --ca /path/ca.pem]

Flag	Default	Description
--mac	—	Supplicant MAC(s), repeatable. Required for MAB; with --peap it only sets Calling-Station-Id
--secret	(required)	RADIUS shared secret (matches the NAS row’s radius_secret)
--server	127.0.0.1	RADIUS server host
--auth-port	1812	Server auth UDP port
--nas-port	GigabitEthernet1/0/1	NAS-Port-Id to claim
--coa-port	3799	Local port to listen for CoA/Disconnect on
--listen / --no-listen	--listen	After authenticating, listen for CoA/Disconnect and ACK
--duration	0.0	Seconds to listen (0 = until Ctrl-C)
--peap	off	Run a PEAPv0 / EAP-MSCHAPv2 supplicant instead of MAB. Requires --username / --password / --ca
--username	—	PEAP inner (tunneled) username
--password	—	PEAP inner (tunneled) password
--ca	—	CA cert PEM validating the server’s TLS cert inside the PEAP tunnel

This host’s source IP must be registered as a device_collector source=radius with a matching radius_secret; the server requires a Message-Authenticator, which this sends.

l2trace radius-coa

Send a CoA-Request (change VLAN / re-auth) or a Disconnect-Request (RFC 5176) to a switch. The server-side trigger the approval loop calls automatically, and a manual operator tool.

l2trace radius-coa --host SWITCH --secret SECRET --mac AA:BB:CC:11:22:33 \
  [--vlan N] [--disconnect] [--port 3799]

Flag	Default	Description
--host	(required)	Switch (NAS) host to send the CoA to
--secret	(required)	Shared secret for that switch
--mac	(required)	Supplicant MAC whose session to change/bounce
--vlan	—	New VLAN (CoA-Request); omit for a bare re-auth
--disconnect	off	Send a Disconnect-Request instead of a CoA-Request
--port	3799	CoA/Disconnect UDP port

l2trace tacacs-lab

A standalone, database-free TACACS+ runner for bringing up a real lab switch — no Active Directory, no Postgres, just a network path to the switch. It serves the production wire protocol against in-memory stand-ins (a static credential map, a static shell-profile resolver, and an accounting sink that logs instead of writing the bitemporal store), so it can prove a Catalyst’s MD5 pad, login dialog, and shell-authorization argument handling agree with l2trace before you provision the full service.

l2trace tacacs-lab --switch-ip 192.0.2.10 --secret SECRET \
  --user name=password[=priv_lvl] [--user ...] \
  [--listen-addr 0.0.0.0] [--port 4949] [--dialect generic]

Flag	Default	Description
--switch-ip	(required)	The lab switch’s source IP — the NAS l2trace accepts
--secret	(required)	Shared secret configured on the switch
--user	(required, repeatable)	name=password[=priv_lvl], e.g. test=Passw0rd=15. Priv is read only as a trailing all-digits field, so a password may contain =
--listen-addr	0.0.0.0	Bind address
--port	4949	Bind port. 49 needs root; default is high, so publish/redirect 49 → 4949
--dialect	generic	Shell dialect: ios | nxos | arista | generic

Enforcement is off: the shell profile is computed and logged, not applied, so the switch keeps its own default privilege. Holds cleartext test passwords in memory — a bring-up tool, not a production path (that’s l2trace tacacs with an identity source). See Bring up TACACS+ on a lab switch.

l2trace nautobot-sync

One-shot: pull the Nautobot device inventory into the device table, registering every Nautobot device (with its mgmt_ip where present) so collectors and the syslog/Loki allowlist have targets.

l2trace nautobot-sync [--url URL] [--token TOKEN] [--insecure]

Flag	Default	Description
--url	NAUTOBOT_URL	Nautobot base URL
--token	NAUTOBOT_TOKEN	Nautobot API token (env fallback; exits 2 if neither is set)
--insecure	off	Skip TLS verification

Idempotent — upserts by hostname. reconcile runs this on a schedule when NAUTOBOT_ENABLED=true; this command is the ad-hoc / first-run path.

l2trace authz-sync

One-shot: pull Nautobot 802.1X authorization intent (authorized/denied + assigned VLAN, per MAC) into the local mac_authorization store the RADIUS path consults, and fire a CoA re-auth for each meaningful change.

l2trace authz-sync [--url URL] [--token TOKEN] [--insecure]

Flag	Default	Description
--url	NAUTOBOT_URL	Nautobot base URL
--token	NAUTOBOT_TOKEN	Nautobot API token (env fallback; exits 2 if neither is set)
--insecure	off	Skip TLS verification

Fail-safe: an explicit operator decision in the local store is never silently overridden — the disagreement is flagged as a conflict instead. reconcile runs this on a schedule when AUTHZ_SYNC_ENABLED=true; this command is the ad-hoc / first-run path.

Configuration via .env

Most flags also have an env-var form. The reconciler reads them on start via Pydantic settings. See .env.example for the full list.

Common ones:

Variable	Default	Effect
DATABASE_URL	(in compose)	asyncpg URL — set by compose to the in-network postgres
NATS_URL	(in compose)	NATS URL — set by compose to the in-network nats
GNMI_AGING_THRESHOLD_SECONDS	300	Compactor closes gNMI rows after this much silence
SNMP_POLL_INTERVAL_SECONDS	60	Compactor uses 2 × this as the SNMP aging threshold
COMPACTOR_INTERVAL_SECONDS	30	How often the compactor pass runs
TUI_THEME	(empty / default)	One of hacker, amber, or empty for terminal default

See also

• Data model reference — what the tables actually look like
• Event envelope reference — the canonical event schema collectors emit