Split collection across hosts

When you need this

“Our switches only answer SNMP from one management box — the ACL names that host and nothing else.”
“The central stack lives in a data center that can’t reach the closet switches directly, but a jump box on that VLAN can.”
“We want the poller’s SNMP communities to stay off the shared database.”

The default topology (docker compose up) runs everything on one host: Postgres, NATS, and a reconciler that spawns every collector. Split-host collection keeps that central stack in place but moves some collectors to a second host that has the network reach the central host lacks. The poller reads device config from the central Postgres, collects, and emits events back to the central NATS. The reconciler on the central host consumes them exactly as if they’d been collected locally.

Nothing about the data model changes. There’s still one device_collector row per device — the split is only about which host spawns which collector.

How the scope split works

ORCHESTRATOR_SOURCES (CSV, empty = all) scopes which sources an orchestrator instance spawns. scoped_factories() filters the built-in factory set to the named sources; a source not listed is simply not spawned on that host.

The central host runs everything except the source the poller owns — e.g. ORCHESTRATOR_SOURCES=gnmi,ssh,netconf,vsphere.
The remote poller sets only its source — ORCHESTRATOR_SOURCES=snmp.

Because the split is per-instance and both hosts read the same device_collector table, a device whose source no host spawns just isn’t collected. Keep the two lists disjoint so nothing is polled twice and nothing is dropped.

1. Expose Postgres + NATS from the central host

Both services are internal-only by default (no host ports). The docker-compose.expose.yml overlay publishes them:

# On the central host:
EXPOSE_BIND_IP=10.0.0.10 docker compose \
  -f docker-compose.yml -f docker-compose.expose.yml up -d postgres nats

EXPOSE_BIND_IP defaults to 127.0.0.1 — a bare -f docker-compose.expose.yml deliberately does not put a database on the wire. Set it to the LAN address the poller reaches. This publishes Postgres on 5432 and NATS on 4222 bound to that address.

2. Firewall the exposed ports — mandatory

Postgres carries a strong password. NATS has no auth, so the firewall is the only thing standing between the event bus and the LAN. Restrict both ports to the poller’s source IP:

# On the central host, lan_nic then poller source IP then ports:
sudo ./scripts/remote-poller-firewall.sh eth0 10.0.0.20 5432 4222

The script inserts rules in the DOCKER-USER chain matched on -i <lan_nic>, so it filters LAN-ingress traffic only and never touches container-to-container traffic on the docker bridges (which would otherwise break the reconciler’s own connection to Postgres). It’s idempotent and multi-source safe — re-run it with a comma-separated list to allow more than one source:

sudo ./scripts/remote-poller-firewall.sh eth0 10.0.0.20,10.0.0.21 5432 4222

Persist the rules with a systemd oneshot so they survive a reboot.

3. Configure the poller’s `.env.poller`

The poller is a standalone stack (docker-compose.poller.yml) with no local Postgres or NATS. It’s driven by a .env.poller file — keep it chmod 600, it holds SNMP communities:

# Point at the central host's published ports (from steps 1–2):
DATABASE_URL=postgresql+asyncpg://l2trace:PASSWORD@10.0.0.10:5432/l2trace
NATS_URL=nats://10.0.0.10:4222

# This host's collection scope. Defaults to snmp — the whole point.
ORCHESTRATOR_SOURCES=snmp
SNMP_POLL_INTERVAL_SECONDS=300

# SNMP communities, referenced by device_collector.auth env:// URIs.
SWITCH_SNMP_COMMUNITY=your-read-community

DATABASE_URL is used both to read device_collector config and to ensure port rows exist; NATS_URL is where collected events are published. Both must resolve to the addresses you exposed and firewalled above.

4. Reference SNMP communities by env:// URI

Store the community as a reference, not a value, on the device_collector row so the secret never lands in the shared database:

{"community": "env://SWITCH_SNMP_COMMUNITY"}

The resolver reads SWITCH_SNMP_COMMUNITY from the poller’s environment at poll time — the actual community string lives only in the poller’s .env.poller, never in Postgres or a pg_dump. See the secrets layer for the full URI scheme set.

5. Start the poller

Use the poller compose file alone, not merged with docker-compose.yml:

docker compose --env-file .env.poller -f docker-compose.poller.yml up -d --build

The container runs l2trace orchestrate — the orchestrator only, no reconciler, compactor, or syslog. It spawns one collector per enabled device_collector row whose source is in ORCHESTRATOR_SOURCES, and emits to the central NATS.

Confirm it’s collecting by watching the logs for the source list it resolved and per-device poll cycles:

docker compose --env-file .env.poller -f docker-compose.poller.yml logs -f poller

Then check the central host — new observations for the poller’s devices should show up in the reconciler’s output and the web UI within a poll interval or two.

Getting the repo onto the poller

If the poller box has no clone credentials for a private repo, rsync the working tree over instead of cloning. Only .env.poller, docker-compose.poller.yml, the Dockerfile, and src/ are needed to build and run.

Failure modes

Symptom	What happened	Fix
Poller logs `connection refused` to Postgres/NATS	Ports not exposed, or `EXPOSE_BIND_IP` still `127.0.0.1`	Re-run step 1 with the LAN address; confirm `docker ps` shows the published port
Poller connects from wrong IP, gets dropped	Its source IP isn’t in the firewall allowlist	Add it via the comma-separated form in step 2
Device polled on two hosts	`ORCHESTRATOR_SOURCES` lists overlap	Make the central and poller source lists disjoint
`secrets: environment variable 'X' is not set` on a device	The env:// community isn’t in `.env.poller`	Add it, restart the poller
Poller stops emitting after a NATS outage	Some NATS client states don’t auto-reconnect after a long window	`docker restart l2trace-poller`