What production HA looks like — and why

A single-host docker compose up is the right default: one Postgres, one NATS, one reconciler, done. This page is about the other shape — the multi-node deployment that survives losing a node — and why it’s built the way it is. It is not a runbook; the actual deploy steps, firewall rules, and failover drills live in the operator’s swarm-deploy runbook. This page is the reasoning behind that runbook’s shape.

The whole design reduces to one goal: losing any single node costs nothing manual. Three stateful concerns — the reconciler’s single-writer guarantee, the database, and the event bus — each fail over on their own, through a different mechanism, without an operator touching a config.

Reference topology

Three nodes. Two of them run the workload; the third runs nothing and exists only to hold a vote.

        ┌───────────┐        ┌───────────┐        ┌───────────┐
        │  node-a   │        │  node-b   │        │  witness  │
        │ (workload)│        │ (workload)│        │  (drained)│
        ├───────────┤        ├───────────┤        ├───────────┤
        │ reconciler│◀──lock─┤   (idle)  │        │           │
        │  web ×1   │        │  web ×1   │        │           │
        │ haproxy   │        │ haproxy   │        │           │
        │ PG leader │◀─sync──┤ PG replica│        │           │
        │ NATS      │========│ NATS      │========│ NATS      │
        │ etcd      │========│ etcd      │========│ etcd      │
        └───────────┘        └───────────┘        └───────────┘

Two managers alone can’t do this — either one dying loses quorum. Three managers hold quorum through any single loss, and the witness is drained so no workload ever schedules onto a box that exists only to vote. The witness needs no dedicated hardware: any stable box on the same low-latency LAN, joined as a manager and drained, works. Its day job (plain compose containers) is invisible to swarm scheduling.

The application runs in swarm. The database does not — it runs as plain compose, on purpose. Swarm’s job is to keep declared replicas alive, which is exactly wrong for a database: after a failover, swarm would happily resurrect the demoted old leader onto a repaired node and fork the timeline. Compose never resurrects a container remotely, so the demoted node stays down until an operator (or Patroni) decides what it becomes.

Layer 1 — the reconciler is a cluster-wide singleton

The reconciler and compactor share one in-memory LiveSet — the hot-path cache that classifies each incoming event as first-sight, continuation, or move. Two reconcilers with independent LiveSets, both writing the same bitemporal log, silently corrupt it: each thinks a MAC is new that the other already has open, and you get overlapping valid_during ranges for the same key (or the mac_obs_no_overlap_per_source exclusion constraint firing on every other event). This is a hard single-writer — not a performance preference, a correctness requirement.

replicas: 1 plus update_config.order: stop-first handles the easy case: rolling updates stop the old task before starting the new one. It does not handle partition rescheduling. If node-a is partitioned from the swarm managers, they declare its task lost and start a replacement on node-b — while the original may still be running and still writing. replicas: 1 counts tasks the managers know about; it can’t count a task they’ve lost sight of.

A Postgres advisory lock closes that gap. On startup the reconciler takes pg_try_advisory_lock on a dedicated connection and keeps that connection open for the life of the process:

conn = await get_engine().connect()
got = (await conn.execute(text("SELECT pg_try_advisory_lock(:k)"), {"k": key})).scalar()

The lock is session-scoped — it lives exactly as long as that TCP session to Postgres, and that’s the whole trick. Consider both sides of a partition:

The old holder that can still reach Postgres keeps its session, keeps the lock, and keeps being the writer. Correct — it’s the one that can still write.
The old holder that lost Postgres has its session reaped by the server; the lock frees; the replacement acquires it. Also correct — a reconciler that can’t reach Postgres can’t write anyway, so handing the role to one that can is exactly right.

A losing acquirer exits, and the supervisor (swarm restart_policy) retries until the holder’s session finally dies. There is never a window where two processes both believe they hold the lock, because the lock’s lifetime is defined by a resource — the TCP session — that only one of them can hold at a time.

The liveness half is hold_lock_forever: a periodic SELECT 1 on the lock connection. Any failure — Postgres restart, a proxy idle-timeout in the path, a failover to a new primary — raises, collapses the process, and the supervisor restarts into a clean re-acquisition against whatever primary now answers:

while True:
    await asyncio.sleep(interval_seconds)
    try:
        await conn.execute(text("SELECT 1"))
    except Exception as exc:
        raise SingletonLockLost from exc  # exit → supervisor restart → re-acquire

The ping does double duty: it’s also traffic that keeps a proxy’s idle timer from silently reaping the otherwise-quiet lock connection. This is what makes it safe to route the lock through a load balancer (Layer 2): a severed or rerouted session becomes a crash-and-reacquire, never a silent lock-less writer.

Layer 2 — the database fails over with no client repoint

Postgres runs as a Patroni-managed pair: a leader taking writes and a sync replica streaming from it, with the leader lease held in a 3-node etcd quorum. On a leader loss the replica self-promotes in tens of seconds. A returning old leader is demoted and rewound into a fresh replica — it does not come back as a second leader.

That handles the database electing a new leader. The harder problem is the clients: naively, every promotion means editing DATABASE_URL on every client and redeploying, which is slow, error-prone, and exactly the kind of manual step this design exists to eliminate.

HAProxy removes it. A pair of HAProxy instances present one stable write endpoint, and route it to whichever node is currently leader by health-checking Patroni’s REST API:

option httpchk GET /primary        # 200 only on the leader (Patroni REST)
on-marked-down shutdown-sessions   # demoted node's connections forced to reconnect
timeout client/server 0            # never sever the idle lock connection

Every client — the reconciler, the web replicas, the external poller — points at that endpoint and never at a node’s Postgres directly. A leader move is then transparent: the GET /primary check starts failing on the demoted node, shutdown-sessions drops the stale connections, and clients reconnect through the same unchanged URL to the new leader. No DATABASE_URL is ever edited on a failover.

Two details are load-bearing:

timeout client/server 0. Any finite timeout eventually severs the reconciler’s idle advisory-lock connection, which would void the Layer 1 singleton guarantee for no reason. The health check follows the leader; it must never be the thing that kills the lock. (And if it ever does, the hold_lock_forever ping turns that into a clean restart rather than a silent double-writer — the two layers back each other up.)
There is one URL, not two. Alembic’s migration path derives its sync psycopg URL from the same DATABASE_URL by swapping the driver. A second, hand-synced URL is a landmine — a repoint that updates one and misses the other splits the app’s view of where the database is.

Layer 3 — the event bus survives a node loss in place

Collectors publish observations to NATS JetStream; the reconciler consumes from it. A single NATS node is a single point of failure for ingest, so the production bus is a 3-node JetStream cluster with streams configured R3 — RAFT-replicated across all three nodes.

R3 is what makes a single-node loss a non-event. The surviving two nodes keep quorum and hold the full stream state, so nothing is lost and nothing has to be rebuilt. Clients seed from a comma-separated list of all three node addresses and reconnect to a survivor automatically; a returning node re-syncs the streams on its own. There is no “move the NATS service” step, because there is nothing to move — the state was already on the survivors.

The cluster runs as plain compose alongside etcd, not as a swarm service, for the same reason the database does: its data lives on each node’s local volume, and a scheduler that relocates the container would strand that state.

One caveat this buys clarity on rather than hides: NATS here has no authentication. The only access control is a firewall allowlist on the client and cluster ports. That’s an acceptable posture for a single-tenant, firewalled bus, but it’s a deliberate choice to know about, not an accident — adding a less-trusted tenant to the cluster would mean adding real auth first.

Why three mechanisms instead of one

Nothing forces the database, the event bus, and the singleton guarantee to fail over the same way, and trying to unify them would make each worse. A database needs a consensus-elected leader and a rewind-on-return story that an event bus doesn’t. An event bus needs replicated append-only streams that a single-writer lock doesn’t. The single-writer guarantee needs a lease whose lifetime is a resource one process can hold — which is neither of the above.

So each layer gets the mechanism that fits it:

Layer	Fails over via	On single-node loss
Reconciler (single-writer)	Postgres advisory lock, session-scoped	Replacement acquires; old holder exits or keeps writing — never both
Database	Patroni leader election + HAProxy write endpoint	Replica promotes; clients reconnect through the same URL
Event bus	3-node JetStream cluster, R3 streams	Survivors hold full state; clients reconnect to one

They share exactly one property: a double-fault threshold. Each is a 3-node quorum (etcd for the DB lease, RAFT for NATS, three managers for the swarm), so each tolerates one loss and stops at two. Losing two nodes stops writes — that’s the honest limit, and it self-heals when a node returns. The point of the design isn’t that nothing can ever go wrong; it’s that the common fault, one node, is absorbed by three independent mechanisms with zero operator action, and the uncommon fault, two nodes, fails in an obvious and recoverable way instead of a silent and corrupting one.

What the operator still does

Almost nothing on a failure, by design — but the runbook covers the handful of things that aren’t automatic: a planned switchover before updating the leader’s image (a clean leader stop reads as a surprise failover, so you sequence it deliberately), the break-glass path when a DB node and etcd quorum are both dead, the health canary that catches “database healthy but the app can’t write to it,” and the placement labels that keep workloads off the drained witness even after it reboots.