The TACACS+ lockout doctrine

TACACS+ controls who may log into your switches and what they may type there. That makes it different in kind from almost anything else l2trace does: a bug here does not lose a data point, it locks an engineer out of the box they need to fix, in the middle of the night, during the incident they were paged for.

So the whole TACACS+ implementation is built around one rule, and this page is that rule and its consequences. If you read one thing before turning on enforcement, read this.

A buggy server is more dangerous than a dead one

A Cisco NAS configured the usual way —

aaa authentication login default group tacacs+ local

— treats the three possible answers from a TACACS+ server very differently:

The server…	…and the switch
is unreachable / never answers	falls through to the next method (`local`) — you still get in
returns ERROR	falls through to the next method (`local`) — you still get in
returns FAIL	stops. denies. no fallback.

Read that table twice. A server that is down is safe, because the switch shrugs and uses its local account. A server that is up but wrong — one that catches a database exception and returns FAIL — has issued an authoritative denial, and the switch obeys it and stops trying. The outage becomes a lockout.

This inverts the usual instinct. The failure mode to engineer against is not “the server crashed”, it is “the server confidently said no for a reason that had nothing to do with the user”.

The rule

On any internal fault — a database error, an unreachable directory, an unparseable packet, an unexpected exception anywhere — return ERROR, never FAIL. FAIL is reserved for the one case where we authoritatively evaluated the user and the answer is genuinely no.

l2trace enforces this structurally, not by hoping every code path remembers it:

The identity-source interface is three-valued (PASS / FAIL / ERROR), so “I could not check” cannot be silently collapsed into “no”. A boolean authenticate() would have no way to distinguish “wrong password” from “AD is down”, and would map both to FAIL — locking everyone out the morning a domain controller reboots.
Every decision path is wrapped so that an escaping exception becomes ERROR, not FAIL.

The one legitimate FAIL in authentication is a wrong password: we checked the credential and it is wrong. That is a real verdict, and the switch is right to stop.

The subtle version of this rule is a credential that won’t even decode. A username or password field that isn’t valid UTF-8 is a transcoding problem of ours, not a verdict on the engineer — so l2trace decodes credentials as strict UTF-8 and answers ERROR on a decode failure, never FAIL. The tempting shortcut here (decode(errors="replace")) silently rewrites the bad bytes, binds the wrong string, checks it, and manufactures an authoritative FAIL out of a bug in our own byte handling — the one direction this doctrine forbids. Better to say “I could not read this” and fall through than to invent a “no.”

The mandatory local account

Because the switch only falls through to local when a local method exists, the doctrine has a deployment half that is not optional:

Configure a local break-glass account on every device, put local in the method list, and verify the local account works before pointing the device at l2trace.

The local account is what stands between “the AAA server had a bad afternoon” and “nobody can log into the fabric”. Test it first, every time.

The three fail directions

Here is the part that surprises people. “Fail safe” is not a direction you choose once. The TACACS+ server makes three different authorization-adjacent decisions, and each one fails in a different safe direction, because the worst outcome is different each time.

Authentication fails CLOSED

A directory outage must never let someone in. So authentication stays strict: if AD cannot confirm the password, the answer is ERROR, and the switch falls through to its local method — where the break-glass account is the intended answer. An outage does not grant access; it routes you to the local account.

Shell authorization fails OPEN (to permit)

Once an engineer has authenticated, a directory outage must never lock them down to nothing. If l2trace cannot fetch the AD groups that decide someone’s privilege level, it applies no profile — the engineer gets a shell at the device’s own default privilege, rather than being denied a shell at all. A degraded login beats no login. Refusing an engineer a shell during an outage is exactly how a fixable incident becomes an unfixable one.

Note this is the opposite direction from authentication, in the adjacent module, on purpose.

Command authorization fails to NEITHER

Per-command authorization is the sharpest case. If l2trace cannot determine which commands a user is allowed to run, it must not fail to permit everything (anyone runs anything during our outage) and must not fail to deny everything (an on-call engineer locked out of the commands they need mid-incident). So it answers ERROR and defers to the device’s own local command rules. “We could not determine this user’s command policy” is answered honestly by handing the question to the switch, not by guessing in either direction.

There is one knob here worth knowing about, TACACS_COMMAND_AUTHZ_FAIL_OPEN. Some switches are configured with no local command-authorization fallback, and on those an ERROR would leave an engineer logged in but unable to type a single command. For those, and only when set deliberately, the flag flips a fault to permit the command. Permitting commands during our own outage is a real blast radius, which is why it is opt-in and never inherited from the shell path’s fail-open reflex.

Decision	Safe direction on our own fault	Why
Authentication	ERROR → `local` (fail closed)	an outage must not let anyone in
Shell authorization	no profile → default privilege (fail open)	an outage must not lock a real engineer down to nothing
Command authorization	ERROR → device’s local rules (neither)	permit-all is a free-for-all; deny-all is a mid-incident lockout

The only way to get this right is to ask, for every single decision, “what is the worst thing that happens if we are wrong, here?” — and never to pick a house style once and apply it everywhere.

Monitor-first, and the triple gate

None of the above is about enforcement being on. By default it is not. The server ships monitor-first and triple-gated:

Monitor mode (default on). Authorization computes the full decision and logs what it would apply, then flattens the reply to a no-op (PASS_ADD with no arguments: permit, add nothing, let the device decide). You can run command authorization for weeks, watching the would-denies accumulate in the Commands view, before it can deny anyone.
The triple gate. A command is only ever denied when three independent conditions all agree: the per-device tacacs_enforce flag, the global TACACS_ENFORCEMENT_PERMITTED arm, and monitor mode being off. A stray enforce flag on one production switch can never, by itself, start denying commands.

Authentication is the exception: it is always answered truthfully (there is no neutral answer to “is this the right password”), so monitor mode governs authorization, not login.

The obfuscation is not a security control

One more thing the doctrine will not save you from. TACACS+‘s body obfuscation is a keyed MD5 stream cipher, and RFC 8907 section 10.1 says plainly that it provides “no meaningful integrity, privacy, or replay protection”. Treat the shared secret as a weak client identifier, not as encryption. The real confidentiality boundary is an isolated management network. l2trace implements the obfuscation because the protocol mandates it, not because it is safe to rely on.

The one thing l2trace does insist on is that the obfuscation is present at all. RFC 8907 §10.5.2 says the unobfuscated body mode must not be used in production, and since a shared secret always exists, a packet that arrives with the unencrypted flag set is either a misconfigured NAS or an on-path attacker who stripped the keying — so by default (TACACS_REJECT_UNENCRYPTED) l2trace drops that connection. A dropped connection reads as “server unreachable,” which — per the table at the top of this page — routes the switch to its local method rather than issuing any verdict. The knob exists to be turned off only for a deliberately-unencrypted lab NAS, never in production.

The same ERROR-not-FAIL reflex governs authentication actions l2trace doesn’t handle. A CHPASS (change your AD password) or a SENDAUTH/SENDPASS (PPP outbound auth, not device admin) request is answered ERROR, not FAIL: an AD password can’t be changed over an LDAP simple bind, so rather than pretend to a verdict, l2trace hands the switch back to local. Only the LOGIN action — the one it can actually adjudicate — proceeds.