What the tests prove

qfence’s entire value proposition is “this provably does not allow two owners of the same availability plan at once.” That claim is only as credible as the adversarial testing behind it (spec §60), so this document lists every split-brain, chaos, and acceptance scenario the spec calls for, whether it’s automated, where, and — just as importantly — what is not yet covered. Treat a regression in this document as seriously as a regression in the code (§46/§60): if test coverage changes, this file changes in the same PR.

How to verify any claim below yourself:

go test ./... -race -count=2

Every scenario cited here is a real, named Go test function you can run individually with go test ./<package>/... -run <TestName> -v. Nothing in this document is aspirational.

CI status: this project’s actual origin is GitLab (gitlab.com/na7r1x/qfence), so .gitlab-ci.yml is the active CI — build, vet, gofmt check, then the full suite twice under -race on every push/MR, mirroring the gate below exactly, and the PostgreSQL type against PostgreSQL 15 and 18 in a docker:dind service once (test_postgres). .github/workflows/ci.yml is kept in the repo, dormant, for the eventual public migration to GitHub, and must be kept in sync with .gitlab-ci.yml whenever either changes. Every job in the pipeline, the docker:dind ones included, passed on 2026-10-28 on the owner’s self-hosted GitLab runner. The project is private until the move to GitHub, so no run is linked here yet; the results below are from local runs unless they say otherwise.

Split-Brain Protection Scenarios (§39)

Covered at two independent fidelity tiers, per §46’s design: a fast, deterministic, seed-based tier for every-commit CI, and a slower tier against a real 3-node Raft cluster with genuine network-partition simulation.

Modeling note on Scenario 5: plain majority-reachability is not sufficient to model this scenario correctly — both sides of an asymmetric partition can numerically reach a majority through a shared witness, so a naive simulation would incorrectly let the challenger through. Real Raft prevents this via pre-vote (a challenger can’t get votes from a voter that’s still hearing from a healthy incumbent). internal/sim.Cluster models the same guarantee explicitly via an incumbency check. This was verified to be load-bearing, not decorative: the fast-tier Scenario 5 test was written and run before the incumbency rule existed, and it failed without it.

Determinism: tests/sim/splitbrain_test.go’s TestSplitBrainScenariosAreDeterministic, plus internal/sim/quorumview_test.go’s TestHasMajorityAgreesWithIndependentReferenceAcrossManySeeds — 2000 random partition topologies (3, 5, or 7 voters) checked against an independently-implemented (union-find) reference algorithm for the majority calculation every safety property in this project ultimately rests on. Runs in ~0.1s.

Critical Acceptance Tests (§47)

All 9, each a self-contained test in tests/e2e/acceptance_test.go.

A real bug this suite found and fixed: resources.CircuitBreaker required N+1 automatic-failover attempts to trip when the spec’s own CAT8 example (“fires 3 times” with max_automatic_failovers: 3) clearly means the Nth attempt itself trips it. The bug had gone unnoticed because the original unit test encoded the same off-by-one rather than the spec’s literal wording. Fixed to >=; both the corrected assumption and the original (now-passing) assumption are covered by separate tests (TestCircuitBreakerAllowsUnderThreshold, TestCircuitBreakerTripsAtThreshold) in internal/plans/circuitbreaker_test.go.

Chaos Tests (§46)

A real precision bug this work found and fixed: a Fencer.Fence() call that timed out was being classified the same as a definite rejection (FAILED). A timeout means “we stopped waiting,” not “the target confirmed anything” — exactly §17’s “cannot determine” case, which must be UNKNOWN. Fixed via errors.Is(err, context.DeadlineExceeded) in internal/fencing/engine.go.

Real Network Transport (§6)

§6 requires quorum voters not share a failure domain, which is only meaningful if they can actually run as separate processes/machines. pkg/quorum/network_test.go’s TestNetworkTransportElectsLeaderAndReplicates proves a cluster built on raft.NewTCPTransport (real loopback TCP sockets, not raft.InmemTransport’s in-process channels) elects a leader and replicates a committed entry to every voter over that real network I/O; TestNetworkNodeConnectPanics locks in that the simulation-only Connect/Partition/Heal methods refuse to silently misbehave if called on a network node. This deliberately runs multiple real TCP listeners within one test process rather than spawning separate OS processes — once RPCs are proven to cross a real socket, an OS process boundary changes nothing raft depends on. NewNetworkNode + BootstrapNetworkCluster/PeerInfo in pkg/quorum/node.go are what a real multi-process deployment uses instead of the in-memory-only NewNode/NewFileNode + BootstrapCluster.

Real Providers (Roadmap Phase 4, §24-§29)

Every provider below is a real client for the documented wire protocol, not a mock — but every one of them was built and tested exclusively against a local double (httptest.Server, or a fake ipmitool script), since no live hardware/cloud account exists in this environment. See “Known gaps” for exactly what that does and doesn’t prove.

Proving the actual Phase 4 exit criterion — driven through the REST API: every integration test above still calls resources.Sequence directly, the same way the Critical Acceptance Tests do — none of them go through api.Server. tests/e2e/lab_realproviders_test.go’s TestPhase4LabTopology_RealProvidersFailoverThroughRESTAPI closes that gap: a real api.Server (auth, routing, audit, the works) on top of a real 3-node raft quorum, wired to the real Redfish + FortiGate + Proxmox clients above against their local doubles, driven by actual HTTP requests through ServeHTTP for a full dc1→dc2 failover. It asserts the BMC double actually received ForceOff calls and reports Off, the FortiGate double’s policy is disabled and its VIP carries the correct new-generation tag, the Proxmox double’s replication endpoint was actually queried, and the resulting audit trail is correct — the same set of properties CAT1/CAT3/CAT4 already prove against resources.Sequence directly, now proven once more against real provider clients through the real REST surface.

A real, safety-relevant bug this work found and fixed: in internal/controllers/infrastructure/ipmi, exec.CommandContext’s context cancellation only kills the direct child process; Cmd.Wait() still blocks until every process sharing its stdout/stderr pipe closes it, so a fenced target’s ipmitool invocation forking a further child could make a context timeout meant to bound fencing latency do nothing at all. Fixed by setting Cmd.WaitDelay on the real command runner, with a regression test using a script that forks a long-lived child.

A real, unrelated bug found and fixed while re-verifying the Phase 7 dependency engine’s full test suite: LocalEncryptedStore’s FileMasterKeySource.MasterKey() applied bytes.TrimSpace to raw master-key bytes before checking whether they were exactly 32 bytes — a uniformly random 32-byte key has roughly a 1-in-20 chance that its first or last byte is an ASCII whitespace value, which TrimSpace would strip, shrinking it below 32 bytes and incorrectly falling through to (and failing) the base64 path. This made TestLocalEncryptedStoreFileMasterKeySource genuinely flaky under repeated -race runs. Fixed by checking the untrimmed content’s length first; trimming now only ever applies to the base64-text path, where “surrounding whitespace tolerated” was always the intended meaning. TestFileMasterKeySourceRawBytesWithWhitespaceEdges (new) forces the whitespace-edge case deterministically rather than relying on randomness to hit it. (Since Phase 11.1 the master key is a key file in the keyring format, ADR 0011 §10, so that source and its test are gone.)

§43 Health Dimensions (Roadmap Phase 4 close-out)

Every dimension the spec asks for is now populated at the REST API layer, not just Quorum/Fencing/Replication: internal/health/provider.go’s ProviderHealth maps any interfaces.Provider.Health() into a dimension status, and internal/health/service.go’s ServiceHealth derives Service from a real ServiceController.Verify() call. api.PlanConfig gained optional ComputeProvider/NetworkProvider fields feeding Compute/Network purely for observability — neither is consulted by the ownership/fencing/promotion sequence itself. The aggregate Provider dimension is the worst of whichever of Compute/Network are actually configured, omitted entirely if neither is (matching how Replication is already omitted when Readiness is nil).

• Unconfigured dimensions stay absent, not defaulted: internal/api/health_test.go’s TestHealthOmitsComputeAndNetworkWhenNotConfigured.
• Real provider signals, correctly aggregated: TestHealthReportsComputeAndNetworkFromRealProviders — a healthy Compute provider and a deliberately-unhealthy Network provider produce compute=HEALTHY, network=DEGRADED, and the aggregate provider=DEGRADED (the worse of the two).
• Service tracks real acquisition, not just configuration: TestHealthServiceReflectsRealAcquisition — service=UNKNOWN before any acquire, service=HEALTHY after a real failover through the REST API.
• Manually re-verified against the actual running cmd/orchestrator binary (not just the test suite) — GET /api/v1/availability-plans/production-app-01 correctly returns compute, provider, and service dimensions from the real demo ComputeProvider.

internal/controllers/infrastructure/fortigate.Fencer now also implements interfaces.Provider (a real GET against FortiOS’s /api/v2/monitor/system/status monitor endpoint, deliberately distinct from Verify, which checks one specific policy’s fence state rather than general reachability) — Proxmox and IPMI already implemented interfaces.Provider and needed no changes to feed Compute.

§34 Provider Config Registry & Admin UI (Roadmap Phase 4 close-out)

Superseded in Phase 10.1 by controllers management (ADR 0010; see “Controllers and availability plans” below). Kept as the record of what was built then.

internal/providers/registry.Registry is a real, config-driven provider registry (§26): RegisterFactory/CreateInstance construct named instances from {id, type, config}, ValidateConfig and TestConnectivity expose interfaces.Provider’s own ValidateConfig/Health methods on demand, SetEnabled tracks an enabled flag, and every entry keeps a capped (10-entry) ring buffer of recent errors. It implements the frozen interfaces.Registry, but that interface’s own RegisterProvider(factory) method — given neither a type name nor an instance id — genuinely cannot support config-driven construction; this is documented plainly as a best-effort compliance shim rather than silently treated as sufficient.

• Full lifecycle, real assertions: internal/providers/registry/registry_test.go — 15 tests covering create/replace (“edit configuration”), unknown-type and factory-failure error recording, validate-with-a-running-instance vs. validate-via-throwaway-instance (works even before any instance exists), connectivity testing (healthy and unhealthy), enable/disable, and the recent-errors ring buffer actually capping at 10 rather than growing unboundedly.
• RBAC, not just plumbing: internal/api/providers_test.go’s TestViewerCannotSetProviderConfig/TestOperatorCannotSetProviderConfig prove provider config is Administrator-only (PermissionAdminister) — a stricter bar than ordinary failover/acquire operations (PermissionOperate), matching §34’s administrative framing.
• Secrets masking is structural, not a filter that could have a bug: the new DTOs (ProviderRegistryEntryDTO etc.) have no field a provider’s raw config could ever populate — TestAdministratorFullProviderLifecycle explicitly asserts neither config nor an example secret-adjacent key appears anywhere in a GET response. (Since Phase 7 one endpoint does return configuration, GET /api/v1/providers/{id}/config, through an allowlist rather than structurally; see “Provider configuration from the console” below.)
• Two provider-tracking surfaces stay in sync: TestSetProviderConfigAlsoAppearsInFlatProviderListing — a registry-created instance is automatically registered into the older, unrelated flat GET /api/v1/providers health listing too, rather than the two silently diverging into different views of “what providers exist.”
• Manually re-verified end-to-end against the real running cmd/orchestrator binary (which now wires a demo factory-backed instance): validate → save → get → test-connectivity → disable, via the exact same HTTP call sequence the new Admin UI page (web/frontend/static/provider.html) makes.

Both gaps originally flagged here are now closed — a providers: YAML bootstrap loader exists, and SetEnabled genuinely gates orchestration — see the next section for what was built and how it was proven.

§26/§28 Provider Config Bootstrap & Enabled-Gating (closing the two gaps above)

Superseded in Phase 10.1: the gate now lives in internal/controllers and covers every binding of every plan, not only the demo’s fencer (see “Controllers and availability plans” below). Kept as the record.

providers: YAML config-file bootstrap loader (§26). internal/config.LoadProvidersFile parses a providers: file shaped exactly like §26’s own example config into plain data (ProviderConfig/ProvidersFile) — rejecting a missing id/type or a duplicate id at load time, since an entry missing either can never be constructed anyway. cmd/orchestrator -config <path> (formerly -providers-config, still accepted) applies each entry (CreateInstance then SetEnabled) to the same registry the REST API/Admin UI already use, failing the whole load on the first bad entry rather than starting with a half-applied provider set.

• Format matches the spec, not an assumption about it: TestLoadProvidersFileMatchesSpecExample parses a file copied from §26’s own sample config verbatim.
• 9 tests total covering the format’s actual edge cases: omitted vs. explicit enabled, missing file, malformed YAML, missing id/type, duplicate id, and an empty (but valid) providers: [].

Provider Enabled genuinely gates orchestration (§28). Registry.IsEnabled(id) plus three wrapper types — EnabledGatedFencer, EnabledGatedReadinessChecker, EnabledGatedServiceController (internal/providers/registry/gated.go) — check enabled-state before delegating to a real provider. The safety-critical core stays completely unaware this concept exists: internal/fencing and internal/plans were not touched at all. A disabled provider produces exactly the failure shape those packages already handle —

• EnabledGatedFencer.Fence/Unfence return a plain error (not wrapping context.DeadlineExceeded), which fencing.Engine’s existing errors.Is check already classifies as FAILED, never UNKNOWN — correct, since “disabled” is a known, deliberate state (§17), not ambiguity about what happened.
• EnabledGatedReadinessChecker.Check returns a plain error and a zero, all-false ReadinessResult, which resources.Sequence’s existing err != nil || !PromotionPossible check already treats as grounds to block (§21) — never a silently optimistic default.

A real prerequisite gap surfaced and was closed along the way: most concrete provider types — mock.Fencer/Readiness/Service, redfish.Fencer, proxmox.Fencer/Readiness, fortigate.Service — did not implement the base interfaces.Provider the registry is typed around at all (only ipmi.Fencer, proxmox.Compute, and fortigate.Fencer did beforehand). Retrofitted onto every one of them, each with a real, working Health() — not a stub — backed by the same underlying check the type already had available (e.g. proxmox.Fencer.Health reuses the VM-status query proxmox.Compute.Health already performs, since both share one Client).

Proof, most-important first:

• TestFailoverBlocksOnDisabledFencer — a real fencing.Engine.Fence call, driven through a real resources.Sequence.Failover against a real 3-node raft cluster, reaches BLOCKED (never ACTIVE) the moment the availability plan’s fencer is registry-disabled. This is the actual claim §28 makes, proven at the same level internal/plans’s own acceptance tests are proven at, not just at the wrapper’s unit level.
• Unit tests per gated wrapper type (enabled passthrough, disabled failure, and — for EnabledGatedFencer.Verify specifically — that disabled reports FAILED via a FenceResult, not a bare error, for any caller invoking it outside the Fence-then-Verify sequence).
• Real factories registered for every provider type (redfish.Factory, fortigate.FencerFactory/ServiceFactory, proxmox.ComputeFactory/FencerFactory/ReadinessFactory, alongside the pre-existing ipmi.Factory), each with its own construction/defaulting/validation tests.
• Manually verified against the real running cmd/orchestrator binary, curl only, no test harness: the demo availability plan’s fencer is itself a registry-managed, EnabledGatedFencer-wrapped instance (id redfish-dc1). A failover to dc2 while enabled reached ACTIVE; POST /api/v1/providers/redfish-dc1/disable followed by an attempted failover back to dc1 returned fencing did not reach FENCED (overall=FAILED, ...) with the availability plan left BLOCKED — the exact behavior the automated test above proves, reproduced independently by hand against the actual binary an operator would run.

Self-Hosted Witness (Roadmap Phase 5, §8/§57)

A witness is an ordinary quorum.Node (built via pkg/witness) with no provider configuration, no fencing credentials, and no availability plan definitions — deliberately no special-cased logic anywhere in the safety-critical core distinguishes it from any other voter.

• Normal operation with a witness in place: tests/e2e/witness_topology_test.go’s TestWitnessTopologyPassesNormalFailover — a real 2-site + 1-witness topology (real network transport, the witness built through the actual pkg/witness code path) reproduces CAT1.
• §8’s availability arithmetic, tested, not just documented: TestWitnessTopologyWitnessAloneDownKeepsQuorum (“witness down, sites up → quorum OK, 2/3”) and TestWitnessTopologyWitnessAndOneSiteDownFailsClosed (“witness + one site down → NO QUORUM, fails closed”). Written up for operators in deploy/witness/README.md, not just here.
• The leadership-decline liveness feature is real, not decorative: since a witness runs no ownership.Engine, raft electing it leader would otherwise block every orchestration client until the next election. pkg/witness/witness_test.go’s TestWitnessDeclinesLeadership proves a witness that acquires leadership transfers it away immediately — observed as leadership spontaneously bouncing between three witnesses with nothing external disturbing the cluster, which cannot happen in a healthy, undisturbed raft cluster without it. This is a pure liveness optimization with zero effect on any safety property (every ownership.Engine caller already retries a “not leader” response as a matter of course).

A real bug this work found and fixed: Witness.Shutdown() panicked on a double-close of its internal done channel when called more than once — a real, expected pattern (an explicit mid-test/mid-operation shutdown followed by a deferred cleanup, or an operator re-running a stop). Fixed via sync.Once.

Roadmap Phase 8.4 addition: cmd/witness gained -tls-cert/-tls-key/-tls-ca flags, wiring the pre-existing pkg/witness.NewMTLS (previously reachable only via internal/witnesshost’s in-process Go API) into the standalone binary for the first time — a self-hosted witness can now itself join a cluster that already includes an mTLS-secured voter (e.g. a vendor-hosted witness, §57’s optional 5-voter mode), not only a plaintext 2-site cluster. validateTLSFlags fails closed on a partial flag set rather than silently starting in plaintext mode when mTLS was clearly intended. The B2 join wizard’s /join flow (Go html/template, zero JS — see below) gained a matching “paste the hosted-witness join bundle” step, each pasted PEM block validated for real (x509.ParseCertificate/key-parse attempts, not just non-empty checks) before being echoed back on the result page for the operator to save to disk and reference via the new flags. This is a deployment-flexibility and UI addition, not a change to any safety-critical decision path — no code in internal/ownership/internal/fencing/internal/plans was touched, and the actual isolation/trust properties this exercises are exactly the ones already proven in “Multi-Tenant Hosted Witness” below.

Known gaps (self-hosted path):

• The self-hosted deployment templates now pass real validation: deploy/witness/terraform/aws and the two Proxmox modules (deploy/witness/terraform/proxmox, deploy/orchestrator/terraform/proxmox) all pass terraform validate/fmt -check against their real providers, and the Proxmox modules’ full terraform plan succeeds completely offline with correct rendered cloud-init output (verified directly). The AWS module’s plan was also attempted with dummy credentials and correctly reached AWS’s real STS endpoint (rejected with a genuine 403 InvalidClientTokenId, confirming the request construction itself is sound) — real AWS credentials are the only remaining gap there. None of the three has been run against real cloud/Proxmox infrastructure end to end (a real apply). The Ansible roles were separately validated with real ansible-playbook --syntax-check and --check dry-run execution, not just read.

Multi-Tenant Hosted Witness (Roadmap Phase 5, §57)

The multi-tenant witness host (internal/witnesshost) and its test record are in the private qfence-managed repo (ADR 0017 §6). The pieces it is built from are core, and are tested here:

• mTLS transport itself, proven at the lower layer: pkg/quorum/mtls_transport_test.go’s TestMTLSTransportElectsLeaderAndReplicates (a 3-node mTLS cluster forms, elects a leader, replicates a committed entry) and TestMTLSRejectsPeerFromUntrustedCA (an intruder holding a certificate from a different CA, explicitly folded into the peer list and reachable on the network, never disrupts the legitimate cluster’s leader election or commits).
• Dynamic membership, proven independent of mTLS: pkg/quorum/membership_test.go’s TestAddVoterJoinsAlreadyRunningCluster (a plain-transport 2-voter cluster with real pre-existing committed state gains a third voter that catches up purely by replaying the log) and TestRemoveVoterDecommissionsCleanly.
• The shipped orchestrator as a customer site: raft over mutual TLS, the runtime membership change that adds a witness, and revocation lists fetched from a URL are tested here with the shipped binaries (see the configuration sources section below). Their run against the hosted witness itself is in qfence-managed.

A genuinely non-obvious finding, not a bug: a TLS 1.3 server’s rejection of an untrusted client certificate does not surface as an error from the client’s initial tls.Dial/handshake call. A TLS 1.3 client can consider its own side of the handshake complete (having validated the server’s certificate) before the server has finished validating the client’s certificate via VerifyPeerCertificate; the rejection only appears on the first subsequent read. This doesn’t weaken the actual security property: raft’s real transport never dials a peer without immediately exchanging RPC data, so the rejection is observed exactly when it matters — proven by TestMTLSRejectsPeerFromUntrustedCA at the full-cluster behavioral level, which asserts on cluster behavior, not a raw dial’s return value.

Hosted Control & Observability Plane (Roadmap Phase 6, §58)

The hosted plane itself and its test record are in the private qfence-managed repo (ADR 0017 §6). What stays here is the orchestrator’s side of §58’s promise: the plane is entirely outside the safety-critical decision path, one-way, downstream of already-decided audit/event history only. pkg/telemetry, the shipping agent embedded in cmd/orchestrator, imports none of internal/ownership, internal/fencing or internal/plans.

• CAT9 (see the acceptance-test table above): TestCriticalAcceptance9_HostedPlaneIsolation drives two failovers through the real REST API with a real pkg/telemetry.Agent shipping to pkg/telemetry/telemetrytest, an ingest double that answers the agent’s request as the hosted plane’s POST /v1/ingest does. telemetrytest.CheckIngest holds both to the same answers: the double in TestTheDoubleKeepsTheContract here, the hosted plane’s handler in qfence-managed. The second failover happens after the double’s real HTTP listener has been closed out from under the already-running agent (not a bad URL configured from the start). Since Phase 12.1’s review the test also proves the outage happened: the queue drains while the endpoint is up, and past a flush after the close the agent’s last success is unchanged, the double took nothing and the records are queued; with the close removed, the test fails. A fifth failover runs while a second agent’s send is stalled for its whole 5s timeout against an endpoint that takes the connection and never answers, and is no slower. CAT9 needs no hosted plane: its claim is all orchestrator-side (ADR 0017 §3). That the records reach the fleet view is the hosted plane’s own test, in qfence-managed. Both failovers succeed identically; the second stays well under the agent’s own 5s client send timeout, and the record it generated ends up in the agent’s queue (bounded, oldest-dropped-first — proven separately by pkg/telemetry/telemetry_test.go’s TestAgentQueueDropsOldestWhenFullWithoutBlocking) rather than blocking anything upstream.
• Core cannot depend on the hosted plane. Since Phase 12.2 the hosted plane is a separate module, go.ivnn.dev/qfence-managed, which requires this one; nothing here requires it.
• Redaction, proven at the unit level: pkg/telemetry/telemetry_test.go’s TestRecordFromAuditTruncatesAndExcludesReason asserts audit.Record.Reason (operator-typed freeform text on a forced operation) never appears in a shipped telemetry.Record’s JSON encoding, and that Error/Detail are bounded to 200 bytes.

Provider configuration from the console (Roadmap Phase 7, §26/§27/§34)

Superseded in Phase 10.1: /api/v1/providers and its screens are gone, and inline credentials are refused in every mode. Phase 10.2 and 10.4 replace them with controller registration and the Controllers section (ADR 0010). Kept as the record.

Administrators can now add a provider and edit an existing one’s configuration from the console, not only enable, validate and test it. Three changes to what the API exposes and accepts come with that:

• Reading configuration back is an allowlist. GET /api/v1/providers/{id}/config (Administrator) returns credentials_ref and the fields the type’s schema declares without SecretRef. Every other key in the stored configuration, whether an inline credential or a key the schema does not describe, is named in withheld and its value is never returned. This is a filter, so unlike the other provider DTOs it depends on something being right: each schema has to flag its credential fields. cmd/orchestrator/schemas_test.go fails if any schema field whose name looks like a credential (password, secret, token, key, passphrase) lacks the flag, unless it is listed as an identifier with the reason (Proxmox token_id). Tests: TestProviderConfigReadIsAnAllowlist (an inline password and an undocumented key are withheld by name, their values appear nowhere in the body, Viewer and Operator get 403).
• An edit cannot silently drop what it was not shown. A save names withheld fields to keep in retain, and the server copies their stored values. A retain naming a field that is not withheld, or one also set in the request, is refused. The console keeps every withheld field by default. Tests: TestProviderConfigRetainKeepsWithheldValues, including a refused save leaving the stored configuration unchanged.
• Credentials only as a reference, in the console. The form has no input for a SecretRef field. Credentials are a credentials_ref, chosen from the names this node’s encrypted store holds (GET /api/v1/providers/credential-refs, names only; the orchestrator’s own orchestrator/ entries are left out) or typed where the store cannot list names (Vault, KMS). The form lists which keys the secret must hold. Before this, the form rendered each SecretRef field as a “reference” text box whose content the server used as the credential itself. The API still accepts inline credentials, as ADR 0001 decided. Tests: TestCredentialRefs, TestProviderRequiredSecretSatisfiedByCredentialsRef (the stored configuration keeps the reference, not the value), ProviderDetail.test.tsx and ProviderNew.test.tsx (no password input, the ref select, retain).

Other changes:

• Field-level validation. Validate, save and create check the configuration against the type’s schema (type of every field present, required fields after credentials_ref is resolved) and return each problem bound to its field. The provider’s own validation still runs after, and its error, which names no field, is returned with field: "". The messages name fields and never quote values. Tests: TestProviderValidateBindsErrorsToFields.
• Create never replaces. POST /api/v1/providers refuses an existing id with 409, checks the id’s shape, and builds the provider before recording anything, so a refused create leaves no entry. Tests: TestCreateProvider.
• Provider changes are audited. provider.create, provider.config.update, provider.enable and provider.disable are recorded with the actor. None of them were before, although disabling a fencer blocks every availability plan that fences with it. Tests: TestProviderChangesAreAudited.

Verified in a browser against a real orchestrator with an encrypted secrets store: a credential stored with secret put, then a Redfish provider added, validated (a bad timeout marked on that field), created, reopened pre-filled with the reference selected and edited. The page never contained the secret value; the audit log held the create and the update. axe-core covers the new screen with a type chosen, in both themes.

Controllers and availability plans (ADR 0010, Phase 10.1)

Providers became infrastructure controllers, and resource groups availability plans. A controller is one connection to one target system, opened from its type’s manifest; a plan binds capabilities of controllers to resources, per site instance. What this changes for safety, and how it was proven:

• The decisions did not change. Before the change, both labs’ configurations were compiled and what the fencing engine and the scheduler receive was recorded (cmd/orchestrator’s TestLabDecisions, testdata/lab-decisions), as was the decision core’s behaviour (internal/sim’s TestDecisionCoreMatchesRecorded). The rewritten configurations reproduce the first with six differences, all target strings, which are now resource ids (ADR 0010 §8): vm-101/vm-102 became 101/102 (the Proxmox fencer acts on its configured VM, never this string), and service:erp-db became erp-db (the Docker port fence learns it acts on a service from its capability, service.unpublish, instead of a prefix). Every name, level, required flag, site, network domain, power feature, shield, readiness check and per-site fence, raise and unfence list is identical. The decision core’s record is byte for byte the same. Both labs then ran a failover and a failback through the API on the new binaries.
• Every binding is gated (§28). A disabled or removed controller blocks every runtime it handed out: fences, shields, readiness checks, the service and dead-man switches. Before, only the demo’s fencer was wrapped; a provider set enabled: false in a configuration file still fenced. A refused fence is FAILED (a plain error, never a deadline), a refused readiness check an error with a zero result. Tests: internal/controllers TestDisabledControllerBlocksWhatBindsIt, TestDeadManRuntimeKeepsPower.
• Levels come from the manifest, and a manifest is checked. A plan no longer types level:; a binding’s level is its capability’s. A type whose manifest breaks the published schema’s rules (a level without a fence or shield role, a compute capability that may be a shield) is refused when it is registered, and every built-in manifest is checked against docs/schemas/controller-type.v1.json itself. Tests: TestManifestsMatchPublishedSchema, TestSchemaValidatorRejects, TestBindFollowsTheManifest.
• The engine reads power from the manifest. A runtime’s features are its capability’s, so power (unfence first, release on return to standby, ADR 0009) cannot drift from what the type declares; every built-in fence capability’s own runtime agrees with its manifest. Test: TestBoundRuntimesAreNamedByTheBindingWithTheManifestsFeatures.
• No inline credentials, in any mode. A controller names its credentials with credentials_ref: local:NAME; a connection or auth block holding a value is refused. Errors name fields, never values. The fingerprint therefore covers references only, and the drift report’s list of uncompared inline credentials is gone. Tests: TestNoInlineCredentials, TestCredentialsThatDoNotResolve, TestControllersListHoldsReferencesNotCredentials, TestPhase8EndpointsNeverLeakSecrets. (Since 10.2 a secret that does not resolve, or holds a field its method lacks, leaves the controller failed instead of refusing the configuration; see below.)
• Every binding has a site. Bindings sit in site instances, so an unscoped target, which every failover fenced, can no longer be configured, and readiness is always per site.
• The old format is refused with an error naming ADR 0010, rather than translated (internal/config TestLoadRefusesTheOldFormat).
• A mock fence cannot be configured outside demo mode (closed 2026-10-03). The mock type’s fences report success without fencing anything. cmd/orchestrator knows the type only in demo mode: -demo, which is always on without -raft-bind (the single-node, in-memory demo). A voter started with -raft-bind and without -demo does not register the type. A controller of type mock is then refused wherever it comes from (a configuration file at start or on reload, a registration through the API, the replicated store), with an error that says why, and the type is not listed in GET /api/v1/controllers/types. Outside demo mode a configuration that defines nothing starts an empty cluster, not the built-in demo, so a production cluster brought up before its configuration was written does not get plans bound to mock fences. Only -demo (QFENCE_DEMO=true in the systemd environment, orchestrator_demo in the Ansible role) brings the demo back. Tests: TestMockRefusedOutsideDemoMode, TestDemoOnlyForAnEmptyConfiguration, TestRefusedTypesSayWhy, and TestNodeStateSurvivesProcessRestart’s check of a real voter without -demo.

Controller registration and authentication (ADR 0010, Phase 10.2)

Controllers are now registered through the API, authenticate with a secret they only reference, and are validated against their target, at registration and every few minutes after. What keeps that safe:

• No credential leaves memory.
  • A secret field takes a reference {provider, path, key?}, never a value. A string that is not a reference is refused with one fixed message, so a password typed in the wrong place is not repeated back.
  • Resolved fields, auth fields and derived tokens print as a placeholder in every format (fmt verbs, JSON, slog).
  • The registry takes a controller’s secret values out of every error, health detail and check detail before passing it on, whatever the type wrote.
  • Tests: pkg/interfaces TestAValueIsNotAReference, TestResolvedFieldsNeverPrint; internal/controllers TestCredentialsAreRedactedEverywhere; internal/controllers/tokens TestTokensNeverPrint.
  • The exit test, internal/api TestNoCredentialReachesAResponseLogOrAudit, uses a type that puts the password in its connectivity details, its check details and its errors on purpose. It then takes every path: registration and its failures, validation, the controller’s pages, re-authentication, enable and disable, export, plans, the dry run, events, the audit log, the node report and the process’s logs. No credential appears. With redaction switched off, it finds ten leaks.
• Read-only resolution.
  • Controllers receive resolved fields; only the registry calls the resolver, and nothing it offers writes, rotates or deletes a secret.
  • The local provider never resolves or lists the orchestrator’s own orchestrator/ entries.
  • Tests: pkg/secrets TestLocalResolverReadsOnly, TestLocalResolverListsNamesOnly.
• A secret that does not resolve fails safe without refusing the configuration.
  • The controller opens unavailable. It is failed, its capabilities failing, and every runtime it hands out fails each call: a fence through it verifies with an error, which the engine reports UNKNOWN, never FENCED. A dead-man switch without the power feature stays one (it does not become a fencer).
  • Before, the configuration was refused, so one orchestrator missing a secret another site needs could not start.
  • Tests: TestUnresolvedCredentialsFailEveryCall, TestCredentialsThatDoNotResolve, and cmd/orchestrator’s TestCredentialsThatDoNotResolveDoNotStopStartup (the real binary: a value in place of a reference is fatal and not repeated; a reference that does not resolve starts with the controller failed).
• What a controller keeps when its secret stops resolving depends on why (ADR 0011 §5, Phase 11.1). The rule is on Registry.Revalidate (and Reresolve, which a secret write runs), applied to each auth block the controller holds: its own credentials, and each capability’s (setup auth) for that capability alone. It replaced, on 2026-10-16, the rule under which every kind kept the held value.
  • unavailable and malformed keep the held value until its own expiry (a lease’s end, a certificate’s); a KV value has none, so it is kept until the controller is opened again. The controller is degraded, not failed, and its reason says “provider unreachable, using the value resolved at T” (or the malformed detail, with the same time); its capabilities are failing, as before, so the attention list shows it, and the API’s credentials.held is true. Every binding, including one built afterwards, keeps carrying that value to the target, so only the target’s answer decides a fence: FENCED if it carries the fence out, UNKNOWN once it refuses the value. Past its expiry, the value is dropped as below.
  • not found and denied drop the value at once, since the provider answered. The runtimes already bound refuse every call from then on (the registry’s gate, health included), and so do the controller’s own health, connectivity test, checks, discovery and binding validation: the target is never sent the value again, and a fence through it is UNKNOWN (an error from verify; a disabled controller still verifies FAILED). The drop is announced (ControllerFailed, CapabilityFailing). The controller is then opened again without it, failed, as one that opened without credentials, and its plans rebuilt.
  • A denied counts only after the provider was signed in to again (interfaces.SecretSignIn, at most once per provider per resolve), so qfence’s own expired session is never read as a revocation. A sign-in that fails is the provider being unavailable: the value is kept. Since Phase 11.4 Vault’s provider signs in again with its certificate or AppRole (at most every 10 s), so a Vault denied after that drops the value; with a static token (development) it cannot sign in again, and its denied keeps the held value.
  • Residual risks: a secret deleted from its store by mistake during an incident makes its fences UNKNOWN at the next revalidation. In local and cluster the secrets API refuses a delete while a controller names the secret, so the risk is in a store qfence does not own (Vault), or the local store changed offline. A held value is lost at a restart during a provider outage: the controller opens without it. While a controller holds its own value through an outage it is not opened again, since that would lose it: a capability’s dropped value is refused by the gate meanwhile, and a capability’s changed value waits for the provider.
  • Tests, through the real fencing engine: internal/controllers TestTheHeldValueByErrorKind (each kind; bound before and after; the target never sent a dropped value; health refused), TestADeniedBeforeSigningInAgainIsNotARevocation (an expired session cleared by signing in: FENCED; a failed sign-in: kept; denied after signing in: UNKNOWN), TestAHeldValueGoesAtItsExpiry (and, since 11.4, a renewed lease’s end moved in place, later or earlier, without opening the controller again), TestACapabilitysOwnCredentialsByErrorKind, TestSignInOncePerProvider; pkg/secrets TestVaultSignsInOnlyWhenItCan; internal/api TestAFailoverThroughAHeldValue (a plan’s failover through validation, reopening and the plan’s rebuild: promoted for the kinds that keep, BLOCKED with the fence UNKNOWN for those that drop) and TestAReloadKeepsHeldCredentials. Making the rule never drop, skipping the sign-in, or letting the gate ignore a drop each fails them.
• Validation grades, and only fails what it proves.
  • Each capability’s checks report proven, inferred, unverifiable or failed. A failed check leaves the capability out at registration and makes it failing afterwards. An unverifiable one warns and advertises, as ADR 0010 decided.
  • A check a type does not report is unverifiable, never passed.
  • Validation runs again every revalidate_seconds (default 5 min, at least 30 s). A validation that passes clears a capability’s failed call.
  • A capability that starts failing stays bound, and shows in its plans’ attention list and as an event (CapabilityFailing, alerting); its fences are still attempted and verified.
  • Tests: TestValidationGradesEachCapability, TestRegistrationSession, and each type’s TestConnectAndValidate against its double.
• Grades are honest about what was read. Two are deliberately not “proven”:
  • FortiGate’s admin-profile check is inferred, because FortiOS does not say which administrator a token belongs to; qfence reads the profile of the api_user the connection names.
  • Redfish and IPMI privileges are inferred, because a BMC may refuse per resource.
  • Proxmox privileges are proven only when held on / or /vms.
• Derived tokens.
  • They live in memory, per orchestrator, and are refreshed on the first use after three quarters of their life, so before they expire.
  • A failed refresh while the token still works degrades the controller; once no token works, the controller is failed.
  • Tests: TestClientCredentialsRefreshBeforeExpiry (an HTTP double), TestFailedRefreshDegrades, and against deploy/keycloak-lab’s real token endpoint, TestClientCredentialsAgainstKeycloak: a 12-second token refreshed 2.4 s before it expired; a wrong client secret fails the controller with the endpoint’s error code and without the secret.
• Registration writes nothing until it commits.
  • A session lives only in the memory of the orchestrator serving it, for 15 minutes from opening. Only the identity that opened it may use it, and an orchestrator holds at most 32.
  • Register writes only capabilities that passed, as the configuration source says (in file mode it returns the file change and writes nothing), audited controller.register.
  • An id already registered, or reserved (types, registrations), is refused.
  • Re-authentication commits new credentials only if every enabled capability still passes with them, so failing credentials never replace working ones.
  • Tests: TestRegistrationSession, TestRegistrationRefusals, TestRegistrationInFileMode, TestEnableDisableAndReauthenticate.
• Calls stay in reach where 10.2 adds them.
  • Validation, registration and re-authentication run only on an orchestrator whose site is in the controller’s reach; elsewhere they are refused, and the controller is shown as remote, with the states the orchestrators at its sites report.
  • Asking another orchestrator whether it resolves a reference goes over the signed peer channel (/api/v1/internal/secret-check, its own signing purpose). The answer says whether the reference resolves, never what to.
  • Tests: TestControllersOutOfReachAreNotCalled, TestPeerSecretCheck.
• Disabling is administration, and names what it blocks. POST .../disable (Administrator) returns the plans that bind the controller, which its gate then blocks (§28), and is refused in file mode. Enable, disable and re-authentication are audited.
• Deleting a controller leaves no plan without it (2026-10-12). DELETE /api/v1/controllers/{id} (Administrator, ?dry_run=true) is refused while a plan binds it or names one of its resources as an instance’s copy, with the bindings in the way, and in local mode while this orchestrator’s configuration file defines it (it would come back at the next restart). File mode refuses it, as every change. What keeps it safe:
  • Checked against what is committed. In replicated mode the checks run on everything committed, and the tombstone is written only if the configuration is still at the revision they read (a raft precondition, now accepted from a follower too). A plan that came to bind the controller in between makes the deletion fail with a conflict, instead of leaving a configuration no orchestrator can apply. A plan’s deletion does the same, and also retires the plan in raft, so an acquire cannot slip in between (see “Deleting a plan”). A follower that cannot apply the replicated configuration refuses, since it cannot tell what binds the controller.
  • Nothing it manages changes. The deletion only forgets the configuration; its credentials stay where their reference names them.
  • What it held open is released. The registry now closes a controller it replaces or removes, and a connectivity test closes the one it opened. Before, a systemd controller’s SSH connection stayed open after its controller was replaced, re-authenticated or tested.
  • It is audited (controller.delete) and published (ControllerDeleted). In local mode the stored configuration goes too, so a restart does not bring it back.
  • Tests: TestDeletingAController; the replicated case in TestReplicatedConfigurationLifecycle (through a follower, and refused when a plan bound the controller after its checks); TestReplacedAndRemovedControllersAreClosed.
• The labs’ decisions did not change. TestLabDecisions and TestDecisionCoreMatchesRecorded pass unchanged.

Secrets management: the store, the write path and the peer channel (ADR 0011, Phase 11.1)

docs/adr/0011-secrets-management.md records the design, and its notes from the build what 11.1 settled. What a controller keeps when its secret stops resolving (§5) is under Phase 10.2 above.

• Every provider only reads; qfence writes only the stores it owns. SecretsProvider and every Rotate are gone, with CloudKMSSecretsProvider. A provider is a SecretResolver (and a SecretLister where it can list); the local store is also a SecretStore (put, delete, list, versions), which only the secrets API uses. Vault’s provider has no write method, and sends Vault nothing but reads of secrets, its own sign-in, and the renewal and revocation of its own token and leases (TestVaultOnlyReads), so its Vault policy can be read-only.

• The master key is a keyring, checked at start, and a wrong one fails closed. -master-key-file holds key ids, the first active; QFENCE_MASTER_KEY is gone, since every program the orchestrator runs inherited it. Each envelope records its key’s id, and a key-check record per key id tells a wrong file from a right one. A wrong file seals the store: the orchestrator starts, holds the file locked, says “the master key does not open this store” in its log, its node report and the voter list, and every local: reference is unavailable, so the fences that need one are UNKNOWN. The check only reads, so the store is left byte for byte as it was, and nothing opens or creates another one. secret rekey (offline, or online through the API, which reads the key file again first) rewraps data keys under the active key without decrypting a value, in one transaction that changes nothing if a key still in use is missing from the file. Tests: TestAWrongMasterKeyFailsClosed, TestRekeyRewrapsDataKeysOnly, TestAStoreFromBeforeKeyIDsIsNotOpened, TestMasterKeyFile, TestOnlineRekey, and cmd/orchestrator’s TestAWrongMasterKeyStartsTheOrchestratorFailedClosed on the real binary.

• Secrets are written into the running orchestrator. PUT /api/v1/secrets/local/{name} and DELETE, Administrator only, from secret put -server, secret delete -server and secret rekey -server. A write answers the reference and version, never the value, and is audited secret.create or secret.update with the reference and version, never the value nor a hash of it. A delete is refused while a controller’s configuration names the secret, naming them, and the refusal is audited. A write re-resolves at once the controllers this orchestrator calls that name the secret and opens again each whose value changed. Tests: TestSecretWrites; TestASecretPutReachesItsControllerWithoutARestart (the real binary and secret put -server against a Redfish double, which sees each new password at its next call; it fails without the re-resolve); and TestNoCredentialReachesAResponseLogOrAudit, extended to the write path, refusals included, and to the common hashes of every value.

• A secret value crosses only a post-quantum key exchange. The API listener offers only TLS 1.3 with X25519MLKEM768, with the certificate it generates and with -tls-cert, so a classical-only client is refused at the handshake. The secrets API also checks each request’s own key exchange, so it takes no value over any other, and none over plain HTTP (-insecure-http). The CLI’s client offers only X25519MLKEM768 too. Tests: internal/api TestTheAPIListenerOffersOnlyThePostQuantumKeyExchange and cmd/orchestrator TestTheAPIListenerTakesOnlyThePostQuantumKeyExchange (both serving paths, on the real binary): Go’s default offer negotiates X25519MLKEM768, and a client offering only X25519, P-256, P-384 and P-521, or TLS 1.2, is refused; giving the listener Go’s default curves fails them.

• Calls between orchestrators verify the orchestrator they call (§13). Each node report pins the SHA-256 of the certificate the orchestrator’s API serves, and of its next one (-tls-next-cert). Every call from one orchestrator to another (a dispatched fence, verify or check; a configuration write forwarded to the leader; a plan read or lease asked of it; a secret check; the report sent to the leader) accepts only a certificate pinned for the node it calls, checked in the TLS handshake before anything is sent, so a man in the middle between sites can neither complete the handshake nor read what the channel carries. The peer channel’s signature, under a key derived from the cluster keyring (ADR 0016 §1, below), still authenticates the caller. A pin counts only while its node is in the raft membership. A call to an orchestrator presenting another certificate fails as not sent, so dispatch tries another executor and never sends a fence twice. Tests: internal/api TestAPeerCallAcceptsOnlyThePinnedCertificate (the server sees no request), TestADispatchSkipsAnExecutorThatPresentsAnotherCertificate, TestARestartedOrchestratorIsReachableOnceItsReportLands and TestAPinGoesWithItsNodesMembership on a real raft cluster over TLS, and cmd/orchestrator TestPeersAcceptARestartedOrchestratorOnceItsReportLands (two orchestrators of the real binary, one restarted with a new generated certificate). Counting a pin failure as sent, skipping the pin check, or skipping the membership check each fails them. Residual risks:

  • An orchestrator that restarts with a generated certificate is unreachable by its peers until its new report lands, in seconds: a fence dispatched to it meanwhile is tried at another executor, or is UNKNOWN if there is none. Replacing a real certificate with no gap needs -tls-next-cert first.
  • A pin is only as trusted as the cluster keyring: whoever holds it could publish a report with another node’s pin, but could already sign calls as any orchestrator.
  • Over plain HTTP (-insecure-http) there is no certificate to pin, and calls are readable on the path, as before.
  • -advertise-api must reach the orchestrator’s own listener, directly or through a proxy that passes TLS through. A load balancer or proxy that ends TLS presents its own certificate, which the other orchestrators refuse: every call to that node fails, so fences dispatched to it go to another executor or are UNKNOWN. Such a deployment worked before 11.1.
  • A removed node’s report is dropped when its removal is committed, and the leader refuses reports from outside the membership, so its old pin cannot come back under its id (pkg/quorum TestARemovedNodesReportGoes, TestAPinGoesWithItsNodesMembership).

• Residual risks: the clients the API listener now refuses. TLS is negotiated before a request’s path is known, so the whole listener refuses a client without X25519MLKEM768, the console and every API client included, not only secret writes:

  • browsers older than Chrome and Edge 131 or Firefox 132, and any browser whose TLS does not offer the exchange;
  • curl, scripts and tools built against OpenSSL before 3.5, such as the curl Ubuntu 24.04 ships (OpenSSL 3.0; checked: curl -k fails with exit 35), and programs built with Go before 1.24;
  • load balancers or monitors that open TLS to the API port to check it, unless their TLS offers the exchange (a TCP check still works), and a TLS-terminating proxy in front of the API, behind which the secrets API takes no value, since the request does not arrive over TLS.

  qfence-orchestrator api is a client for scripts, and the labs use it in place of curl; peer calls between orchestrators use Go’s TLS, which offers the exchange by default.

Secrets management: the cluster store, scopes and revert (ADR 0011, Phase 11.2)

The opt-in cluster store: secrets in the raft log, sealed under a secrets keyring only the orchestrators hold, so one write reaches every orchestrator and each resolves from its own copy. ADR 0011’s notes from the build record what 11.2 settled.

• Opt-in: without a secrets keyring, raft carries nothing new. Until an orchestrator started with -secrets-key-file writes to the store, nothing of it is in the log, a snapshot or a node report (TestTheSecretStoreSurvivesASnapshot checks an unused store leaves the snapshot as it was).
• The seal. Each secret’s name, scope and value are sealed with AES-256-GCM under a key of the secrets keyring, with associated data binding each ciphertext to its part, its entry, its version and the key’s id; an entry’s key in raft is the HMAC-SHA256 of its name under the same key. Keys are exactly 256 bits; a key file holding a key of any other length, or one other users can read, is refused. The keyring reads the configuration keyring’s file format but is a Go type of its own, so neither stands in for the other. Tests that fail if the seal changes: TestTheSealIsAES256GCMUnderA256BitKey (a value opens with the standard library’s AES-256-GCM under the key file’s key and the specified associated data, and under no other entry, version, key id, part, or a 128- or 192-bit key), TestTheSealsKnownAnswer (byte for byte, checked with Python’s hmac and pyca/cryptography over OpenSSL), and TestTheKeyringTakesOnly256BitKeys.
• Writes are sealed where they are received, and go through the leader with compare-and-set. The orchestrator whose secrets API takes the value seals it before it goes anywhere; a follower forwards the sealed command over the peer channel, which carries ciphertext under opaque keys only. The FSM refuses a write based on an older version of the entry, a write under any key but the store’s, and a write or rekey while an orchestrator’s latest report does not list the key, so none receives a secret it cannot read; the orchestrator checks first, so the refusal names them. A deletion leaves a tombstone whose versions go on, so no old ciphertext can be replayed onto the entry. Tests: pkg/quorum TestSecretWritesAreCompareAndSet, TestSecretWritesNeedEveryOrchestratorToHoldTheKey, TestASecretRekeyMovesEveryEntry; pkg/secrets TestTheClusterStoreAcrossThreeVoters, TestAnOrchestratorWithoutTheKeyFailsClosed (the FSM refuses a write the store’s own check would have, too), TestRekeyingTheClusterStore; internal/api TestClusterSecretsThroughTheAPI.
• Reads come from each orchestrator’s own copy, across a restart while cut off. Every orchestrator watches its copy: when the store changes, the controllers it calls that name a cluster: secret resolve again, and it takes a raft snapshot. hashicorp/raft (v1.7.3) does not persist the commit index: a node that restarts without a leader restores its latest snapshot and applies nothing after it, so without that snapshot a site restarted while cut off would read an older copy, or none. Shown on real processes: cmd/orchestrator TestTheClusterStoreOnARealThreeVoterCluster stops dc1 and the witness and restarts dc2 alone, which opens its controller with the secret from its own copy and cannot write; with the snapshot disabled, dc2 never sent the password.
• The witness holds no secret’s name or value. The same test reads the witness’s raft.db and its snapshot after writes through both orchestrators: both hold the store (its writes, its entries), and neither holds the name, either value or the value’s JSON, as they are, JSON-escaped, in hex, or in base64 at any alignment (TestHoldsFindsEveryForm checks the search finds each form). Found by it and fixed: a controller’s reason in its node report, which raft carries to the witness, named the secret it could not resolve (“secret cluster:NAME: not found”); every name a controller’s configuration references is now withheld from that reason (TestNodeReportsWithholdSecretNames, and TestNoCredentialReachesAResponseLogOrAudit now checks raft’s copy holds neither a cluster secret’s value nor its name).
• An orchestrator without the store’s key fails closed. With no secrets keyring, with other bytes under the store’s key id (a sealed key check tells them apart; otherwise a name’s HMAC would find nothing and read as not found, which drops a held value), or without that id, every cluster: reference is unavailable there, never not found, so a held value is kept (ADR 0011 §5) and fences through it are UNKNOWN only once it expires or the controller is opened again. Its node report and the voter list say why, and its report leaves out a key that does not open the store, so writes are refused while it could not read them (TestAnOrchestratorWithoutTheKeyFailsClosed).
• Every orchestrator can read every cluster secret, every site’s. The store has one key, and every orchestrator holds it; every voter, the witness included, holds the ciphertext. So whoever controls one orchestrator, or holds a copy of the secrets key file and of any voter’s raft data (the witness’s disk is enough), can read every cluster: secret: dc1’s orchestrator can read dc2’s BMC password. This is the design ADR 0011 accepted (S3), for one-step entry and resolution during a partition; it is not narrowed by scopes, which stop an Administrator from having a secret sent to another host, not a compromised orchestrator from reading it. What follows for an operator:
  • Keep a credential that only one site’s orchestrators use, and whose exposure at another site matters, in local on those orchestrators (or in that site’s own Vault, which only that site’s orchestrators define, Phase 11.4): local is never replicated, and a controller is called only from the sites in its reach (ADR 0010 §9).
  • Treat the secrets key file as the most sensitive file on an orchestrator: with any voter’s raft data, it opens every site’s credentials. Keep it off the witness and out of backups taken with the raft data (ADR 0011 §12).
  • A hosted witness’s operator holds the ciphertext. It stays unreadable without the key, now and against a future quantum computer (ADR 0011 §2), but a leak of the key file is then enough on its own.
• A scope per secret, checked at resolve, in local and cluster. Every write names the controller types and the addresses the secret may be sent to; the resolver is told the asking controller’s type and the addresses its credentials go to, and refuses any other use as denied, saying which use and never the value. In local the scope is bound into the value’s associated data, in cluster sealed with it, so it changes only with a new value (TestAScopeIsCheckedAtResolve: a scope widened on disk without the key does not open; TestASecretScope; TestCredentialAddresses). Ports in scopes came with Phase 11.2b, below. The swarm lab’s scopes are checked by TestLabDecisions, which fails if a lab controller is refused its credentials.
• A refusal is audited, and so is a first use. Every resolve a controller makes goes through an audited resolver: a controller’s first use of a reference (since the orchestrator started) and every refusal (denied) are audited secret.resolve, naming the reference, the controller, its type and its endpoint’s host, never a value; routine re-resolves are counted in the controller’s detail (credentials.resolve_counts). Registration checks resolve through it too, so registering a controller at a host a secret’s scope does not name leaves a refusal in the audit log of every orchestrator asked, and the controller.register and controller.reauthenticate records name the endpoint host. Tests: TestAResolveIsAuditedAtFirstUseAndEveryRefusal, TestRegistrationSession.
• One previous version, and revert. A write keeps the version it replaces; a revert stores the previous value and scope again as the next version, so a second revert undoes the first; older versions are dropped (TestRevertKeepsOnePreviousVersion, TestSecretWrites, and the CLI against the real binary in TestASecretPutReachesItsControllerWithoutARestart). Audited secret.revert with the version whose value it restored.
• Rekey. secret rekey -server ... -store cluster reads the secrets key file again and moves the whole store under its active key in one command against the store’s revision, refused until every orchestrator reports the key; the versions and tombstones go with it (TestRekeyingTheClusterStore, TestClusterSecretsThroughTheAPI).
• Residual risks:
  • A write counts an orchestrator once its first report is in raft: a voter that has never reported cannot be told from a witness. An orchestrator joining without the key is unavailable for cluster: references and says so; writes are refused from when its report lands.
  • An orchestrator’s latest report counts however old it is, so a site that is down does not stop the other from entering a credential mid-incident. If it comes back with a key file that lacks the store’s key, its cluster: references are unavailable until the file is mended; the file is read again when it changes, with no restart.
  • The witness sees sizes, timing, key ids and versions, and at a rekey which new entry each old one became.
  • Old ciphertext stays in the raft log until it is compacted (a snapshot keeps the last 10,240 entries by default; -raft-snapshot-threshold and -raft-snapshot-interval set when), and in the two snapshots kept. A key that leaves every key file retires it; a deleted secret’s value stays readable there to whoever holds the key it was sealed under.
  • A rekey opens every value in the memory of the orchestrator that runs it and seals it again; only ciphertext leaves it. The master key’s rekey, by contrast, rewraps data keys only.
  • The watcher looks once a second: a store change applied and then lost to a crash before its snapshot is missing from a restart that happens while cut off. This is caught, and fails closed: each orchestrator records the raft index of the newest store change it has seen, and one whose copy is older makes every cluster: reference unavailable, so a secret deleted or replaced since is never used again, until a leader brings it up to date (see “A restart while cut off from quorum” under ADR 0001; TestAnOrchestratorBehindItsConfigurationFailsClosed, TestReplicatedConfigurationAcrossARestartWhileCutOff).
  • Losing every copy of the secrets key file loses every cluster: secret; they are entered again.
  • A scope matches a host as the controller’s configuration names it: a scope naming a DNS name does not allow an endpoint given by address, and the reverse. * gives the protection up. A Vault secret’s scope is Vault’s own policy.
  • A rekey forwarded to the leader carries every entry; the peer channel takes up to 32 MiB, about 180 secrets of the largest size (64 KiB, with their previous versions). A rekey run on the leader has no such bound.

What the witness may read: the cluster keyring, sealed reports and tighter scopes (ADR 0016, Phase 11.2b)

docs/adr/0016-what-the-witness-may-read.md records the design, and its notes from the build what 11.2b settled.

• The cluster keyring (§1). -cluster-key-file replaces -cluster-secret-file, whose file took any 32 bytes, a passphrase included, with no key ids, rotation or permission check, and was optional. It is pkg/keyring’s format as a Go type of its own (clusterkey.Keyring), so neither the configuration keyring nor the secrets keyring stands in for it: keys of exactly 256 bits, the first active, a file others can read refused. An orchestrator with -raft-bind refuses to start without it; the single-node demo takes none, and the witness never holds it. Tests: internal/clusterkey TestTheClusterKeyringTakesOnly256BitKeys, TestAClusterKeyFileOthersCanReadIsRefused; cmd/orchestrator TestAVoterRefusesToStartWithoutTheClusterKeyring (the real binary, also refusing the old flag).
• Derived keys, each for one use. The peer key and the seal key are HKDF-SHA256 of each cluster key, with no salt and the labels qfence/cluster/v1/peer-hmac and qfence/cluster/v1/seal as info; neither is the key itself, so a key leaked from one use does not give the other (TestTheDerivedKeysAreHKDFSHA256, a known answer computed with Python’s hmac and hashlib, RFC 5869 written out).
• A peer call names its key. Every call between orchestrators (a report to the leader, a dispatched fence, verify or check, a configuration write forwarded to the leader, a plan read or lease asked of it, a secret check) carries the id of the key it is signed under, and the HMAC covers the call’s purpose, that id, the time and the body. Every internal endpoint is served through one wrapper that verifies the call with that key from the receiver’s own file, and refuses an id it does not hold, before the endpoint runs. A refusal (401) therefore means nothing ran, so dispatch counts it as not sent and tries another executor. An answer the leader signs is signed under the key the question named, and the asker refuses one under any other. Tests: internal/api TestThePeerSignatureNamesItsKey (a body, purpose, key id or time changed on the way, or a key id the receiver lacks, is refused); TestAnAnswerMustComeUnderTheQuestionsKey; TestAPeerCallUnderAKeyTheReceiverLacksIsRefusedBeforeItActs (a fence dispatched under a key the executor lacks is never run there: with a second executor at the site that holds the key, it runs there; with none, it is UNKNOWN; once the first executor’s file holds the key, it runs there); the lease and plan-read tests (a question under a key id the leader lacks is refused); internal/configsync TestAPeerReportUnderAKeyTheLeaderLacksIsRefused.
• Rotation without a restart. Each orchestrator reads its key file again when it changes (looked at every second), signs with the file’s first key and verifies with any key in it, and sends its report again at once when its active key changes. Rotation: add the new key to every orchestrator’s file, then put it first in every file, then remove the old one. Test: TestAClusterKeyRotatesWithoutARestart (a follower’s report lands at every step, with the leader and follower part-way through, and is refused once the leader no longer holds the key it was signed under). A change is a change of the file’s permissions or of a hash of its contents, so a chmod that mends a refused file, or two lines swapped, is seen. A file that changed and cannot be read, or that others can read, leaves the keys read before in use and says so in the log once (TestAClusterKeyFileIsReadAgainWhenItChanges). The secrets key file is watched the same way since this step: before, it was read again only when its modification time or size changed, so a chmod alone was missed.
• Node reports are sealed by the orchestrator that writes them (§2). In plain stay only the node id, the leader’s receive time, the secrets key ids the cluster store’s key rule reads, and the cluster key id; the rest (API URL and pins, fingerprints, configuration key ids, site, automatic-failover flag, self-fencing status, controllers, secrets providers, the time of the report) is sealed with AES-256-GCM under the seal key derived from the writer’s active cluster key, before the report goes anywhere. The associated data binds the part, the node id, the key id and the secrets key ids, so a body cannot be moved to another node’s report and a plain field cannot be changed beside it. The leader sets its receive time outside the seal and commits a follower’s report opening nothing; the state machine stores a report only sealed. Tests: internal/clusterkey TestAReportIsSealedWithAES256GCMUnderTheSealKey (opened with the standard library’s AES-256-GCM under the derived key and the associated data written out, and under no other key, node, key id or list), TestASealedReportCannotBeMovedOrChanged, TestTheSealsKnownAnswer (byte for byte, checked with pyca/cryptography over OpenSSL); pkg/quorum TestAReportIsStoredOnlySealed.
• A report that does not open is absent, never guessed. Every reader on an orchestrator (the scheduler, the voter and controller views, dispatch, registration checks, peer pins) opens reports through one cache in the ownership engine, which opens a report again only when it or the keyring changes, and never serves a body whose plain fields changed. A report sealed under a key the reader does not hold, or that does not open, is absent: its node has no pin, so a call to it fails at the handshake before anything is sent, saying its report does not open here; dispatch never picks it; it counts as not reporting for drift, as ADR 0016 decided, and the drift view lists it with the reason, so the configuration key’s rotation and the migration checks refuse rather than skip it; its self-fencing status is missing, as for a node that never reported, so it backs no failover. The voter list says why (report_unreadable) and which key each report is under (report_key_id), from one read of the state. Tests: internal/ownership TestReportsOpenThroughTheEnginesKeyring; internal/api TestADispatchSkipsAnExecutorWhoseReportDoesNotOpen (an executor that would have verified the call is never sent it; with none left the fence is UNKNOWN and nothing runs), TestNoPinForANodeWhoseReportDoesNotOpen; internal/configsync TestAPeerReportUnderAKeyTheLeaderLacksIsRefused (an orchestrator without the leader’s key cannot even read where the leader is).
• Fence debts’ targets and reason are sealed by the leader that records them (§3). In plain stay the plan id, site, generation, time and key id; the detail is sealed bound to plan, site, generation and key id. A debt whose detail does not open is still owed: the leader fences every target of the site from its configuration, as before, and clears it once confirmed; the plan detail, the console and the audit say the detail cannot be read there, and guess no reason. A debt is never refused or dropped for want of its detail: one recorded with no cluster keyring, or written by an older binary, is stored with its site and no detail, and owed all the same. Tests: internal/api TestADebtWhoseDetailDoesNotOpenIsStillFenced; internal/clusterkey TestADebtsDetailOpensOnlyForItsPlanSiteAndGeneration; internal/ownership TestReportsOpenThroughTheEnginesKeyring (a debt recorded with no key); pkg/quorum TestFenceDebtRecordClearAndPromotion; the console’s FenceDebts.test.tsx.
• What the witness may read is a list, and a test holds it (§4). quorum.WitnessReadable lists every text field of a command or snapshot the witness may read; spec §8 says the same in words. TestTheWitnessReadsOnlyWhatIsListed fills every field of a command of every kind and of a snapshot and fails on any text field neither listed nor sealed (and on a field it did not fill); TestTheStructuralTestFailsOnAFieldAddedUnsealed shows it fails on a field added to a report, a debt, a command or the snapshot unsealed; it also failed, as expected, when a plain api_url was put back on the sealed report by hand.
• Proven on a real three-voter cluster. cmd/orchestrator TestWhatTheWitnessReadsOnARealThreeVoterCluster: two orchestrators and the witness binary report a controller whose endpoint cannot be reached and one whose target has a known identity, and a failover leaves a fence debt naming its target. The cluster key is rotated on both with no restart (c2 added second, then first, then c1 removed once the voter list shows every report under c2), and a write through the follower is still forwarded to the leader. Then every committed command in the witness’s raft.db and every snapshot is walked against the list, and the files are searched for the endpoint, the failure text, the identity, both API addresses, both pins, the fence target, the debt’s reason and a controller id, as they are, JSON-escaped, in hex and in base64: none is there. It failed, as expected, when each orchestrator’s API URL was leaked into a plain field by hand.
• A scope may name a port (§6). A scope’s host entry is a host, which allows any port of it, as before, or host:port, which allows that port only (an IPv6 address with a port bracketed, [fd00::1]:8443); * still allows any host. The resolver is told every address the credentials go to as host:port, as the type declares them (below): a URL’s port or its scheme’s (443, 80), a host field’s port field or that field’s default (PostgreSQL 5432, SSH 22), a fixed port (IPMI 623). A Unix socket is localhost with no port, which only a host entry allows. Hosts compare without case; a port is a number from 1 to 65535; brackets go only around an IPv6 address. A use is allowed only if every address is. The scope is bound and sealed as before. Tests: TestASecretScope (another port of a host named with its port is refused, a host alone allows any, a Unix socket only by a host entry, every address must be allowed), TestAScopeIsCheckedAtResolve, TestCredentialAddresses, and on the real binary TestASecretPutReachesItsControllerWithoutARestart, whose scope names the BMC’s port, and whose last value, scoped to another port of the same host, is refused to the controller and never sent.
• Every type declares the addresses it contacts (§7). A manifest’s addresses name, for each address the type contacts, the field it is read from (a connection field, an auth method’s setting, a capability’s setting), how its port is found (url, a port field and its default, or a fixed number), and whether credentials go there. The resolver checks a secret’s scope against every address the connection and the auth method being resolved declare as receiving credentials, read from the controller’s configuration (an unset field’s default included); one that does not parse is kept as written, which no scope allows. A type whose auth methods take a secret and that declares no credential address is refused at registration, as is a declaration naming a field the type lacks, saying nothing of its port, or sending credentials to a capability’s address. Every built-in type declares its addresses; Keycloak’s management interface (the base URL’s host at management_port) is declared as receiving none, so a scope need name only the base URL’s port. Tests: TestCredentialAddresses, TestADeclarationIsChecked, TestAnAddressFieldsDefaultCountsAndOneThatDoesNotParseFailsClosed, TestATypeTakingASecretWithNoCredentialAddressIsRefused, TestManifestsMatchPublishedSchema.
• A check holds the declarations. Every built-in type opens its connections through the dialer qfence gives it (ControllerConfig.Dial); PostgreSQL hands it the configured host, not one it resolved first, and FortiGate no longer goes through a proxy from the environment. internal/controllers/addresscheck runs a type against doubles with a recording dialer and fails on any address it contacted that no declaration gives for its configuration; a double declared as receiving no credentials is wrapped so that a credential reaching it, as it is, in base64 or inside a Basic authorization, fails the test. IPMI runs ipmitool, so it is checked on the -H/-p address each command line names. Infrastructure types run it from their own tests (TestRedfishContactsOnlyItsDeclaredAddresses, and the same for Proxmox, FortiGate, Docker and IPMI); application types run it in the conformance kit, whose addresses.declared result fails an undeclared address (Keycloak’s fake, PostgreSQL’s real servers, systemd’s host; Keycloak’s management double refuses credentials). TestTheCheckFailsATypeThatContactsAnUndeclaredAddress and TestTheKitFailsAnUndeclaredAddress show it fails a type that strays, TestACredentialReachingADoubleDeclaredForNoneFails the guard. A Docker tcp:// endpoint must name its port: its client would reach a port-less one on 80, or 443 with TLS, which no declaration could say.
• Run on the swarm lab (2026-10-19, up --build --fresh) with the cluster keyring: its plans were made active at dc1 by real failovers dispatched over the key-id peer channel; a switchover moved erp-db to dc2, sso and web-portal following; its cluster key was rotated in three steps with no restart, every report under the new key within seconds and a plan read through the follower still answered. The witness’s raft.db, 85 committed commands, held nothing outside the list and none of the lab’s controller endpoints, controller ids, service pins, API addresses or API pins in any form (TestAWitnessDataDirHoldsOnlyWhatSpec8Lists, run with QFENCE_WITNESS_DATA on any stopped witness’s data directory). TestLabDecisions reproduces both labs’ recorded decisions.
• Residual risks:
  • The check runs before release, against doubles: nothing refuses, at run time, a connection to an address no declaration gives (ADR 0016 considered it and left it). A type that dialed some other way than through the dialer it is given would contact addresses the check never sees; the kit fails a type that declares addresses and records none.
  • An external controller’s service makes its own connections; the kit holds it to nothing about them (its addresses.declared is n/a). qfence sends it no secret (ADR 0005 §4).
  • The leader seals each open debt’s detail again under the new key within a second of it being put first, in the background, never holding up fencing (TestADebtIsSealedAgainUnderTheNewKey; TestAFenceDebtIsResealedOnlyWhileItIsTheSame: only while the debt is exactly the record it read, ciphertext included, so never bringing back one cleared meanwhile nor overwriting one recorded again at the same generation; and on the real cluster in TestWhatTheWitnessReadsOnARealThreeVoterCluster). A debt whose detail does not open on the leader cannot be sealed again, and the leader’s log says so: if its key is removed everywhere, its targets and reason cannot be read from raft, though the site still owes its fence and is fenced. They are still in the audit log of the orchestrator that recorded the debt (plan.fence.debt.recorded), which never enters raft and is not shipped to the hosted plane (it drops a record’s reason), and the plan detail, the leader’s log and the console point there; the console links to this orchestrator’s records (TestADebtWhoseDetailDoesNotOpenIsStillFenced, FenceDebts.test.tsx). The operator docs say to keep the old key until such a debt is cleared.
  • An unreadable report from a third orchestrator does not block automatic failover as drift would: ADR 0016 counts it as not reporting, as a node that is down is. It is listed, with the reason, in the drift view and the voter list.
  • Sizes and timing still show: how large each report is, and when it changes.
  • A witness that held reports from before Phase 11.2b keeps them on disk until its raft data is replaced.
  • A key must be in every orchestrator’s file before any puts it first. A call signed under a key the receiver lacks is refused: a fence dispatched under it is UNKNOWN, a report under it does not land.
  • Whoever holds the cluster keyring can sign calls as any orchestrator, as with the secret it replaces, and with any voter’s raft data, the witness’s included, read every report and debt detail.
  • A node removed and re-added under the same id is pinned only by the report it sends then, as before; a report that does not open under any key the reader holds leaves its node unreachable from that reader until it reports again.

The console and secrets: its policy, write-only fields and resolution per orchestrator (ADR 0011, Phase 11.3)

docs/adr/0011-secrets-management.md §6 and §8 record the design, and its notes from the build what 11.3 settled.

• The console runs under a strict Content-Security-Policy, set before any field takes a value. Its page (index.html, and every deep link that falls back to it) is served with default-src 'none'; script-src 'self'; style-src 'self' 'nonce-…'; img-src 'self'; font-src 'self'; connect-src 'self' (and the OIDC issuer’s origin, when the console signs in with OIDC, since the token exchange runs in the browser), form-action 'self'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'. No script runs inline or through eval: the theme script that ran inline in index.html is its own file, /theme.js, and the boot shell’s styles are in the stylesheet. Two libraries add a <style> element as the page runs (Radix Select’s viewport, and the scroll lock of every dialog, whose text depends on the window and cannot be hashed): each carries a nonce of 128 random bits that the orchestrator writes into the page, new in every response. Every other response (the API’s JSON, the assets) carries default-src 'none'; base-uri 'none'; frame-ancestors 'none', so one opened as a document runs and loads nothing; every response carries X-Content-Type-Options: nosniff and Referrer-Policy: no-referrer. The no-JS fallback page runs no script and allows its one style element by its hash, taken from the page as the template renders it. Tests: internal/api TestTheConsolePageIsServedUnderAStrictPolicy (every directive, no weak source, the nonce new in each response and the one the page carries, compressed or not), TestTheConsolePolicyNamesOnlyTheIdentityProvidersOrigin, TestTheEmbeddedConsolePageRunsNothingInline (the page the binary embeds, built or the placeholder of a Go-only checkout, holds no inline script, style, handler or javascript: URL; it failed on the old placeholder), TestEveryResponseCarriesTheSecurityHeaders, TestTheFallbackPageAllowsOnlyItsOwnStyle; and Playwright against the production build the binary embeds (playwright.secure.config.ts): every route in both themes, a select’s viewport and a dialog with nothing refused (it failed when the select’s nonce was left out), markup injected into the page neither running a script or handler nor applying a style, the fallback page under its own policy, and every response of the console’s session over TLS 1.3 with X25519MLKEM768 as the browser negotiated it.
• The console takes a value only into the cluster store, in a write-only field, and keeps none of it. Where this orchestrator can write the cluster secrets store, A7a’s step 2 offers, for each auth block with secret fields, to enter the secret now, and the Secrets page (/secrets, Administrator) to store a new secret or a new value; the console never writes a value into local, and without the cluster store it takes no value and shows secret put. On the Secrets page, Revert and Delete, which carry no value, act on the cluster store and on the serving orchestrator’s own local store, and say which (the local ones change that orchestrator’s store only). The fields are uncontrolled password fields (a masked text area for a PEM value), never pre-filled, with autocomplete, spell checking and password managers turned away. A value is read from the page only when it is sent, after the form’s own checks pass, and every field is emptied in the same step, so it is in that one request (PUT /api/v1/secrets/cluster/{name}, which takes a value only over X25519MLKEM768, and seals it before it goes anywhere) and nowhere else: a write that fails says to type it again. The answer is the reference, version, scope and the controllers opened again. The scope offered is the server’s (POST /api/v1/secrets/scope: the type and every address the type declares its credentials go to, with the port), and each address may only be widened to its host. GET /api/v1/secrets lists names, versions, scopes and the controllers that name each, never a value, Administrator only, and is in the no-credential-leak test. Tests: TestTheSecretsPageListsNamesNeverValues, TestTheScopeANewSecretIsOffered, TestNoCredentialReachesAResponseLogOrAudit (with the listing); vitest ControllerRegisterSecret.test.tsx (the value in one request, the fields emptied after a write and after a failed one, nothing read before the checks pass, no value field without the store), Secrets.test.tsx, secret-entry.test.ts; and Playwright against the embedded build: a Redfish controller registered by typing its key, by keyboard alone, in both themes with axe clean and no policy violation, the double receiving the typed key, the value in exactly one request from the browser and in no response, console message, local or session storage, IndexedDB, Cache Storage, cookie, orchestrator log, event, audit record or controller detail; and a secret stored on the Secrets page the same way.
• Where each binding’s secret resolves, per orchestrator, is reported and decides nothing (ADR 0011 §6). Each orchestrator puts, for each controller it calls and each auth block with a secret, the provider the reference names, the site that provider is defined at there, and whether it resolves, is held through a failed resolve (ADR 0011 §5) with when it was resolved and when it stops being valid, or cannot resolve with the kind, into its sealed node report: never a secret’s name or a value. The dry run of a failover or switchover lists, for every binding of the plan with a secret and every orchestrator at a site its controller is called from, this orchestrator’s own view and the others’ reports (a report older than the freshness window says so and nothing more), and warns when the provider is defined at another site than the orchestrator: “if dc1 is cut off, dc2 holds this value only until it restarts”. The plan’s attention list names a secret that cannot be resolved on an orchestrator, one held there and until when, and one resolved through a provider at another site. Nothing reads it to decide: the dry run’s verdict and reasons are the same whatever it says, and an automatic failover is never refused for it, since a fence whose secret does not resolve is UNKNOWN when it runs and another binding may confirm the site. Tests: internal/controllers TestHowASecretStandsIsReportedPerAuthBlock (resolves, held with its expiry, dropped, opened without it, not called here); internal/api TestResolutionPerOrchestratorInTheDryRunAndTheAttentionList (this orchestrator, a report that cannot resolve, one held through an outage of a provider at another site, a stale report, the attention items, and the verdict unchanged); vitest for the dry run’s section.
• An orchestrator whose report does not open here is shown where an operator looks (ADR 0016 §2). Such a report counts as not reporting here: drift does not compare it, so drift does not hold automatic failover back on it, it has no pin, and nothing is dispatched to it. GET /api/v1/cluster now carries the cluster’s attention list, naming each such orchestrator with the reason its report does not open; the Cluster screen raises it at the top, marks the voter’s row and shows the reason in its Report sealed under column, which also shows every voter’s report key, the one a key rotation’s last step waits on; the sidebar counts it and the dashboard lists it. Nothing decides from it. Tests: TestTheClusterAttentionNamesAReportThatDoesNotOpenHere (and gone once the report opens), vitest for the screen and the dashboard, and Playwright on a real cluster of two orchestrators and the witness (tests/secure/cluster.sh: dc1 leads with its report sealed under a key dc2 lacks), dc2’s console flagging dc1 with the voter list’s reason, in both themes, axe clean.
• Run on the swarm lab (2026-10-19, reset, then up): erp-db’s dry run through either orchestrator lists every one of its bindings with a secret on every orchestrator at a site its controller is called from, the FortiGate’s bindings on both, each resolving through the cluster store at its own site; dc1’s console shows it in both themes (tests/swarm/secrets-per-orchestrator.spec.ts), axe clean. The Keycloak registration spec on the lab passes again, the picker listing the cluster store’s names.
• Residual risks:
  • An orchestrator whose report never landed in raft, because it cannot open the leader’s report to learn where to send its own, is not flagged as unreadable: it shows “no report” under Report sealed under, as a voter that has not reported does.
  • An orchestrator reports how its secrets stood at its last report, sent within seconds of a change and at least every five minutes; one that is down reports nothing new, and its rows say when it last did. An orchestrator whose report does not open here is not listed at all, since its site is not known here; the Cluster screen flags it.
  • A browser extension, or the browser’s own password manager despite the hints, can read or keep what is typed into a page; the console cannot stop either. The operator’s browser is in the trust boundary, as their terminal is for secret put.
  • The value stays in the browser’s memory until it is collected (a JavaScript string cannot be cleared), and in the serving orchestrator’s while it seals it: the handler clears the request’s bytes and the encoded value, but the decoded field strings stay until Go collects them.
  • A script from this origin is allowed. The origin serves only the console’s files and the API, whose JSON is never read as a script (nosniff, application/json); a future endpoint that served text a caller chose, as a script, would undo that.
  • The nonce is readable by any script on the page, as it must be for the libraries to use it; it guards against markup injected without script, which cannot read it.
  • Trusted Types are not required: React and Radix write innerHTML (Radix Select’s style), which a Trusted Types policy would refuse.

Vault, defined per orchestrator (ADR 0011, Phase 11.4)

docs/adr/0011-secrets-management.md §3, §4 and §11 record the design, and its notes from the build what 11.4 settled.

• Each orchestrator defines its own providers. -secrets-config is a YAML file of this orchestrator’s providers beside local and cluster. Each entry gives the provider’s id (the name references use), its type (vault), the site it serves, its address, CA and mounts, and its auth. Every piece of secret zero is named by a file, never given as a value. A field the file does not know, such as a token typed into it, is refused without being repeated, and so is a definition of local or cluster. A reference names exactly one provider, and the resolver asks only that one. A reference to a provider an orchestrator does not define is unavailable there. It is no longer refused as configuration, since the configuration is the cluster’s and each orchestrator defines its own providers. Tests: TestLoadProviders, TestADefinedProviderIsReportedWithItsSite, TestCheckRefusesWhatTheManifestDoesNot (a vault: reference accepted), TestRegistrationRefusals (a session waits at step 1, “no secrets provider”).
• Node reports carry each provider’s id, type, site and whether it answers, never its auth material. A provider is available while it holds a token that has not ended and Vault answered its last request, or its health check (sys/health, made when nothing else asked Vault for 30 s). The detail says why not, without a certificate, key, token or id. The dry run warns of a binding that resolves through a provider at another site than the orchestrator’s (ADR 0011 §6). A sealed Vault, which answers every read with 503, does not read as answering. Tests: TestADefinedProviderIsReportedWithItsSite; TestVaultCertificateAuth (a refused certificate’s detail holds no PEM); TestASealedVaultDoesNotReadAsAnswering; cmd/orchestrator TestTheOrchestratorResolvesThroughItsOwnRealVault (the voter list carries those fields and no other).
• Secret zero: a client certificate by default, AppRole, or a development token.
  • Certificate auth presents a client certificate; qfence-orchestrator pki issue -client issues one valid as a TLS client only, naming no host. The docs say to issue it from a CA of its own, never the raft CA, so no voter’s certificate (the witness’s included) can sign in, and to pin its common name in Vault’s role. AppRole reads a role id and a secret id from their files. A static token is for development only, warned of at start.
  • Certificate auth and AppRole need an https address. A static token over plain HTTP is warned of as well: the token and every value cross the network readable.
  • A secret-zero file that others can read is refused, at start and at each sign-in. The files are read at each sign-in, on a connection of its own, so a renewed certificate or a new secret id needs no restart.
  • The token qfence earns lives in memory only. It is renewed at two thirds of its TTL, and replaced by signing in again when Vault renews it short of its TTL (near its max TTL), cannot renew it, or answers that it is not valid. A token replaced while still valid is revoked once nothing read under it is held, and every one at shutdown.
  • A Vault that cannot be reached when the orchestrator starts leaves only its own references unavailable; it is signed in to once it answers.
  • Tests: TestNewVaultRefusesBadConfig (each refusal, a key file mode 0644 among them), TestVaultKeepsItsTokenRenewed, TestVaultCertificateAuth, TestVaultAppRole, TestVaultUnreachableAtStart, pkg/mtls TestAClientCertificateIsAClientOnly; against a real Vault, TestRealVaultCertificateAuth (reads go on past the first token’s max TTL; a certificate from another CA is refused, as unavailable) and TestRealVaultAppRole.
• A Vault denied counts only after a fresh sign-in (ADR 0011 §5). The registry’s sign-in after a denied makes the provider sign in again, unless it did within 10 s. A denied after that is Vault’s answer, so the held value is dropped and its fences are UNKNOWN. A static token cannot sign in again, so its denied keeps the value, as through an outage. Tests: TestVaultCertificateAuth, TestVaultSignsInOnlyWhenItCan, and internal/controllers TestADeniedBeforeSigningInAgainIsNotARevocation.
• qfence stays read-only at Vault, and the provider’s guards stay.
  • It sends Vault reads of secrets and, beyond them, only its own sign-in, its token’s renewal and revocation, the renewal and revocation of its own leases, and the health check. TestVaultOnlyReads lists every request and fails on any other.
  • A lease is renewed and revoked by the path forms, sys/leases/renew/ID and sys/leases/revoke/ID, so a policy can grant them for qfence’s own lease prefix alone, as the vault lab’s does.
  • A reference’s path must start with one of the provider’s mounts, which may not overlap. A path on none, or with an empty, . or .. segment, is refused before any request. No redirect is followed, a sign-in’s included. A lease id that is not a clean path on its mount is refused, since the token is sent to it.
  • Tests: TestVaultOnlyReads, TestVaultNeverLeavesItsMount (removing either guard fails it), TestVaultReadsSeveralMounts, TestALeasedEngineThatGivesNoLeaseIsMalformed.
• Leased reads, one lease per orchestrator (ADR 0011 §4).
  • A path on a leased engine (Vault’s database engine, say) is one lease on each orchestrator, whichever references name it: the fields of #username and #password come from one credential, and no credential is shared between orchestrators. Concurrent resolves share one read. A read that comes with no lease is malformed, never a value held without an end.
  • It is renewed at two thirds of its TTL. It is read again when the renewal fails, when Vault renews it short of its TTL (its max TTL), when it is not renewable, or when the token it was read under is replaced, since Vault revokes a lease with its token. Its end is the earlier of its own and its token’s.
  • The orchestrator is told of each path whose lease changed, and the controllers it calls that name the path resolve it again at once. A renewal moves the held value’s end in place, later or earlier, without opening the controller again, so a renewal never rebuilds a plan. A lease read again opens them again with the new credential, and the old lease is revoked after that.
  • A lease no controller here names any more is revoked rather than renewed, and every lease is revoked at shutdown.
  • A lease that cannot be renewed or read again because Vault does not answer is kept until it ends, then dropped: nothing is held for it, the controllers that name it drop the value, and a fence through it is UNKNOWN. A read again that Vault answers with not found or denied drops it at once.
  • The dry run shows when a lease ends: “resolves, its lease ends T unless renewed”, or “held until its lease ends T”.
  • Tests: TestALeaseIsOnePerOrchestratorAndRenewedAtTwoThirds, TestALeaseIsReadAgainWhenRenewalFails (revoked at Vault; renewed short of its TTL), TestALeaseThatEndsWhileVaultIsUnreachableLeavesNothing, TestLeasesAreRevokedAtClose (and an unnamed lease revoked), TestLeasesEndWithTheirToken, TestALeaseReadIsShared; internal/controllers TestAHeldValueGoesAtItsExpiry (an end moved in place, later or earlier; kept to the end the provider last gave and no longer); internal/api TestALeaseThroughTheOrchestrator (the dry run’s end, a renewal without reopening, a new credential opened at once, a lease that ended leaving the controller failed); vitest for the dry run’s wording.
  • Against a real Vault and PostgreSQL: TestRealVaultLeases (one lease for two references; the login signs in; renewed, then read again at its max TTL, with the old login’s role dropped by Vault; dropped at Close) and TestRealVaultALeaseEndsWithVaultUnreachable (Vault paused past the lease’s end: unavailable, then read again once Vault answers). cmd/orchestrator TestTheOrchestratorResolvesThroughItsOwnRealVault runs the binary with its site’s Vault in -secrets-config. A BMC double receives the password vault kv put wrote, a leased credential is one PostgreSQL role, and SIGTERM revokes the lease, so the role is gone.
• No disk cache of a Vault value. Values and leases live in memory only, never past a lease’s end, so a restart during a Vault outage starts without them (ADR 0011 §5). A site that must resolve across a restart while cut off uses cluster, whose copy is on disk, or its own Vault.
• In CI. test_vault runs deploy/vault-lab in a docker:dind service: Vault 1.21.4 over TLS with a qfence-orchestrator pki certificate, certificate auth, AppRole, and a database engine on PostgreSQL 17, seeded with the vault CLI. The RealVault tests run against it under -race. The job has been run locally in CI’s shape (golang:1.26, a separate docker:dind), and passes on a real runner (2026-10-28).
• Run on the swarm lab (2026-10-19, up --build --fresh). Each site has its own Vault on its own network, and each orchestrator’s -secrets-config defines it as vault. fgt-edge, which every shield and VIP binding uses, names vault:secret/qfence/lab-fortigate.
  • Both orchestrators report vault at their own site answering. erp-db’s dry run, through either orchestrator, lists both shields and the VIP resolving through vault at each orchestrator’s own site, with no warning.
  • dc1 was failed (the simulator’s Fail: its orchestrator, its nodes and its Vault down). dc2 kept resolving the FortiGate’s token from its own Vault, with no refusal. erp-db and web-portal failed over to dc2 on dc1’s self-fence at the bound, and dc2 raised their shields through the FortiGate with that token (“shields raised at dc2”).
  • On Restore, dc1’s Vault came back, and dc1 resolved through it within seconds.
• Residual risks:
  • qfence does not insist on a post-quantum key exchange with Vault, as its own API listener does: values read from Vault cross whatever TLS Vault, or a balancer in front of it, negotiates. Vault 1.21.4 negotiates TLS 1.3 with X25519MLKEM768 with qfence’s client (measured on the swarm lab, 2026-10-20), so what is read from it is protected against harvest-now-decrypt-later; a balancer or an older Vault in front may not be, and nothing reports it.
  • A static token, and Vault over plain HTTP, are for development: both are warned of at start, and neither is refused.
  • The client certificate is valid for 90 days. Once it expires, sign-ins fail; a periodic token keeps being renewed without one, but an orchestrator that restarts, or whose token cannot be renewed, is then unavailable at Vault. Issue the next certificate before then; it is read at the next sign-in.
  • A token whose max TTL ends replaces itself, and every lease read under it is read again under the new one, which opens again the controllers that use them. A periodic token (token_period) never does that, and the docs recommend one.
  • An AppRole secret id with a use limit or a TTL makes a later sign-in fail; the docs say to give one without.
  • While a lease is read again, the orchestrator briefly holds two credentials until the old one is revoked.
  • A KV value has no expiry, so one held through a Vault outage is kept until the controller is opened again (a restart drops it), and a revocation at Vault takes effect only when Vault answers again.
  • A Vault standby that redirects (307) is never followed, and its answer reads as denied. Point address at the active node, or at a load balancer that forwards; Vault’s standbys forward requests by default.
  • Namespaces (Vault Enterprise, HCP Vault) are sent, and checked only against the double.
  • -secrets-config is read at start: a change needs a restart. The files it names are read at each sign-in.

AWS Secrets Manager, a reader with the host’s identity (ADR 0011, Phase 11.5)

docs/adr/0011-secrets-management.md §1, §3 and §11 record the design, and its notes from the build what 11.5 settled. No AWS account has been used. Everything here was checked against a double, and the check against a real account is DEFERRED (real hardware/platform; Known gaps).

• Defined per orchestrator, like Vault. An entry in -secrets-config of type: aws-secrets-manager gives its id, site and region, and optionally an https address (a VPC or FIPS endpoint) and its ca_file. A reference names the secret: aws:qfence/fgt, or aws:qfence/fgt#token. Tests: TestLoadProviders, TestNewAWSRefusesBadConfig (plain HTTP, a path, a user in the URL, a region that is not one).
• The host’s own identity, and nothing else (ADR 0011 §11).
  • The only credentials are the EC2 instance role’s, read from the instance metadata service with IMDSv2 only: a session token first, then the role, then its credentials. IMDSv1 is never tried.
  • The metadata address must be on the host’s own link (loopback, link-local, or fd00:ec2::254), since the credentials come back over it in plain HTTP.
  • The file takes no credential of any kind: a token, key or certificate file is refused by its field’s name, and a key typed in is refused as an unknown field, without being repeated.
  • No AWS environment variable, ~/.aws file, container credentials endpoint or web identity token is read. With the metadata service down, nothing reaches Secrets Manager at all.
  • The credentials live in memory only. They are read again at two thirds of the time they had left, and at once, with the request sent again, when AWS answers that they are expired or unknown. Nothing is revoked at shutdown: the credentials are the instance’s.
  • Tests: TestAWSUsesTheHostsIdentityOnly (canaries in AWS’s credential environment variables, a shared credentials file, a web identity token file and a container credentials endpoint that records being asked), TestAWSKeepsItsCredentialsFresh, TestAWSUnreachableAtStart, TestLoadProviders (each refusal), and cmd/orchestrator TestTheOrchestratorReadsAWSSecretsManager, whose environment holds other credentials.
• Read only. It sends Secrets Manager GetSecretValue and nothing else, and the metadata service only IMDSv2’s token and the role’s credentials. It is neither a store nor a lister: ListSecrets cannot be granted for a prefix, so references to it are typed. Its role needs secretsmanager:GetSecretValue on qfence’s secrets alone, and kms:Decrypt on a customer managed key. Tests: TestAWSOnlyReads (every request listed, any other fails it), and the binary’s test.
• Every answer is a typed resolver error (ADR 0011 §5).
  • Not found: no such secret, or one marked for deletion. Denied: the role’s policy or the secret’s KMS key refused. Malformed: a binary secret, a value that is not a JSON object of string or number fields, or an invalid request.
  • Unavailable: credentials refused again once read again, throttling, a server error, a redirect (never followed), an answer that cannot be read, an error code it does not know, and no answer.
  • A denied counts after the credentials were read again. No error carries AWS’s message (it names the role’s ARN and the account), a body, a credential or a value.
  • Tests: TestAWSErrorsAreTyped (22 answers), TestAWSSignsInOnlyWhenNeeded, TestAWSResolvesWhatSecretsManagerHolds.
• Node reports. It is available while it holds the instance’s credentials and Secrets Manager answered its last request (neither throttled nor a server error), or its health check: a TLS handshake with the endpoint when nothing asked it for 30 s, which asks AWS nothing. The detail never holds a credential. Test: TestAWSStatus.
• What the double rests on. The double’s check of Signature Version 4 is written apart from the reader’s signer, and the signer gives the signatures of AWS’s published test suite (TestSigV4MatchesAWSExamples). botocore 1.34.46, AWS’s own Python SDK, was run against the double on 2026-10-20 (TestTheDoubleAgreesWithBotocore, QFENCE_BOTOCORE=1). Its IMDSv2 read and signed calls passed the double’s check, a wrong secret key failed it, and its errors parsed to the double’s codes. The API’s shape is botocore’s model of the service. None of this shows how a real account answers (Known gaps).
• The tests fail when the reader breaks. Each of these mutations, made by hand and then undone, failed a test (2026-10-20): the signer leaving the region out of its key, no retry on a refused credential, a read without IMDSv2’s token, not found read as unavailable, and credentials never read again before their expiry.
• Residual risks:
  • It has never run against a real AWS account (Known gaps).
  • It is for orchestrators that run on EC2. One elsewhere has no instance role to sign in with; IAM Roles Anywhere is out of scope (ADR 0011).
  • Every process on the instance can read the same role’s credentials from the metadata service. Give the role only what qfence reads, and do not run other workloads on the orchestrator’s instance.
  • In a container on EC2, IMDSv2’s token answer needs a response hop limit of 2; with the default of 1 the reader cannot reach the metadata service and is unavailable.
  • IAM’s “revoke active sessions” on the role denies the instance’s current credentials. The metadata service keeps handing them out until it renews them, so until then every read is denied: held values are dropped and fences through them are UNKNOWN. That is the deliberate revocation ADR 0011 §5 means.
  • Secrets Manager is regional. A site cut off from its region keeps a held value only until the orchestrator restarts (there is no disk cache, as for Vault). A fence a cut-off site needs belongs in cluster.
  • A value has no expiry, so one held through an outage is kept until the controller is opened again, and a removal at AWS takes effect only once AWS answers again.
  • The health check is a handshake only. An endpoint that completes it but cannot serve reads as answering until a request fails.
  • A secret whose name has +, = or @, or one in another account (named by ARN), cannot be referenced.
  • Whether AWS’s endpoint negotiates X25519MLKEM768 with qfence’s client has not been measured.
  • -secrets-config is read at start: a change needs a restart.

Site executors, discovery and bindings (ADR 0010, Phase 10.3)

Each site’s orchestrators now make that site’s controller calls, so no controller is reached across datacentres; bindings are checked against the resources their controllers discover. What keeps it safe:

• The decisions did not change. The engine, the failover sequence and the scheduler get the same runtimes. Each is wrapped in a route that either calls the controller (this orchestrator’s site is in its reach) or sends the call to an orchestrator at a site that is (ADR 0010 §9). TestLabDecisions and TestDecisionCoreMatchesRecorded pass unchanged.
• A call that cannot be answered is never a fence.
  • The sender tries a second executor only when the first could not be reached at all, so a fence that may have run is never sent twice.
  • A lost answer, an answer that arrives after the sender stopped waiting, a target slower than the fence timeout, and a site with no executor all leave the fence verified through the same path: UNKNOWN, never FENCED (§17).
  • Tests: TestLostAndLateRepliesAreUnknown (the executor did act, and the leader still reports UNKNOWN), TestSlowTargetAndMissingExecutorAreUnknown (no executor fails fast).
• Calls between orchestrators close their connections. Every peer call (dispatch, lease answers, plan reads, configuration writes, secret checks) builds its client on a new transport, so a connection kept for reuse was never reused or closed. On the swarm lab, after 19 hours, a follower had left about 14,000 connections open to the leader, and the host was running out of local ports; the peer channel would then have failed, and with it every dispatched fence (UNKNOWN). Since 2026-10-16 each connection closes once its answer is read. Every call already opened its own connection, so what a lost or late answer means is unchanged. Test: TestPeerCallsLeaveNoConnectionOpen (10 of 10 stayed open before the fix).
• A deposed leader cannot act.
  • An executor refuses a command from an older raft term, and a fence, unfence, acquire or release from anyone but the leader of the current term.
  • It runs its own copy of the binding and refuses one configured differently from the sender’s, so drift between orchestrators is refused, not acted on. Each of those refusals leaves the fence UNKNOWN.
  • Tests: TestExecutorRefusesStaleTermsAndOthers.
• The right site runs it, and says so. On a three-voter raft cluster with orchestrators at two sites, over the real signed channel, the leader at dc2 fenced dc1 through dc1’s executor and never called dc1’s controller itself; its own site’s targets it called directly. Each change an executor runs is audited there (executor.fence). Readiness at dc1 is read at dc1. Registration tests a dc1 controller at dc1 when opened through dc2, and dc2 never calls dc1’s controller even for health: that comes from dc1’s report. Tests: TestFenceRunsAtTheControllersSite, TestRegistrationAndHealthAtTheControllersSite.
• Dead-man switches stay at their site. Its own site’s orchestrator arms them, cut off or not, so they are never sent elsewhere, and a plan whose dead-man controller cannot be called from that site is refused.
• Only what was checked is bound. A binding created or changed through the API must use a capability advertised where its check runs, and a resource the controller’s discovery lists. A resource that no discovery covers is accepted as typed, marked unverified, with a warning.
  • A binding already in the plan stays when its capability starts failing, as ADR 0010 §4 requires: the plan shows it, and a fence through it is attempted and verified.
  • Tests: TestBindingsAreValidatedPerResource, TestBindingValidationPerResource.
• Per-resource checks that matter. A Proxmox vm.stop binding is checked for the VM being on the node it names (a fence sent to another node does not stop it), and for the privilege on that VM. A Docker lag command is checked by running true in the workload.
• Discovery reads inventory with the controller’s credentials, so it is Administrator only, cached 30 s, refreshed at most every 5 s, never run for a disabled controller or from outside its reach (TestDiscoveryAPI, TestDiscoveryIsCachedAndRateLimited).
• §14 checks declared levels. A binding’s level is its capability’s, and a level written in a plan is refused rather than dropped. An application-level fence never satisfies the compute rule, and never confirms a site that has a compute target (TestSection14ChecksDeclaredLevels, TestApplicationLevelNeverSatisfiesCompute).
• Setup before use. A capability that still needs a required setting or its own credentials is needs-setup and cannot be bound. Its own credentials that fail to resolve fail that capability alone (TestCapabilitySetup).

Verified on the swarm lab (lab.sh up --fresh --register; each orchestrator’s API reached through a simulator proxy cut with its site, every controller but the FortiGate reached only from its own site):

The cost, as ADR 0010 §9 accepts: a site that has lost power or its orchestrator can no longer be proven off by its BMC from the other site, so a failover that took about 30 s on that proof now waits for the bound (about 4 min 15 s). Listing the other site in the BMC controller’s reach brings the proof back, where the network allows it.

The console for controllers and bindings (ADR 0010, Phase 10.4)

The console now registers controllers and edits plans. What keeps that safe is that it decides nothing:

• The server grades, the console shows. Every controller and capability state, check grade, level and binding verification on screen is the API’s, each with its own badge. The console filters lists and counts states; it never works out a level, a grade or whether a binding is verified.
• Only what the server offers can be bound. A binding’s capability comes from the server’s bindable list for its role at the instance’s site, its resource from that capability’s discovery (or typed, and then shown unverified), its parameters from the manifest. No level can be typed: a plan document that carries one is refused (10.3).
• Nothing is written unchecked. A plan is saved only after the server validated it and the person saw the field changes and each new binding’s check. A controller is registered only through the session’s steps, and a capability is enabled or re-authentication committed only once its checks pass.
• Secrets are never shown, and entered only into the cluster store. A form names a secret by reference, and shows the secret put command for an orchestrator that lacks one. Tests check that no request a form sends carries anything but a reference. Amended in Phase 11.3 (ADR 0011 §8): the registration wizard’s step 2 and the Secrets page also take a value in a write-only field, sent once to the cluster secrets store and never read back; see “The console and secrets” above.
• Deleting a plan stops nothing and forgets nothing that matters. DELETE /api/v1/availability-plans/{id} is refused while a site owns the plan (its workload would run unmanaged and its dead-man switches lose the site that renews them), while a site owes it a fence (the only record that a site may still run it), while a forced failover awaits approval, while another plan depends on it, and in local mode while the configuration file defines it. Ownership is read linearizably on the raft leader; a follower asks it (ADR 0012 §5), and refuses when it cannot. The ownership record stays, so a plan re-created with the same id never reuses a fencing token; its lease is refused afterwards, so a workload that asks stays standby (TestDeletingAPlan, the replicated case in TestReplicatedConfigurationLifecycle). Two races close it (2026-10-12):
  • A plan that comes to depend on it. In replicated mode the tombstone is written only while the configuration is still at the revision the checks read. A plan that came to depend on it in between makes the deletion fail with a conflict, where it used to leave a configuration no orchestrator could apply (tested through a follower).
  • An acquire. A deleted plan is retired in raft, and raft refuses an acquire of a retired plan, forced or not. Retiring is the deletion’s commit point, and the FSM refuses it while a site owns the plan or owes it a fence. Retiring and acquiring are ordered by the log, so an acquire committed after the checks makes the deletion fail with the reason and nothing deleted, and an acquire after the deletion is refused. In replicated mode the tombstone retires the plan in the same write; in local mode the orchestrator retires it through the leader (forwarded by a follower) before it drops its own copy, and reinstates it if that fails. Creating the plan again reinstates it (in replicated mode, writing its entry does), and its generation goes on. A change of configuration source lifts every retirement, since the new source decides which plans exist. Tests: TestRetiringAPlanOrdersDeletionAndAcquire, TestReplicatedTombstoneRetiresThePlan, TestRetiredPlansSurviveASnapshot (pkg/quorum); TestDeletingAPlanOrdersWithAnAcquire, TestLocalDeletionThroughAFollower, and the replicated case in TestReplicatedConfigurationLifecycle (internal/api).
  • In local mode the retirement is cluster-wide, while a deletion drops only this orchestrator’s copy. Another orchestrator that still holds the plan cannot acquire it until it is deleted there too, or created again through the API (saving it reinstates it). Before, it could acquire it while the orchestrator that deleted it, perhaps the owner site’s, no longer renewed its dead-man switches. A plan an orchestrator loads from its configuration file is not reinstated by loading it: save it through the API.
• A follower shows the leader’s view, or says it cannot (10.7). A follower answers the plan reads by asking the raft leader over the signed peer channel, with a nonce its signed answer must echo (ADR 0012 §5). Only the plan GET routes are answered that way, as a read-only identity: nothing that changes state is forwarded. When the leader cannot be asked, the follower answers from what it holds and the owner is UNKNOWN with the reason, never a stale owner from its own memory (TestPlanReadsAnsweredByTheLeader).
• A plan’s document is readable by every role, and changed only through review (10.7). GET .../config is open to Viewers so the editor can show the plan read-only: a plan document holds no secret and no secret reference, which live on controllers. A change still needs an Administrator, the server’s validation, the person’s review of the field changes and each binding’s check, then one write. In file mode the server writes nothing: it returns the replacement file, and the configuration changes only when a person deploys it (TestMigrationLocalToFileAndBack).
• The §14 override is deliberate and recorded. Allowing network-only fencing needs the plan’s id typed (paste refused), and the change that turns it on is audited as a reduced guarantee (TestShieldsRequiredByProofStrength).

Application health and local recovery (ADR 0005, Phase 9.1a)

A plan can now watch the application at its owner site, not only whether the site is reachable. A health binding that reports the application unhealthy on every check for failover.application_health.unhealthy_for_seconds starts an episode. The owner site’s recovery binding restarts the application, up to max_attempts within window. When the restarts do not hold, or there is no recovery binding, the plan escalates: with fail_over it fails over, and otherwise a person is asked. A restart is an action at a live site, and a failover on application health fences a site that is still up, so both are fenced in:

• UNKNOWN never counts (§17, §48). Only a binding that reaches its target and finds it unhealthy counts. An error, a timeout or a target the controller cannot reach is UNKNOWN, and starts the count again, so a check that cannot run never restarts or fails anything over. Held for unhealthy_for, it is an attention item (“cannot see the application’s health”). After a restart’s grace, UNKNOWN leaves the episode waiting: neither another restart nor an escalation (TestUnknownNeverTriggers).
• A restart only where the plan is active, and never during a change of owner. The leader restarts only at the owner site of an ACTIVE plan, under the plan’s group lock (a failover or fence of it holds that lock), after a linearizable read, carrying the owner’s generation. An executor at the site refuses a restart unless its own raft state shows that site owning the plan, ACTIVE, at that generation, and only the leader of the current term may send one. A restart refused either way is not made and does not count. Tests: TestRecoveryNeverAtANonOwner, TestRecoveryPausesDuringAFailover, TestExecutorRestartsOnlyTheActiveOwner (two orchestrators: the restart runs at dc1’s executor, never across sites; a stale generation and a site that no longer owns the plan are refused).
• Restarts never touch the circuit breaker (§41, D9). The breaker counts failovers. Restarts are bounded by their own window, attempt limit and doubling backoff. A failover that recovery escalates to is an ordinary automatic failover: it passes the dependency check, the configuration gate and the breaker, is reserved with the breaker before it runs, fences the old owner in §19’s order, and needs readiness at the target. Tests: TestRecoveryRunsOutThenFailsOver (two restarts and the failover leave a breaker of two armed), TestApplicationEscalationFailsOverThroughTheGates (a tripped breaker refuses the escalated failover, audited once).
• A failover on application health fences a live site. The owner is up and its fences are applied as in any failover. A plan whose application shares a host with others should scope its compute fence to the application’s VM or container. The dry run shows what will be fenced.
• A restart loop that never runs out is still seen. An application that fails every so often and recovers on the first restart never uses up a window. Recovery in flag_after_episodes episodes within flag_period raises an attention item and ApplicationRecoveryRepeated, and changes nothing else (TestRepeatedRecoveryIsFlagged).
• Everything is recorded. Every change of the owner site’s health raises ApplicationHealthChanged. Crossing unhealthy_for is audited as plan.health, saying what follows. Every restart is audited as plan.recovery.attempt and raises ApplicationRecoveryStarted, then Succeeded or Failed; running out raises ApplicationRecoveryExhausted. An executor that runs a restart for the leader audits it as executor.recover. A restart by hand (POST .../bindings/{binding}/restart, Operator) counts in the window like any other (TestRestartByHand).
• Configuration that cannot work is refused: recovery bindings without failover.application_health, application_health with no health binding, fail_over without automatic_failover, a recovery binding at a site no health binding watches, two at one site, and a recovery policy outside qfence’s limits (TestHealthAndRecoveryConfiguration). A recovery capability must declare the recovery policy as its policies (ValidateManifest).

Verified on the swarm lab (2026-10-05), on the plan orders-api: a service the swarm never restarts (restart condition none), with service.health as its health binding, service.restart as its recovery binding (two restarts within 10 minutes, 10 s backoff, 20 s grace), and application_health of 15 s with fail_over. lab.sh crash kills its task.

The plan page and the editor on that plan pass axe-core in both themes.

Known limit. The health answers, the restart window and the episode history are the leader’s memory. A new leader starts afresh, so a leader change resets the window and the flag count.

External controllers, application fences and switchover (ADR 0005, Phase 9.1b)

An operator’s own application controller is a service qfence calls over the external controller protocol (HTTPS with mutual TLS; Writing an application controller). Its code runs nowhere near the orchestrator, and qfence treats everything it says as untrusted:

• No reply is UNKNOWN (ADR 0005 §3). Every call has a deadline: the controller’s timeout, shortened by the plan’s fence timeout. A reply that is late, cut off, over 64 KiB, malformed, carries an unknown field or a value outside the protocol, or echoes another command id or token counts as no reply: UNKNOWN for a fence or a verify, not ready for readiness, no signal for health. Driven through the real fencing engine against a misbehaving TLS service, case by case (TestRepliesThatCountAsNone); the conformance kit runs the same check against qfence’s own client every time (client.unknown). A controller has at most max_concurrent_calls calls in flight; one more waits within its own deadline (TestConcurrencyIsBounded).
• Ready means ready. Promotion needs ready, reachable, healthy and replica_current together; ready alone never promotes, and UNHEALTHY is a signal while UNKNOWN is not (TestAnswersAreTyped).
• The service is pinned, and holds its own secrets (§4). qfence presents a client certificate named by credentials_ref and sends a service nothing unless it presents the certificate pinned at registration, or the next one an administrator pinned ahead of a renewal; a service not yet pinned is shown, never called (TestConnectPinsAndFetches, TestNextPinLeavesNoGap). No command carries a secret.
• A certificate change is never silent (2026-10-09). Before, a renewed certificate failed the controller with nothing but a generic ControllerFailed, and nothing said a certificate was about to expire. Now each test records the certificate presented and the pin it matched. One within 14 days of expiry publishes CertificateExpiring; one that matches no pin publishes CertificateNotPinned; both alert, are announced once per certificate (again daily while they last), and are attention items in the plans that bind the controller, as is a service that now presents its next certificate. New pins are tested against the service before they are saved, and the service must present one of them (TestCertificateRotation). Pinning a certificate never takes a new manifest.
• The manifest cannot change unnoticed (§2). It is fetched and fingerprinted at registration and kept in the controller’s configuration; one written by hand is refused. A service that later serves another fails the controller (and so every fence through it is UNKNOWN) until it is registered again, which is refused while a binding would lose its capability, role or level (TestChangedManifestFails, TestExternalControllerRegistration). An external manifest must be an application controller’s: kind: application, level application only, application roles and observations only, no settings or credentials of its own (TestServedManifestRules, ValidateManifest).
• Application-level fences never prove a site. A demotion is reported by application code; §14 exists because such reports can be wrong. Under ALL and ANY alike, a site whose required targets are all application-level is UNKNOWN whatever they report; an application fence never confirms a site; a required one is one more condition, which an escalation runs even after the site is confirmed; and a configuration where a failover could fence a site only through application fences is refused when it is loaded (TestApplicationFencesNeverProveASite). Before this phase it was not so: a site whose only required fence was application-level counted as proven and confirmed under either policy. No shipped type had an application-level capability, so no configuration could hit it. The dry run says what a required application fence costs: it must report FENCED too, and since it usually cannot answer for a site that is down, losing that site ends a failover BLOCKED.
• A switchover keeps §19’s order (§7). graceful demotes the old owner through its application-level fence and verifies each target, waits for the target’s readiness to report a measured lag of 0, then applies and verifies the plan’s required fences, confirms readiness and promotes, exactly as a failover. A demotion that does not verify, or no zero lag within the binding’s readiness_timeout, ends BLOCKED before any compute fence or promotion; a compute fence that fails after a verified demotion ends BLOCKED, unpromoted, with the service unmoved; a check that measures no lag never satisfies the wait (TestGracefulSwitchoverKeepsTheOrder, TestSwitchoverReadinessTimeoutBlocks, TestSwitchoverNeverPromotesBeforeTheFences). A switchover that cannot start (the plan not ACTIVE elsewhere, nothing to demote through) uses no generation and changes nothing (TestSwitchoverRefusalsChangeNothing). fence-then-promote is exactly a failover (TestFenceThenPromoteIsAFailover).
• Promotion is the last step before the service. A failover or switchover to a site with a switchover binding promotes its application through it after every other site is verified fenced, readiness is confirmed and the site’s own targets are unfenced, with the new generation token; a refusal blocks with the service unmoved (TestFailoverPromotesThroughTheSwitchoverBinding). The executor runs it only for a switchover binding of the command’s site, from the leader of the current term.
• The protocol refuses stale tokens twice. qfence’s executors refuse a command from a deposed leader; a service refuses a fence, unfence, recover or switchover whose token is older than the highest it has seen for the plan. The Go SDK does the second before a controller’s code runs (TestHandlerRefusesBeforeTheController), and the kit checks every external controller for it, requiring the stale_token refusal.
• An unfence carries its generation (2026-10-09). Before, Fencer.Unfence took no token, so an unfence sent by a leader that was deposed while it was in flight could undo the next leader’s fence, and only the executor’s term check stood in its way. Now promotion unfences under the new generation, and lowering shields or a standby fence under the plan’s current one. An executor refuses an unfence older than the plan’s generation as its site has it (TestExecutorRefusesAStaleUnfence); the promotion’s unfence carries the new generation (TestUnfenceOnPromoteReleasesOnlyThePromotedSite); the SDK and the kit’s unfence.stale-token hold external controllers to it. What remains: an unfence already past the executor when a newer leader fences the same target. A target that keeps tokens refuses it; the infrastructure fencers (Proxmox, Redfish, IPMI, Docker, FortiGate) ignore tokens, since their platforms have nowhere to keep one, so for them the window is that one call’s flight.
• An accepted data loss reaches the service (2026-10-14). A failover whose readiness an operator’s accepted loss covers (ADR 0006) sends switchover the accepted bound as accepted_lag_seconds, and a service that refuses a copy lagging past its own limit promotes one within it. Before, nothing told the service, so such a failover through an external controller was refused at promotion and ended BLOCKED with the service unmoved. The value reaches the service (TestAnswersAreTyped); the SDK refuses it on any other action, negative or beyond a duration’s range (TestHandlerRefusesBeforeTheController); and the kit holds every external controller to it (switchover.accepted-loss).
• The conformance kit (§9) runs against built-in types through their doubles (TestBuiltinHarness) and against a service through its conformance hooks (orchestrator conformance, TestExternalHarness); it catches a controller whose verify reports what it sent, whose health calls a stopped application healthy, that promotes while lagging, that refuses a lag an operator accepted, or that refuses old tokens without saying why (TestKitCatchesBrokenControllers).

Exit test (2026-10-09). A controller for an in-memory key-value store was written in Python, standard library only, by an author who read only the guide and the published schemas, never qfence’s source. It passed all 20 checks on its third run. The two failures before that were the kit’s: it left out lag_seconds when it meant 0, and its stale-token checks accepted any refusal (one passed on a not_ready refusal of a token that was not stale). Both are fixed. The author’s list of gaps in the guide was worked into it.

Verified on the demo lab (2026-10-09), plan orders-db: hosts pve-dc1-03 and pve-dc2-03 in the simulator, fenced through their BMCs, and the orders application through the mock application controller (-demo). A graceful switchover from dc1 to dc2 demoted dc1’s application, powered off pve-dc1-03, powered on pve-dc2-03 and promoted dc2’s application (generation 2, audited plan.switchover, and at dc2 executor.fence, executor.unfence and executor.switchover); a failover back promoted dc1’s through its own switchover binding. erp-db and web-portal were untouched. The plan page, the switchover dialog and the controller page pass axe-core in both themes.

The systemd application controller (ADR 0005, Phase 9.2)

The systemd type manages an application run as a systemd unit, on a host the site’s orchestrators reach over SSH (its guide). Nothing runs on the host but the commands qfence sends, as a dedicated user, through sudo rules that name each unit.

• No answer is UNKNOWN (ADR 0005 §3). A host that takes the connection and says nothing, or stops answering on an open one, is an error within the call’s deadline: UNKNOWN for a fence or a verify, not ready for readiness, no signal for health. After a timeout the connection is dropped and the next call connects again (TestHostThatHangs). A failover while the host hangs ends BLOCKED with its fence UNKNOWN, and the new owner’s copy is never started (the acceptance slice’s “the host stops answering”, against a real container frozen with docker pause). A controller runs at most max_concurrent_calls commands at once; one more waits within its deadline (TestConcurrencyLimit).
• The host key is pinned, and is the controller’s identity. A host whose key is not pinned, or is not the pinned one, is refused before any command is sent. The test reports the key it presented, so it can be checked against the host’s own and pinned (TestHostKeyIsPinned, TestSystemdControllerRegistration, and the console’s ControllerRegisterSystemd.test.tsx).
• sudo is checked for each unit, and must be narrow. A binding’s validation asks sudo (sudo -n -l) for exactly the systemctl commands its capability sends for its unit. It also asks about a unit nobody manages, and fails the binding if sudo would allow it: such rules make the qfence key a root key on the host. A plan whose fence names a unit the rules leave out is refused (TestValidateBinding, TestSystemdControllerRegistration).
• Verify reads the unit. FENCED only while the unit reads masked and inactive (or failed), durable only while the mask is the persistent one in /etc, never from what was sent (TestVerifyReadsTheUnit; the kit’s verify.reads-state). A unit whose file is in /etc/systemd/system cannot be masked, and its fence binding fails unit.maskable.
• Stale generations are refused on the host itself. A fence, unfence or restart runs as one script, holding a lock on a record per plan and unit in the user’s home directory. The script refuses a generation older than the record, and otherwise records it before acting, whichever of the site’s orchestrators sends the command. After a failover on a real host, the deposed generation’s unfence and restart of the old owner’s copy, and its fence of the new owner’s copy, are each refused and change nothing (the acceptance slice’s CAT1; the kit’s four stale-token checks). Unlike the infrastructure fencers, which keep no tokens, this type therefore also refuses an unfence that was already past the executor when a newer leader fenced the unit. Each plan’s generations are its own (TestGenerationsArePerPlan).
• Recovery cannot undo a fence. systemd refuses to restart a masked unit (TestRestartLeavesAFencedUnitDown).
• command.check is an argument vector, under a timeout. Each word reaches the host as it was given: quotes, $, backticks, ; and newlines included (TestCommandsAreArgumentVectors). It runs under timeout(1), which kills it and what it started when it overruns. Exit 0 passes and any other exit fails; an overrun (124, 137), a command that cannot run (125–127) and a host that does not answer are UNKNOWN (TestCommandCheck). It has no fence, recovery or switchover role.
• An application fence still never proves a site. The acceptance slice fences each site with a compute fence, and with the systemd fence as a further required condition. When sudo refuses the old owner’s fence, the failover ends BLOCKED, the old owner’s copy runs on, and the new owner’s is never started (CAT3). A stale replica blocks before promotion (CAT4).
• Where it was run (2026-10-09). The conformance kit and the acceptance slice (CAT1, CAT3, CAT4, stale generations, a host that hangs, through plans.Sequence on a real three-node raft) run in CI against an in-process SSH server with a model of systemd, sudo and the guard script (TestConformanceKit, TestAcceptance). With QFENCE_SYSTEMD_IT=1 they run against systemd 252, OpenSSH 9.2 and sudo 1.9.13 in a Debian 12 container on the local Docker context (TestConformanceKitInContainer, TestAcceptanceInContainer), and both passed: 18 of the kit’s 21 checks, the three switchover checks not applying.

The PostgreSQL application controller (ADR 0013, Phase 9.3)

The postgresql type manages a primary and its streaming standbys over SQL alone, with TLS and the server’s certificate pinned (its guide). Its role is never a superuser.

• Nothing is sent before the pin matches, and the environment cannot weaken the connection. The certificate is checked in the TLS handshake, before the role’s name is sent; an unpinned or other certificate is shown for checking and pinning (TestObservation’s “connect pins the server”, the console’s ControllerRegisterPostgres.test.tsx). No PG* environment variable can change the host, the credentials, TLS or fallbacks. A superuser role fails privileges, and each missing grant is named with its statement.
• No answer is UNKNOWN; a refusal is UNHEALTHY. A frozen server is an error within the call’s deadline: UNKNOWN for a fence, not ready, no health signal (TestObservation, and the acceptance slice’s “the primary stops answering”: BLOCKED, its fence UNKNOWN, the standby not promoted). The server’s own error, or a TCP connection its host refuses, is a proven UNHEALTHY.
• Health depends on the owner. At the owner site a standby or a read-only server is UNHEALTHY; elsewhere a standby is healthy. The caller says which site owns the plan, executors included (TestHealthChecksKnowTheOwner, TestExecutorRestartsOnlyTheActiveOwner).
• The lag is the bound on what a promotion could lose, read on one clock per term. It grows from the primary’s last message once the primary is gone, and a lag that cannot be read is not ready (TestLagOf, and TestObservation’s silent and gone primary). A WAL receiver’s reading counts only once the primary has spoken: before its first message, a receiver reports its own start as both times (TestLiveReading). The last reading survives an orchestrator restart, written at most every 15 s, so a restart can only make the lag larger (TestLastReadingSurvivesARestart). With replication cut, the lag counts up from the last message (acceptance slice).
• A primary is ready only where the plan was last ACTIVE (ADR 0013 §3, as amended). Raft records that site; the sequence tells readiness the one its acquire read, and every other caller reads it from raft. An old primary not rejoined, or a standby promoted outside qfence, is not ready, so a failover back to it cannot serve diverged data (TestReadinessAtAPrimary, TestReadinessIsToldWhereThePlanWasLastActive, TestLastActiveOutlivesOwnership; the graceful switchover case checks dc1’s old primary).
• The fence is read from the server. db.readonly is FENCED only while default_transaction_read_only is on from the configuration file and no application session holds a transaction ID, durable because it survives a restart (TestActuation; a fenced server restarted stays read-only in the slice). It ends every non-superuser application session, identified by role, not by an application name a client can set.
• Stale commands are refused on the server. Each plan’s newest generation lives in the server’s own configuration, read through pg_file_settings under an advisory lock on a connection whose close releases it. A deposed generation’s fence, unfence or promotion is refused at either server after a failover (CAT1; the kit’s three stale-token checks). A value qfence did not write refuses every guarded command rather than being overwritten.
• Zero lag is measured. A graceful switchover verifies the demotion again each round, reading where the old primary’s WAL ends, then reads how far the standby has replayed, and promotes only once it has replayed all of it. With an application writing throughout, no committed row was lost (the acceptance slice’s first case). A read-write transaction held open after the fence blocks it.
• Promotion checks its own readiness, and runs once. db.promote refuses a lag beyond its limit unless the operator accepted that loss for this failover (ADR 0006; TestAcceptedLossReachesThePromotion, the kit’s switchover.not-ready and switchover.accepted-loss). The same promotion sent twice at once promotes once (CAT1).
• Where a cluster manager runs, qfence only observes. A declared manager, or a sign of one, fails no-cluster-manager, so db.readonly and db.promote are not advertised (TestActuation, TestManagerWithdrawsActuators).
• Where it was run (2026-10-14). The conformance kit (19 of 21 checks, recovery not applying) and the acceptance and chaos slice (a graceful switchover with a writer, CAT1 with stale generations and a double promotion, CAT3, CAT4, a frozen primary, replication cut, a held read-write transaction, a fenced server restarted), through plans.Sequence on a real three-node raft, against a TLS primary and streaming standby of PostgreSQL 15 and of 18 on the local Docker context (QFENCE_POSTGRES_IT=1), and in a docker:dind service as CI runs them on every push (test_postgres).

Residual risks.

• The fence is soft. A session that asks for BEGIN READ WRITE can still write after db.readonly until the plan’s required fence stops the server. One that keeps its transaction open blocks a graceful switchover; one that commits short transactions between the wait’s rounds can pass it, and what it writes after the fence is lost with the old primary. An application-level fence never proves a site, so the required fence is what makes the failover safe.
• Rejoining is manual. An old primary keeps its unreplicated transactions until an operator rejoins or re-seeds it, and reads not ready until then.
• Finding no cluster manager is inferred. A manager that leaves no sign over SQL is not found: declare it.
• The lag bound is optimistic by one network trip.

The Keycloak application controller (ADR 0014, Phase 9.4)

The keycloak type observes Keycloak over HTTPS and changes nothing in it (its guide). It reads the base URL and the management interface, with both certificates pinned. Restarting, fencing and starting Keycloak stay with the Docker or systemd controller that runs it, and its database is its own plan, which the Keycloak plan depends on at the same site.

• Nothing is sent before the pin matches. Each endpoint’s certificate is checked in the TLS handshake before any request, so no sign-in or secret reaches another server (TestNothingSentWithoutThePin, TestConnect). A proxy from the orchestrator’s environment is never used, and a redirect is never followed. No error carries the probe user’s password or the admin account’s secret, even when Keycloak repeats what it was sent (TestValidate, TestDiscover).
• Only Keycloak’s own failures are UNHEALTHY. These are: readiness DOWN, a sign-in or sign-out that fails on the server, a realm that does not exist, and a connection the pinned host refuses. A timeout, or a reply cut off, is UNKNOWN. So are refused probe credentials: a wrong or rotated password, a missing or locked-out user, a missing client. They say nothing about Keycloak, and a rotated password must never fail a plan over (TestRealmHealth; the chaos slice).
• Readiness alone misses a database that refuses writes; the probe user does not. Measured on Keycloak 26.7.4, readiness stayed UP while sign-ins failed with HTTP 500 after 10.1 to 10.4 s. With a probe user, realm.health reads UNHEALTHY; without one, HEALTHY (the chaos slice). The probe has 15 s. Every application health check now has 20 s, whatever its interval: it had the interval, 10 s by default, which would have read that failure UNKNOWN (TestHealthCheckDeadline).
• A Keycloak stopped by design is not a broken controller (ADR 0014 §3). Once registered, a connection its pinned host refuses reads not running. That is UNHEALTHY, which at a standby site triggers nothing. The controller stays ready, with its checks unverifiable and Not running shown beside its state. An operator’s test, at registration, when a capability is enabled or validated, or when its pins or credentials change, needs Keycloak running. So a stopped Keycloak cannot be registered with a fingerprint typed in (TestConnect, TestValidate, TestNotRunningOnceRegistered).
• The probe user and the admin account can change nothing. Registration refuses a probe user with any role beyond the realm’s defaults, read from the token its own sign-in returns. It refuses an admin account with any role except the view-* and query-* roles of realm admin clients. Both are proven against real Keycloak (TestValidateBinding, TestRegistrationAgainstKeycloak).
• A health check leaves no session. The sign-out follows every sign-in, even when the token cannot be read (TestRealmHealth, and against real Keycloak, TestChaos’s count).
• Where it was run (2026-10-14).
  • Conformance kit: every check that applies passes; fence, recovery, readiness and switchover do not apply.
  • Chaos slice:
    • Keycloak stopped: UNHEALTHY, not running.
    • Keycloak frozen: UNKNOWN.
    • Its database stopped: UNHEALTHY 27 ms after the stop, readiness’s database check DOWN.
    • Its database read-only: UNHEALTHY after 10.4 s with a probe user, HEALTHY without one.
    • Keycloak restarted on a read-only database: UNHEALTHY, and never ready.
    • The realm deleted: UNHEALTHY.
    • The probe user’s password changed: UNKNOWN.
  • Against: Keycloak 26.7.4 on PostgreSQL 18, on the local Docker daemon and through a docker:dind daemon as CI runs them (test_keycloak).
  • On the swarm lab: the sso plan runs Keycloak on erp-db’s cluster, fenced to no replica at the standby, recovered by service.restart and depending on erp-db. Its standby read not running throughout, its controller ready. After a graceful switchover of erp-db, sso followed through health alone with the defaults: three restarts, then a failover, 4 min 43 s without sign-ins. Its registration in the console against that Keycloak ran end to end in a real browser (tests/swarm/register-keycloak.spec.ts).
  • Found there and fixed: an application’s new owner was restarted 5 s after its promotion. Its health binding had read UNHEALTHY for the 9 min 36 s its site was a standby, the application stopped by design, and that time counted toward unhealthy_for. A new owner’s health now counts from the change of owner (TestNewOwnerCountsFromItsPromotion).

Residual risks.

• Sign-ins are down after the database moves: about a minute with follow, about 5 minutes without. With follow (ADR 0015), the Keycloak plan fails over once its database’s plan is ACTIVE at the new site: 59 s and 64 s measured on the swarm lab, held longer by its breaker’s cooldown if the database moved within it. Without, its old site’s Keycloak fails, recovery’s restarts cannot help, and the plan fails over once they run out: 4 min 43 s measured with the defaults (ADR 0014 §7).
• Healthy is measured from the orchestrator. It does not prove users reach Keycloak through their load balancer or proxy, or that realms other than the bound one work.
• No same-cluster check. A Keycloak promoted on the wrong database is caught only when the probe’s realm is missing there (UNHEALTHY) or its user is (UNKNOWN, then an attention item).
• A restart restarts every realm, so a probe failing in one realm restarts them all.
• Direct sign-in is not in OAuth 2.1. If a later Keycloak drops it, realm.health falls back to readiness alone and no longer catches a database that refuses writes.

A plan follows its same-site dependency (ADR 0015, Phase 9.6)

A same-site dependency with follow: true makes the raft leader fail the dependent over to the site where the dependency is ACTIVE, as an automatic failover (ADR 0015). It adds no new way to promote: a follow is the scheduler’s automatic failover, aimed at that site.

• Opt-in, and inert unless automatic failover runs. follow is refused without same_site (TestFollowNeedsSameSite). It does nothing unless the plan has automatic_failover: true and the leader runs -automatic-failover, and the plan’s status says which is missing. same_site without follow is the gate it was (TestSameSiteWithoutFollowStaysAGate).
• The trigger is committed state, read on the leader. A followed dependency ACTIVE at another site, read through the same verified read as the dependency gate. A read that fails triggers nothing. Only the leader’s scheduler acts, and a change of leader loses nothing, since the trigger is a state rather than an event (TestChainFollowsItsRootLinkByLink, on a real three-node raft).
• Never in place of site mode or recovery. An unreachable owner is site mode’s. An owner found fenced, or a recovery in flight, is automatic recovery’s. A dependency moving, being recovered, or ACTIVE at an owner found fenced holds the follow until it settles (TestFollowWaitsForEveryOtherCondition).
• Nothing is fenced before the target’s readiness answers. Readiness at the target is checked read-only, with 30 s, before the sequence acquires or fences anything. A refusal fences nothing and holds the follow for 15 s (TestFollowChecksReadinessFirst).
• The same gates and the same breaker. The dependency gate at the target (every same-site dependency ACTIVE there), the configuration-drift gate, then the plan’s own breaker: a follow is held by its cooldown, and counted toward max_automatic_failovers. The cooldown’s refusal now says when it ends (TestFollowIsHeldByTheBreaker).
• A dependent never moves past a dependency that stayed. In a chain, each link follows once the link below it is ACTIVE at the new site, one link per round, under its own breaker; a link held by its tripped breaker keeps the links above it (TestChainFollowsItsRootLinkByLink).
• Every automatic failover aims where its dependencies are. A lost site, an application’s escalation and a follow all target the first reachable site where the plan’s same-site dependencies are ACTIVE, and otherwise the first reachable site, as before. With two sites this changes nothing (TestAutomaticFailoverGoesWhereItsDependenciesAre).
• It is shown before and after. The plan read’s follow status (idle, following, waiting with the scheduler’s reason, or inert); a dependent left at a site where a same-site dependency is no longer ACTIVE is an attention item, on its dependency’s link and in A1 and A2; the dry run of a failover or a switchover lists the dependents that will follow and those that will be left, and never refuses (TestFollowIsShown, TestPortalFollowsErpThroughTheScheduler; the console’s unit tests and tests/e2e/10-follow.spec.ts, against the real orchestrator with axe-core in both themes).
• The recorded lab decisions gain follow on the swarm lab’s sso and web-portal, and are otherwise reproduced; the demo lab’s are unchanged (TestLabDecisions).
• On the swarm lab (2026-10-15), with sso and web-portal following erp-db, real binaries and real controllers: lab.sh switchover erp-db dc2, then back to dc1. Each time, within about a second of erp-db becoming ACTIVE at the new site, the leader failed sso over after it (“erp-db is ACTIVE at dc2: following it”), then web-portal. Nothing was restarted at the old site, both databases held every order, and each breaker counted two of its three. Sign-ins (lab.sh signins) were down for 64 s, then 59 s, against 4 min 43 s without follow (9.4b, same lab, dc2 to dc1): the switchover (19 s, 22 s), the follow (8 s, 9 s), and Keycloak’s start (37 s, 28 s).

Residual risks.

• A plan that follows moves while its own site and application are healthy, and its old copy is fenced. A dependency moved by mistake takes its followers with it, each within its own breaker. The dry run lists them first.
• A follow that fails after fencing leaves the plan BLOCKED, stopped at both sites. Checking readiness first removes the common cause, not every cause: readiness can still change between that check and the sequence’s own.
• A dependency moved twice within a dependent’s cooldown leaves the dependent behind until the cooldown ends. The attention item says when.
• Followed dependencies ACTIVE at different sites leave no site where all of them are: the plan stays, refused with that reason.
• A dependency on another orchestrator’s plan never follows. It reads as never ACTIVE here, which fails closed (see “Availability plan Dependency Engine”).

New capabilities on FortiGate, Proxmox and Docker (ADR 0010, Phase 10.5)

Phase 10.5 adds network fences that cut a firewall interface or VLAN, or a VM’s NIC; compute fences that stop a whole Proxmox node or drain a swarm node; readiness from firewall probes and from the Proxmox guest agent; and Proxmox password login. Each fence can do damage if it is pointed at the wrong thing, so each carries a check. What keeps them safe:

• A firewall fence never cuts qfence’s own path (not-management-interface, §15). interface.down and vlan.down look at the interface and at everything that goes down with it: the VLANs on it, and an aggregate or switch it is a member of. The fence is refused when any of those carries the address qfence reaches the firewall at, is dedicated to management, is an HA management interface, or is where the firewall’s route back to this orchestrator leaves.
  • The check runs at binding (a failure refuses the binding) and again before every fence. The fence is also refused when the interfaces cannot be read. A refused fence changes nothing, so the interface verifies up, the fence is FAILED and the failover is BLOCKED.
  • Each site’s executor makes the check itself, since the fence runs there.
  • Tests: TestInterfaceFenceRefusesTheManagementPath (one case for each way an interface can be on the path), TestEndToEndManagementInterfaceFenceBlocksPromotion (through the real sequence: BLOCKED, management port still up, VIP not moved).
• A node fence is never sent through the node it fences (endpoint-not-on-node). Before every fence, node.fence asks which node answers the controller’s API (/cluster/status’s local node). It refuses when that is the target, and when it cannot tell. Binding validation refuses such a binding too (TestNodeFenceRefusesTheEndpointNode, TestE2E_NodeFenceFailoverAndEndpointRefusal).
• A fence is judged by what the target reports, never by the command it accepted.
  • node.fence is FENCED only while the node answers with no guest running. A node the cluster sees offline is UNKNOWN, because a node that shut down and one cut off from the cluster look the same. Once mode shutdown has taken a node down, only another binding, such as a BMC’s power evidence, can prove it off (TestNodeFenceShutdown).
  • vm.link-down reads the configuration the VM runs now (current=1). A change Proxmox holds as pending is FAILED, with the reason (TestLinkDown).
  • node.drain is FENCED only once the node is drained and the swarm reports no live task on it. A down node that still has tasks is UNKNOWN (TestDrainFencerCannotSeeADownNode).
  • Each fence repeated is a no-op (§37).
• Durable only when a restart keeps it. A stopped node’s fence is durable only when none of its guests starts on boot or is one Proxmox HA wants started (TestNodeFenceStopGuests). Interface and NIC fences, and a drain, are saved configuration, so they are durable.
• Moved work is named. Draining a swarm node moves its tasks to the swarm’s other nodes, and Proxmox HA can start or move a guest it manages. So binding validation says where a drained node’s tasks will go (swarm.other-nodes), and warns when HA manages a guest on a node being fenced (ha.not-managed, unverifiable). A site fence drains every node of the site (ADR 0010 §4).
• Probes prove readiness or block it. A link monitor that is not alive on every interface it watches, a real server the firewall reports down, an agent that does not answer, and a command that exits non-zero are all “not ready”. A monitor, server or VM that is not found is an error. Both block promotion (CAT4’s rule): TestEndToEndDeadLinkMonitorBlocksPromotion, TestE2E_FailingServiceProbeBlocksPromotion.
• A VIP’s ownership is read from the firewall (found in 10.4, fixed). vip.own verifies that the VIP is enabled and carries this generation’s tag. It no longer checks this process’s memory of having acquired it, so every orchestrator gives the same answer after a leader change or a restart. Acquire, release and verify also refuse a token older than the generation the firewall’s tag names (§40), which also catches an orchestrator that restarted and remembers no later generation (TestServiceVerifiesFromTheFirewallsTag, TestServiceRefusesWhatTheFirewallsTagMakesStale).
• Password login keeps the ticket in memory only. The Proxmox ticket and its CSRF token are fetched again after 90 of the ticket’s 120 minutes, or once when the cluster refuses them. A failed login degrades the controller. The password is never in an error, and an account that needs a second factor is refused (TestPasswordLogin).

Verified on the swarm lab (2026-10-03):

node.drain also ran against a real Docker Engine, a one-node Docker-in-Docker swarm (TestAgainstRealSwarm, QFENCE_DOCKER_IT=1). The drain took 9.2 s. The service’s task stopped, the node carried qfence.availability=active, and after undraining the task ran again.

Residual risk. No new FortiGate or Proxmox capability has run against the real system. Each follows the vendor’s documented API and has been tested against a double, so treat them as protocol-correct on paper (see “Known gaps”). The route-back part of not-management-interface is an inference behind NAT. It also covers only the path of the orchestrator that runs the check, which is the one that fences.

Return to standby (ADR 0009 §6)

A site that was fenced off, its hosts powered off by a fence or a watchdog, used to wait for someone to power it on, check that its workloads came up in their standby role, and wait for replication. ADR 0009 §6 does this, keeping the site fenced throughout. What keeps it safe:

• Never an unfence. Only the site’s power fences are released: targets whose fencer switches power, and the hosts of its dead-man switches. Every other fence, such as its workload’s standby shape, stays. Principle (c) holds: only promotion unfences.

• Only where a standby fence exists. A site whose only required fence is its power would be left unfenced. The action refuses it, and return_to_standby: automatic is rejected at validation.

• Confirmed before it counts. Once power is back, the site is fenced again, power excluded, until it is confirmed (fenced durably) or 5 min pass. Its hosts boot first. If it is never confirmed, it is fenced again with the whole ladder, power included, and the action fails, audited.

• Shields first. The owner’s shields are raised before anything is powered on.

• Serialized. The action holds the group’s lock, so no failover, check or returning-site fence runs meanwhile. A debt the site owed is cleared only once it is confirmed.

• Readiness decides “ready”. The site is standby-ready when its readiness passes. The site checks keep asking for confirmed sites, and the console shows each site as fenced, confirmed, or ready to be promoted.

• Not forced. An Operator may do it (POST .../sites/{site}/standby); it is audited as plan.site.standby.

• Automatic, opt-in (fencing.return_to_standby: automatic). The leader does the same by itself, at most once per site and generation, retried after 5 min if it fails. All of these must hold:

  • the site is held off: it owes a fence, or it is fenced but not confirmed in its standby role (its power is off);
  • its orchestrator is reachable and its power fences answer;
  • it is not marked “being prepared”;
  • the owner has a required shield;
  • it cannot start in its active role: it was confirmed fenced in its standby role at this generation before its power went (leader memory), or the group declares boots_fenced. “In its standby role” means by a fence that stays once power is back, such as its shape. A durable power fence confirms a site, but this action releases it;
  • when it switches power, no other group at the site could start in its active role there either. Each other group with the site among its sites is held by that site, was never granted, was confirmed there in its standby role at its generation, or boots fenced. qfence is not told which groups share hosts, so every group at the site counts.

  Otherwise it waits for an operator.

Tests (internal/api/standby_test.go):

• an Operator’s return releases only the power fence, confirms the standby shape, is ready, and is audited as not forced;
• refusals for the owner, a Viewer, and a site whose only fence is its power (never released);
• validation of automatic for such a site;
• the automatic return of a site confirmed and then powered off, which clears its debt;
• no automatic return while another group at the site could start active there, until it boots fenced (TestAutomaticReturnToStandbyWaitsForOtherGroups);
• a site proven only by a durable power fence: no automatic return unless the group boots fenced, and then it returns (TestAutomaticReturnAfterAPowerFence).

Verified on the swarm lab (return_to_standby: automatic for both groups):

• dc1 was partitioned. The leader proved it fenced by its database’s shape and failed over 40 s later, and dc1’s watchdogs powered it off at 4 min.
• Once dc1’s links were restored, the leader returned it to standby by itself within 50 s: it released bmc-dc1, powered on both nodes through IPMI, confirmed the standby shape, and cleared its debt.
• 51 s later dc1 was ready to be promoted, a standby streaming from dc2.

Found on the way: after the lab was restarted with its state, its writer had stopped, and the standby had replayed no transaction since it started, so its lag was unknown. The early proof did not fence, correctly, since the standby was not ready. The backed failover at the bound fenced dc1 and stopped at BLOCKED on readiness, as designed.

Found while building §7, and fixed:

• Other groups on the same hosts. Returning one group’s site powered on hosts that ran another group’s workloads, whose shape at that site was never confirmed. Only that group’s shield and its fence debt stood between them and a second active copy. Automatic return now looks at every group at the site.
• Confirmed by power. A site proven by a durable power fence (always-off) counted as “confirmed at this generation”, though its shape was never checked. Powering it on could have started it active. Only a standby fence counts now.
• Never returned. A site held off by a durable power fence was never due, since only debts and transient fences triggered. It is due now, and still needs a standby confirmation or boots_fenced.

Residual risk:

• The boot window of a site that lost power before its standby fence was applied: its workloads start in their last role for the seconds until it is confirmed. The shields keep them from the owner’s resources, and §7’s lease closes the window for workloads that ask.
• “Confirmed at this generation” is leader memory. After a leader change, the automatic return waits for an operator unless the group boots fenced.
• A manual return to standby is the operator’s decision: it does not check the site’s other groups. The confirmation dialog warns that workloads may start in their last role.

Booting fenced: the lease (ADR 0009 §7)

A site that lost power before qfence could fence it into its standby role comes back with its workloads in their last role. For a primary database, that is a second primary for the seconds until qfence fences it. The lease lets a workload ask first:

• The question. GET /api/v1/availability-plans/{id}/lease?site=S with a Viewer token. S may start active when it owns the group and the group is PROMOTING or ACTIVE there.
• Only a linearizable read answers. The leader verifies its majority before it reads. A leader that cannot is “unknown”.
• Followers ask the leader. The question is signed on the peer channel under the cluster keyring (ADR 0016 §1), bound to its purpose, and carries a nonce. The leader signs its answer under the key the question named, and the answer must echo the nonce. An answer that is forged, replayed from an earlier question, or signed under another key is rejected, and the follower says “unknown”. A follower without a reachable leader, or whose key the leader does not hold, also says “unknown”. The answer relies on the signature, not on transport security, although the call is also pinned to the leader’s certificate since Phase 11.1 (ADR 0011 §13).
• No answer, no active role. A workload that gets “unknown”, or cannot ask, does not start in its active role. This is the §40 fencing token applied at boot.
• Declared, not assumed. fencing.boots_fenced: true says a group’s workloads ask. It relaxes automatic return to standby (§6): such a site may be powered on even when its shape was never confirmed. qfence cannot check the declaration; a group that declares it and does not ask has the §6 boot window, without the operator in the loop.

Tests:

• internal/api/lease_test.go, on a real three-voter cluster: the leader’s answer for the owner and another site, a site not in the group, a follower before it knows the leader’s address, a forwarded answer, a follower without the secret, one with another secret, an answer replayed by something between follower and leader, and a leader cut off from its majority.
• internal/labsim/lease_test.go: the lab relay asks the site’s own orchestrator for its own site, answers “unknown” without a token or an orchestrator, and logs each change once.

Verified on the swarm lab (both groups boots_fenced: true; lease.sh in erp-db, erp-app and web-portal):

• On the lab’s first start, before qfence granted anything, the workloads waited for an answer, then kept their deployed role (generation 0).
• dc1, the owner, lost power (Fail), and erp-db failed over on the BMC’s proof, leaving dc1 a transient debt. On Restore dc1’s nodes powered on with its database still ROLE=primary. It asked for its lease 4 s after dc1’s orchestrator started and was refused (“dc2 holds the group”). It re-seeded from dc2 as a standby and was read-only 2 s later. qfence paid the debt 19 s after the refusal. Without the lease, dc1 would have run a second primary for about 20 s, kept from clients only by dc2’s shields.
• dc2, then the owner, was blacked out. erp-db failed over at the bound on dc2’s dead-man switches (4 min 33 s), and web-portal 5 s later. On Restore qfence paid both attested debts within 6 s, before dc2’s tasks started, so its database came up as a standby without asking. The debt fence and the lease are two layers: whichever comes first holds.

Residual risk:

• Workloads that do not ask. The lease is opt-in for each workload, and boots_fenced is only a declaration.
• The lease is asked at boot. A workload already running is stopped by fencing, not by the lease.

Shields by proof strength, and only promotion unfences (ADR 0009 §4, §5)

A promotion can rest on something weaker than proof that the old site stopped:

• an attestation, where a person stands in for the proof;
• network-only fencing, which proves the site isolated, not stopped;
• a self-fence promise in the software tier.

In each case the site that took over must be able to protect itself. ADR 0009 §4 makes a required shield mandatory for all three:

• Network-only fencing: a group with allow_network_only_fencing: true needs a required shield at every site. Its configuration is rejected otherwise.
• The software tier: a group whose self_fence.backs_failover can fail over away from a site without dead-man switches needs a required shield at every other site. Its configuration is rejected otherwise.
• Attestation is allowed by default, so it is checked when used, not at validation. An attested failover without a required shield at the promoted site is refused, and the failover wizard’s dry run says why. requires_shield: false is rejected.
• Only promotion unfences (§5, principle c). No API unfences a site that does not own the group: an unfenced site may serve, which is a split brain. Returning a site to standby (§6) keeps it fenced.

Tests:

• internal/api/backing_test.go (TestShieldsRequiredByProofStrength): the software tier and network-only fencing without shields are rejected, and accepted with them.
• internal/api/shields_test.go (TestAttestationRequiresShield): the refusal, and false rejected.
• internal/api/durability_test.go (TestOnlyPromotionUnfencesASite): manual fences, prepare marks, checks, drift and debt fences, a recovery of the owner and a dry run never unfence dc2; promoting it does, once.

Verified on the demo lab, which gained shields (FortiGate policies) for this:

• A blackout of dc1 was recovered by an attested failover that first raised dc2’s shield.
• dc1 then owed a fence. When its BMCs answered again, qfence powered its host off durably within 2 s.

Durable power fences (ADR 0009 §3)

A BMC that powers a host off proves it off now. Its power-restore policy decides whether the host starts again when power returns, so a power fence was transient on most servers. ADR 0009 §3 makes it durable. What keeps it safe:

• Set before power-off. A Redfish or IPMI power fence first sets the policy to always-off (PowerRestorePolicy: AlwaysOff, chassis policy always-off), then powers the host off. A policy that cannot be set does not stop the power-off. The fence then reads as transient, and §1’s debt takes over.
• Verified, not assumed. The fence counts as durable only when the BMC reports always-off.
• Undone by unfence. Before powering on, unfence sets unfenced_power_restore_policy: LastState for Redfish, previous for IPMI, both configurable. This is best effort: a BMC without the setting still powers on, and a promotion never fails over a policy.
• Scope.
  • Power evidence (fence_action: none) never changes the policy.
  • durable_power_fence: false turns this off for a provider, leaving its power fences transient.
• A persistent change to hardware. A host set to always-off stays off after a facility power loss until someone, or a promotion, powers it on. That is the point: a fenced site stays fenced (principle a). The simulator page marks such hosts “stays off”.

Tests:

• internal/controllers/infrastructure/redfish/fencer_test.go (TestDurablePowerFence):
  • a fence sets AlwaysOff and verifies durable;
  • unfence sets LastState and powers on;
  • opting out and power evidence leave the policy alone;
  • an unknown policy is rejected.
• internal/controllers/infrastructure/ipmi/ipmi_test.go (TestDurablePowerFence): the same for IPMI.
• internal/labsim/links_test.go (TestPowerRestorePolicy): the simulator honours the policy when a site’s power returns.

Verified on the demo lab:

• A failover of dc1 set both its hosts to AlwaysOff and forced them off, and dc1 was confirmed fenced with nothing owed.
• Restore left the hosts off.
• After they were powered on by hand, a failback set LastState again and powered on and started the database.

Residual risk: a site whose only required fence is its power leaves no fence in place once it is powered on. The checks report that as drift, and returning such a site to standby is refused (ADR 0009 §6).

Least force: escalation under ANY (ADR 0009 §2)

Under success: ANY, one proof per site is enough, but qfence used to fence every required target anyway. A BMC set to power off did, even when the database’s demotion had already proved the site stopped. Now each site’s required targets are tried in their configured order, and the first that confirms the site (§1: fenced durably) ends its ladder. What keeps it safe:

• The verdict is unchanged. A site still needs one proof. A transient proof does not end the ladder, so a power-off never cuts short a durable fence further down.
• A network fence cannot end the ladder of a site with a Level 1 target (§14), since it proves isolation, not that the workload stopped.
• Secondary targets still run. Since 2026-10-12 a site’s secondary application fences run before its required ones and its other secondary targets after them (see the next section).
• Time is still bounded. execution: PARALLEL fences sites in parallel, and the targets of one site one after another. The self-fencing time W counts every rung of the site’s required ladder, so the dead-man timeout and the backing bound grow with it (4 min and 4 min 12 s for the lab’s erp-db).
• Unfence restores power first: targets whose fencer switches host power (Redfish, IPMI) come before everything else, since nothing on a host can be unfenced while it is off.
• Opting out. fencing.escalate: false fences every required target, as before; it is rejected outside ANY.

Tests:

• internal/fencing/escalation_test.go:
  • the first confirming target ends the ladder and the power fence is never used;
  • a transient proof and a network fence do not end it;
  • sites escalate independently, in parallel;
  • escalate: false fences every target;
  • unfence restores power first.
• internal/api/fencingmodes_test.go (TestEscalateSettingAndLadder): the setting round-trips, is rejected under ALL, and a failover leaves the BMC alone once the shape confirms the site.

Verified on the swarm lab, where erp-db lists its database’s standby shape before its BMC: a partition of dc2 failed erp-db over 40 s after the cut, confirmed by the shape, and the BMC was never asked.

Residual risk: the order is the operator’s. A power fence listed before a shape fence still powers the site off first; the editor and the documentation say to list the gentlest first.

Secondary application fences first (ADR 0009 §2, decision 14, 2026-10-12)

A site’s secondary application fences (a systemd unit.stop, a database demotion) ran after its required fences. When the required fence powered the application’s host off, the application fence found nothing to talk to: in the demo lab, billing’s unit.stop read UNKNOWN on every failover and never masked the unit. They now run first. What keeps it safe:

• The verdict is unchanged. An application fence never proves a site (ADR 0005 §7), and a secondary one never counts at all, so moving it changes no promotion. The required fences run exactly as before, escalation included, once it is done.
• The wait is bounded. The step takes at most fencing.secondary_application_timeout_seconds (default 10 s, at most 600), however many targets there are: every site’s run at once, each site’s in configured order. One cut short is UNKNOWN with the reason, and the next of its site is not tried; the required fences then start.
• W counts it. A site’s self-fencing time, and with it its dead-man timeout and the self-fence backing bound, grows by the timeout when the site has a secondary application fence, so a dead-man switch never fires while the active self-fence may still be within its time.
• Only application fences move. Secondary network and resource fences act on infrastructure that a power-off does not take away, and still run last.
• The fencing overview marks them “runs first”, and the plan editor has the timeout.

Tests:

• internal/fencing/escalation_test.go: TestSecondaryApplicationFencesRunFirst (application fence, then the required power fence, then a secondary network fence, under ALL and under an escalation); TestSecondaryApplicationFencesAreCutShort (a hung fence is UNKNOWN after the timeout, the next of its site is not tried, another site’s runs at the same time, and the site is proven all the same).
• internal/api/backing_internal_test.go: W counts the timeout only for a site that has a secondary application fence.
• internal/api/fencingmodes_test.go: the setting round-trips and a negative value is refused.

Residual risk:

• A failover whose old site cannot be reached waits up to the timeout before its required fences start: 10 s more by default, for the plans that have a secondary application fence.
• A fence cut short may still finish on its host: an SSH command already sent runs on. It does what a fence does, stopping the application, under the generation it was sent with (§40).

Durable fences: a site stays fenced when it restarts (ADR 0009 §1)

A fence proves a site stopped now. Whether it stays stopped when the site restarts depends on the fence: a service whose spec asks for its standby shape does, but a host powered off may be powered on again by its BMC when power returns. ADR 0009 §1 makes the difference part of the verdict. What keeps it safe:

• Observed, not assumed. Each fencer reports whether a FENCED target’s state is durable from what it reads, never from what it sent:

  Fencer	Durable when
  Docker service	always: the spec asks for the fenced shape
  Docker network	always: the change is in the spec
  Docker container	its restart policy is no or unless-stopped
  Redfish	off, with PowerRestorePolicy AlwaysOff
  Proxmox	stopped, and not set to start on boot
  FortiGate	always: saved configuration

  A fencer that cannot tell reports transient. IPMI does, until ADR 0009 §3.

• Confirmed means durable. A site is confirmed fenced only when one of its FENCED targets that can prove it (§14: Level 1 where it has one) is durable. A proof is enough to promote; a transient one leaves the site owing a durable fence (a transient debt, beside ADR 0008’s attested ones).

• Every path records the debt. It is recorded for a site fenced only transiently by:

  • a failover of any kind, before promotion (a failure to record it blocks);
  • a site check or a drift fence;
  • a manual fence of that site.

• Paid only by durability. The returning-site loop fences the site every 5 s while it is fenced only transiently, and clears the debt once it is confirmed. A manual fence clears it only when it confirms the site. While a debt is open, the survivor’s unproven shields stay up.

• Self-fencing aims for durability too. A site that fenced itself only transiently tries again every 10 s, and its report says whether it is confirmed.

Tests:

• internal/fencing/fence_error_test.go: TestSitesConfirmedOnlyByDurableProof, including that a durable network fence does not confirm a site with a compute target.
• The fencers’ own tests: container restart policy, Redfish power-restore policy, Proxmox onboot.
• internal/api/durability_test.go:
  • a transient failover leaves a transient debt that only a durable fence pays;
  • a retry waits 5 s;
  • a site check and a manual fence record one, and a durable manual fence pays it.
• internal/api/selffence_test.go: TestSelfFenceRetriesUntilConfirmed.

Verified on the swarm lab, whose database BMCs give power evidence only (fence_action: none, no restore policy): a power loss at dc1 failed erp-db over on the BMC’s proof and left dc1 owing a durable fence. When dc1’s nodes came back, the loop fenced its database into its standby shape within seconds. Before, the next check found it running as a primary up to a minute later.

Residual risk:

• The seconds between a site’s workloads starting and its fencers answering remain, for workloads that do not ask for a lease (ADR 0009 §7).
• A fencer’s durability is only as good as what it can read. For example, a Proxmox VM managed by HA may be restarted by HA whatever onboot says.

Self-fencing: a site cut off from quorum stops itself (ADR 0008 §4)

Remote fencing needs the survivor to reach the site it fences. §16 asks for a second line of defense that does not. ADR 0008 §4 builds it into the orchestrator, opt-in per group with fencing.self_fence. What keeps it safe:

• It only ever stops things. A site’s orchestrator that has been out of quorum contact for after_seconds fences its own site’s targets, for each self-fencing group its last-known state says it owns. It uses the group’s own fencers and policy, and retries until they verify as fenced. It never unfences: bringing a service back belongs to the leader once quorum returns.

• Quorum contact is measured on the monotonic clock. A follower is in contact while a leader reaches it; a leader while it can verify its majority. An orchestrator that starts counts from its start, so one that comes back still cut off fences its site too (decision 12).

• Dead-man switches (decision 8), with the IPMI BMC watchdog first:

  • the owner arms its site’s switches with after_seconds plus its self-fencing time, and renews them every third of after_seconds while it holds quorum (decision 13): its own fence goes first and leaves the workloads in their fenced shape, and the switch is the backstop;
  • they stop being renewed when it is cut off, dies or hangs, and power the host off without it;
  • they are disarmed when the site stops owning the group, and a site that does not own it disarms its switches once whatever it remembers: a run of its orchestrator that died while it owned the group may have armed them;
  • only after 5 s of unbroken contact does it arm, renew or disarm them, because a node that rejoins hears the leader before its copy of the state has caught up.

• A failover backed by a self-fence (decisions 7 and 11). The leader may treat an unreachable owner as fenced by its own promise only when all of the following hold:

  • the owner’s last self-fencing report, replicated in raft, said self-fencing was armed, and was at most 30 s old when contact was lost;
  • the configuration is the same on every node;
  • the owner has been unreachable for at least (after_seconds + its site's self-fencing time) × 1.01 + 10 s, by which time its switches have fired.

  The failover still fences everything that answers. The promise stands in only for targets that do not reach FENCED, and not at all where ANY already has a proof for that site. When the promise is used, the failover is audited as forced (plan.failover.self-fence-backed) with the report as evidence, and the old site owes a fence (§1).

• A proof comes first (decision 14). From primary_unhealthy_for on, before the bound, the leader tries to prove the owner fenced. It fences the owner’s targets only once everything the failover needs holds: dependencies, the configuration gate, the circuit breaker, and the promoted site’s readiness. It does so outside the failover sequence, with the current generation, as a manual site fence does. If they verify, the group fails over at once, with no promise used. If not, nothing has changed and the group is never BLOCKED; the leader retries every 15 s (plan.fence.early).

• Two tiers.

  • Hardware: a dead-man switch armed at every one of the owner’s sites; it fences even a dead orchestrator.
  • Software: active self-fencing only. The promoted site needs a required shield, raised before it serves.

• Recovery when quorum returns (decisions 9 and 10).

  • Owner found fenced: the leader audits plan.owner.fenced, sets service health FAILED and shows it on the group page.
  • Recover: an Operator may recover the group to either site, only while the owner verifies as fenced, through the normal sequence.
  • auto_recover: the leader does the same by itself, but only when the owner reported fencing itself at the current generation, so it never undoes an operator’s fence. It promotes a site where the group’s dependencies hold and whose readiness passes, the previous owner first. A group whose dependency’s owner is fenced too waits for that group’s recovery, then follows it.
  • Recover never strands a group: a recovery the dependencies would refuse is refused before the owner is released.
  • Owner that cannot be asked: an owner that reported fencing itself but whose targets do not answer is reported too (plan.owner.self-fenced-unverified, service health FAILED). Nothing recovers it by itself, because nothing proves it stopped: the operator fails it over with an attestation.

• Counts as the §16 watchdog. A group with self_fence is no longer DEGRADED for lacking one.

Tests:

• internal/api/selffence_test.go: fencing when cut off, only for the owner, and at startup without quorum; the armed report; dead-man switches armed, renewed and disarmed, and not before contact has settled; configuration validation.
• internal/api/backing_test.go: the bound; the software tier needs a shield; refusals without fresh evidence or with configuration drift; the hardware tier; the backing unused, and not claimed in the audit, when the fence works.
• internal/api/recover_test.go: an owner found fenced is loud; Recover to either site, refused while the owner may serve; auto_recover to the owner or, when it is not ready, the other site; never after an operator’s fence; an unverified self-fenced owner is loud and not recovered; the record goes with its generation.
• internal/fencing/fence_error_test.go (TestAttestationUnusedWhereAnySiteIsProven), internal/ownership/contact_test.go, internal/controllers/infrastructure/ipmi/deadman_test.go (against a fake ipmitool), internal/api/automatic_test.go (TestProbeTreatsAnImmediateCloseAsUnreachable).
• internal/labsim/links_test.go: the lab’s raft link proxies and IPMI watchdogs.

Verified on the swarm lab, where the simulator carries raft between the orchestrators and emulates the nodes’ BMC watchdogs:

Found and fixed while verifying:

• The voter probe: it counted a proxy that accepts a connection and drops it as a reachable voter, so an isolated site never looked down.
• Attestation under ANY: an attestation was marked used for a site that another of its targets proved fenced, which recorded a needless debt.
• The audit reason: it claimed a backing the failover did not use.
• Rejoining nodes: a node that rejoined renewed its watchdogs once from its stale copy of the state.
• Silent unverified owners: an owner self-fenced but unverified raised no alert.
• Stale owner-fenced records: the record was shown after the group had moved on.
• Always shields: they were never re-checked between failovers (see Shields).
• Switches armed by an earlier run: a site that came back as a non-owner never disarmed the watchdogs its previous run had armed, and one powered a restored site off.
• Recovery and dependencies: a portal could be recovered to a site its database was not at, and a recovery refused by its dependencies left the group released.

Residual risk:

• Availability for safety. With the witness down and the link between sites cut, both sites lose quorum and the owner stops. after_seconds trades false stops against recovery time.
• The software tier trusts a live orchestrator. An orchestrator that died with its workloads running fences nothing, but the survivor assumes it did. The shield keeps the survivor’s resources safe, and §1 fences the old site when it answers, but the old site’s own clients can write into its diverging copy meanwhile. The hardware tier does not have this gap.
• A site proven fenced only by being powered off can start in its active role when it is powered on, until qfence fences it. Since ADR 0009 §1 it owes a durable fence, paid within seconds of its fencers answering (see “Durable fences”), instead of at the next check. Before that, on the lab, dc2’s own writer wrote into its database for about a minute. The shields keep it out of the VIPs throughout.
• A failover that must rely on the promise still waits for the whole bound: 4 min 12 s for erp-db on the lab, whose ladder has two rungs with a 90 s fence timeout each. Shorter fence timeouts, or fewer rungs, shorten it.
• An early proof fences an owner that may be alive. An owner that is only unreachable (its orchestrator crashed, say) is fenced at primary_unhealthy_for if it can be, exactly as an automatic failover without self-fencing does.
• A site that loses power has no chance to fence itself into its shape, so when it comes back its workloads start as they were, until it is fenced again (see the power-off risk above).
• The IPMI dead-man switch has only been run against the lab’s emulation and a fake ipmitool, not a real BMC.

Continuous checks: sites that do not own a group stay fenced (ADR 0008 §3)

A site can return to running a group’s workload without a failover: a power restore, someone starting it by hand, an automation. The leader now verifies every site that does not own each group, every fencing.recheck_seconds (default 60). What keeps that safe:

• Read-only. A check only calls each required target’s Verify (fencing.Engine.Check) and judges the result by the group’s policy (ALL, or ANY per site), exactly as a fence would. It never fences by itself, and a test proves it.

• What is checked.

  • Checked: only groups with an owner, and only sites other than the owner.
  • Not checked:
    • a site that owes a fence, which §1 handles;
    • a site marked “being prepared”;
    • a group whose failover this node is running, which holds the group’s lock.

• Drift is loud. A site observed not fenced is drift. It is announced once, when it starts:

  • an event;
  • a plan.fence.drift audit record with what was observed, which is a failure and so a hosted-plane alert;
  • the group’s fencing health DEGRADED;
  • the group page highlighting it.

  A site that does not answer is “not verified”, not drift: a dark site reads that way.

• Fencing it is opt-in (fencing.on_drift: fence; default alert, decision 5). The default is alert because some fences, like powering a host off, conflict with preparing a site to take a plan back. Shape fences keep a fenced site usable, and groups built on them should choose fence. Each fence is audited (plan.fence.drift.fenced).

• “Being prepared” is a forced exemption.

  • POST .../sites/{site}/prepare needs PermissionForce, a reason of at least 20 characters and at most 24 hours. It is audited as forced.
  • It is refused for the owner and for a site that owes a fence.
  • Only the leader that accepted it holds it: a leader change drops it, so checks resume (fail loud).

Tests:

• internal/fencing/fence_error_test.go (TestCheckVerifiesWithoutFencing).
• internal/api/sitechecks_test.go covers:
  • drift detected and alerted once, the owner never checked, and alert never fencing;
  • on_drift: fence fencing the site without touching the owner;
  • the prepare mark: its permission, validation, exemption, audit and early end;
  • debts and unowned groups skipped;
  • configuration round trip.

Verified on the swarm lab with on_drift: fence: dc2’s portal was started and its database set to primary by hand while dc1 owned both groups. Drift was detected at the next check and both were fenced 5 s later.

That database then stayed down, safely. Its promotion by hand had put it on a newer timeline than dc1’s primary, so it refused to re-seed from an older one. The lab’s reseed command is the operator’s explicit way out.

Residual risk:

• Drift is found only at the next check, so it can run for up to recheck_seconds. Shields (§2) keep it out of the owner’s resources meanwhile.
• A check proves what the fencing providers can see: a workload they do not cover is not checked.

Shields: the site that took over protects itself (ADR 0008 §2)

A shield is a target at a site that is raised when that site is promoted and lowered when it stops being the owner. Examples are a firewall policy disabled at the survivor’s edge, or a database refusing the other site’s clients. It keeps the other sites out of the survivor’s resources and needs nothing from them, so it works when fencing cannot: during an attested failover, or while a returning site’s workloads start before qfence can fence it. What keeps it from weakening anything:

• Never a fence, structurally. Shields (role: shield) are parsed into their own list, apart from the fencing targets. No fence verdict, ANY group, §14 level rule, attestation plan or fence debt ever sees them. A shield must name the site it protects and must be network or resource level.
• Shields follow proof (decision 4, revised). With mode: unproven, the default, the owner’s shield is up while another site cannot be confirmed fenced: it owes a fence, a check observed it not fenced, or a check could not reach it. It comes down only after two checks in a row confirm every other site fenced, so a flaky check cannot make a broad shield flap. The leader raises and lowers these between failovers. With mode: always, a shield is up whenever its site owns the group. Shields belong to the owner only.
• Raised before the promoted site serves.
  • The promoted site’s always shields, and its unproven shields when another site is unconfirmed (the fence was attested, or a site owes a fence), are raised and verified after readiness and before it is unfenced. They are raised under ALL, whatever the group’s fence policy.
  • A required shield that is not raised blocks the promotion (BLOCKED), and nothing at the promoted site is unfenced.
  • A secondary shield is best effort.
• Verified, not assumed. At every check, the leader reads the owner’s shields that should be up (every always shield, and the unproven ones while they apply). It raises any it finds down and audits that (plan.shield.drift): a firewall that reboots or is edited by hand can drop one between failovers, and a new leader knows nothing of what the old one raised. On the swarm lab, the simulator’s FortiGate forgot every policy when it restarted, and dc1’s shields stayed down while dc1 owned both groups. That is how this was found.
• Lowered only when its site stops owning (decision 4). Every other site’s shields are lowered once the failover has verified their fence, one at a time and best effort: lowering never decides a failover.
• Attestation requires one (decision 3, made unconditional by ADR 0009 §4). An attested failover is refused unless the promoted site has a required shield, which is then raised before promotion like any other. requires_shield: false is rejected.
• Visible: the dry run lists the shields raised, and the fencing overview shows each shield’s last RAISED or LOWERED.

Tests:

• internal/api/shields_test.go covers:
  • raised at the promoted site and lowered at the others;
  • a required shield failing blocks before unfence;
  • a secondary shield or a failed lowering does not block;
  • a shield never counts as a fence, even under ANY (a mutant counting shields as fences fails it);
  • unproven shields not raised by a verified failover, raised by an attested one and by drift, and lowered only after two confirmed checks;
  • configuration validation and round trip;
  • an attested failover refused without a shield and allowed with one; requires_shield: false rejected.
• internal/labsim/linked_test.go covers a shield raised through qfence’s real fortigate-fencer taking a site out of the VIP.

Verified on the swarm lab, where each site’s shield disables the FortiGate policy that lets the VIPs reach the other site:

• A failback raised dc1’s shields and lowered dc2’s.
• An automatic failover on a partition flipped them.
• dc2’s portal, started by hand while dc1 owned the group, answered health checks but got no traffic: every request went to dc1.

Residual risk:

• A shield protects the survivor’s resources, not the old site’s own clients or data.
• It must still let through what the old site needs to become a standby. qfence cannot check that; the operator configures it.

Sites that owe a fence (ADR 0008 §1)

After an attested failover the old site was never fenced, and when it returns it runs what it ran before. ADR 0008 §1 makes qfence fence it by itself. What keeps that safe:

• Recorded in raft, before promotion.
  • An attested failover records that each site with an attested target owes a fence (CommandFenceDebtRecord), after the fence verdict and before the promotion.
  • If it cannot be recorded, the failover blocks: unrecorded, nothing would fence the site later.
  • A leader change keeps the debt.
  • Voters older than it reject the command and keep no copy; ownership is unaffected, as with the circuit breaker.
• Only a site that is not the owner, only its own targets.
  • The leader fences a site that owes a fence using the targets that name that site, never an unscoped one, which could reach the owner.
  • It never fences the current owner.
  • It skips a round while a failover of the same group runs on this node: a per-group lock serialises a failover and a returning-site fence, so the fence cannot land after the site is promoted.
• Judged like any fence. Clearing needs the site to verify as fenced under the group’s policy (ALL, or ANY per site).
  • A dark site is retried every 5 s, since its refusals cost nothing.
  • A site that answers but does not fence backs off from 15 s to 60 s.
• No waiver. Only these clear a debt:
  • that verified fence, automatic or a manual fence of the site;
  • promoting the site, which the FSM clears in the same log entry that makes it ACTIVE;
  • deleting the group through the replicated configuration.
• Visible.
  • The group’s fencing health is DEGRADED, and its page says which site owes a fence and since when, with a manual fence link for force-capable roles.
  • Audit records: plan.fence.debt.recorded, plan.fence.returning-site, and plan.fence.debt.stale once a debt is older than fencing.debt_alert_after_seconds (default an hour). The stale record is a hosted-plane alert.

Tests:

• pkg/quorum/fencedebt_test.go covers record and clear, promotion clearing, snapshots, and a leader change.
• internal/api/fencedebt_test.go covers:
  • an attested failover leaves a debt, and a verified one leaves none;
  • a dark site is retried, then fenced when it answers, without touching the owner;
  • the owner is never fenced, and nothing runs while a failover holds the group;
  • manual fence and promotion clear it;
  • one stale alert;
  • configuration round trip;
  • an attested target without a site creates no debt.
• Mutating out the owner check or the group lock fails those tests.

Verified on the swarm lab: dc1 blacked out, both groups recovered by attestation (each recording that dc1 owes a fence), then dc1 restored with nothing else touched. qfence fenced dc1 for both groups about 7 s after its power returned, before its workloads answered: the VIPs never saw dc1, and its database came back a standby re-seeded from dc2. In the same scenario before §1, both VIPs flagged a split brain until a person fenced dc1.

Residual risk:

• The window between a site’s workloads starting and its fence endpoints answering remains. The lab’s is shorter than the time its workloads take to start, but that is a property of the lab, not a guarantee. ADR 0008 §2 (shields) protects the survivor during it. §4 narrows it for a site that returns still cut off from quorum: its orchestrator fences its own site after_seconds after it starts.
• An attested target without a site creates no debt.

Docker and Docker Swarm providers, proof from any one mechanism per site (ADR 0007)

ADR 0007 adds the first providers tested against the real system on every change, and a lab where qfence moves a real Postgres between two real swarms. What protects the guarantee:

• Verification judges what runs, not what was asked.

  • A service is FENCED only once its spec asks for the fenced shape and every task that may still run carries the fenced shape’s environment. A task still shutting down counts as running.
  • A task of another shape on a node the swarm reports down cannot be seen, so it makes the result UNKNOWN, not FAILED and not FENCED.
  • A fenced shape that keeps replicas must name environment entries that tell its tasks apart, and the two shapes may not share an entry. Otherwise the factory refuses the configuration.
  • A missing container or service is UNKNOWN: a typo must not read as proof.

• Waiting is bounded, and a deadline is UNKNOWN. Fences wait for the change to take effect until fencing.fence_timeout_seconds (new, with verify_timeout_seconds). The deadline error wraps context.DeadlineExceeded, which the engine reports as UNKNOWN (§17).

• ANY is per site, and Level 1 first.

  • fencing.success: ANY (now configurable, with execution) needs one FENCED required target per site being fenced. Targets without a site form one more group, so proof that one site stopped never covers another.
  • In a group with a Level 1 target, only a Level 1 proof counts, unless allow_network_only_fencing is set (§14). Before, ANY let a network fence alone succeed while the compute fence failed; ANY was never reachable from configuration, and that engine test was changed.
  • All required targets are still attempted.

• Power evidence. A Redfish fencer with fence_action: none never switches power. It proves only a host that is already off: FENCED for Off, FAILED for On, UNKNOWN when the BMC does not answer.

• Readiness that cannot measure is an error.

  • A lag_command that fails or prints anything other than a number of seconds makes the check an error, so the lag is unknown and promotion blocks (ADR 0006 cannot accept it). The command’s stderr is in the error.
  • The lab’s check reports a primary as current only while a standby streams from it. A site that returns from a blackout still running an old primary therefore cannot be promoted: verified on the lab.

• A fence command that errors proves nothing. When the command fails (connection refused, credentials rejected, a timeout), the engine still asks the target for its state, as after every fence:

  • FENCED: the target is fenced, for example a host that was already off.
  • FAILED: only when the target is observed running after a non-timeout error. After a timeout, the command may still be on its way, so the result stays UNKNOWN.
  • UNKNOWN: when the target cannot be asked either, as for a dark site or bad credentials. Both errors are kept.

  Before, any fence error other than a timeout read as FAILED, although it observed nothing (§17). Both block, so safety is unchanged. Tests: internal/fencing/fence_error_test.go. Verified on the swarm lab: a powered-off node’s Docker API reads UNKNOWN.

• Fencing a site that comes back. A manual fence may name a site other than the owner. It then fences only that site’s own targets, never an unscoped one, which could reach the owner. The audit reason carries [site X], and the console offers it on the fence screen.

• Credentials. tls_cert/tls_key come through credentials_ref, and the schema guard now treats any key word in a field name as a credential.

Tests:

• internal/controllers/infrastructure/docker/docker_test.go runs against an HTTP double of the Engine API: shapes, tasks still of the old shape, tasks on down nodes, a deadline, idempotent repeats, a stale spec version, configuration refusals, network fencing of services and containers, and readiness errors.
• integration_test.go runs against a real Docker-in-Docker swarm with QFENCE_DOCKER_IT=1.
• internal/fencing/engine_test.go covers ANY per site, unscoped targets as their own group, and Level 1 first.
• internal/controllers/infrastructure/redfish/fencer_test.go covers power evidence.
• internal/api/fencingmodes_test.go covers the modes’ round trip, validation, the dry run, and a failover proven by the BMC while the Docker fence is UNKNOWN.
• internal/api/sites_test.go covers a named-site fence touching only that site.
• internal/labsim/linked_test.go covers BMCs switching machines, partitions, virtual servers and split-brain detection.

Verified on the swarm lab (deploy/swarm-lab), against real Postgres replication:

• Planned failover: 16 s for the database, and failback with every order.
• Dependency: a refused portal move before its database.
• Node failure: rescheduled by swarm, and qfence did nothing.
• Partition: automatic failover through the Docker API in about 45 s.
• Power loss: erp-db failed over on the BMC’s power evidence while the portal blocked.
• Blackout: erp-db blocked, then attestations recovered both groups.
• Return of the blacked-out site: it came back as an old primary, and both virtual servers logged a split brain. qfence refused to promote it, and a fence of that site made it a standby that re-seeded from the newer timeline.

Residual risk:

• The Docker API is in band. The lab gives it a management path apart from the wan, and network_domain declares it (§15), but a real deployment must do the same or the fence shares fate with the workload.
• A site that returns after an attested failover runs its last spec until someone fences it: see Known gaps.

Accepted data loss (ADR 0006)

When the only replica left is older than the readiness policy, promotion is refused, and that replica only gets older. ADR 0006 lets a person accept losing the data since its last sync to bring the service back. What keeps it narrow:

• Only a measured loss, within a stated bound. The failover proceeds past readiness only if the check answered, the replica itself is healthy, and the lag measured at promotion is at most the max_lag_seconds the operator typed (and the group’s cap, if set). An unreachable replica, a readiness check that errors, or an unhealthy replica still blocks: an unknown loss cannot be accepted. Replication that reports errors can be accepted when a last sync was measured, and the error is shown and audited.
• Nothing else changes. Quorum, ownership, fencing and its verification come first as always; the acceptance only replaces the readiness verdict. The request is also refused before anything is fenced when the replica is already unreachable, unhealthy or behind by more than the bound.
• Authority, friction and audit as for attestation: PermissionForce, a statement of at least 20 characters, the forced flow in the console, readiness_override {allowed, approvals, max_lag_seconds} per group, and a forced audit record plan.failover.data-loss-accepted with the bound and the lag measured at promotion, plus a hosted-plane alert. Combined with an attestation in one failover, each override is checked by its own rules, the larger approval count applies, and each gets its own audit record at the same generation.
• Never automatic, never stored. The acceptance lives in one failover call; automatic failover blocks on a stale replica.

Tests: internal/plans/sites_test.go (TestAcceptedDataLoss: within and above the bound, erroring replication, unreachable, unhealthy and erroring checks), internal/api/dataloss_test.go (Operator refused, bound below the measured lag refused without fencing, dry run allowed and never fencing, allowed: false, cap, unreachable and unhealthy replicas refused, erroring replication accepted and audited, the lag recorded at promotion rather than at the dry run, combined with an attestation under two approvals with one audit record per override, configuration round trip), internal/scheduler/sitemode_test.go (TestAutomaticFailoverNeverAcceptsDataLoss), and the console’s FailoverWizard.test.tsx.

Residual risk: the loss accepted is real, and the bound is on replication lag as the provider measures it. A provider that under-reports lag would under-state the loss.

Operator-attested fencing (ADR 0003)

When a site is dark, its BMCs do not answer, fencing cannot be proven, and every failover blocks, which is correct. ADR 0003 adds a narrow, audited way for a person to stand in for the fence verdict. What keeps it narrow:

• Per failover and per target. An attestation names required targets on one failover request. It is never stored, never set by configuration, and never used by automatic failover; the scheduler’s policy has no attested set. A target of the site being promoted, or a secondary target, cannot be named.
• Fencing still runs. Attested targets are fenced like any other; one that reaches FENCED is recorded as fenced, not attested. One that does not counts as fenced by attestation, with its observed state kept (never rewritten to FENCED), and the fencing overview marks it. A failing required target that is not named still blocks.
• Only after an observed failure. A target can be attested only once a real fence attempt has failed on the leader at least min_failing_seconds ago (60 by default). A dry run’s status query does not count, and the record resets on restart or a leader change.
• Authority and friction. Attesting needs PermissionForce, a statement of at least 20 characters and the forced-operation flow in the console. It is audited as forced (plan.failover.attested, with the statement and each target’s observed result) and raises a hosted-plane alert. fencing.attestation.allowed: false refuses it for a group.
• Two approvals, if configured. With approvals: 2, the request does nothing until a different identity approves it. The request is held only in the leader’s memory and dropped on expiry (10 minutes), on any change to the group’s state, generation or configuration, and on a leader change, so nothing that authorises a promotion survives an incident unattended. Both identities and statements are in the audit record.
• The dry run never fences. It reports each target’s eligibility and asks the target for its state with the read-only Verify, so an operator is told when a target answers after all.

Tests: internal/fencing/sites_test.go (TestAttestationStandsInOnlyForNamedRequiredTargets), internal/api/attestation_test.go (Operator refused, promoted-site target refused, short statement refused, no observed failure refused, too recent refused then accepted, allowed: false, a named target that fences recorded as fenced, the dry run never fences and reports an unreachable target, two approvals with the same identity refused and a different one accepted, a pending request dropped when the group changes, on cancel and on a leader change, configuration round trip and validation, automatic failover carrying no attestation), and the console’s FailoverWizard.test.tsx and PendingAttestation.test.tsx. Verified on the demo lab: dc1 blacked out, the automatic failover blocked on bmc-dc1, attestation was refused while the failure was 2 s old, then after 60 s, a force release and an attested failover, erp-db ran at dc2 with the audit record carrying the statement and “bmc-dc1: attested, observed FAILED”.

Residual risk, stated plainly: if the statement is wrong, both sites may run the application. See Known gaps.

Site-scoped fencing, promotion by unfencing, and automatic failover (ADR 0002)

ADR 0002 changes what a failover fences and adds the first thing that fails over without an operator. What protects the split-brain guarantee in each part:

• A failover never fences the site it promotes, and still fences every other site: targets scoped to another site and every unscoped target. A configuration whose failover to some site would fence no required target, or break the §14 level rule, is refused when it is loaded; at run time a scoped set with no required target reports UNKNOWN (the engine already treated an empty required set as UNKNOWN) and blocks. Configurations without site behave exactly as before. Tests: internal/fencing/sites_test.go, internal/plans/sites_test.go (TestScopedFailoversFenceOnlyTheSiteGivenUp), internal/api/sites_test.go (validation cases, dry run, a failover there and back through the API counting every fence and unfence).
• Unfencing the promoted site happens only after every other site is verified fenced and readiness is confirmed, and before the service moves; a failure blocks. It is opt-in (unfence_on_promote) and never touches unscoped targets. Tests: TestUnfenceNeverHappensBeforeReadinessIsConfirmed, TestUnfenceFailureBlocksBeforeTheServiceMoves.
• Readiness covers the promoted site or refuses. With several entries or any scoped one, a failover to S needs every entry covering S; configured readiness that covers nothing for S refuses (§48). Test: TestSiteReadinessChecksOnlyThePromotedSite.
• Automatic failover adds no new way to promote. It is off unless the orchestrators are started with -automatic-failover. It runs only on the raft leader, only for ACTIVE groups with automatic_failover: true, and only through the same sequence, behind the drift gate, dependencies and the replicated circuit breaker (limits left at zero take the §41 defaults rather than meaning “no limit”). Its signal: the owner site has had no reachable voter for primary_unhealthy_for_seconds; a site with no voter gives no signal, so nothing happens (§48). A lasting refusal is audited once per reason. Tests: internal/scheduler/sitemode_test.go (fails over after the delay, a short dip resets it, leader only, no reachable target, a non-ACTIVE group left alone, one audit record per refusal), internal/api/automatic_test.go.
• Verified on the demo lab (real binaries, real providers against the simulator): dc1 failed (orchestrator stopped, hosts off, BMCs answering) led to an automatic failover to dc2 about 22 s later, with dc1 forced off through Redfish before dc2’s VMs started and the VIPs moved; failback to dc1 after restoring it; with dc1 blacked out (BMCs unreachable) the automatic failover stopped at BLOCKED with nothing started at dc2.

Every fence is recorded for the fencing overview and the fencing health dimension, target by target: a manual fence and every failover, manual or automatic (Sequence.OnFence), so a site-scoped failover does not blank the targets it did not fence. Tests: TestFencingOverviewRecordsFailoverFences, TestAutomaticFailoverReportsItsFence.

The ACQUIRING dead end is now explained rather than removed: Acquire only takes ownership, the console says so and links to Release, and the failover dry run tells how to get out.

Availability plan Dependency Engine (Roadmap Phase 7, §3.2)

§3.2 reserves a depends_on field (“app tier fails over only after the DB tier is ACTIVE”) but explicitly forbids implementing evaluation before Phase 7, since it changes the safety analysis for automatic failover across dependent groups. This was treated as a design task first, not straight implementation, per this phase’s own prompt.

The core safety decision, proven from the state machine itself, not just judgment: a dependency-blocked failover attempt is refused outright — no state transition, no fencing, no generation consumed — never routed through BLOCKED. pkg/models/state.go’s validTransitions[StateBlocked] is exactly {StateStandby}; chained with pkg/orchestration/ownership.go’s ApplyAcquire requiring CanTransitionTo(StateAcquiring) (false from BLOCKED), routing a dependency block through BLOCKED would mean even an ordinary retry once the dependency resolves fails, requiring an operator’s emergency ForceAcquire (§24) just to recover a transient, self-resolving condition. There is direct precedent for “refuse outright, no transition” already in internal/plans/sequence.go: when Ownership.Acquire itself fails, Sequence.Failover returns the error directly without transitioning — the dependency check (internal/plans/dependencies.go’s CheckDependencies) slots into that exact precedent, one step earlier, before Acquire is ever called.

Zero new distributed-systems machinery needed: a dependency’s state is read via the same ownership.Engine/raft cluster the dependent availability plan itself lives on — Engine.CurrentState already answers “what state is this ID in” for any plan ID it has ever seen, with no concept of api.PlanConfig at all. This is also the feature’s one structural limitation: a depends_on naming an availability plan that actually lives on a genuinely different orchestrator/raft cluster would hit the same engine’s documented “never seen this ID” fallback (STANDBY, no error) and read as permanently unsatisfied — fails closed correctly (§48 is not violated), and CheckDependencies itself has no way to tell that case apart from an operator’s plain typo, since both produce an identical “unresolvable STANDBY.”

Mitigation for the above, added once the gap was flagged: api.Server.CheckDependencyReferences() is a startup-time diagnostic — called once in cmd/orchestrator/main.go, right after every RegisterPlan call at bootstrap has completed — that walks every registered availability plan’s DependsOn list and logs a warning (log.Printf, not a startup failure) for any target ID never registered on this same server. It is deliberately never consulted by CheckDependencies/Sequence.Failover themselves and never blocks startup: a real, deliberate cross-orchestrator dependency is a legitimate configuration this must not break, so a dangling reference is surfaced for an operator to judge, not rejected outright. This turns an otherwise-silent misconfiguration (most likely a typo) into a visible one at deploy time, without changing any runtime safety behavior. Tested by internal/api/api_test.go’s TestCheckDependencyReferencesFlagsDanglingID and TestCheckDependencyReferencesSilentWhenResolved (including the forward-reference case, which must never be flagged once the referenced availability plan is registered).

Wired into every path that can promote an availability plan:

• resources.Sequence.Failover — checked first, before Acquire. Covers every manual failover, a move back to a site the plan left included, with no direction-specific logic needed.
• §45 dry-run (api.DryRunReport.Dependencies) — PASS / “PASS (none configured)” / FAIL, feeding the same Promotion/Reasons every other precondition already does.
• A new §43 health dimension (health.DimensionDependency/DependencyHealth) — since a blocked attempt never transitions state, this is the dashboard-visible signal that would otherwise be invisible outside the audit log, computed live on every GET /availability-plans/{id} exactly like FencingHealth/ProviderHealth already are.
• internal/scheduler’s automatic-failover loop — checked before the circuit breaker, and deliberately not counted as a failed attempt (no Breaker.Allow/RecordAttempt/Sequence.Failover call at all): fencing/readiness never ran, so an unmet dependency must not contribute to the anti-flap trip threshold, which exists to catch a malfunctioning failover mechanism, not “still waiting on a normal precondition.”

Config-time safety net: api.Server.RegisterPlan now returns an error and rejects a depends_on cycle — including direct self-dependency, which is simply a cycle of length one, needing no separate check — via a single DFS from the newly-registered availability plan (sufficient because every prior registration was itself validated, so any new cycle must pass through the node whose edges just changed). A dependency naming an availability plan not yet registered is correctly treated as a legal forward reference (ordinary bootstrap ordering, e.g. an app tier registered before its DB tier), not an error.

Deliberately not built: any automatic cross-plan cascade (“when the DB tier becomes ACTIVE, auto-trigger the app tier”). §3.2’s text describes a gate, not a trigger. (Since Phase 9.6, ADR 0015: an opt-in follow on a same-site dependency is such a trigger, under the rules in “A plan follows its same-site dependency”.) Force operations (ForceAcquire/ForceRelease) remain untouched — they already bypass fencing/readiness as an existing, audited emergency mechanism (§24), and this phase did not add dependency-awareness to that path.

Tests (no mockable “unit” tier exists for anything touching a real raft-backed ownership.Engine — every test below uses the identical real-3-node-cluster harness this codebase’s other resources/scheduler/api tests already use):

• internal/plans/dependencies_test.go — empty/satisfied/unsatisfied(never-acquired)/unsatisfied(mid-failover, the literal “what does BLOCKED mean for a group whose dependency is mid-failover” scenario)/read-error(quorum down)/multi-dependency AND-semantics.
• internal/plans/sequence_test.go’s TestFailoverBlockedByUnsatisfiedDependencyNeverCallsAcquireOrFencer — the single most important test: proves the availability plan’s own ownership state is byte-identical before and after a blocked attempt (so Acquire provably never ran), plus zero fencer/service invocation. TestFailoverProceedsOnceDependencyBecomesActive covers the positive path.
• internal/scheduler/scheduler_test.go’s TestSchedulerSkipsAutomaticFailoverWhenDependencyUnsatisfied — proves the circuit breaker never trips across several unhealthy episodes with an unmet dependency, the decisive evidence that a dependency block isn’t counted as a failed attempt.
• internal/api/api_test.go’s TestFailoverRejectedWhenDependencyUnsatisfied (409 through the real HTTP handler chain, state unchanged via a follow-up GET, health.dependency=DEGRADED), TestFailoverDryRunReportsDependencyStatus, TestFailoverSucceedsAfterDependencyBecomesActive, TestRegisterPlanRejectsCycle/RejectsSelfDependency/AllowsForwardReference.
• tests/chaos/dependency_test.go’s TestDependencyCheckFailsClosedWhenQuorumUnavailable — a real raft majority taken down mid-test, proving Sequence.Failover fails closed with zero fencer/service invocation when the dependency check itself can’t be verified.
• Manually verified against a real running server (two availability plans, one depending on the other, driven exactly like the automated REST tests but over real HTTP/curl): 409 with a clear “unmet dependencies: production-db-01: requires ACTIVE, actual STANDBY” error and unchanged STANDBY state before the dependency is ACTIVE, health.dependency DEGRADED then HEALTHY, and a real 200/ACTIVE once the dependency genuinely becomes ACTIVE.

Known gaps:

• Cross-cluster dependency IDs are mitigated (a startup-time warning, described above) but not fully closed — a genuine cross-orchestrator dependency and a typo still can’t be distinguished automatically, only surfaced identically for a human to judge.
• The dependency check is a point-in-time snapshot, not continuously enforced — nothing re-verifies it once Sequence.Failover has started fencing/promoting. This is consistent with, not a regression from, how Readiness.Check already behaves in the same method (checked once, never re-verified before Service.Acquire).

Production Web UI — Phase 8.0 API Prerequisites (Roadmap Phase 8.0, UI_DESIGN.md §8.2)

Sixteen API additions (N1-N16) the three Phase 8 consoles need, built and tested entirely before any UI code — a UI can only render what the API returns (this phase’s own rule), so a real bug in one of these is a bug the UI would otherwise have had to work around or hide. Every one goes through the existing withAuth/RequireViewer RBAC gate (§55); none is a new privileged path.

SSE backpressure (N1) needed no new mechanism, only a consumer for one that already existed. events.Bus.Publish’s per-subscriber send was already non-blocking (select { case ch <- e: default: }), so internal/api/stream.go’s handleStream inherits “a slow consumer can never back-pressure the bus” for free by subscribing through the same Bus.Subscribe. What is new: TestStalledStreamConsumerIsDroppedWithoutAffectingOrchestration proves this end-to-end through the HTTP layer — a subscriber that never drains its channel, plus 40 real acquire/release cycles through the API that all complete normally while the stalled channel saturates at its bound (16) rather than growing or blocking anything.

N7 (plan config write) is the one item in this phase that mutates safety-relevant configuration, per this phase’s own ground rules (“keep N7 sequential and reviewed”). Two safety-motivated departures from §29’s illustrative YAML, both because the alternative would have meant guessing at something safety-relevant:

• Every fencing target requires an explicit level field. fencing.TargetConfig.Level (§14) determines whether an availability plan is relying on network-only fencing, which §14 treats as a reduced-safety configuration requiring explicit acknowledgement — inferring it from a provider’s Capabilities().Features was considered and rejected: real providers report inconsistent, non-classifying feature strings (redfish → "redfish", ipmi → only "fencing", proxmox’s fencer → "compute"), so guessing risked silently misclassifying a compute fencer as network-only or vice versa.
• Exactly one readiness/service provider is accepted; a second entry in either list is a validation error (FieldError), never silently dropped. The real PlanConfig Go model holds one Readiness/Service field, not a list — accepting a list on the wire but quietly using only the first entry would mean an operator’s config could specify a second provider that is never consulted at all, silently.
• Provider references are resolved against s.Registry (the same config-driven registry Phase 4 built) via a live type-assertion (interfaces.Fencer/ReadinessChecker/ServiceController) — an unresolvable or wrong-role reference is a structured field error, never a partially-built availability plan.

N4’s clock_skew_seconds is always null — an honest gap, not a placeholder value. §10: “Preferred default for v1: do not implement wall-clock leases at all” — clock skew only matters as a safety-relevant health dimension once a lease-based fast-read path exists to bound it, and this codebase deliberately has none. Separately, no peer-to-peer channel exists between orchestrator nodes today even for pure observability (ClusterInfo.Peers is a list of raft server IDs, not dialable HTTP addresses) — building one would be new distributed-systems infrastructure disproportionate to a UI-prerequisite phase. NodeHealthDTO.ClockSkewSeconds is a *float64, structurally incapable of carrying a fabricated number; it is always nil today, documented as such.

N6’s last_transition is node-local and best-effort, not raft-replicated. Adding a timestamp to models.PlanState (the actual consensus-replicated struct) for a pure display convenience would have been a consensus-state schema change; instead Server.lastTransition is an in-memory map updated on every successful mutation, resetting on restart — the same allowance pkg/audit.Log and pkg/events.Bus already have documented for exactly this class of data.

N10’s cursor pagination is header-based (X-Next-Cursor), not an envelope, specifically so every pre-existing consumer of GET /audit/GET /events’s plain-JSON-array response (including three years’ worth of this codebase’s own tests) keeps working unchanged.

N11 closed a real, pre-existing gap, not just added new plumbing: audit.Record.Reason already existed and was already being populated — but only for force-acquire/force-release, and AuditRecordDTO never had a Reason field at all, so §31/§35’s required “reason” audit field was being silently dropped at the API response boundary for every operation, forced or not. Fixed by adding the field to the DTO and accepting an optional reason on the non-forced mutating endpoints too.

N14’s idempotency cache is scoped to method+path+key, not to §37’s broader idempotency guarantee. §37 already requires every orchestration operation to be safe to retry on its own terms (fencing DC1 twice is safe by construction). The Idempotency-Key header is a narrower, additive guarantee: a client retry with the same key gets back the exact same recorded response rather than a second independent attempt through the real handler — useful because some operations (e.g. acquire on an already-ACQUIRING availability plan) are safe-but-not-identical on a genuine retry (the second call correctly 409s). Concurrent duplicates with the same key are serialized on one cache entry (a per-entry mutex), so only the first ever actually reaches the handler; a 10-minute TTL bounds memory rather than retaining request/response history forever.

N15’s join wizard cannot itself join anything — this is inherent to pkg/witness’s own design, not a limitation introduced here. pkg/witness.New’s doc comment already states a non-bootstrapping witness “does not attempt to discover or join anything on its own.” The three-step wizard (cmd/witness/join.go, Go html/template, zero JS) is therefore a configuration/documentation generator: its result page states plainly which AddVoter(...) call an administrator on the existing cluster’s leader must run to actually complete the join, alongside the §8 availability-arithmetic table (generalized to N voters) for the resulting topology. Gated by a single-use, crypto/rand-generated bootstrap token (constant-time compared, invalidated on first successful completion or process restart) so an unauthenticated visitor to the status port can’t extract cluster topology information through the wizard.

N16, the hosted plane’s API additions, and their test record are in the private qfence-managed repo (ADR 0017 §6).

Tests: every new endpoint in internal/api has a real-HTTP-layer test in internal/api/phase8_prereqs_test.go (using the same httptest-against-api.Server convention as every other test in that package), including TestPhase8EndpointsNeverLeakSecrets (the TestNoSecretFieldEverAppearsInAnyResponse discipline extended to every new response type, run against a registry populated with a real credentials_ref-shaped provider config). cmd/witness/main_test.go and pkg/witness/witness_test.go cover the join wizard and extended status against a real raft cluster. Full suite verified with go build ./... && go vet ./... && gofmt -l . clean and go test ./... -race -count=1 -timeout 300s -p 2 green.

Production Web UI — Phase 8.1 Design Layer (Roadmap Phase 8.1, web/console only)

Two findings from this phase are directly safety-relevant, not just design polish, because of what they touch: §17/§48’s “UNKNOWN must never look like fine or like failed” rule, and §4.1’s dangerous-operation confirmation flow.

A real WCAG AA contrast failure in the vendored colour tokens, found only by actually running axe-core against a live, rendered page — not by code review. --state-ok/warn/danger/accent/neutral, used as small-text foreground colour, fall short of 4.5:1 in light theme (measured, not estimated: contrastRatio: 2.49 for the worst case, --state-warn against its own tint). This matters here specifically because StateBadge is the one component every plan/fence/health/provider state in the product renders through — a low-contrast label is a real, if partial, erosion of “state must never be communicated by colour alone” for anyone with low vision, independent of the colour-blind case §4.2 already designs for. Fixed by moving the tone colour onto the glyph icon (decorative, exempt from the text-contrast success criterion) and the background/border, leaving the label text itself in the always-compliant foreground colour — confirmed with a second axe-core run showing zero violations of any severity, in both themes, not merely zero serious/critical.

DangerousAction — the component every failover/failback/fence/force- operation in the product will route through once Phase 8.2/8.3 wire it up — was initially built with a real gap against §4.6’s “the server’s error message is shown verbatim”: it only unwrapped ApiError instances, silently substituting a generic placeholder for any other thrown error’s real text.* Caught by a unit test that rejected with a plain Error rather than an ApiError and asserted the exact message appeared on screen. Fixed to unwrap any Error. This is exactly the class of bug §48’s “prefer a false negative to a false positive” principle warns about at the UI layer: an operator mid-incident reading a paraphrased “The operation failed” instead of “dial tcp: connection refused” has strictly less information to act on.

Every other §4.1 rule is enforced structurally in DangerousAction, verified with dedicated tests, not left to each future call site to remember correctly: Execute is absent (not merely disabled) when a fresh plan’s PROMOTION is BLOCKED; a live 60-second plan-freshness timer re-disables Execute; a forced operation requires the EmergencyOperator role and an exact typed match of the availability plan id and a reason of at least 10 characters before Execute enables; one Idempotency-Key is generated once per attempt and reused, never regenerated, across a retry; onDone fires after both a successful and a failed attempt, so the caller’s re-read of real state is unconditional (§4.1: “never optimistic… a failed operation may still have changed something”).

Environment note: this sandbox has no system Chromium and no root access to install one — real-browser verification (the mechanism the two findings above were actually found with) was obtained anyway via apt-get download (works without root) plus dpkg-deb -x extraction plus LD_LIBRARY_PATH, not skipped. See ROADMAP.md’s Phase 8.1 section for the exact package list; worth reusing verbatim for Phase 8.2+’s Playwright integration tests rather than re-deriving it.

Production Web UI — Phase 8.2 Orchestrator Console P0 Screens (Roadmap Phase 8.2, web/console)

A3 (availability plan detail) crashed outright, rendering nothing at all, for any availability plan with zero secondary fencing targets — found only by running the Playwright suite against a real cmd/orchestrator, not by unit tests against hand-written mock data. Go’s encoding/json marshals a nil []string as null, not []; internal/api/plans.go’s detailDTO builds fence_required/fence_secondary with var required, secondary []string, so a real availability plan with no secondary targets genuinely sends fence_secondary: null on the wire — every unit-test fixture had always supplied a non-empty array for both fields, so this never surfaced until real backend data was involved. This is directly safety-relevant, not cosmetic: A3’s Fencing panel is exactly where an operator checks required/secondary fencing coverage during an incident (§14), and a hard crash denies that information entirely rather than rendering it wrong — worse than either. Fixed by typing fence_required/fence_secondary string[] | null in src/api/types.ts (an honest transcription of the real Go shape, matching this file’s own stated rule that these types must never drift from what the server actually sends) and defaulting with ?? [] once in PlanDetail.tsx, matching the existing codebase convention for other nullable arrays. Covered by a regression test asserting the component renders (not crashes) when fence_secondary is null.

The A4 failover/failback wizard — the highest-stakes screen in the console, per UI_DESIGN.md §4.1 — was verified end-to-end against a real raft-backed cmd/orchestrator, not a mocked API layer, across all five scenarios the roadmap’s exit criteria name: a golden path reaching ACTIVE with a strictly higher generation; a dependency-blocked plan (reusing the Phase 7 dependency engine to reach a genuine, stable BLOCKED state, since the mock fencer has no fault-injection hook) rendering zero Execute/Force controls, not merely disabled ones; a real Viewer token (issued via the live token API, not a fixture) receiving the server’s actual 403 message inline, never a blank page; the SSE stream’s own real reconnect/backoff/polling-fallback/recovery state machine (not a mocked timer); and the entire flow — open wizard, choose target site, dry-run, acknowledge the fence warning, execute, close — completed with only Enter/Space/ArrowDown/Tab, never a simulated click, which is a direct, executable check against §4.1/§4.7’s keyboard-accessibility requirement for the one flow in this product that can take a system offline.

Runtime performance budgets beyond entry-chunk gzipped size were deferred from here to Phase 8.6 and are now enforced; see that section below.

State and persistence (Phase 8.6)

What each component keeps, where, and what a restart does to it. “Durable” means written (and fsynced) before the change is visible; a failed write fails the change, except where noted.

Why configuration is node-local by default. Every raft voter, including a witness, stores the whole log. A witness must hold no plan definitions, controller configuration or credentials in readable form (§8, as amended by ADR 0001 and ADR 0011). So in local and file mode, configuration made through one orchestrator’s API is durable on that orchestrator only. Replicated mode and the cluster secrets store are the two exceptions, and the operator has to choose each: what goes through raft is sealed, and the witness holds ciphertext. See the configuration source modes below, and the cluster store above.

Tests: pkg/statestore, pkg/audit/persist_test.go, pkg/auth/token_persist_test.go, internal/api/nodestate_test.go, pkg/quorum/breaker_test.go (including restoring a pre-Phase-8.6 snapshot and a leader change), and a test that runs the real binary across a SIGTERM restart: cmd/orchestrator/restart_test.go (as a cluster voter). The hosted plane’s are in qfence-managed. Several assert that no private key, token secret or provider credential reaches disk in the clear.

Configuration sources and drift (ADR 0001)

Each orchestrator makes failover decisions from its own copy of the plan and controller configuration, so two orchestrators with different copies could make different decisions. docs/adr/0001-configuration-source-modes.md records the design. All five stages are implemented.

Drift detection (every mode). Each orchestrator computes a SHA-256 fingerprint of its effective configuration (canonical JSON of every controller and availability plan; credentials are references, so a reference is covered and no value can be) and reports it into the raft state on change and at least every 5 minutes. The report holds the fingerprint, never the configuration, so a witness still learns nothing about availability plans (§8). Followers cannot write to raft, so they send their report to the leader over HTTPS, signed on the peer channel under the cluster keyring (-cluster-key-file, held by orchestrators only, ±2 minutes of clock skew allowed; ADR 0016 §1). A report signed under any other key, or under a key id the leader does not hold, is refused with 401 and never reaches raft.

• Automatic failover is refused while a fresh report (under 15 minutes old) differs from this node’s configuration. Missing and stale reports do not block, because a node that is down is exactly when automatic failover is needed. This is fail-closed only toward positive evidence of disagreement.
• A manual failover dry run is not blocked. It shows config_consistency and a warning naming the orchestrators that differ.
• GET /api/v1/config/drift and the console’s Cluster screen show every orchestrator’s fingerprint, its age, and whether it matches.

File mode. One configuration file (-config: controllers: and availability_plans:) is deployed to every orchestrator and is the only source. In file mode:

• Every configuration write through the API returns 409: create or replace an availability plan. validate_only still works. The server enforces this; the console only follows it.
• Credentials are not in the file. A controller’s auth names a secret with credentials_ref: local:NAME. qfence-orchestrator secret put stores it (a JSON object of the auth method’s fields) in that node’s encrypted secrets store, through the running orchestrator’s secrets API (-server), which re-resolves it at once, or offline. Each orchestrator resolves it at start. Writing a secret is allowed in file mode: a secret is not configuration, and the reference that names it still goes through the file (ADR 0011 §8). Since Phase 10.2 a reference that does not resolve no longer stops the start: the controller is failed on that orchestrator, and its fences are UNKNOWN there (another site may be the one that needs the secret). A value where a reference belongs is still fatal.
• Configuration that was changed through the API in an earlier local-mode period is discarded when a node starts in file mode. Otherwise it would come back on top of a newer file if the cluster later switched back to local mode.
• GET /api/v1/config/export (Administrator, audited) renders the running configuration as a file. It holds references only, as the configuration does. A node that loads the export has the same fingerprint as the node that produced it (tested).

Switching modes. POST /api/v1/config/migration/plan then .../execute (Administrator; also Settings → Configuration in the console). Switching never changes the configuration in force. A plan passes only if it runs on the raft leader and no orchestrator’s fresh report differs. To switch to file mode, it also requires every orchestrator’s loaded file to equal the configuration in force. Execute recomputes the plan and refuses with 409 if its plan_id no longer matches, so a plan made before any change to the configuration, the files or the reports cannot be executed. Both switch directions are audited as config.source.migrate. The mode lives in the raft state, so it survives restarts and applies to every orchestrator.

Tests: internal/configsync/reporter_test.go (drift across a real leader and follower, the gate, the dry-run warning, a report under a key the leader lacks refused, a cluster key rotated without a restart), internal/api/configsource_test.go (credentials_ref resolution and refusals, export without credentials and its round trip, local → file → local including a wrong and a stale plan_id, 409 on every write in file mode), pkg/quorum/breaker_test.go (mode survives a snapshot), and cmd/orchestrator/filemode_test.go. That last one runs the real binary through export, secret put (which refuses rather than hangs while the orchestrator holds the store), a restart with -config, the switch to file mode, a restart in file mode, and the switch back.

People: OIDC, with local tokens as break-glass (stage 3). cmd/orchestrator -oidc-issuer ... -oidc-audience ... accepts the identity provider’s access tokens, so who may do what is decided in one place for every orchestrator and nothing about people needs replicating. Endpoints come from the issuer’s discovery document, whose issuer must match exactly. Tokens must be RS256, from that issuer, for that audience, and carry exp (30 seconds of clock leeway). The role comes from -oidc-role-claim, which may be a string or a list such as a groups claim, optionally through -oidc-role-map. A token that maps to no role, or to more than one, is refused rather than defaulted. When the IdP rotates its keys, a token with an unknown kid triggers a re-fetch of the JWKS, at most once every 10 seconds. The key set is also re-fetched every 15 minutes (-oidc-jwks-refresh), in the background, so a key the IdP removed stops being trusted. A failed re-fetch keeps the cached keys, and no request ever waits on the IdP for a key it already holds. Each node’s local API tokens keep working (mode both) so an IdP outage cannot lock operators out of failover. -oidc-local-tokens=false turns them off; then no bootstrap token is issued and the token API returns 404. An IdP that is unreachable at start does not stop the orchestrator. It logs an error and retries every 30 seconds. Until the keys load, no OIDC token is accepted and the console is not sent to the IdP. In the audit log an IdP user appears as oidc:<preferred_username> [<sub>], or oidc:<sub> without a name. A local token may not be issued with that prefix, so an IdP account can never pass for a break-glass token. Tests: pkg/auth/oidc_discovery_test.go (discovery and issuer mismatch, key rotation and its rate limit, required exp, group mapping, the chain) and cmd/orchestrator/oidc_test.go (the real binary against an in-process IdP: OIDC and break-glass together, OIDC only, IdP down at start, audit actors). Against a real Keycloak (deploy/keycloak-lab: 26.7.4, dev mode, in-memory storage):

• cmd/orchestrator/keycloak_test.go covers Keycloak’s real tokens with roles from groups, an unmapped user and a tampered token refused, and the audit actor. It also covers signing-key rotation: a token from a new key is accepted within seconds, and a token from a removed key stops being accepted.
• web/console/playwright.keycloak.config.ts runs the console’s Authorization Code + PKCE sign-in in a real browser through Keycloak’s own login page. It covers the administrator and operator roles and an unmapped user.

These tests found three gaps, now fixed:

• a removed signing key stayed trusted until restart;
• tokens from a freshly rotated key could be refused for up to a minute;
• a user the orchestrator refused was silently returned to the sign-in page. The console now explains why.

These tests run only when Keycloak is available (QFENCE_KEYCLOAK_URL), not in CI. A live stream (/api/v1/stream) authenticated with an OIDC token stays open past the token’s expiry. It is read-only.

Raft over mutual TLS (stage 4). cmd/orchestrator -raft-tls-cert/-raft-tls-key/-raft-tls-ca runs raft on the same mutual-TLS transport the witness already had (quorum.NewMTLSNetworkNode). Every connection, in both directions, must present a certificate that chains to the cluster’s CA; peers are identified by CA, not by address. A partial set of flags is refused at start rather than silently falling back to plaintext. Plaintext raft in cluster mode still works, with a startup warning. qfence-orchestrator pki init|issue is a minimal offline CA for this: it refuses to overwrite an existing CA, writes keys 0600, and issues 90-day certificates. Go’s TLS 1.3 negotiates a hybrid post-quantum key exchange by default. The self-hosted witness’s deploy templates now carry its matching -tls-* flags. Tests (cmd/orchestrator/raft_tls_test.go): two orchestrators form a quorum over mTLS while a third voter whose certificate comes from another CA never joins or sees quorum; the shipped cmd/witness and an orchestrator form a quorum over mTLS with the other site down and commit an acquire through it; and a partial flag set is refused. Still missing: certificate renewal short of a restart.

Certificate revocation on every voter. A revoked raft certificate is refused by every voter, not only by a hosted witness:

• Signed lists. Revocations travel as standard X.509 CRLs signed by the cluster’s CA.
  • A hosted witness’s operator signs one per tenant, and serves it at a URL the sites fetch.
  • qfence-orchestrator pki revoke -dir ... -name dc2 (or -serial) does the same for a self-managed CA. It writes crl.pem, and pki crl re-signs it.
  • The CRL number is the count of revocations, which only ever grows.
• Sources. An orchestrator reads a list from -raft-tls-crl (a file, re-read on SIGHUP and every -raft-tls-crl-refresh), -raft-tls-crl-url (e.g. the tenant CRL), POST /api/v1/cluster/crl (Administrator, leader only, audited) and the raft state.
• Verification. Each voter verifies every list against its own trusted CA and enforces only the newest valid one. A list signed by another CA, or older than the one in force, is refused.
• Distribution. The leader publishes a newer list into raft, where the state machine only accepts a higher CRL number. Every voter picks it up, including the self-hosted witness, which also accepts -tls-crl, and a site that cannot reach the URL.
• Enforcement. A revoked peer is refused at the handshake, and any connection already open to or from it is closed on its next read or write.
• Restarts. Each voter stores the list it enforces beside its raft data and loads it before raft starts, so a restarted voter enforces it from its first connection.
• The remaining window. A voter that has never seen a list (down since before the revocation, and with no file) accepts the revoked peer until it syncs with a peer that has the list. The flags close that window: -raft-tls-crl-url on sites, -tls-crl on a witness.

GET /api/v1/cluster/crl shows the list in force (number, revoked serials, source, and whether its next-update date has passed) and the one in raft. An expired list stays enforced, because revocations are cumulative.

Tests:

• pkg/mtls/crl_test.go: newest signed list wins, another CA’s list is refused, re-signing is accepted.
• pkg/quorum/crl_test.go: raft only moves forward, and survives snapshots.
• cmd/orchestrator/revocation_test.go, with the real binaries. With a pki CA, the list is published on the leader; the leader and the witness then keep committing. Stopping the witness makes the leader lose quorum even though the revoked site is still running, which proves the site refuses it. The restarted witness enforces its stored list, and without the leader the revoked site and the witness never form a quorum. That a revocation at the hosted witness’s operator reaches the sites through the tenant CRL URL is tested in qfence-managed, with these binaries.

While writing these tests, a flaw in the first version turned up: a witness restarted before it had applied the list trusted the revoked site, and the two formed a quorum once the leader stopped. It was closed by storing the list on each voter and by the file flag.

Replicated mode (stage 5). Configuration is edited through the API and console of any orchestrator and applied to all of them through raft.

• Encryption. Every provider and availability plan is one entry, sealed with AES-256-GCM under a 256-bit key from -config-key-file. Every orchestrator holds the key; the witness never does. The key file must be 0600 or the orchestrator refuses to start. qfence-orchestrator config-key new creates a key. The associated data binds each ciphertext to its namespace, entry, version and key ID, so a ciphertext cannot be moved to another entry, relabelled or replayed onto another version. Versions only increase: deletions are tombstones.
• Compare-and-set. A write names the version it replaces. The raft state machine applies a batch all-or-nothing and refuses a stale write (409), so two concurrent edits cannot silently overwrite each other.
• Followers. A follower forwards the already-encrypted write to the leader over the peer channel. The signature is bound to that endpoint, so a signed request cannot be replayed to another one. The forwarded body is ciphertext.
• Validation. Before committing, a provider configuration is built with the provider’s own factory and an availability plan is validated against the live providers. A write every orchestrator would then fail to apply is therefore refused up front.
• Failing closed. An orchestrator that cannot decrypt or apply the replicated configuration refuses every operate and force request with 503 and the reason, and refuses automatic failover. This covers a missing or wrong key, a key it does not hold, or a credentials_ref it cannot resolve. Reads keep working. It never falls back to a file or to node-local state. At start, a node in replicated mode applies neither its configuration file nor the demo groups, and discards configuration persisted through the API in local mode.
• A restart while cut off from quorum (fixed 2026-10-18). hashicorp/raft (v1.7.3) does not persist the commit index: a node that starts without a leader restores its latest snapshot and applies nothing after it until a leader tells it which entries are committed. Before this fix, an orchestrator restarted while cut off started from its last snapshot’s configuration source and entries, or with none, and then from local mode and its own file, against ADR 0001 §3. Now each orchestrator snapshots its raft state when the configuration source or entries change (as it does for the cluster secrets store and the revocation list), and records on its own disk the raft index of the newest configuration change it has seen, before it applies it, and of the newest cluster secrets store change. One whose restored state is older (a crash before the snapshot, a snapshot that failed or was lost) knows it: it applies no configuration and guesses no source, refuses every change and operation with 503 and why (reads go on), reloads no file, and says so in its log and GET /api/v1/config/source; it applies its configuration once a leader has brought it up to date. A copy of the cluster secrets store older than the one recorded makes every cluster: reference unavailable the same way, rather than resolve a value since deleted or replaced. A record beyond the node’s own raft log is from raft data since replaced (a cluster built again with -state-dir kept), and is dropped with a warning rather than hold the node back for ever. Tests: pkg/quorum TestTheConfigIndexIsTheLastConfigurationChange; internal/api TestAnOrchestratorBehindItsConfigurationFailsClosed; cmd/orchestrator TestReplicatedConfigurationAcrossARestartWhileCutOff (two orchestrator binaries and the witness binary: dc2 cut off and restarted keeps the replicated configuration from its snapshot; with its snapshots deleted it fails closed, and applies the configuration once dc1 and the witness return; without the snapshot on change, or without the guard, it fails).
• Switching in. A plan to replicated mode requires every reporting orchestrator to hold the active key, and warns about inline credentials that would enter the raft log, encrypted. Executing it encrypts the configuration in force and switches the mode in a single raft command, refused if the replicated store changed since the plan.
• Switching out. Going to file mode needs every orchestrator’s file to match the configuration in force. Going directly to local mode is refused: a restarted node would silently lose configuration that exists only in raft.
• Key rotation. Put the new key first in every key file and restart the orchestrators one at a time. Then run POST /api/v1/config/rotate-key (or the console’s “Re-encrypt” button). It is refused until every orchestrator reports holding the new key, then re-encrypts every entry in one compare-and-set command. The old key can then be removed.
• Tests.
  • internal/configcrypt covers binding to entry, namespace, version and key ID, tampering, wrong and unknown keys, rotation keyrings, and key-file parsing and permissions.
  • pkg/quorum/configentries_test.go covers compare-and-set, the all-or-nothing batch, the revision check with an atomic source switch, refusal of plaintext, tombstones against replay, and snapshots.
  • internal/api/replicated_test.go uses a real three-voter cluster. It covers import from local mode, an edit made on a follower and applied everywhere, and a raft voter without an orchestrator (the witness’s position) holding no plaintext. It also covers a node with no key or the wrong key refusing operations and automatic failover while reads still work, rotation refused and then completed, an old-key-only node locked out afterwards, replicated → local refused, and replicated → file.
  • cmd/orchestrator/replicated_test.go runs the real binary: the switch, an API change surviving a restart from the raft log, operations refused without the key, and a world-readable key file refused.
• Limits. The witness learns entry names (plan and provider IDs, which ownership state already exposes), sizes and change times. Anyone holding the key and a copy of the raft log can read the configuration. A provider instance is only rebuilt when its configuration changes, so a credentials_ref secret updated on one node is used there only after its next restart or configuration change.

Changing membership at runtime. POST /api/v1/cluster/voters/plan then .../execute (Administrator; also Cluster → Change membership in the console) add or remove a raft voter, for example to join a hosted witness or replace a site that is gone. The plan passes only if:

• this node is the raft leader;
• for an add, the ID and address are new, and something is listening on the address (the voter must already be running, not bootstrapped);
• for a remove, the voter is a member and not the current leader;
• after the change, the voters this node can reach right now still form a majority of the new membership.

It warns about an even number of voters and about a single remaining voter. Reachability is a TCP connection to each voter’s raft address; over mutual TLS a stranger cannot get further. Execute recomputes the plan and refuses (409) if the membership or any voter’s reachability changed, so a plan cannot be executed after a voter it counted on went away. Changes are audited (cluster.voter.add/remove) and published as MembershipChanged events. GET /api/v1/cluster and the voter table now read membership from the live raft configuration. Before this, a node restarted without -raft-bootstrap listed only itself. Tests (cmd/orchestrator/voters_test.go, the real binaries over mTLS):

• A running two-site cluster adds a separately started witness, keeps quorum and commits with one site down, refuses to remove the witness then (it would lose quorum), refuses to remove the leader, and removes the failed site.
• The full hosted onboarding, the same change against a hosted witness’s tenant, is tested in qfence-managed with these binaries.

Reloading the configuration file. In file mode an orchestrator re-reads its file on SIGHUP (systemctl reload), POST /api/v1/config/reload (Administrator) or the console’s reload button. The new file is first built in full on a scratch registry, exactly as a start would build it. A file that cannot be applied in full is refused and changes nothing (422). Otherwise the running configuration is brought to match it: changed providers are rebuilt (credentials and construction are checked before anything changes), changed availability plans re-registered, and providers and groups the file no longer lists removed. Each reload reports what changed and is audited (config.reload, success or failure, with the actor or signal:SIGHUP). The node’s file fingerprint is updated, so the drift check sees the new file at once. Each orchestrator reloads its own file, so until every one has, drift detection reports them as different and automatic failover waits. Reload is refused outside file mode. In local mode the file sits under API changes; in replicated mode the replicated store is the only source. The replicated applier uses the same reconcile step. Tests: cmd/orchestrator/reload_test.go (the real binary: SIGHUP, the API, a broken file changing nothing, refusal in local mode, the audit trail). An availability plan removed from the file keeps its ownership state in raft.

Found by the demo lab

Building deploy/demo-lab (the real providers against a simulator of their vendors’ APIs) found two defects that no existing test covered:

• Availability plans configured through the API or a configuration file could never fence. planConfigFromDoc built the fencing policy field by field and left FenceTimeout and VerifyTimeout at zero. Every fence therefore ran with an already-expired deadline, reported UNKNOWN, and blocked the failover. It failed closed, since nothing was promoted, but no such group could fail over. Only groups built in Go with fencing.DefaultPolicy(), which is what every earlier test used, worked. The policy now starts from DefaultPolicy(), and the engine treats an unset timeout as the default. Regression tests: internal/fencing (TestUnsetTimeoutsUseTheDefaults, with a fencer that honours its deadline as network fencers do) and internal/api (TestConfigDocGetsTheDefaultFencingTimeouts). The lab then showed the full sequence through real providers: failing replication blocks the promotion; with healthy replication qfence powers off the dc1 host (Redfish), stops its VM (Proxmox) and enables the VIP (FortiGate); and a dependent group follows only once its dependency is active.
• The FortiGate provider’s health check failed against a FortiOS-shaped response. Monitor endpoints return results as an object, but the client required a list. Fixed, with a regression test using the FortiOS monitor shape.

Adding the systemd hosts to the lab (Phase 9.2, 2026-10-09) found two more things:

• The lab could no longer fail over a lost site. Phase 10.1’s conversion gave each BMC controller its hosts’ site, site: dc1, and since Phase 10.3 a controller is called only from the sites in its reach. When dc1 was lost, no orchestrator could call its BMCs, so every proof was UNKNOWN and erp-db and web-portal stopped at BLOCKED, while the lab’s walk-through promised a failover in 20 s. It failed closed, but the lab did not show what it said. The BMC controllers now reach both sites, as BMCs on a management network do, and losing the site that owns them fails every plan with automatic failover over to the other (recorded decisions unchanged). Nothing in the code was wrong; no test runs the lab’s failovers.
• A secondary application fence on a host its own power fence switches off never acts. A site’s secondary fences run after its required ones (ADR 0009 §2), so billing’s unit.stop, secondary as ADR 0005 §7 advises, reaches a host its BMC fence has just switched off, and reads UNKNOWN. It is not a hole: billing’s unit is not enabled, the power fence is durable, and on_drift: fence fences a host that is powered on again (verified: powered on by hand, it was off again within two seconds, with billing still down). But the mask that would keep billing down whatever happened is not applied. A required unit.stop runs first, in configured order, and then a site whose host cannot be reached blocks every failover; the lab tried it, and both a lost site and a second failover after the first had powered the standby off stopped at BLOCKED. Decided 2026-10-12: secondary application fences now run before a site’s required ones (see “Secondary application fences first”).

What the UI will and will not let you do

This section is for operators and auditors deciding how far to trust the three web consoles. It describes what the running UIs actually enforce, as checked in the code and by tests, not what the design documents intend. One rule sits under all of it: the UI never decides safety. Every state, health value, owner and generation shown on screen is what the API returned. The UI never infers one, never predicts one, and never updates optimistically: after every operation it re-reads state from the server, including after a failure.

Failover and switchover (orchestrator console): dry-run gated

• Execute stays disabled until a dry-run plan for the exact parameters on screen has been fetched and rendered. Changing the target site discards the plan and disables Execute again.
• A plan older than 60 seconds is stale. Execute disables itself while the dialog is open, and the plan says to re-run it.
• A BLOCKED plan has no Execute control at all (removed from the page, not greyed out). The server’s reasons are listed in its place.
• A fence warning must be acknowledged (“This operation will fence dc1”) before Execute enables.
• Double-submit cannot happen. One Idempotency-Key covers the whole attempt, including retries, and the button shows a pending state.
• Tested against a real cmd/orchestrator: web/console/tests/e2e/01-golden-path, 02-blocked-path, 05-keyboard-only, 07-keyboard-focus, plus unit tests of DangerousAction.

Operations with no server-side dry-run: confirmation only

Acquire, release, provider enable/disable and resetting a tripped circuit breaker (Phase 8.9) have no dry-run endpoint on the server, so the console does not pretend to show a plan for them. Each one gets an explicit confirmation dialog, an idempotency key, and a re-read of state afterwards, including after a refusal. They are gated to the Operator role (Administrator for providers), the same permission the server enforces. The breaker reset takes an optional reason, recorded in the audit log.

The group page also says when a group is set to fail over automatically but the raft leader was not started with -automatic-failover, so nothing will fail it over by itself (Phase 8.9). The server reports each orchestrator’s flag in its node report; the page does not guess, and says “not known yet” before the leader has reported.

Forced operations: maximum friction, always audited as forced

Manual fence is the one forced operation the console offers. It fences the current owner at its current generation outside the quorum → ownership → fence → promote sequence, so on an ACTIVE group the live site stops serving and nothing takes over. Since Phase 8.6 the server treats it that way: POST /fence requires the force permission (EmergencyOperator or Administrator; before 8.6 a plain Operator could call it), requires a reason, and audits it as forced with that reason. The console matches this. Operators and Viewers don’t see the Fence action at all. The form and dialog are red-bordered and titled with the sequence being bypassed, and state that the action will be recorded as forced. The Fence button stays disabled until there is a reason of at least 10 characters and the plan id has been typed exactly (paste blocked). There is still no dry-run, because the server has none for fence. Tested by internal/api/manual_fence_test.go (403 for Viewer/Operator, 400 without a reason, the fencer never reached on refusal, audit row forced with reason) and web/console/tests/e2e/09-forced-path.spec.ts (the full flow against a real orchestrator, through to the audit row marked FORCED).

Force release is offered on the group page since ADR 0003, for leaving BLOCKED once the blocked site is known to be stopped, with the same friction as manual fence. force-acquire is not reachable from any console. It exists only in the REST API, which requires the force permission and records it as forced. The console’s shared dangerous-action component already implements the same friction for them, gated on the same force permission (Phase 8.6 aligned it to the server instead of the single role name UI_DESIGN.md once used), but no screen offers them. The console’s Audit screen can filter to forced operations in one click, whatever issued them.

The witness console holds nothing to escalate

The witness status page is public by design and contains no secrets. The join and config pages require the witness’s single-use bootstrap token, printed only to its own stdout. The join wizard never contacts another node: it produces the exact commands for the existing cluster’s leader to run.

Console parity (Roadmap Phase 8.9)

The fencing features of ADRs 0002–0009 were built API-first. Phase 8.9 made what they decide visible and operable from the console. Every addition is a read of state the server already keeps; no decision path changed. The console renders what the API returns: the waits, verdicts and meanings are worked out on the server.

• Automatic failover. The group page says when a group is set to fail over automatically but the leader does not run the scheduler (-automatic-failover, now in each node report). While the owner is unreachable it shows since when, when a failover may be tried, until when only a proof moves the group, and the last proof and refusal, from the scheduler on the leader. A tripped circuit breaker can be reset by an Operator, with a reason kept in the audit record.
• Self-fencing. Each site’s report: its age, whether it can fence itself, its dead-man switches and their timeout, its last self-fence, and what a failover relying on it would find. For an owner the scheduler found cut off, the report is judged as it was when contact was lost, as the backing check does.
• Fences. The Fencing tab and overview show each target’s site, role, ladder step and last result: durable or not, attested, or skipped by escalation. The last fence gives each site’s verdict. GET .../last-fence replaces the untagged struct POST .../verify returned.
• Readiness and attention. Each site’s readiness status, and in the group list and dashboard, the reasons a group needs a person.

Found on the swarm lab, and fixed:

• A site that had just lost its power read as “a standby ready to be promoted”. Its last site check predated the fence debt, and a site owing a fence is not checked. It is now listed as owing a fence, with no earlier verdict. This predates Phase 8.9.
• A non-owner’s dead-man switches read as “not armed: needs a shield”. Only the owner arms them.
• A cut-off owner’s report read as “stale: a failover cannot rely on it”. It was judged against now instead of at the moment contact was lost.

Residual:

• The scheduler’s view, the site checks and the last fence are node memory. A follower says where they are rather than showing them.
• The last fence and per-target results reset when an orchestrator restarts.

Tests: internal/api/automatic_status_test.go, visibility_test.go (last fence, automatic status, self-fencing, readiness by site, attention, dead-man listing, shield modes), internal/scheduler TestSiteModeStatus, cmd/orchestrator/schemas_test.go (every config key in a schema; list items declared), and the console’s AutomaticFailover, SelfFencing, PlanDetail, FencingOverview, SiteChecks, PlanList, Dashboard, ProviderNew, PlanEditor and Cluster tests.

Production Web UI — Phase 8.6 Hardening (Roadmap Phase 8.6)

What is now tested, per app, and what the testing found. Every suite runs against real binaries (cmd/orchestrator, cmd/witness), and all except the opt-in memory soak are wired into CI (web_build, web_e2e). The counts are Phase 8.6’s, when the hosted plane’s console was in this repo too; since Phase 12.2 its suites run in qfence-managed.

Measured on a 2-vCPU development VM: FCP 576ms (budget 1.2s), dashboard interactive 440ms cold (2.0s) and 259ms warm (400ms), route change 17ms (150ms), SSE→DOM 16ms median and 25ms worst (250ms, measured as an upper bound from request send), 400-row scroll 44–59fps (the low end while the 8h soak shared the same two vCPUs) with zero main-thread long tasks in every run. The scroll gate is “no main-thread task over 50ms” plus a 30fps gross-regression floor, not a literal 60fps. On a shared 2-vCPU runner, compositor starvation drops frames with no long task at all, so a 60fps gate would measure the runner, not the app.

Soak (live tailing): 3-minute smoke at 20 events/s: heap ~20MB, flat once the 2000-row ring buffer filled. Run 2 at 2 events/s: 7h25m of the planned 8h (the run was stopped when the development session hosting it exited, 35 minutes short), 53,333 real events. Peak JS heap 22.3MB against the 150MB budget; the event list never exceeded 2,000 rows; the hourly median heap drifted from 20.0MB to 21.6MB and was flattening (+0.5MB over the last four hours), within the soak’s own 1.25× growth gate (1.10×). The stream was LIVE at every sample but one, a single RECONNECTING at 141 minutes, recovered by the next sample, which coincided with full -race and e2e suites saturating the same two vCPUs. Run 3, the full 8h at 2 events/s: 57,552 real events; every one of 474 samples LIVE; rows capped at 2,000; peak JS heap 22.6MB. Hourly mean heap 20.7MB in hour 2, then 21.0, 21.1, 21.4, 21.6, 21.8 and 21.8MB, levelling off. The growth gate compares the median in the first third after the ring buffer filled (20.8MB) with the maximum in the last third (22.6MB): 1.09× against the 1.25× limit. Every memory gate passed, but Playwright still failed the run on its time limit. The perf config traced the whole run, and writing an 8h trace (every API call and every sample’s snapshot of a 44,000-node page) when the browser context closed took about 14 minutes, past the 490-minute limit. The soak now runs with tracing off; a 3-minute check with that change finishes promptly. Run 5, the full 8h at 2 events/s (2026-10-03), passed in the test runner: 57,539 real events; all 478 samples LIVE; rows capped at 2,000; peak JS heap 17.6MB. Hourly mean heap 14.2MB in the first hour, then 16.1, 16.3, 16.5, 16.8, 17.0, 17.2 and 17.0MB. Growth gate: first-third median 16.3MB against last-third maximum 17.6MB, 1.08× against the 1.25× limit. (Run 4 was lost with its session; before run 5, the soak’s plan had to be created in the ADR 0010 format, which it still did not use.) Found on the server side: the orchestrator’s event bus and audit log are unbounded in memory (both append forever), so a long-running orchestrator’s own memory grows with every event even though the browser’s stays flat. Both have since been capped (the audit log at -audit-retention, 100,000 records by default; the event bus at a 10,000-event ring).

Idempotency (§37)

Covered at every layer it applies to, not just the provider boundary the spec states it for: ownership engine (TestReleaseByActualOwnerIsIdempotent, TestReacquireBySameActiveOwnerIsIdempotent), mock provider (TestFencerRepeatedFenceIsIdempotent, TestServiceAcceptsRetryAtSameGeneration, TestServiceReleaseTwiceIsSafe), and the REST API itself (internal/api/api_test.go’s TestReleaseTwiceViaAPIIsSafe).

Security-relevant tests

• Fencing-token enforcement (§40): mock.Service is the reference implementation; TestServiceRejectsStaleGeneration and the chaos-tier TestDelayedStaleCommandIsRejectedAfterRealFailover.
• Secrets never leak into API responses (§27/§49): internal/api/api_test.go’s TestNoSecretFieldEverAppearsInAnyResponse scans every response type for “secret”/“password”/“credential”.
• OIDC/JWT verification (§50): pkg/auth/oidc_test.go — tested against a local JWKS server (no external IdP needed). Explicitly covers the negative cases that matter most for this class of vulnerability: alg:none bypass, a token signed by an untrusted key while claiming a trusted kid, wrong issuer/audience, expiry, and an unrecognized role claim (rejected, not defaulted — §48 applied to authorization).
• TLS (§49): pkg/tlsutil/selfsigned_test.go proves the generated certificate completes a real TLS handshake with a client that verifies it, and that a client with no trust in it is rejected. cmd/orchestrator defaults to HTTPS. Since Phase 11.1 its listener offers only TLS 1.3 with X25519MLKEM768 (TestTheAPIListenerTakesOnlyThePostQuantumKeyExchange), so curl -k succeeds only with curl built against OpenSSL 3.5 or later; qfence-orchestrator api verifies the generated certificate against the copy in <state-dir>/api-cert.pem.

Known gaps

Listed here deliberately rather than left as scattered comments, per §60’s instruction to “surface any known gaps honestly… rather than implying full coverage.”

• Delayed network packets (chaos) — see the Chaos Tests table above.
• OIDC’s browser login flow (Phase 8.2’s web/console, src/auth/oidc.ts) is a real hand-rolled Authorization Code + PKCE implementation (state/nonce/token exchange, crypto.subtle S256 challenge) but has only ever been exercised against a mocked token endpoint (oidc.test.ts), never a real IdP — this sandbox has none to test against. The old web/frontend Admin UI it’s replacing (retired in Phase 8.3) never had a browser OIDC flow at all, only bearer-token verification server-side; that older gap is now superseded by this narrower one. Browser session management follows §4.5 exactly: the OIDC access token lives in memory only (lost on refresh, by design), the bearer-token fallback lives in sessionStorage only, and the two are never held simultaneously. CSRF protection remains out of scope for the same reason it always was: the UI’s bearer-token-via-header pattern isn’t subject to classic CSRF the way a cookie session would be.
• JWKS parsing supports RSA keys only. An OIDC provider using EC or Ed25519 signing keys isn’t supported yet (pkg/auth/oidc.go).
• Real controller types exist (Phase 4, ADR 0010), but, apart from Docker (ADR 0007, tested against real Docker Engines), only ever validated against local doubles, never live hardware. Redfish, IPMI, FortiGate, and Proxmox (internal/controllers/infrastructure/redfish, /ipmi, /fortigate, /proxmox) are real REST/CLI clients implementing the documented wire protocols, each tested against an httptest double (or, for IPMI, a fake ipmitool script) standing in for the real system — no live BMC, firewall, hypervisor, or RMCP+/IPMI v2.0 wire-protocol behavior has been exercised anywhere. Treat these as protocol-correct-on-paper, not field-proven, until run against real infrastructure. This holds for Phase 10.5’s capabilities too: FortiGate interface.down, vlan.down, probe.ping and probe.service, and Proxmox node.fence, vm.link-down, the guest-agent probes and password login. The exact FortiOS fields and endpoints they read are as documented and not yet confirmed: the route lookup, the link-monitor and load-balance monitors, ha-mgmt-interfaces. So are Proxmox’s local flag, current=1, agent/exec’s JSON body, the VE 9 privilege names and HA’s handling of an API stop. Docker’s node.drain ran against a real Engine.
• Bindings in a configuration file are not checked against their resources. Binding validation per resource (Phase 10.3) runs for bindings created or changed through the API and on request (POST .../bindings/validate). A file’s bindings are compiled as written, and a wrong resource shows only when a fence or check runs (UNKNOWN or an error, never FENCED).
• A site whose orchestrator is down cannot be fenced from another site unless the controller lists that site in its reach (ADR 0010 §9, by design). Its fences report UNKNOWN, and the failover rests on its dead-man switches and self-fencing.
• An orchestrator that declares no site calls every controller itself, as before sites existed. A multi-site cluster must set -site on every orchestrator; every orchestrator in a cluster has the cluster keyring (-cluster-key-file), which executors are reached with.
• A local: reference is resolved from each orchestrator’s own store. A name missing on one node leaves the controller failed there (registration and the node reports show which orchestrators cannot resolve it). secret put -server has to be run against every orchestrator that calls the controller, since each holds its own local store; it no longer needs the orchestrator stopped (Phase 11.1). A cluster: reference (Phase 11.2) is written once, through any orchestrator, where the cluster store is in use.
• The IPMI password is an ipmitool argument (-P), so another local user can read it from the process list while a command runs. The orchestrator’s host should not be shared.
• AWS Secrets Manager has not been checked against a real AWS account (Phase 11.5): DEFERRED (real hardware/platform). The reader is tested against pkg/secrets/awsdouble, which botocore, AWS’s Python SDK, accepts as AWS, and its signer against AWS’s published test vectors. That proves the requests are signed and shaped as AWS’s SDK makes them. It does not prove how a real account answers: that each refusal comes with the code the double gives (a KMS permission refused as AccessDeniedException, a disabled key as DecryptionFailure, a secret marked for deletion as InvalidRequestException), the metadata service’s timing and hop limit, a VPC endpoint, or the TLS key exchange AWS’s endpoint negotiates. The check waits for the owner’s go-ahead and an account. Its residual risks are in “AWS Secrets Manager, a reader with the host’s identity”.
• No human screen-reader pass (Phase 8.6). The failover flow has been checked at the accessibility-tree level (names, roles, states, live-region placement), but nobody has used NVDA, JAWS or VoiceOver on it. How announcements actually sound and in what order they arrive is untested.
• The systemd type’s runs against a real host are not in CI. They need privileged containers on a Docker daemon (QFENCE_SYSTEMD_IT=1), as the Docker type’s do. CI runs the same kit and acceptance slice against a model of the host, which could drift from real systemd, sudo or OpenSSH; the model refuses any script but the guard script the type ships.
• A systemd fence reads the unit, not the application. A unit that reads masked and inactive may still leave processes running that systemd does not track: ones started outside the unit, or a unit with KillMode=none. It is an application fence, so it never proves a site; the compute fence does.
• An external controller’s honesty is checked, not proven. qfence counts no reply as UNKNOWN and refuses stale commands itself, but a service that answers in the protocol’s terms can still lie: a verify that says FENCED without reading anything. The conformance kit catches the common ways (a double undoes the fence and verify must notice), and an application fence never proves a site in any case; but a controller that passes the kit and then misbehaves in production is not caught by anything but its own operators.
• A service’s renewed certificate fails the controller unless it was pinned ahead. The pin is the service’s own certificate, not a CA. An administrator pins the renewal as the next certificate before the service changes to it, or pins it after (POST /api/v1/controllers/{id}/certificates, TestCertificateRotation). Until one of the pins matches, every call through the controller is refused before anything is sent, and its fences are UNKNOWN; CertificateNotPinned says why. A connection made before the renewal is reused until it closes.
• Automatic failover watches applications only where a plan has health bindings. Since ADR 0002 its signal is “the owner site has no reachable raft voter”. A plan with health bindings and failover.application_health also acts on its application (ADR 0005, Phase 9.1a; see “Application health and local recovery”). A plan without them is not failed over when its application dies at a live site, and stays ACTIVE at a site whose VM stopped.
• A site that returns after an attested failover can run briefly before qfence fences it. qfence now fences it as soon as its fence endpoints answer (ADR 0008 §1, above), but its workloads may start first. Shields (ADR 0008 §2) keep it out of the survivor’s resources in the meantime, and continuous checks (§3) find a non-owner site running its workload within a minute. Self-fencing (ADR 0008 §4) narrows the window only for a site that returns still cut off from quorum. A site whose failover proved it fenced by power alone owes no fence, and is found only at the next check (see “Self-fencing”, Residual risk).
• An attested failover can cause a split brain if the statement is wrong. Operator-attested fencing (ADR 0003, below) lets a person stand in for a fence verdict qfence could not reach. qfence cannot check the statement; fencing tokens (§40) and watchdogs (§16) only contain the damage. Two approvals compare identities, not people: one person holding both an SSO login and a break-glass token counts as two.
• Configuration is per orchestrator node. API-made plan configuration persists on the node it was made on and is not replicated (by design, see State and persistence), so nodes can diverge unless the operator keeps them in step.
• Mixed-version clusters do not share circuit-breaker state. A voter older than Phase 8.6 rejects the breaker commands; ownership state is unaffected, but its copy of breaker state stays empty until every voter is upgraded.
• Tokens minted by SSO logins never expire, and with persistence they now accumulate on disk (as hashes). Token expiry is not implemented.
• No CI pipeline run is linked here yet (see CI status note above). Every job passed on 2026-10-28 on the owner’s self-hosted GitLab runner, but the project is private; the move to GitHub (Phase 12.3) links a passing GitHub Actions run.

Test tiers, at a glance

• internal/*/**_test.go — unit and package-level tests.
• tests/sim (+ internal/sim/*_test.go) — fast, deterministic, seed-based. Run on every commit.
• tests/chaos — real Raft clusters, real timing, real network-partition simulation. Slower.
• tests/e2e — the 9 Critical Acceptance Tests, one canonical test per CAT.