The swarm lab

Real controllers against real workloads, on one machine (ADR 0007, ADR 0010). Two Docker Swarm sites run as Docker-in-Docker containers. A real Postgres runs as primary at dc1 and streams to a standby at dc2. qfence moves it, and the applications on top of it, between the sites through each site’s docker controller.

The demo lab simulates a Proxmox estate. It needs no privileged Docker, and this lab does not replace it.

deploy/swarm-lab/lab.sh up        # first start: about 2 minutes
deploy/swarm-lab/lab.sh status
deploy/swarm-lab/lab.sh orders    # the orders each site's database holds
deploy/swarm-lab/lab.sh reseed dc2  # discard a non-owner site's database copy (it re-seeds)
deploy/swarm-lab/lab.sh crash     # kill orders-api's task at its owner (qfence restarts it)
deploy/swarm-lab/lab.sh signins   # sign the probe user in once a second at each site's Keycloak
deploy/swarm-lab/lab.sh switchover erp-db dc2   # a planned move, as an operator asks for one
deploy/swarm-lab/lab.sh down      # stop; state is kept
deploy/swarm-lab/lab.sh reset     # delete the nodes and all state

It needs Docker with privileged containers, about 2.5 GB of RAM, openssl, GNU tar, and Go to build the binaries. It pulls docker:28-dind, postgres:17-alpine, quay.io/keycloak/keycloak:26.7.4 and hashicorp/vault:1.21.4. A lab created before erp-db moved to the postgresql controller (Phase 9.3), before sso (Phase 9.4), before its credentials moved to the cluster secrets store (Phase 11.2), before the cluster keyring (Phase 11.2b: .lab/secrets/cluster.keys, and node reports sealed in raft), or before each site had its Vault (Phase 11.4), needs lab.sh up --fresh: its configuration, certificates and secrets are made when its state is.

The controllers’ credentials are in the cluster secrets store (ADR 0011 §2), cluster:lab-*. Once both orchestrators are up, lab.sh stores each once, through dc1’s API, with secret put -ref cluster:NAME -scope-type TYPE -scope-host HOSTS: dc1 seals it under the secrets key in .lab/secrets/secrets.keys, which both orchestrators hold and the witness does not, and raft carries the ciphertext to every voter. Each orchestrator reads it from its own copy, so a site cut off from the other still resolves its credentials, across a restart. secret versions, secret revert and secret rekey -store cluster work against either orchestrator’s API.

The FortiGate’s token is in each site’s own Vault instead (ADR 0011 §3), vault:secret/qfence/lab-fortigate. Each site runs a Vault 1.21.4 (dev mode, in memory, seeded with the vault CLI at every start) on its own network, over TLS. Each orchestrator’s -secrets-config (.lab/dc1-secrets.yaml, .lab/dc2-secrets.yaml) defines vault as its own site’s, and it signs in there with a client certificate from .lab/pki/vault-clients, a CA of its own, never the raft CA. Each Vault admits only its own site’s orchestrator, to a policy that reads secret/qfence/* and nothing else. So the one reference resolves through dc1’s Vault on dc1 and through dc2’s on dc2, and the shields that use it at one site never depend on the other site’s Vault.

What runs

            dc1 (network qfswarm-dc1)                dc2 (network qfswarm-dc2)
  dc1-mgr   erp-db  primary, :5432 published   ─┐ ┌─ dc2-mgr   erp-db  standby, unpublished
            erp-app (1), web-portal             ├─┤            erp-app (0), web-portal (0)
            sso (1), :8443 and :9000            │ │            sso (0)
  dc1-wkr   web-portal                          │ │  dc2-wkr
                                    network qfswarm-wan (replication)

On the host:

lab.sh up prints the sign-in tokens.

How qfence sees it

• erp-db (ADR 0013, the PostgreSQL controller’s guide).

  • Its controllers: pg-dc1 and pg-dc2, of type postgresql, reach each site’s Postgres over TLS with its certificate pinned, as the role qfence (never a superuser). They go through the site’s erp-db-mgmt relay, so they reach a fenced standby, whose client port is unpublished.
  • Fencing: under success: ANY, either of two Level 1 proofs is enough per site. One is the site’s Docker API: the service runs its fenced shape, a standby, re-seeding from the other site once that site is promoted. The other is the manager’s BMC: power evidence (fence_action: none) that the database’s host is off.
  • Secondary targets: db.readonly makes the database read-only and ends its application sessions; as a secondary application fence it runs before the others (ADR 0009 decision 14). Unpublishing the port (docker-network-fencer, Level 2) takes the site out of the VIP’s rotation. Scaling erp-app to zero stops the writer.
  • Readiness: replica.lag, the standby’s lag as it measures it from the primary’s last report, at most 300 s. A primary is ready only where erp-db was last ACTIVE, which is how the lab’s first activation at dc1 passes.
  • Promotion: db.promote, through each site’s switchover binding, graceful. A planned switchover makes the old primary read-only, waits until the standby has replayed every byte of its WAL, fences the old primary’s site, then promotes: no committed order is lost, and lab.sh orders shows the same last order at both sites once the old primary has re-seeded.

• web-portal. It is fenced by scaling to zero, and only the Docker API proves that: the portal spans two hosts, so no single BMC can. Readiness is that the site’s swarm is up. It depends on erp-db being active at the same site, and follows it there (ADR 0015): once erp-db is ACTIVE at the other site, the leader fails web-portal over after it.

• orders-api (ADR 0005, application health and local recovery). It is fenced by scaling to zero. Its health binding (service.health) checks every 5 s that its task runs on the manager; its recovery binding (service.restart) replaces the task with a forced update. failover.application_health acts after 15 s of proven unhealthy: a restart, then 20 s of grace; a second restart 10 s later if the first did not hold; and when two restarts within 10 minutes have been used, a failover to dc2. lab.sh crash kills the task; the swarm leaves it dead, so only qfence brings it back. Crash it, watch the plan page’s recovery row go RECOVERING and back to READY; crash it twice more within ten minutes and recovery runs out, ApplicationRecoveryExhausted, and orders-api fails over.

• sso (ADR 0014, the Keycloak controller’s guide).

  • Its controllers: kc-dc1 and kc-dc2, of type keycloak, reach each site’s Keycloak on the manager, with its certificate pinned. The probe user qfence-probe signs in to the realm erp through the public client qfence-probe. The admin account qfence-admin, which may only view the realm erp, lists the realms. lab.sh up makes all three on the first start.
  • Health: realm.health, every 10 s at the owner: Keycloak’s readiness, then the probe user’s sign-in and sign-out. It catches a database that refuses writes, which readiness alone does not.
  • Recovery and fencing are the swarm’s. service.restart replaces Keycloak’s task. Fenced, sso runs no replica: it could not start against a standby. The standby site’s Keycloak is therefore stopped by design, and its controller reads it as not running, which triggers nothing.
  • It depends on erp-db at the same site, and its health and recovery use the defaults: 60 s of proven UNHEALTHY, then three restarts within 10 minutes (30 s backoff, doubling; 60 s grace each), then a failover.
  • When erp-db moves, sso follows it (follow, ADR 0015): once erp-db is ACTIVE at the other site, the leader fails sso over there, before its old site’s Keycloak is restarted for want of a writable database. lab.sh signins prints, once a second, which site signs users in; run it while lab.sh switchover erp-db dc2 moves the database, and read the outage (scenario 8).

• Shields (ADR 0008 §2). Each group has a shield at each site: the FortiGate policy that lets its VIP reach the other site’s servers. The owner’s shield disables it, so a site that comes back running gets no client traffic. On the simulator page it shows as “kept out”. fgt-edge’s token comes from the calling orchestrator’s own site’s Vault: the dry run shows each orchestrator resolving it through vault at its own site.

• Self-fencing (ADR 0008 §4). Both groups set fencing.self_fence with after_seconds: 30, backs_failover and auto_recover.

  • Cut off: a site’s orchestrator that has been cut off from quorum for 30 s fences its own targets for the groups it owns.
  • erp-db, hardware tier: its dead-man switches are the four nodes’ BMC watchdogs, driven through the lab’s ipmitool. The owner arms and renews them while it holds quorum; unrenewed, they power the site’s nodes off.
  • web-portal, software tier: it has no switch of its own, so a failover on its self-fence relies on its shield.
  • Timing: from primary_unhealthy_for (20 s) the leader tries to prove the unreachable owner fenced, and fails over as soon as it can. It relies on the self-fence only after the bound: (after_seconds + the site's fencing time) × 1.01 + 10 s, 4 min 12 s for erp-db, whose ladder has two rungs of 105 s each (ADR 0009 §2), and 1 min 46 s for web-portal.
  • Watchdogs: they run for after_seconds plus the site’s fencing time (4 min for erp-db), so a site that can fence itself does so first, leaving its database in its standby shape, before the watchdog powers it off.

• Booting fenced (ADR 0009 §7). Every workload asks qfence for its group’s lease before it starts in its active role (workloads/lease.sh). It asks through the simulator’s relay on its site’s network, which asks the site’s own orchestrator.

  • A database that is refused starts as the other site’s standby.
  • erp-app and web-portal wait until they get their lease.
  • With no answer, each waits.

  Both groups therefore set boots_fenced: true, which lets the leader return a site to standby even when its standby shape was never confirmed.

• The simulator.

  • BMCs: a Redfish system is a node container, and powering it off stops the container. The same systems answer IPMI watchdog commands; a watchdog that runs out powers its node off, and Restore leaves such a node off, as a real one would.
  • Raft links: the orchestrators reach each other through proxies in the simulator (127.0.1.x), so Partition cuts a site’s orchestrator off while it keeps running, and single links can be cut from the page.
  • API links: each orchestrator’s API is reached by the other through a proxy too (127.0.1.11:18454, 127.0.1.21:28454, advertised with -advertise-api), and each dials from its site’s address (-peer-source), so Partition and Fail also cut the commands the leader sends to that site’s executor.
  • FortiGate: a VIP is a real TCP listener. While qfence has the VIP enabled, it forwards to whichever site’s real server passes a health check. If two sites answer, the page flags a split brain.

The database’s roles

Changing ROLE restarts the task, and the entrypoint acts on the new role:

• Primary: it first asks for its lease. If another site holds erp-db, it starts as that site’s standby instead; with no answer it waits. It then promotes a standby (a new Postgres timeline). Before qfence has made either site active (generation 0), it keeps its deployed role.
• Standby: it keeps an existing standby as it is. Otherwise it re-seeds from the other site with pg_basebackup, but only from a primary on a newer timeline. A site that returns from a blackout still running an old primary is therefore never used as a source.
• The lag check: it trusts a primary only while the other site is its standby.

Re-seeding discards what the old primary had that the standby did not, which is the loss a failover accepts. A switchover without restarts, pg_rewind, and fencing at the database level belong to Phase 9’s application controllers.

Registering a controller through the API

lab.sh up --fresh --register starts the orchestrators without the controller docker-dc1 and without availability plans (which bind it). Everything else is then done through the API, as an operator would with curl (ADR 0010 §6):

1. The cluster switches to replicated configuration (POST /api/v1/config/migration/plan, then .../execute), so one change reaches both orchestrators.
2. On dc1’s orchestrator (the controller’s site; dc2’s refuses: ADR 0010 §9) a registration session authenticates with the reference cluster:lab-docker (step 1 checks every orchestrator at dc1 can resolve it), tests connectivity (the swarm’s cluster id and the Engine’s version), validates service.shape, service.unpublish, service.lag and engine.ping (each swarm.manager check inferred), and registers.
3. Once both orchestrators have reported, the plans are created with PUT /api/v1/availability-plans/{id}/config, and POST .../bindings/validate must show every binding verified: each resource found by its controller’s discovery (services.list, systems.list, policies.list) at the controller’s site.

lab.sh prints each step. Then the lab runs as usual: both groups are made active at dc1 with a failover. GET /api/v1/controllers/docker-dc1 shows the controller, the orchestrators’ views of it and its bindings; the audit log holds controller.register. A lab started this way restarts in replicated mode.

Every orchestrator validates the controllers it calls when it starts and every 5 minutes: GET /api/v1/controllers shows each one’s state, and the plans’ attention lists show a capability that starts failing. The simulator’s BMCs, FortiGate and IPMI answer the checks; the FortiGate controller names its REST API administrator (api_user: qfence) so its admin profile can be read.

Site executors

Every controller but fgt-edge is called only from its own site (ADR 0010 §9). The leader sends a binding’s action, with the plan, binding, generation and raft term, to an orchestrator at the controller’s site over the API link; that executor refuses an older term and audits what it did (executor.fence, executor.unfence). So when dc2 leads, dc1’s fences run on dc1’s orchestrator and appear in its audit log. A command whose reply does not arrive in time is UNKNOWN, never fenced. The dead-man switches are armed only by their own site’s orchestrator.

Registering and binding in the console

The console does the same: Controllers → Register a controller runs the registration session as a wizard, and Availability plans → New availability plan binds what the controllers advertise, each resource picked from discovery at the controller’s site. Playwright checks both on a running lab (cd web/console && npx playwright test -c playwright.swarm.config.ts). It registers a Docker controller for dc1’s swarm and a plan that binds only what reads: the BMCs’ power evidence and the Engine’s ping. Each run adds a controller and a plan with fresh ids, and only a fresh lab removes them: there is no delete.

Scenarios

Run failovers on the raft leader’s console (lab.sh status names it).

1. Planned failover. Fail erp-db over to dc2. Its dry run lists sso and web-portal, which follow it.

   • qfence restarts dc1’s database as a standby, unpublishes it, and stops the writer.
   • It then checks dc2’s lag, promotes dc2, publishes its port, and starts the writer there.
   • Once erp-db is ACTIVE at dc2, the leader fails sso and web-portal over after it (plan.failover.automatic, “erp-db is ACTIVE at dc2: following it”).
   • lab.sh orders shows dc2 primary with every order.
   • curl 127.0.0.1:28080 answers from dc2.
   • Moving them back is the same, in reverse.

2. Node failure. Power off dc1-wkr on the simulator page. Swarm moves its portal replica to the manager, and qfence does nothing.

3. Partition. Press Sites → dc1 → Partition. dc1’s orchestrator keeps running but loses its raft and API links, and its nodes lose the wan.

   • The leader cannot reach dc1’s executor, so its early proofs report dc1 UNKNOWN (plan.fence.early, “Post https://127.0.1.11:18454/api/v1/internal/execute: EOF”). It never calls dc1’s Docker API or BMC itself.
   • At 30 s dc1 fences itself (its own audit log has plan.fence.self): its database takes its standby shape, its portal scales to zero, and its watchdogs are no longer renewed.
   • At the bound, about 4 min 14 s (the watchdogs’ 4 min plus margins), erp-db fails over on dc1’s self-fence (plan.failover.self-fence-backed, dead-man armed), then web-portal. dc1 owes both a fence, and its watchdogs power its nodes off.
   • Restore reconnects dc1 but leaves its nodes off. The leader returns it to standby by itself (return_to_standby: automatic, ADR 0009 §6). It powers dc1’s nodes on through their BMCs, confirms its database’s standby shape, which was in place before the power went, and reports dc1 ready to be promoted once it has caught up (plan.site.standby.automatic). Nobody powers anything on or fences anything.

4. Power loss. Press Sites → dc1 → Fail. dc1’s orchestrator stops with its nodes, so no executor can call its BMC (bmc-dc1 is reached only from dc1) and every proof is UNKNOWN. Both groups fail over on dc1’s dead-man switches at the bound, about 4 min 20 s: they were armed, so its nodes are off whatever happened to its orchestrator. dc1 owes both a fence. dc1 had no chance to fence itself, so on Restore its nodes power on with its database still ROLE=primary. The database asks for its lease, is refused (“dc2 holds the group”), and starts as dc2’s standby; the simulator’s action log shows the lease line. qfence then pays both debts within seconds of dc1’s orchestrator and Docker API answering (plan.fence.returning-site).

   • Add reach: [dc1, dc2] to bmc-dc1 and the BMC’s proof comes back: dc2’s orchestrator may then call it, and erp-db fails over about 30 s after the power goes. The lab keeps BMCs on their own site, the case ADR 0010 §9 is written for: no network path between the sites’ management networks.
   • Fail a site right after restarting its orchestrator and the backing is refused: the new process has not reported yet. Both groups block (force-release them and attest dc1’s fence after 60 s).

5. Blackout. Press Blackout. Nothing answers, not even the BMCs. Since 10.3 this differs from Power loss only on the simulator page: neither has a reachable executor at dc1.

   • After the bound, erp-db fails over on dc1’s dead-man switches: they were armed, so its nodes are off whatever happened to its orchestrator. dc1 owes a fence. web-portal follows.
   • On Restore, qfence fences dc1 through its executor within seconds of its orchestrator and Docker API answering, often before its tasks start. A database that starts first asks for its lease, is refused, and starts as dc2’s standby. dc2’s shields keep it out of the VIPs meanwhile: no split brain.

6. Drift. While dc1 owns both groups, start dc2’s portal by hand (lab.sh docker dc2-mgr service scale web-portal=2).

   • The next check, within a minute, finds dc2 not fenced: an alert, and, since the lab sets on_drift: fence, a fence back to 0.
   • The same applies to setting dc2’s database to ROLE=primary by hand. Its promotion puts it on a newer timeline than dc1’s primary, so after the fence it will not re-seed from an older one: lab.sh reseed dc2 discards that copy, on purpose.

7. No quorum anywhere. Cut all three raft links on the simulator page (Orchestrator links). No site has quorum.

   • At 30 s the owner fences itself: its database takes its standby shape and its portal scales to zero. Nothing can fail over.
   • Restore the links within 4 min, before its watchdogs fire. The leader finds both groups’ owner fenced (plan.owner.fenced), and auto_recover promotes a site whose readiness passes, the owner first. The owner’s demoted database has nothing to follow, so the other site is promoted (plan.recover.automatic).
   • web-portal waits for its database, then follows it. The old owner’s database re-seeds from the new primary and becomes its standby, and nobody had to act.
   • Restore them later instead, and the watchdogs have powered the owner off. The recovery is the same; power the nodes on afterwards.

8. sso follows its database. Run lab.sh signins in one terminal, then lab.sh switchover erp-db dc2 in another.

   • erp-db moves gracefully, losing no committed row. dc1’s Keycloak is left on a database that is read-only: its sign-ins fail.
   • Once erp-db is ACTIVE at dc2, the leader fails sso over after it: it checks dc2’s readiness, fences dc1’s Keycloak (no replica) and starts dc2’s. lab.sh signins shows dc2 signing users in. web-portal follows the same way.
   • Measured on 2026-10-15: 64 s without sign-ins, then 59 s on the way back (lab.sh switchover erp-db dc1), against 4 min 43 s on 2026-10-14, when sso did not follow and failed over only once recovery’s three restarts had run out (ADR 0014 and ADR 0015, build notes).
   • Each follow counts toward sso’s circuit breaker and waits out its cooldown, 60 s here: move erp-db back within it, and sso stays behind until it ends, with an attention item saying so.

9. A site cut off with its Vault. Press Sites → dc1 → Fail (or run lab.sh site down dc1). dc1’s orchestrator stops, and its Vault with it.

   • dc2’s orchestrator never reads dc1’s Vault: its report keeps vault at dc2 answering, and fgt-edge’s token resolving there.
   • At the bound, about 4 min 20 s, erp-db fails over to dc2 on dc1’s dead-man switches, then web-portal follows, and dc2’s shields go up through fgt-edge, with the token from dc2’s Vault: the leader’s audit has the shields’ actions, and dc2’s dry run shows them resolving through vault at dc2.
   • Restore starts dc1’s Vault (seeded again) and orchestrator, which signs in and resolves the token again.
   • lab.sh vault down dc1 alone leaves dc1’s orchestrator running: it keeps the token it resolved (a KV value is held until the controller is opened again), and its dry run shows it held, unavailable (ADR 0011 §5).

Several automatic failovers in a row trip a group’s circuit breaker, which then blocks automatic failover until an Operator resets it with POST /api/v1/availability-plans/{id}/circuit-breaker/acknowledge. The group page shows the breaker OPEN but cannot reset it yet (ROADMAP Phase 8.9).

Files