The demo lab

One command brings up a complete, connected qfence environment on one machine, managing simulated infrastructure through qfence’s real controllers.

deploy/demo-lab/lab.sh up --build     # first time: build everything, then start
deploy/demo-lab/lab.sh up             # later: start again, reusing binaries and state
deploy/demo-lab/lab.sh status         # what is running, and which voter leads
deploy/demo-lab/lab.sh logs dc1       # follow one component's log
deploy/demo-lab/lab.sh app status     # billing on the two systemd hosts
deploy/demo-lab/lab.sh down           # stop everything (state is kept)
deploy/demo-lab/lab.sh reset          # stop and delete all lab state

up options: --build rebuilds the web console, every binary and the systemd host image first (otherwise they are built only if missing), --fresh starts from new state (new CAs, keys and cluster), --no-keycloak runs without Keycloak (the orchestrators then accept only their local tokens), and --no-systemd creates the lab without its two systemd hosts and the billing plan. Whether a lab has the systemd hosts is decided when it is created; --fresh changes it. A lab created before the cluster keyring (Phase 11.2b, .lab/secrets/cluster.keys) needs --fresh.

Needs: Go, curl, Docker for Keycloak and for the systemd hosts (privileged containers, on the local Docker daemon), and npm for --build. A first up --build takes about two minutes. Lab use only: every credential is generated for the lab and printed, and Keycloak’s admin is admin/admin.

What runs

How they are connected:

• Raft. dc1, dc2 and the witness form one raft cluster over mutual TLS. The certificates come from a lab CA made with qfence-orchestrator pki, and every voter enforces the CA’s signed revocation list (.lab/pki/raft/crl.pem).

• Sign-in. People sign in to the orchestrators through Keycloak (OIDC). Keycloak groups map to roles:

  User	Password	Role
  alice	alice-password	Administrator
  bob	bob-password	Operator
  carol	carol-password	Viewer

  Each orchestrator also has a break-glass token, printed by up.

• Telemetry. Off unless asked for. Run up with LAB_INGEST_ENV naming a file that sets INGEST_URL (an ingest endpoint, ending /v1/ingest), INGEST_TOKEN_dc1 and INGEST_TOKEN_dc2, and for a private CA INGEST_CA (its PEM file). up keeps a copy in .lab/ingest.env, and each orchestrator then ships its audit records and events there with -hostedplane-endpoint, as deployment dc1 or dc2. What it sends is pkg/telemetry.Record.

• Configuration. Both orchestrators load the same configuration file (.lab/qfence-lab.yaml, copied from qfence-lab.yaml). They report its fingerprint for drift detection over a shared cluster secret, and hold a key for replicated configuration mode.

• Controllers. The configuration defines real Redfish, Proxmox and FortiGate controllers pointed at the simulator, and the availability plans bind their capabilities per site (ADR 0010). Their credentials are in each orchestrator’s encrypted secrets store, referenced by credentials_ref: local:NAME. Each site’s BMCs can be reached from both sites, so the surviving site fences a lost one.

• systemd hosts. app-dc1 and app-dc2 are real hosts in containers, managed by the systemd application controller (its guide): qfence logs in over SSH as the user qfence, with a key up made and stored as local:lab-ssh. Each host has its own host key, whose fingerprint the configuration pins, and sudo rules that name billing.service and nothing else. Their power is the simulator’s: their BMCs (app-dc1, app-dc2 under Hosts) start and stop the containers.

The simulated estate

dc1 (primary)                        dc2 (disaster recovery)
pve-dc1-01  VM 101 erp-db            pve-dc2-01  VM 101 replica
pve-dc1-02  VM 102 web-portal        pve-dc2-02  VM 102 replica
pve-dc1-03  orders application       pve-dc2-03  orders application
app-dc1     billing (systemd)        app-dc2     billing (systemd)
Replication 101-0/102-0 copies dc1 -> dc2 while the VMs run at dc1,
            101-1/102-1 copies dc2 -> dc1 while they run at dc2.
FortiGate VIPs vip-erp-db and vip-web-portal point clients at the active site.

The simulator follows cause and effect where a failover depends on it: a host that loses power stops its VMs; a VM cannot start on a host that is off; a replica syncs only while its source VM runs and both hosts are on, so it goes stale five minutes after its source site is lost.

Four availability plans, all active at dc1 when the lab starts (up runs a real failover to dc1 for each, which stops the dc2 replicas and enables the VIPs):

• erp-db:

  • Fencing is scoped to sites (ADR 0002): a failover fences only the site it gives up. dc1 is fenced out of band through its hosts’ BMCs (Redfish ForceOff, required) with a Proxmox VM stop as secondary; dc2 is fenced by stopping its replica VM.
  • Promotion unfences the promoted site (unfence_on_promote): it powers the host on and starts the VM there.
  • Readiness: the replica at the site being promoted must be healthy and less than 5 minutes old.
  • Service: vip-erp-db on the FortiGate.
  • Automatic failover after dc1 has had no reachable voter for 20 s. The lab starts the orchestrators with -automatic-failover; it is off by default.

• web-portal: the same shape, and it may only move to a site once erp-db is active at that same site (same_site).

• orders-db: an application qfence watches and controls itself (ADR 0005), through the mock application controller (orders-app-dc1, orders-app-dc2): an application in each orchestrator’s memory, called through the external controller protocol as your own controller would be. It is known only in demo mode, so the lab’s orchestrators run with -demo.

  • Fencing: each site’s host through its BMC (required), and the application’s own demotion (an application-level fence: secondary, and never what proves a site stopped).
  • Readiness and health come from the application: its lag, and whether it runs.
  • Switchover: each site has a switchover binding, graceful. A switchover from dc1 to dc2 demotes dc1’s application and verifies it, waits for dc2 to report zero lag, powers off pve-dc1-03, powers on pve-dc2-03 and promotes dc2’s application. A failover promotes through the same binding, without the demotion and the wait.

  On the plan’s page, Switchover (beside Failover) shows the dry run’s plan: the strategy, the demotion, the wait, then the fences as for a failover. The controllers’ Resources tab shows each copy of the application as primary, standby or demoted.

• billing: a systemd unit on app-dc1 and app-dc2, through the systemd application controller (app-dc1, app-dc2). The unit is not enabled: only qfence starts it, where billing is active.

  • Fencing: each site’s host through its BMC (required), which stops the container, and unit.stop (mask, then stop: secondary). A site’s secondary application fences run first, for at most 10 s (ADR 0009 §2), so a failover masks and stops billing, then powers its host off. When the host cannot be reached, as when its site is lost, the systemd fence reads UNKNOWN and the BMC proves the site on its own.
  • Health: the unit is active (unit.active), and its heartbeat is at most 10 s old (command.check running billing-check on the host), checked every 5 s. Unhealthy for 10 s, billing is restarted (unit.restart, at most twice in 10 minutes), then failed over (application_health, fail_over: true).
  • Promotion powers the new owner’s host on, waits for it to answer over SSH, then unmasks and starts billing there. Billing keeps no data, so no replica is checked.
  • Continuous checks every 15 s: a site that does not own billing and is found unfenced (a host powered on by hand, say) is fenced again (on_drift: fence).

The lab simulator page

http://127.0.0.1:8600 shows, refreshing every 3 seconds:

• qfence: both orchestrators (up, raft leader), each availability plan’s state, owner and generation as the leader sees them, and the scheduler’s latest automatic-failover decisions.
• Sites: Fail stops the site’s orchestrator (through lab.sh site down) and cuts its hosts’ power while their BMCs still answer; Blackout does the same and makes the BMCs unreachable too; Restore brings the BMCs back, powers the hosts on (their VMs stay stopped) and starts the orchestrator.
• Hosts, VMs, replication, FortiGate: power, BMC reachability, VM state, replication overrides (Stall, Fail) and FortiGate reachability, one device at a time. app-dc1 and app-dc2 are marked container: their power is their containers'.
• Recent actions: every call qfence’s controllers made, and every operator change.

A walk-through

1. lab.sh up --build --fresh, then open the simulator page and, in another window, the console of the raft leader (lab.sh status), signed in with SSO as alice. Both groups are ACTIVE at dc1.
2. Automatic failover. On the simulator, press Fail for dc1. dc2 and the witness keep quorum and dc2 becomes leader. About 20 s later qfence fails erp-db over to dc2 on its own: the action log shows dc1’s hosts forced off through Redfish, VM 101 started at dc2 and vip-erp-db moved; web-portal follows as soon as erp-db is active at dc2.
3. Moving back. Press Restore for dc1. Its BMCs and orchestrator come back, but its hosts stay off: the failover’s fence set their power-restore policy to AlwaysOff, so they do not start again when power returns (ADR 0009 §3). Power them on under Hosts: their VMs stay stopped, and replication into dc1 (101-1) resumes. The checks now report dc1 not fenced, an alert only, since a host that is on is what dc1’s fence rules out. In the console, fail erp-db, then web-portal, over to dc1: the dry run shows what will be fenced (the dc2 VM) and unfenced (dc1’s host, its policy set back to LastState, and its VM).
4. A failover that must block, and an attestation. Press Blackout for dc1. This time the BMCs do not answer: nothing can prove dc1’s hosts are off, so the automatic failover stops at BLOCKED and nothing starts at dc2; the error names the BMC that did not answer. web-portal stays where it was, since it may only follow erp-db. Suppose you have confirmed by other means that dc1 is dark (ADR 0003). Sign in as alice (Administrator) and, on erp-db’s page, Force release, then Failover to dc2. The wizard lists bmc-dc1 as a target that cannot be fenced. Once it has been failing for 60 s, tick it and write what you checked. The attested failover still tries the BMC, raises dc2’s shield (the FortiGate policy that lets the VIP reach dc1; an attested failover is refused without one, ADR 0009 §4), then starts erp-db at dc2 and moves its VIP; it is audited as a forced operation with your statement, and the fencing overview marks the target as attested. dc1 now owes a fence: when it comes back, qfence fences it, and the shield comes down two checks after that. web-portal follows by the same route after its own failed attempt. If the blackout lasts more than 5 minutes, the dc2 replica is also older than the readiness policy: the wizard then offers to accept the data loss as well (ADR 0006). Type the most loss you accept and why; the failover promotes the replica only if it is still within that bound, and records the lag measured at promotion. Set approvals: 2 under fencing.attestation to see the request wait for a second person (bob cannot approve: it needs the EmergencyOperator or Administrator role). Without attesting, the way back is to Restore dc1, start VM 101 on pve-dc1-01 in the simulator (the database restarted by hand), then force-release erp-db and fail it over to dc1. After an attested failover dc1 owes a fence: when its BMCs answer again, qfence powers its hosts off and sets them to stay off.
5. Readiness. Set a replication job to Stall or Fail and dry-run a failover to that job’s site: refused.
6. An application on systemd hosts. Open billing’s page.
   • lab.sh app hang dc1 stops billing’s process: its unit stays active, but its heartbeat stops. About 20 s later the heartbeat check turns UNHEALTHY and qfence restarts billing; the Activity tab shows the episode.
   • lab.sh app break dc1, then lab.sh app crash dc1: billing dies and cannot start again. qfence’s restart fails, the restarts are used up, and qfence fails billing over to dc2 by itself, within about a minute. It masks and stops billing at dc1, powers app-dc1 off, powers app-dc2 on, waits for it to answer and unmasks and starts billing there. The fencing overview shows the systemd fence as “runs first”, and FENCED. lab.sh app status shows both hosts.
   • Power app-dc1 on under Hosts: billing stays down there (masked), and within 15 s qfence finds dc1 unfenced and powers it off again.
   • lab.sh app fix dc1, then fail billing back to dc1 in the console.
   • lab.sh logs app-dc2 follows billing’s and sshd’s journal on a host.

Other things to try:

• Switch configuration source. Settings → Configuration switches the cluster to file or replicated mode.

• Reload the configuration file. Switch to file mode first; reloading is refused in the other modes. Then edit .lab/qfence-lab.yaml and reload it on each orchestrator, with kill -HUP or the console’s reload button.

• Register a systemd controller. On the leader’s console, Controllers → Application → Register: type systemd, the site whose host runs billing now, host 127.0.0.1, port 18022 for app-dc1 or 28022 for app-dc2, user qfence, secret reference local:lab-ssh. The first test shows the host’s key; compare it with docker exec qfence-lab-app-dc1 ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub, then pin it. The wizard then warns that the lab’s own controller for that host (app-dc1 or app-dc2) already reaches it. The lab’s configuration source is local, so Register adds the controller to that orchestrator only. Delete it afterwards from its page (Delete…): nothing binds it, so it goes, and its SSH connection with it. The lab’s own controllers cannot be deleted: billing binds them and the configuration file defines them, and the dialog says so.

• Revoke a certificate. Revoke dc2’s raft certificate with the lab CA, then publish the signed list on the leader, with api set up as in Get started:

  deploy/demo-lab/.lab/bin/qfence-orchestrator pki revoke -dir deploy/demo-lab/.lab/pki/raft -name dc2
  jq -Rs '{crl_pem: .}' deploy/demo-lab/.lab/pki/raft/crl.pem | api -data-file - POST /api/v1/cluster/crl

  Every voter then refuses dc2’s certificate, including on connections already open, and dc2 drops out of raft. api GET /api/v1/cluster/crl shows the list each orchestrator enforces. up --fresh makes new certificates.

To start the story again, run lab.sh up --fresh. The simulator also forgets its state whenever it restarts. lab.sh site down|up dc1|dc2 stops or starts one orchestrator from the command line.

Limits

• The automatic-failover signal is narrow. A site counts as down when none of its raft voters answers.
• Recovering from BLOCKED is manual, by design (§11): force-release, then fail over, once someone has checked the old site; attest its fence targets if they still cannot be fenced (ADR 0003), and accept the data loss if the replica is older than its 5-minute limit (ADR 0006).
• qfence does not watch erp-db’s or web-portal’s application. Those groups stay ACTIVE at a site whose VM stopped; only a lost site (no reachable voter) triggers anything automatically. orders-db’s application is watched, but the plan does not fail over on it (no application_health); the swarm lab’s orders-api shows that.
• The systemd fence cannot act on a lost site. When billing’s site is lost, its host does not answer, so the systemd fence reads UNKNOWN and the BMC proves the site on its own. It is secondary for that reason: a required one would block the failover, as ADR 0005 §7 says. In the lab the call fails at once, since the lost site’s orchestrator is down; a real outage that drops packets makes the failover wait up to the 10 s the secondary application fences may take.
• A powered-off host’s controller is failed. The standby’s host is off, so its systemd controller fails its connectivity tests until the host is powered on again, and reads ready within 30 s of it.
• The orders application is a stand-in. Its copies live in each orchestrator’s memory, so restarting an orchestrator puts its copy back as it started (running, standby).
• Simulated hardware. The simulator follows the vendors’ public API references, like the controller types’ own test doubles. It has not been checked against real equipment, and its replication model is deliberately simple.
• Local only. Everything runs on 127.0.0.1, and the HTTPS certificates come from a lab CA (.lab/pki/https/ca.pem). Accept the browser warning, or trust that CA.