Skip to content
Get started

Get started

qfence runs each of your applications at one site at a time, and moves it to another site when the first is lost, without ever letting two sites run it at once. It decides through a raft quorum across your sites and a third voter, the witness, and it promotes a new site only after it has proven the old one stopped (concepts).

Try it on one machine

The demo lab runs a whole deployment on one Linux machine: two sites’ orchestrators and a witness, the three voters of one raft cluster, managing a simulated estate of Redfish BMCs, Proxmox VE hosts and a FortiGate through qfence’s real controllers, with Keycloak for signing in.

You need Git, Go 1.26, Node.js 24 with npm, curl, and Docker on the local daemon, for Keycloak and two systemd hosts in privileged containers. Then:

git clone https://github.com/ivnn-dev/qfence
cd qfence
deploy/demo-lab/lab.sh up --build

The first up --build builds the console and the binaries, then starts everything and makes each plan active at dc1, in about two minutes. It ends with where everything is:

qfence demo lab is up.

  Orchestrator dc1   https://127.0.0.1:18443
  Orchestrator dc2   https://127.0.0.1:28443
  Witness status     http://127.0.0.1:38081
  Lab simulator      http://127.0.0.1:8600    (watch the hosts, VMs and VIPs; inject failures)
  Keycloak           http://127.0.0.1:8180/admin    (admin / admin)

and which voter leads. lab.sh status shows the processes and the three voters:

deploy/demo-lab/lab.sh status
  voter dc1  leader=true  reachable=true
  voter dc2  leader=false  reachable=true
  voter witness  leader=false  reachable=true

Fail over

Open the leader’s console, accept the lab CA’s certificate warning, and sign in with SSO as alice, password alice-password. On erp-db, choose Failover, then dc2. The dry run shows what will be fenced and checked. Acknowledge the fence warning and execute it. On the simulator page, dc1’s host powers off, VM 101 starts at dc2, and the FortiGate VIP moves.

Or through the API, from the clone:

L=deploy/demo-lab/.lab
api() { $L/bin/qfence-orchestrator api -server https://127.0.0.1:18443 -token-file $L/secrets/dc1-admin.token -ca-file $L/pki/https/ca.pem "$@"; }
api -data '{"target_site":"dc2"}' POST "/api/v1/availability-plans/erp-db/failover?dry_run=true"
api -data '{"target_site":"dc2","confirmed":true}' POST /api/v1/availability-plans/erp-db/failover

This api points at dc1. When dc2 leads, use port 28443 and dc2-admin.token: only the leader changes ownership. The Operate pages use the same api.

Lose a site

On the simulator page, press Fail for the site that runs erp-db. Its orchestrator stops and its hosts lose power. The other site and the witness keep a majority, and about 20 seconds later qfence fails erp-db over by itself: it powers the lost site’s hosts off through their BMCs, starts the VM at the survivor and moves the VIP. Then try Blackout, where the BMCs do not answer either, so nothing can prove the site is off and qfence stops at BLOCKED.

The demo lab’s page walks through these and more: moving a plan back, attesting a fence, an application on systemd hosts.

Then

  • Concepts: what qfence does, and why.
  • Install: two sites and a witness on your own hosts.
  • Safety: what the tests prove, and the known limitations.

deploy/demo-lab/lab.sh down stops the lab and keeps its state; lab.sh reset deletes it.