Skip to content
Install
Two sites and a witness

Two sites and a witness

This guide brings up the topology most deployments use: two sites, each running qfence-orchestrator, and a witness in a third failure domain. The three are raft voters, so one site can be lost and the other still has a majority to fail over with. Why the third voter is needed, and what it does and does not buy you, is in the witness and the availability arithmetic. If you already have a third independent voter, use it in the witness’s place.

    flowchart LR
  o1["dc1, 10.0.0.1<br/>qfence-orchestrator"]
  w["witness, 10.0.0.5<br/>qfence-witness"]
  o2["dc2, 10.0.0.2<br/>qfence-orchestrator"]
  o1 <-->|raft, TCP 7000| w
  w <-->|raft, TCP 7000| o2
  o1 <-->|raft, TCP 7000| o2

Operators reach each site’s API and console at TCP 8443.

The steps below install each voter by hand with systemd. The Ansible roles and the Terraform modules for Proxmox and AWS run the same install script with the same settings. The keyring and the raft certificates are yours to copy whichever you use: none of them handles keys.

To see a working topology first, without any of this, run the demo lab on one machine.

What you need

  • Three Linux hosts with systemd, one per failure domain. The examples call them dc1 (10.0.0.1), dc2 (10.0.0.2) and witness (10.0.0.5).
  • Between all three: TCP 7000, for raft.
  • To each site: TCP 8443, for the API and the console, from your operators and from the other site.
  • A workstation with a clone of this repository, Go 1.26 and Node.js 24, to build the binaries. There are no published releases yet.

1. Build the binaries

On the workstation, in the clone:

(cd web/console && npm ci && npm run build)
CGO_ENABLED=0 go build -o qfence-orchestrator ./cmd/orchestrator
CGO_ENABLED=0 go build -o qfence-witness ./cmd/witness

The first line builds the console, which the orchestrator serves from inside its binary. Without it, the console’s address shows a page saying it was not built. CGO_ENABLED=0 makes static binaries, which run on a host whatever its C library. Build on the hosts’ architecture, or set GOARCH (arm64, say).

2. Make the cluster keyring and the raft certificates

On the workstation, or better on a machine that stays offline:

./qfence-orchestrator cluster-key new -id c1 > cluster.keys
chmod 600 cluster.keys
./qfence-orchestrator pki init -dir qfence-ca
./qfence-orchestrator pki issue -dir qfence-ca -name dc1 -hosts 10.0.0.1
./qfence-orchestrator pki issue -dir qfence-ca -name dc2 -hosts 10.0.0.2
./qfence-orchestrator pki issue -dir qfence-ca -name witness -hosts 10.0.0.5
  • The cluster keyring goes to both orchestrators and never to the witness (ADR 0016 §1). Keys derived from it sign every call between the orchestrators, and seal what they keep in raft that the witness must not read. Every orchestrator in a cluster refuses to start without it. Copy it out of band, or over SSH with a post-quantum key exchange (OpenSSH 9.9 or later), and back it up apart from the hosts.
  • The certificates make raft run over mutual TLS: each voter accepts only peers with a certificate from this CA. They are valid for 90 days. Keep qfence-ca/ca-key.pem offline; you need it to issue the next ones.

3. Install each site

Copy to the dc1 host: qfence-orchestrator, the directory deploy/orchestrator/systemd/ from the clone, cluster.keys, and ca.pem, dc1.pem and dc1-key.pem from qfence-ca/. Then, as root, from the directory you copied them to:

sh systemd/install.sh ./qfence-orchestrator
install -m 0600 -o qfence -g qfence cluster.keys /etc/qfence-orchestrator/cluster.keys
install -m 0644 ca.pem /etc/qfence-orchestrator/raft-ca.pem
install -m 0644 dc1.pem /etc/qfence-orchestrator/raft.pem
install -m 0400 -o qfence -g qfence dc1-key.pem /etc/qfence-orchestrator/raft-key.pem

install.sh creates the user qfence, installs the binary in /opt/qfence/ and a systemd unit, and writes /etc/qfence-orchestrator/orchestrator.env. It enables the service but does not start it. The env file explains every setting. Set these:

QFENCE_NODE_ID=dc1
QFENCE_SITE=dc1
QFENCE_RAFT_BIND=10.0.0.1:7000
QFENCE_RAFT_PEERS=dc1=10.0.0.1:7000,dc2=10.0.0.2:7000,witness=10.0.0.5:7000
QFENCE_RAFT_BOOTSTRAP=true
QFENCE_RAFT_TLS_CERT=/etc/qfence-orchestrator/raft.pem
QFENCE_RAFT_TLS_KEY=/etc/qfence-orchestrator/raft-key.pem
QFENCE_RAFT_TLS_CA=/etc/qfence-orchestrator/raft-ca.pem
  • QFENCE_RAFT_BIND is an address the other voters reach this one at, never 0.0.0.0.
  • QFENCE_RAFT_PEERS is the same on all three voters: a mismatch leaves a cluster that never elects a leader, or two clusters.
  • QFENCE_RAFT_BOOTSTRAP=true on all three forms the cluster on their first start. A voter that already has raft state ignores it, so it can stay set.

To try the topology before you configure any real controller, also set QFENCE_DEMO=true. The orchestrators then serve two demo plans on mock controllers, whose fences fence nothing. Never leave it on a cluster that protects anything.

Install dc2 the same way, with its own certificate and QFENCE_NODE_ID=dc2, QFENCE_SITE=dc2 and QFENCE_RAFT_BIND=10.0.0.2:7000.

4. Install the witness

Copy to the witness host: qfence-witness, the directory deploy/witness/systemd/, and ca.pem, witness.pem and witness-key.pem. Not the cluster keyring. Then, as root:

sh systemd/install.sh ./qfence-witness
install -m 0644 ca.pem /etc/qfence-witness/raft-ca.pem
install -m 0644 witness.pem /etc/qfence-witness/raft.pem
install -m 0400 -o qfence-witness -g qfence-witness witness-key.pem /etc/qfence-witness/raft-key.pem

In /etc/qfence-witness/witness.env, set:

QFENCE_WITNESS_ID=witness
QFENCE_WITNESS_BIND=10.0.0.5:7000
QFENCE_WITNESS_PEERS=dc1=10.0.0.1:7000,dc2=10.0.0.2:7000,witness=10.0.0.5:7000
QFENCE_WITNESS_BOOTSTRAP=true
QFENCE_WITNESS_TLS_CERT=/etc/qfence-witness/raft.pem
QFENCE_WITNESS_TLS_KEY=/etc/qfence-witness/raft-key.pem
QFENCE_WITNESS_TLS_CA=/etc/qfence-witness/raft-ca.pem

5. Start the three voters

In any order:

systemctl start qfence-orchestrator   # on dc1 and dc2
systemctl start qfence-witness        # on the witness

A voter waits for the others, so the order does not matter. Within a few seconds of the last one starting, they elect a leader.

6. Check the cluster

On the witness, its status names the leader and lists the three voters:

curl -s http://127.0.0.1:8081/status

Each orchestrator printed an administrator token once, on its first start. It is in the service’s journal; save it in a file only you can read:

journalctl -u qfence-orchestrator | grep -A1 'bootstrap administrator token' | tail -1 | awk '{print $NF}' > admin.token
chmod 600 admin.token

The API takes only TLS 1.3 with the post-quantum key exchange X25519MLKEM768, which curl has only when built against OpenSSL 3.5 or later. The orchestrator’s own client, qfence-orchestrator api, always works. Without a certificate of its own (QFENCE_TLS_CERT), the orchestrator makes one at each start and writes it to /var/lib/qfence-orchestrator/api-cert.pem, which the client verifies the API against. On dc1:

api() { /opt/qfence/qfence-orchestrator api -server https://127.0.0.1:8443 -token-file admin.token -ca-file /var/lib/qfence-orchestrator/api-cert.pem "$@"; }
api GET /api/v1/cluster

It answers with "quorum_available":true and the three voters as peers.

7. Fail over a demo plan

With QFENCE_DEMO=true, on the leader the witness’s status named, with that orchestrator’s own token and api set up as above:

api -data '{"target_site":"dc1"}' POST "/api/v1/availability-plans/production-app-01/failover?dry_run=true"
api -data '{"target_site":"dc1","confirmed":true}' POST /api/v1/availability-plans/production-app-01/failover
api GET /api/v1/availability-plans

The dry run lists each check and what would be fenced. The failover makes dc1 the owner of production-app-01 at generation 1. The other orchestrator refuses a failover: only the leader changes ownership. The console, at https://10.0.0.1:8443/ on either site, takes the same token.

Then take QFENCE_DEMO out of both env files, restart both orchestrators, and configure your own controllers and plans (see Configure).

Before it protects anything

  • A certificate for the API. Set QFENCE_TLS_CERT and QFENCE_TLS_KEY on each orchestrator.
  • Sign-in. People sign in through your identity provider: QFENCE_OIDC_ISSUER and the settings beside it. The administrator tokens stay as break-glass.
  • Credentials. Controllers name their credentials by reference. Each orchestrator keeps its own encrypted store (QFENCE_MASTER_KEY_FILE), and can share one through raft (QFENCE_SECRETS_KEY_FILE), or read its site’s Vault (QFENCE_SECRETS_CONFIG). The env file shows each.
  • Automatic failover is off until you set QFENCE_AUTOMATIC_FAILOVER=true on both orchestrators. Until then, nothing fails over without an operator.
  • Certificates expire. The raft certificates last 90 days: issue the next ones and restart each voter, one at a time, before then.

Shipping telemetry (optional)

An orchestrator can ship redacted telemetry, its audit records and events, one way to a hosted control and observability plane (§58). Give each site its own -hostedplane-endpoint (the plane’s .../v1/ingest URL), -hostedplane-token (that site’s ingest token) and -deployment-id, and -hostedplane-ca if the plane’s certificate comes from a private CA, through QFENCE_EXTRA_FLAGS in the env file. The token is then on the orchestrator’s command line, which other users of the host can read: keep the host to qfence alone. The witness ships nothing. What leaves is pkg/telemetry.Record, and cutting the plane off changes nothing local: Critical Acceptance Test 9, in tests/e2e/acceptance_test.go.

What this proves, and what it does not

This topology runs the real raft consensus, transport and API across separate machines: the mechanism qfence exists to get right. With QFENCE_DEMO=true its fences fence nothing. Fencing real hardware needs real controllers in the configuration file, pointed at real BMCs, hypervisors and firewalls; the controller types for those are tested against doubles of their vendors’ APIs, not yet against live equipment (see Known gaps in SAFETY.md).

The orchestrator