The witness

With two sites, the witness is the third raft voter, in a third failure domain. It takes part in raft and nothing else: no controller configuration, no credentials, no plans. What it is for, what it can read, and what stays available when it or a site is down is in the witness and the availability arithmetic: read that before you place one. Two sites and a witness installs the whole topology step by step.

Ways to install it

All four run the same install script and give the same result:

• systemd/: the install script and its unit, run by hand on a Linux host with systemd. The other three call it.
• ansible/: an Ansible role that runs the same script, with the binary built as the install guide’s step 1 does. Its variables, witness_*, are in roles/qfence_witness/defaults/main.yml.
• terraform/aws/: makes the VM on AWS, with a security group that admits only your two sites and your admin range, then installs through cloud-init. Its guide says how to port it to another cloud.
• terraform/proxmox/: the same on Proxmox VE, beside the sites’ module.

What is where

The service runs as the user qfence-witness, which cannot log in. The witness never gets the cluster keyring.

A brand-new cluster is formed by its three voters’ first start, each with its bootstrap setting on (QFENCE_WITNESS_BOOTSTRAP=true here) and the same full peer list. A voter that already has raft state ignores the setting, so it may stay on.

Monitoring

The witness serves plain HTTP on QFENCE_WITNESS_STATUS_LISTEN (127.0.0.1:8081 by default; reach it through an SSH tunnel or a local monitoring agent):

• GET /healthz answers ok while the witness runs: point your monitoring at it.
• GET /status is JSON: this voter’s state (Follower, Candidate, Leader or Shutdown), the leader, the raft term and commit_index, and each voter in peers with its address, whether it leads, and how long since the leader last heard from it (last_contact_ms).
• GET / is the same as a page, with no JavaScript.

It is not TLS: it carries no configuration, credentials or plan data, only raft’s health. Never base a safety decision on it yourself; qfence bases them on quorum, fencing and readiness alone.

Each start prints a new single-use token, in the service’s journal, for its join wizard at /join and its configuration page at /config. The wizard prints the commands for adding this witness to sites that are already running (below); it contacts nothing itself.

Adding a witness to sites that are already running

A witness can join a cluster that already runs, such as a second witness for five voters, or one run for you by a provider. Start it with QFENCE_WITNESS_BOOTSTRAP=false: it waits to be added. Its raft certificate must come from the CA the sites’ certificates come from, since every voter must trust every other.

Then, on the sites’ raft leader, add it with its id and raft address: the console’s Cluster → Change membership, or the API, with api set up as in step 6 of the install guide. First the plan:

api -data '{"action":"add","id":"witness2","address":"10.0.0.6:7000"}' POST /api/v1/cluster/voters/plan

It lists its checks, and is not ok unless this orchestrator is the leader, the new voter is listening, and the cluster keeps a reachable majority afterwards. A fourth voter gets a warning: four tolerate no more failures than three, so add a fifth. Then carry out the plan, with the plan_id it returned:

api -data '{"action":"add","id":"witness2","address":"10.0.0.6:7000","plan_id":"<plan_id>"}' POST /api/v1/cluster/voters/execute

"action":"remove", with the voter’s id alone, takes a voter out the same way.

If the CA is someone else’s, as with a provider’s witness, run each site with -raft-tls-crl-url (QFENCE_RAFT_TLS_CRL_URL) pointing at its revocation list. The sites fetch the signed list, check it against the CA, publish it through raft, and refuse a revoked certificate from each other too, including on connections already open. GET /api/v1/cluster/crl on a site shows the list it enforces.