Skip to content
Install
The orchestrator

The orchestrator

Each site runs one qfence-orchestrator. It is a raft voter, it serves the REST API and the console, and it calls the controllers its site reaches. Two sites and a witness installs a whole topology step by step; this page is about the orchestrator’s part.

Ways to install it

All three run the same install script and give the same result:

  • systemd/: install.sh and a systemd unit, by hand on any Linux host with systemd. The other two call it.
  • ansible/: a role around install.sh, for hosts you already manage with Ansible. Its variables, orchestrator_*, are documented in roles/qfence_orchestrator/defaults/main.yml and fill in the env file below.
  • terraform/proxmox/: makes the VM on Proxmox VE, then installs through cloud-init.

What is where

PathWhat it is
/opt/qfence/qfence-orchestratorThe binary
/opt/qfence/run-orchestrator.shTurns the env file into the binary’s flags
/etc/qfence-orchestrator/orchestrator.envThe settings, each explained in orchestrator.env.example
/etc/qfence-orchestrator/cluster.keysThe cluster keyring, the same on every orchestrator
/var/lib/qfence-orchestrator/Raft’s data and the node’s own state: the audit log, API tokens (as hashes), its secrets store and plan configuration made through the API
/var/lib/qfence-orchestrator/api-cert.pemThe API’s certificate, when the orchestrator generates one

The service runs as the user qfence, which cannot log in. systemctl reload qfence-orchestrator reads the configuration file again (file mode only; all or nothing, and audited).

Started without a configuration file, an orchestrator has no controllers and no plans. With QFENCE_DEMO=true it serves two demo plans on mock controllers instead, whose fences fence nothing.

Rotating the cluster keyring

The keyring (ADR 0016 §1) can be rotated with no restart. Each orchestrator reads its file again when it changes.

  1. Add a new key to every orchestrator’s file, after the old one: qfence-orchestrator cluster-key new -id c2 >> /etc/qfence-orchestrator/cluster.keys on one, and copy the line to the others.
  2. Move it first in every file. The first key is the one in use.
  3. Wait until GET /api/v1/cluster/voters shows every orchestrator’s report under the new key (report_key_id). Within a second, the leader seals the open fence debts again under it: fence_debts[].key_id on a plan’s detail shows which key each is under.
  4. Remove the old key from every file.

A fence debt whose detail the leader cannot open cannot be sealed again; its log says so. Keep the old key until that debt is cleared. If the old key is lost everywhere, the site still owes its fence and is fenced: the debt’s targets and reason are in the audit log of the orchestrator that recorded it (GET /api/v1/audit?action=plan.fence.debt.recorded&plan_id=<plan>), which never enters raft. A report or a debt’s detail sealed under a key an orchestrator does not hold is absent there (ADR 0016 §2, §3), and the voter list says why (report_unreadable).

Two sites and a witnessThe witness