The systemd application controller

The systemd controller type manages an application that runs as a systemd unit on a Linux host. qfence’s orchestrators at the host’s site reach it over SSH. Nothing is installed on the host apart from a user, its key and its sudo rules. ADR 0005 §1 and §9 set out the design.

One controller is one host. Use it to:

• watch a unit’s health and restart it in place before failing over;
• stop the copy at a site qfence gives up, so it cannot start again until qfence starts it;
• check, with your own command, whether a copy can be promoted.

1. What it can do

unit.stop is an application-level fence. It never proves that a site has stopped: the plan still needs a compute or network fence for each site, such as the host’s BMC (power.off) or its VM (vm.stop). It defaults to required: false. A required one is a further condition for promotion. It usually cannot reach a site that is down, so it turns the loss of a site into BLOCKED, and the plan editor and the dry run say so.

The type has no switchover capability: a plain unit has nothing to promote. A failover starts the new owner’s copy by unfencing it.

2. What the host needs

• systemd as PID 1, and systemctl at the path the sudo rules name (/usr/bin/systemctl by default).
• An SSH server the site’s orchestrators can reach. They must be orchestrators at the controller’s site or in its reach.
• sudo, sh, flock (util-linux) and timeout (coreutils). All of them are present on every mainstream systemd distribution.
• A dedicated user for qfence, with a writable home directory.

3. Set up the host

The user and its key. Create a user that does nothing else, and give it a key qfence alone holds:

useradd --create-home --shell /bin/sh qfence
install -d -m 700 -o qfence -g qfence ~qfence/.ssh
ssh-keygen -t ed25519 -N '' -C qfence -f qfence-db1     # on your workstation
install -m 600 -o qfence -g qfence qfence-db1.pub ~qfence/.ssh/authorized_keys

The sudo rules. Name each unit qfence manages, each command it needs, and nothing else. unit.stop needs mask, stop, unmask and start; unit.restart needs restart. Write one line per unit in /etc/sudoers.d/qfence, then check the file with visudo -cf /etc/sudoers.d/qfence:

qfence ALL=(root) NOPASSWD: /usr/bin/systemctl mask orders.service, /usr/bin/systemctl stop orders.service, /usr/bin/systemctl unmask orders.service, /usr/bin/systemctl start orders.service, /usr/bin/systemctl restart orders.service

qfence sends each command with exactly these words, so the rules match them exactly. Rules that allow systemctl for any unit are refused. A binding’s validation asks sudo whether it would allow a unit nobody manages. If it would, the binding fails sudo.units: such rules make the qfence key a root key on the host.

Where the unit file lives. systemctl mask puts a link to /dev/null in /etc/systemd/system. It cannot do that for a unit whose own file is already there, so a binding of unit.stop fails unit.maskable for such a unit. Move the file to /usr/local/lib/systemd/system/, where systemd also reads units, and run systemctl daemon-reload. Drop-ins in /etc/systemd/system/<unit>.d/ are fine.

The host key. Read the fingerprint of the host’s key on the host itself, to compare with the one qfence shows at registration:

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

qfence asks for an Ed25519 key first, then ECDSA, then RSA, so a host with several keys always presents the same one.

4. Store the key on the orchestrators

The private key is a secret, so configuration only names it. Put it in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of the ssh_key method’s secret fields:

jq -n --rawfile k qfence-db1 '{private_key: $k}' |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref db1-ssh \
    -scope-type systemd -scope-host db1.example.com

-scope-type and -scope-host are the controller types and the addresses that may use the secret: the controller’s host, which allows any port of it, or db1.example.com:22, its SSH port, which allows that port only (ADR 0016 §6). Any other controller that names it is refused it (ADR 0011 §8).

Run it against each of those orchestrators while it runs (writing a secret): the controllers that use the key pick it up at once.

Add passphrase to the object if the key has one. The user name is not secret. It goes in the controller’s auth.config.

5. Register the controller

In the console: go to Controllers, then Application, then Register. Choose systemd (over SSH), enter the host and pick the secret reference. The first connectivity test is refused, and shows the key the host presents. Compare it with the fingerprint you read on the host, then choose Pin this host key and test again. The pinned key becomes the controller’s identity, host-key:SHA256:…. If two controllers pin the same key, they reach the same host.

In a configuration file (ADR 0001):

controllers:
  application:
    - id: db1
      type: systemd
      site: dc1
      connection:
        host: db1.dc1.example.com
        host_key_sha256: "SHA256:vzulm2xA/wt9+aD8Zip1AtW4o8v9izQy795XSI2HDqw"
      auth: {method: ssh_key, credentials_ref: "local:db1-ssh", config: {user: qfence}}
      capabilities: [unit.active, unit.stop, unit.restart, command.check, units.list]

Validation. For the controller, sudo.units passes as inferred when sudo answers the user without a password, and unit.maskable is inferred. Both are proven for each unit when a binding names it.

6. Bind it to a plan

Each site’s copy of the application usually runs on its own host, so each has its own controller (app1 at dc1, app2 at dc2). This plan fences each site through its host’s BMC (power.off, the fence that proves the site) and stops the site’s copy as an extra fence. It watches the owner’s copy twice over: its unit, and a probe of the service itself. It restarts the copy twice before failing over:

availability_plans:
  - id: orders
    sites: [dc1, dc2]
    fencing: {unfence_on_promote: true, fence_timeout_seconds: 60, on_drift: fence}
    instances:
      - id: orders-dc1
        site: dc1
        resource: {controller: app1, resource: orders.service}
        bindings:
          - {id: orders-stop-dc1, controller: app1, capability: unit.stop, resource: orders.service, role: fence}
          - {id: bmc-app1, controller: bmc-dc1, capability: power.off, resource: app1, role: fence, required: true}
          - {id: orders-unit-dc1, controller: app1, capability: unit.active, resource: orders.service, role: health}
          - {id: orders-probe-dc1, controller: app1, capability: command.check, role: health,
             params: {command: [/usr/local/bin/orders-probe], timeout: 5s}}
          - {id: orders-restart-dc1, controller: app1, capability: unit.restart, resource: orders.service, role: recovery,
             params: {max_attempts: 2, window: 10m}}
      # orders-dc2: the same, with app2 and bmc-dc2.
    failover:
      automatic_failover: true
      application_health: {unhealthy_for_seconds: 60, fail_over: true}

With unfence_on_promote, a failover to dc2 powers dc2’s host on, then unmasks and starts dc2’s copy. The unfence waits, within the fence timeout, for the host to answer over SSH, so fence_timeout_seconds must cover the host’s boot.

Do not enable the unit. qfence starts it where the plan is active (the unfence) and restarts it there (recovery). A unit that is not enabled stays down when a fenced host is powered on again, whatever happened to its mask.

Where the systemd fence runs. A site’s secondary application fences run before its required ones, for at most fencing.secondary_application_timeout_seconds (default 10 s; ADR 0009 §2). A secondary unit.stop therefore masks and stops the unit while its host is still up, and the host’s power fence follows. When the host cannot be reached, as when its site is lost, the unit.stop is cut short at that limit and reads UNKNOWN, and the required fences prove the site without it; a failover then takes up to that long more. A required unit.stop runs in the configured order, but a site whose host cannot be reached then cannot be fenced at all: the failover blocks (ADR 0005 §7). Keep it secondary unless the host stays up when its site is fenced.

Readiness on a standby that is off. A standby site stays fenced while it waits. When its compute fence powers its host off, nothing on that host can answer a readiness check, so a command.check readiness binding there would block every promotion. Check readiness from somewhere that answers while the host is off, or bind none for a service that keeps no data.

command.check. command is the program and its arguments, as a list. qfence sends each word single-quoted, so the login shell passes it unchanged: nothing is expanded, split, piped or redirected. If you need those, put them in a script on the host and name the script. The command runs as the qfence user, under timeout(1). When it overruns timeout (default 10s, at most 10m), it and the processes it started are killed. How each exit is read:

It is for health and readiness only. To fence or recover with your own code, write an external controller (Writing an application controller).

Try it in the demo lab. deploy/demo-lab/lab.sh up runs two systemd hosts in containers, app-dc1 and app-dc2, whose power the simulated BMCs switch, and a plan, billing, that binds every capability above; lab.sh app upsets billing behind qfence’s back (deploy/demo-lab/README.md).

7. Stale commands

Every fence, unfence and restart carries the plan’s generation (§40). On the host, qfence runs it as one short script that holds a lock on a record for the plan and unit, <state_dir>/<plan>/<unit>.generation. The script refuses a generation older than the record. Otherwise it brings the record up to date before it acts. A command a deposed leader sent late therefore cannot undo what a newer leader did, whichever of the site’s orchestrators sent either command. The script’s arguments are passed as positional parameters, never written into its text.

Each plan’s generations are its own. If you delete a plan and create another with the same id, its generations start again from 1, and the host refuses them as stale (§8).

8. Troubleshooting

9. Limits

• An application fence is not a compute fence. A unit that reads masked and inactive may still have processes that systemd does not track: something started outside the unit, or a unit with KillMode=none. The plan’s compute fence is what makes a failover safe.
• The generation records live on the host. Anyone with root on the host can change them, as they can change anything qfence manages there.
• Unit names with escapes (\x2d) are not accepted: the sudo rules would have to spell them out. Name the unit without one.
• Tested against systemd 252 (Debian 12) in a container, with OpenSSH 9.2 and sudo 1.9 (QFENCE_SYSTEMD_IT=1 go test ./internal/controllers/application/systemd/).