Controllers and bindings
qfence acts on nothing directly. A controller is one system it acts through: a BMC, a Proxmox VE cluster, a firewall, a Docker Engine, a database server. An availability plan says what to do with them, site by site, in bindings (ADR 0010).
A plan’s parts
A plan protects one resource that can run at any of its sites. Each site has an instance: that site’s copy of the resource, and the bindings that fence it, check it and promote it there. A binding names a controller, one of its capabilities, the resource the capability acts on, and the role it plays. The plan’s service bindings, such as a virtual IP, are not a site’s: they follow whichever site owns the plan.
flowchart TB
plan["Plan<br/>erp-db"]
plan --> i1["Instance at dc1<br/>VM 101"]
plan --> i2["Instance at dc2<br/>VM 101"]
plan --> svc["Service<br/>vip.own"]
i1 --> b1["power.off<br/>fence, required"]
i1 --> b2["vm.stop<br/>fence"]
i1 --> b3["replication.lag<br/>readiness"]
i2 --> b4["the same<br/>at dc2"]
b1 --> c1["Controller<br/>bmc-dc1"]
b2 --> c2["Controller<br/>pve-dc1"]
b3 --> c2
svc --> c3["Controller<br/>fgt-edge"]
c1 --> s1[("Secret<br/>lab-redfish")]
c2 --> s2[("Secret<br/>lab-proxmox")]
c3 --> s3[("Secret<br/>lab-fortigate")]
That is the demo lab’s ERP database. At dc1, its bindings power off the host pve-dc1-01 through the BMC controller bmc-dc1 (Redfish), stop VM 101 through the Proxmox VE controller pve-dc1, and read its replication job 101-1 there; the service binding enables the virtual IP vip-erp-db on the FortiGate controller fgt-edge. A controller is configured once and bound by any number of plans. Its credentials are never in configuration: it names a secret, which each orchestrator that calls it resolves from its own store (secrets).
Roles
| Role | What the binding does | When it runs |
|---|---|---|
fence | Stops the resource at a site the plan is given up by: powers a host off, stops a VM, disables a policy, demotes a primary. Its check says FENCED, FAILED or UNKNOWN. required: true makes it count toward the site’s verdict; one that is not is secondary. | In a failover, for every site but the target; then continuously, to check that sites which do not own the plan stay fenced. Its unfence runs when the site is promoted, with unfence_on_promote. |
shield | Protects the site it belongs to from the others, such as a firewall policy that cuts the old site off. | Raised at the promoted site while another site is not confirmed fenced (returning sites). |
readiness | Says whether the site’s copy may be promoted: a replica recent enough, an Engine up. | Before promotion, at the target site. |
health | Says whether the owner’s copy is healthy. | Every interval_seconds, at the owner site. |
recovery | Restarts the copy where it runs. | When health has been bad for long enough, before any failover (application controllers). |
switchover | Promotes the application at the new owner. | After the other sites are fenced and the target is ready. |
dead-man | A switch that powers the site’s host off unless it is renewed, such as a BMC watchdog. | Armed and renewed by the owner site while it holds quorum (returning sites). |
service | Points clients at the owner: enables a VIP or a route. | Last in a failover, after every fence is verified. |
Each capability says which roles it may take, and a fence’s capability says its level: compute, network or application (fencing levels). The plan editor offers only what fits, and the configuration refuses the rest.
Where a controller is called from
Each site’s orchestrators make that site’s calls, so no controller is reached across sites unless you say so (ADR 0010 §9). A controller’s site is where it lives, and its reach lists further sites that may call it. The leader decides every action; when it is not at an allowed site, it sends the call, signed and carrying the plan’s generation and the leader’s term, to an orchestrator that is, which refuses it if that leader has since been deposed.
sequenceDiagram
participant L as Leader, at dc2
participant E as Orchestrator at dc1
participant P as pve-dc1<br/>site dc1
participant B as bmc-dc1<br/>site dc1, reach dc2
L->>E: check pve-dc1-erp (generation 7, term 3)
E->>P: VM 101's status
P-->>E: stopped
E-->>L: FENCED
L->>B: power off pve-dc1-01 (dc2 is in its reach)
B-->>L: accepted
L->>B: power state?
B-->>L: Off: FENCED
When dc1 is lost, nothing at dc1 answers: pve-dc1’s bindings have no orchestrator to run them and read UNKNOWN. bmc-dc1 lists dc2 in its reach, so dc2 still fences dc1’s hosts through it. That is why a fence that must prove a lost site stopped needs a controller the other site can reach: a BMC on a management network both sites route to, or a firewall in front of both (infrastructure controllers).
Registration, validation, discovery and the dry run run from an allowed site too, so what they prove is the path a fence will take.
Registering and validating
A controller is registered in four steps, in the console or by its configuration (ADR 0010 §6):
- Its type and connection, its site and reach.
- Its credentials, as a reference to a secret.
- A connectivity test, from an allowed site. It reads the target’s identity, such as a BMC’s UUID or a cluster’s name, so a second controller for the same system is warned of. For a target whose certificate qfence pins, the first test shows it for you to compare and pin.
- Its capabilities, each checked for what it needs: the account’s privileges, the target’s settings.
Each check has a grade:
| Grade | Meaning |
|---|---|
| proven | The target showed it: the token holds the privilege on that VM. |
| inferred | Everything visible says yes, but the target may still refuse: a role that grants the privilege, which the device may restrict further. |
| unverifiable | qfence could not tell: the device does not show the account its own role. Nothing is wrong; the first real use is the proof. |
| failed | It cannot work as configured. The capability is not advertised, so no plan can bind it; registration goes on with the rest. |
Checks that need the bound resource, such as whether a VM is on the node a binding names, run when a binding names it, and some run again before every fence.
The two kinds
Infrastructure controllers act on what the application runs on: power, VMs, containers, networks. Their fences are the ones that prove a site stopped. Application controllers act on the application itself, through its own interface. Their fences, a demoted primary or a masked unit, never prove a site stopped: they run first, as secondary fences, while the site is still up. Each type has a guide: infrastructure and application.