Infrastructure controllers
An infrastructure controller acts on what an application runs on: a server’s power, a VM, a container, a firewall. Its fences are the ones that prove a site has stopped, so every plan needs at least one at each site (fencing levels). Five types are built into the orchestrator, each with a guide:
| Guide | One controller is | Fences | Also |
|---|---|---|---|
| Redfish | A BMC’s Redfish service | power.off, power.evidence (compute) | |
| IPMI | A BMC, through ipmitool | power.off (compute) | watchdog.arm, a dead-man switch |
| Proxmox VE | A Proxmox VE cluster’s API | vm.stop, node.fence (compute); vm.link-down (network) | replication lag, guest-agent probes |
| FortiGate | A FortiGate’s REST API | policy.disable, interface.down, vlan.down (network) | vip.own, route.own; link-monitor and real-server probes |
| Docker | A Docker Engine, or a swarm manager | container.stop, service.shape, node.drain (compute); container.disconnect, service.unpublish (network) | restarts; health and lag probes |
Three things every fence controller needs
The other site must reach it. A site is fenced when it is lost, by the orchestrators of the site taking over. So a controller that fences dc1 must answer from dc2, and list dc2 in its reach: only orchestrators at its site and the sites in its reach call it (ADR 0010 §9). A BMC on an out-of-band management network that both sites route to is the usual answer. A hypervisor’s API at the lost site is not: it is lost with the site.
Its path should not be the workload’s. network_domain names the network a controller is reached through (spec §15). A fence reached over the same network as the workload fails, or cannot be verified, exactly when that network is the problem. qfence cannot see where a network runs, so it relies on what you declare: the dry run, and each fence’s result, warn of a compute or network fence with no network_domain.
Its fence should last. A fence is durable when what it stopped stays stopped after a restart: a host whose BMC will not power it back on, a VM that does not start on boot, a container whose restart policy does not start it (ADR 0009 §1). A fence that holds only until a restart is transient, and the site then owes a durable one: qfence shows the debt and alerts after fencing.debt_alert_after_seconds (ADR 0008 §1). Each guide says when its fences are durable.
The steps, for every type
- Give qfence its own account on the device, with only the rights its capabilities need. Each guide lists them, and validation checks them.
- Store the account’s secret in the encrypted secrets store of every orchestrator that will call the controller, scoped to the type and the device’s address (secrets).
- Register the controller, in the console (Controllers, then Infrastructure, then Register) or in a configuration file. The connectivity test reads the device’s identity, so a second controller for the same device is warned of. Where the console writes it depends on the cluster’s configuration mode (Configure): in replicated mode, to every orchestrator; in file mode, nowhere, giving you the entry to add to every orchestrator’s file; in local mode, as in the demo lab, only to the orchestrator you are signed in to.
- Bind its capabilities in a plan, and run the plan’s dry run.
How they were tested
The Docker type is tested against a real Docker Engine. Redfish, IPMI, Proxmox VE and FortiGate are tested against doubles built from each vendor’s published API, and against the demo lab’s simulator, never against the equipment itself (limitations). Each guide’s last section says what was and was not checked.