The Redfish infrastructure controller

The redfish controller type switches a server off and on through its BMC’s Redfish service: Dell iDRAC, HPE iLO, Lenovo XClarity, Supermicro and most current BMCs speak it. Powering a host off is the strongest fence there is, the compute level (fencing levels). It is a REST client over HTTPS; nothing runs on the server.

One controller is one BMC endpoint. A binding’s resource is one of its computer systems, by id: the last part of /redfish/v1/Systems/<id>, such as System.Embedded.1 on iDRAC or 1 on iLO. Use it to:

• power a site’s hosts off when the site is given up, and keep them off when power returns;
• prove a host is off without switching it off (power.evidence);
• power the new owner’s hosts on when it is promoted.

1. What it can do

A fence returning only means the BMC accepted the command. qfence then reads the power state back, and only Off is FENCED.

Durable power fences. A host that is merely off starts again when its power returns, if the BMC’s policy says so. With durable_power_fence (the default), power.off sets the policy to AlwaysOff before it switches the host off (ADR 0009 §3). A BMC that refuses the policy, or has none, still powers the host off: the fence is then transient, and the site owes a durable one. power.evidence never changes the policy.

2. What the BMC needs

• A Redfish service over HTTPS on the management network, reachable from the orchestrators of the controller’s site and of every site in its reach (§6).
• An account for qfence whose role grants the Redfish privilege ConfigureComponents. The standard Operator role does; ReadOnly does not.
• Optional: a certificate the orchestrator hosts trust. qfence checks the BMC’s certificate against the host’s trust store. A BMC with a self-signed certificate needs insecure_skip_verify: true: there is no CA file or certificate pin for this type.

3. Set up the account

Create a local account on the BMC, through its web interface or with Redfish itself, and give it the Operator role:

curl -sk -u admin https://bmc-dc1.example.com/redfish/v1/AccountService/Accounts \
  -H 'Content-Type: application/json' \
  -d '{"UserName": "qfence", "Password": "…", "RoleId": "Operator", "Enabled": true}'

Some BMCs number their account slots and expect a PATCH of a free one instead; their documentation says which. An account that also signs in to the web console is fine, but one that does nothing else is easier to audit and to rotate.

4. Store the password on the orchestrators

The password is a secret, so configuration only names it. Put it in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of the basic method’s secret field:

jq -n --arg p "$PASSWORD" '{password: $p}' |
  qfence-orchestrator secret put -server https://orchestrator-dc2:8443 -token-file admin.token -ca-file ca.pem -ref bmc-dc1 \
    -scope-type redfish -scope-host bmc-dc1.example.com

-scope-type and -scope-host are the controller types and the addresses that may use the secret: the host of the controller’s endpoint (ADR 0016 §6). Any other controller that names it is refused it (ADR 0011 §8). Run it against each of those orchestrators while it runs (writing a secret): the controllers that use it pick it up at once.

The user name is not secret: it goes in the controller’s auth.config. Keep it out of the secret, since qfence hides a secret’s values wherever it shows text, and a word as common as a user name would be hidden in other text too, such as a device’s name.

5. Register the controller

In the console: go to Controllers, then Infrastructure, then Register. Choose Redfish BMC, its site, and the further sites in its reach; enter the endpoint and pick the secret reference. The connectivity test reads the service root and lists the systems, which proves the account can sign in. It shows the BMC’s vendor, product and Redfish version. The identity is uuid:<the service root's UUID>: if two controllers name the same BMC, the second is warned.

In a configuration file (ADR 0001):

controllers:
  infrastructure:
    - id: bmc-dc1
      type: redfish
      site: dc1
      reach: [dc2]
      network_domain: oob-mgmt
      connection: {endpoint: https://bmc-dc1.example.com, timeout: 5s}
      auth: {method: basic, credentials_ref: "local:bmc-dc1", config: {username: qfence}}
      capabilities: [power.off, systems.list]

timeout bounds each request to the BMC (10 s by default). The plan’s fence_timeout_seconds and verify_timeout_seconds bound the whole fence and its check: raise them for a BMC that is slow to answer. Every field is in the reference.

Validation. role.ConfigureComponents reads the account’s role from the BMC’s account service. It passes as inferred when the role grants ConfigureComponents, because a BMC may still refuse a reset by its own rules, and fails when it does not. A BMC that does not list the account to itself leaves it unverifiable: the first fence is then the proof.

6. Bind it to a plan

A site’s BMCs must answer from the other site, since that is where a lost site is fenced from: list the other site in the controller’s reach, and put the BMCs on a network both sites route to. This is the demo lab’s ERP database at dc1: its host’s power is the required fence, its VM’s stop runs first as a secondary one, and a firewall policy is the site’s shield:

instances:
  - id: erp-db-dc1
    site: dc1
    resource: {controller: pve-dc1, resource: "101"}
    bindings:
      - {id: bmc-dc1, controller: bmc-dc1, capability: power.off, resource: pve-dc1-01, role: fence, required: true}
      - {id: pve-dc1-erp, controller: pve-dc1, capability: vm.stop, resource: "101", role: fence, params: {node: pve-dc1-01}}
      - {id: shield-dc1, controller: fgt-edge, capability: policy.disable, resource: "11", role: shield, required: true}

With fencing.unfence_on_promote, a failover back to dc1 powers the host on, then starts its VM: power fences unfence first, then the others in their order, each within fence_timeout_seconds. The power-on returns once the BMC accepts it, not once the host has booted (§8).

Power evidence under ANY. When qfence should never switch a host off itself, bind power.evidence beside the fence that does the work, required, under fencing.success: ANY and escalate: true (the default). The gentler fence is tried first; when the site is lost, its fence cannot answer, and the BMC proves the host off (the swarm lab does this).

Settings, in the capability’s config:

capabilities:
  - id: power.off
    config: {durable_power_fence: true, unfenced_power_restore_policy: AlwaysOn}

unfenced_power_restore_policy is what the unfence sets before it powers the host on: AlwaysOn, AlwaysOff or LastState (the default).

7. Troubleshooting

8. Limits

• Not tested against a real BMC. The type follows DMTF’s published ComputerSystem schema and is tested against a double of it (internal/controllers/infrastructure/redfish), and against the demo lab’s simulator, whose hosts go off at once (limitations). This guide was followed there: registration, validation, a durable fence, and an unfence with each restore policy. How long a real BMC takes to report Off, and vendor differences such as one that refuses ForceOff for a system already off, have not been seen.
• One read of the power state. A fence does not wait for the host to report Off: qfence reads it once, right after the BMC accepts the command, and a host still going down is not yet FENCED (§7).
• The unfence does not wait for the boot. A binding unfenced after the power-on, such as a VM’s start, is sent while the host may still be booting. A systemd unit’s unfence waits for its host to answer; a Proxmox VE VM’s start does not, and the promotion blocks if the node refuses it (Proxmox VE).
• Basic authentication only. Each request carries the account’s credentials; Redfish sessions are not used.
• One BMC, its systems. A chassis whose BMC manages several servers is one controller with one binding per system.