The IPMI infrastructure controller

The ipmi controller type switches a server off and on through its BMC over IPMI v2.0 (RMCP+), for BMCs without Redfish or where IPMI is what is enabled. It runs ipmitool, as Pacemaker’s fence_ipmilan does, on the orchestrator’s host; nothing runs on the server. It also arms the BMC’s watchdog as a dead-man switch, which powers the host off unless it is renewed.

One controller is one BMC, which manages one host. Use it to:

• power a site’s hosts off when the site is given up, and keep them off when power returns;
• power the new owner’s hosts on when it is promoted;
• make a site that is cut off from quorum power itself off, even when its orchestrators are dead (watchdog.arm).

1. What it can do

A binding’s resource is that name: the connection’s host.

Durable power fences. With durable_power_fence (the default), power.off sets the BMC’s power restore policy to always-off before it powers the host off, so the host stays off when power returns (ADR 0009 §3). A BMC that refuses the policy still powers the host off: the fence is then transient, and the site owes a durable one.

The dead-man switch. It closes the gap a dead orchestrator leaves. With fencing.self_fence on, a site that is cut off from quorum for after_seconds fences its own targets. But an orchestrator that has died cannot. The watchdog does it instead: its countdown is after_seconds plus the time the site’s own fences take, the owner’s orchestrators renew it every third of after_seconds while they hold quorum, and when the renewals stop, the BMC powers the host off. With backs_failover, the other site’s automatic failover may then count on the cut-off owner having stopped, without an attestation (returning sites).

2. What the orchestrators and the BMC need

• ipmitool on the host of every orchestrator at the controller’s site and reach, on its $PATH or at binary_path. A controller whose ipmitool cannot be found fails to open, with ipmitool not usable, when the configuration is applied, not at the first fence.
• The BMC’s LAN channel enabled for IPMI over LAN, at UDP port 623, reachable from those orchestrators.
• A BMC account for qfence at privilege level OPERATOR, which may control chassis power and the watchdog. ADMINISTRATOR works too, but grants more than qfence needs.

3. Set up the account

On the host itself, with the local ipmitool and root, pick a free user slot (5 here) on the LAN channel (usually 1):

ipmitool user list 1
ipmitool user set name 5 qfence
ipmitool user set password 5          # asks for the password
ipmitool channel setaccess 1 5 link=on ipmi=on callin=on privilege=3
ipmitool user enable 5

Privilege 3 is OPERATOR. Then check from an orchestrator’s host that the account signs in at that level:

ipmitool -I lanplus -H bmc-dc1.example.com -U qfence -P "$PASSWORD" session info active | grep -i privilege

4. Store the password on the orchestrators

Put it in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of the basic method’s secret field:

jq -n --arg p "$PASSWORD" '{password: $p}' |
  qfence-orchestrator secret put -server https://orchestrator-dc2:8443 -token-file admin.token -ca-file ca.pem -ref ipmi-app1 \
    -scope-type ipmi -scope-host bmc-app1.dc1.example.com

-scope-host is the BMC’s address, the connection’s host, which receives the credentials at port 623 (ADR 0016 §6). Run it against each of those orchestrators while it runs (writing a secret).

The user name is not secret: it goes in the controller’s auth.config. Keep it out of the secret, since qfence hides a secret’s values wherever it shows text, and a word as common as a user name would be hidden in other text too, such as a device’s name.

5. Register the controller

In the console: go to Controllers, then Infrastructure, then Register. Choose IPMI BMC, its site and reach, enter the BMC’s address and pick the secret reference. The connectivity test runs mc guid and mc info: it shows the BMC’s IPMI version, firmware, manufacturer and product. The identity is guid:<the BMC's GUID>.

In a configuration file (ADR 0001):

controllers:
  infrastructure:
    - id: ipmi-app1
      type: ipmi
      site: dc1
      reach: [dc2]
      network_domain: oob-mgmt
      connection: {host: bmc-app1.dc1.example.com}
      auth: {method: basic, credentials_ref: "local:ipmi-app1", config: {username: qfence}}
      capabilities: [power.off, watchdog.arm, systems.list]

interface is ipmitool’s -I, lanplus by default. Every field is in the reference.

Validation. privilege.operator runs session info active and reads the session’s privilege level. OPERATOR, ADMINISTRATOR or OEM passes as inferred, since a BMC may still restrict a command; USER or lower fails.

Its health shows only that ipmitool can run. It does not call the BMC between fences, so a BMC that stopped answering shows at the next fence, its check, or a connectivity test. Run the plan’s dry run after any change to the management network.

6. Bind it to a plan

A site’s BMCs must answer from the other site, since that is where a lost site is fenced from: list it in reach. Each site’s host has its own controller:

availability_plans:
  - id: orders
    sites: [dc1, dc2]
    fencing:
      unfence_on_promote: true
      self_fence: {enabled: true, after_seconds: 60, backs_failover: true}
    failover: {automatic_failover: true}
    instances:
      - id: orders-dc1
        site: dc1
        resource: {controller: app1, resource: orders.service}
        bindings:
          - {id: power-app1, controller: ipmi-app1, capability: power.off, resource: bmc-app1.dc1.example.com, role: fence, required: true}
          - {id: watchdog-app1, controller: ipmi-app1, capability: watchdog.arm, resource: bmc-app1.dc1.example.com, role: dead-man}
      # orders-dc2: the same, with ipmi-app2.

backs_failover lets the other site’s automatic failover count on the watchdog, so it needs failover.automatic_failover. A watchdog’s countdown, after_seconds plus the site’s fencing time, must stay under 6553.5 s, the longest IPMI holds. The swarm lab arms its nodes’ watchdogs this way (deploy/swarm-lab/README.md).

7. Troubleshooting

8. Limits

• The password is on ipmitool’s command line (-P) while each call runs, so another user of the orchestrator’s host can read it in the process list. Give the orchestrator’s host no other users.
• Not tested against a real BMC. The type is tested against a scripted fake ipmitool, and against the lab simulator through the swarm lab’s ipmitool shim, which understands exactly the commands qfence sends (limitations). This guide was followed there: registration, validation, a durable fence, its unfence, and a watchdog armed and renewed by the owner. How long a real BMC takes to report off has not been seen: a fence does not wait for it, and reads the power state once.
• One host per BMC. For a chassis whose BMC manages several nodes, use the Redfish type.