The Proxmox VE infrastructure controller

The proxmox controller type works through a Proxmox VE cluster’s REST API. It stops and starts VMs, stops every guest on a node, takes a VM’s network link down, and reads storage replication and the QEMU guest agent. Nothing is installed on the nodes or in the guests beyond what Proxmox VE already runs.

One controller is one cluster, reached through one node’s API. Use it to:

• stop a site’s VM when the site is given up, and start the new owner’s;
• check, before a promotion, that the VM’s replica at the site taking over is recent enough;
• cut a VM off the network while it keeps running, as a fence or a shield;
• probe a guest through its agent.

1. What it can do

Durable VM fences. A stopped VM that starts on boot runs again when its node restarts. vm.stop does not change onboot: a VM bound to it should have it off, which also keeps a node that restarts from starting the copy qfence fenced (ADR 0009 §1). With onboot: 1, the fence is transient, and the site owes a durable one.

Proxmox VE HA. Leave the VMs a plan protects out of Proxmox VE’s HA manager, or set them to ignored or disabled: HA would start a VM that qfence stopped, on the same node or another. A node.fence binding checks it (ha.not-managed, unverifiable while HA manages a guest on the node), and its fence reports a guest HA wants started as transient.

2. What the cluster needs

• Proxmox VE 8 or 9, whose privilege names the table below uses, with its API (port 8006) reachable from the orchestrators of the controller’s site and reach.
• An API token for qfence, with the privileges below on the paths it acts on.
• For probe.agent-ping and probe.service: the QEMU guest agent installed in the guest and enabled in the VM’s options.
• For replication.lag: a storage replication job for the VM (ZFS storage).

3. Set up the token

On any node, create a role with the privileges the capabilities you enable need, a user, and a token for it. This one covers stopping VMs, taking links down, replication and discovery:

pveum role add QfenceFence --privs "VM.PowerMgmt VM.Config.Network VM.Audit Sys.Audit"
pveum user add qfence@pve --comment "qfence orchestrators"
pveum acl modify /vms --users qfence@pve --roles QfenceFence
pveum acl modify /nodes --users qfence@pve --roles QfenceFence
pveum user token add qfence@pve orchestrators --privsep 0

The last command prints the token’s secret once: keep it for §4. The token’s id is qfence@pve!orchestrators. With --privsep 0, the token has its user’s privileges; with privilege separation on, give the token the same ACLs with --tokens 'qfence@pve!orchestrators'.

An ACL on /vms covers every VM. To limit qfence to the VMs it protects, put the ACL on /vms/101 and the rest instead, or on a resource pool. Validation then infers the privilege for the controller and proves it for each VM when a binding names one.

The basic method, a user’s password, also works. qfence signs in for a ticket and keeps it in memory. An account with a second factor cannot sign in this way: use a token.

4. Store the token on the orchestrators

Configuration only names the token’s secret. Put it in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of the api_key method’s secret field:

jq -n --arg s "$TOKEN_SECRET" '{token_secret: $s}' |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref pve-dc1 \
    -scope-type proxmox -scope-host pve1.dc1.example.com

-scope-host is the host of the controller’s base_url (ADR 0016 §6). Run it against each of those orchestrators while it runs (writing a secret). The token’s id is not secret: it goes in the controller’s auth.config.

5. Register the controller

In the console: go to Controllers, then Infrastructure, then Register. Choose Proxmox VE, its site and reach, enter the API’s URL on one node, and pick the secret reference. The connectivity test reads the cluster’s version and status. The identity is the cluster’s name: two controllers for one cluster are warned of.

In a configuration file (ADR 0001):

controllers:
  infrastructure:
    - id: pve-dc1
      type: proxmox
      site: dc1
      network_domain: pve-mgmt-dc1
      connection: {base_url: "https://pve1.dc1.example.com:8006"}
      auth: {method: api_key, credentials_ref: "local:pve-dc1", config: {token_id: "qfence@pve!orchestrators"}}
      capabilities: [vm.stop, vm.link-down, replication.lag, vms.list, vm.nics, replication-jobs.list]

The connection checks the API’s certificate against the orchestrator host’s trust store. Proxmox VE’s own certificate is self-signed: give the nodes a certificate the hosts trust (ACME, or your CA), or set insecure_skip_verify: true. Every field is in the reference.

Validation. qfence reads the token’s privileges (/access/permissions). One held on /, /vms or /nodes is proven for every object; one held only on some objects is inferred, and proven for each VM or node when a binding names it. A binding of vm.stop also checks that the VM is on the node it names (vm.node). The guest-agent probes check that the agent answers (guest-agent.running).

node.fence and the node you reach. A node asked to fence itself could not be asked afterwards whether it did. So node.fence reads which node answers the API, before each fence, and refuses when it is the node being fenced, or when it cannot tell (endpoint-not-on-node). Point base_url at a node you will not fence with it, and set its mode in the capability’s config:

      capabilities:
        - id: node.fence
          config: {mode: stop-guests}

stop-guests stops every guest and leaves the node up. shutdown also shuts the node down; after that, the node is offline, its fence reads UNKNOWN, and only another path, such as its BMC, can prove it off.

6. Bind it to a plan

A cluster at a site is lost with the site, so vm.stop cannot prove a lost site stopped: pair it with the host’s BMC, required, and let the VM’s stop run as a secondary fence while the site is up. This is the demo lab’s ERP database:

instances:
  - id: erp-db-dc1
    site: dc1
    resource: {controller: pve-dc1, resource: "101"}
    bindings:
      - {id: bmc-dc1, controller: bmc-dc1, capability: power.off, resource: pve-dc1-01, role: fence, required: true}
      - {id: pve-dc1-erp, controller: pve-dc1, capability: vm.stop, resource: "101", role: fence, params: {node: pve-dc1-01}}
      - {id: pve-dc1-erp-replication, controller: pve-dc1, capability: replication.lag, resource: 101-1, role: readiness,
         label: erp-db replica at dc1, params: {node: pve-dc1-01, max_replication_lag: 300s}}
  - id: erp-db-dc2
    site: dc2
    resource: {controller: pve-dc2, resource: "101"}
    bindings:
      - {id: pve-dc2-erp, controller: pve-dc2, capability: vm.stop, resource: "101", role: fence, required: true, params: {node: pve-dc2-01}}
      - {id: pve-dc2-erp-replication, controller: pve-dc2, capability: replication.lag, resource: 101-0, role: readiness,
         label: erp-db replica at dc2, params: {node: pve-dc2-01, max_replication_lag: 300s}}

With fencing.unfence_on_promote, a failover to dc1 powers the host on, then starts the VM: power fences unfence first, then the others in their order. The VM’s start is sent once, right after the power-on returns (§8).

Here dc2’s VM is fenced by vm.stop alone, so when dc2 is lost, nothing can prove it stopped, and a failover back to dc1 waits for an operator’s attestation (ADR 0003). Give each site a fence the other site can reach to avoid that.

replication.lag reads the job’s last sync, not the data: a job that synced at 12:00 and failed since is not ready, whatever its lag. Set max_replication_lag to the data you are prepared to lose; an operator can accept more for one failover (ADR 0006).

vm.link-down as a shield. Bound with role: shield at the site it protects, it takes down a NIC of a VM there, such as a router VM’s link towards the other site, while the other site cannot be confirmed fenced (returning sites). It must leave up what the other site needs to become a standby, such as its replica’s connection.

7. Troubleshooting

8. Limits

• Not tested against a real Proxmox VE. The type is tested against a double of the Proxmox VE API built from its published API, and against the demo lab’s simulator (limitations). This guide was followed in the demo lab for vm.stop, replication.lag and discovery. node.fence cannot be: the simulator does not say which node answers its API, so endpoint-not-on-node refuses it there, as it should. How long a real stop task takes has not been seen: vm.stop does not wait for it, and reads the VM’s status once. node.fence does wait, until no guest runs.
• Storage replication runs within one cluster. Proxmox VE replicates a VM’s disks between nodes of one cluster. The demo lab gives each site a cluster and replicates between them, which a real Proxmox VE does not. How a cluster whose nodes are at both sites reports its jobs, read from the node params.node names, has not been checked.
• A VM’s start does not wait for its node. When the site’s host was powered off by its BMC, a promotion powers it on and then starts the VM at once. A real node is still booting then and refuses the start, and the promotion blocks. The demo lab’s nodes are up at once, so the lab does not show it.
• A basic ticket lasts two hours. qfence signs in again before it expires; a sign-in that fails shows on the controller.