The sites on Proxmox

Makes the two site VMs of a two-sites-and-a-witness topology on Proxmox VE, each running qfence-orchestrator as a raft voter, with the API and the console. The witness’s module makes the third voter: run both against the same Proxmox cluster for the whole topology.

What has been checked. The module passes terraform validate and terraform fmt -check with the bpg/proxmox provider (v0.113.1 at the last check). A terraform plan with dummy values succeeds offline, for both sites and for one site picked with create_sites (below), and each time the cloud-init it renders lists all three voters in QFENCE_RAFT_PEERS. The module was reviewed against the provider’s documented schema. It has never been applied to a real Proxmox cluster: check it against your Proxmox version, and disk.interface against your template (below).

What the first boot does

cloud-init clones the repository at qfence_repo_ref and builds qfence on the VM: the console with Node.js 24 from nodejs.org, then the orchestrator with the Go version go.mod names, from go.dev. It installs them as step 3 of the install guide does, removes both toolchains, and writes the env file. The VM needs outbound HTTPS for this: to your git host, go.dev, proxy.golang.org, nodejs.org and registry.npmjs.org. Ubuntu’s own Go package is too old to build qfence, so it is not used. This first-boot script has been run on Ubuntu 22.04, the template below, in a container.

The env file has no raft TLS settings, so raft runs in plaintext. To run it over mutual TLS, as the install guide does, copy each VM its certificate (step 2) and add the QFENCE_RAFT_TLS_* lines to both sites’ env files and the QFENCE_WITNESS_TLS_* lines to the witness’s, then restart all three. The three voters must agree.

Prerequisite: a cloud-init-enabled template

This module clones an existing template — creating one is a standard, one-time Proxmox operation this module deliberately doesn’t automate (it’s well documented elsewhere and varies more by preference than anything qfence-specific). The short version, from a Proxmox VE host:

wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
qm create 9000 --name ubuntu-2204-cloudinit --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
qm importdisk 9000 jammy-server-cloudimg-amd64.img local-lvm
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot c --bootdisk scsi0
qm set 9000 --serial0 socket --vga serial0
qm template 9000

Note the disk ended up on scsi0 — that’s what this module’s disk.interface assumes. If your template uses a different interface (e.g. virtio0), change it in main.tf.

Usage

module "orchestrator_sites" {
  source = "github.com/REPLACE_ME/qfence//deploy/orchestrator/terraform/proxmox"

  proxmox_endpoint  = "https://pve.example.lan:8006/"
  proxmox_api_token = "terraform@pve!qfence=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  proxmox_insecure  = true # typical for a self-signed on-prem Proxmox cert

  target_node    = "pve1"
  template_vm_id = 9000

  sites = {
    dc1 = "10.0.0.1/24"
    dc2 = "10.0.0.2/24"
  }
  gateway = "10.0.0.254"

  # From the witness module's own output, once provisioned:
  witness_peer = "witness=10.0.0.5:7000"

  qfence_repo_url = "https://gitlab.com/REPLACE_ME/qfence.git"
  qfence_repo_ref = "main" # pin a real tag/commit for anything beyond a first test

  raft_bootstrap = true # the founding voters form the cluster together; a voter with raft state ignores it
}

Each site VM needs the cluster keyring (ADR 0016 §1), the same file on both and never on the witness. This module does not pass it through cloud-init, so it stays out of the Terraform state and the snippet store. Make it once with qfence-orchestrator cluster-key new -id c1, copy it to /etc/qfence-orchestrator/cluster.keys on each site VM (mode 0600, owned by qfence) over SSH with a post-quantum key exchange (OpenSSH 9.9 or later), then systemctl start qfence-orchestrator. cloud-init starts the service itself only if the file is already there.

Both sites and the witness form the cluster together on their first start, with raft_bootstrap = true and witness_bootstrap = true. A voter that already has raft state ignores the flag, so a later apply with it set changes nothing. deploy/README.md has the sequence across both modules.

Single Proxmox host vs. a real cluster

The example above puts both sites’ target_node at the same value — fine for an initial lab test, but defeats the point of a 2-site topology if that one Proxmox node is also a single point of failure.

var.sites is deliberately used for two things at once: it’s both “every site in the topology” (used to compute the complete raft peer list every VM’s cloud-init needs, via qfence_repo_*/witness_peer/etc.) and, by default, “every site this module call creates a VM for.” For a real cluster with sites on different Proxmox nodes, keep sites identical (the full map) across both calls, but use create_sites to say which one each call actually provisions — otherwise each call would try to create every site in the map itself, duplicating VMs:

locals {
  all_sites = {
    dc1 = "10.0.0.1/24"
    dc2 = "10.0.0.2/24"
  }
}

module "site_dc1" {
  source = "github.com/REPLACE_ME/qfence//deploy/orchestrator/terraform/proxmox"
  # ...same provider/witness_peer/qfence_repo_* args as the first example...
  target_node    = "pve1"
  template_vm_id = 9000
  sites          = local.all_sites
  create_sites   = ["dc1"]
  gateway        = "10.0.0.254"
}

module "site_dc2" {
  source = "github.com/REPLACE_ME/qfence//deploy/orchestrator/terraform/proxmox"
  # ...same provider/witness_peer/qfence_repo_* args as the first example...
  target_node    = "pve2"
  template_vm_id = 9000
  sites          = local.all_sites
  create_sites   = ["dc2"]
  gateway        = "10.0.0.254"
}

Both calls compute the same complete raft_peers string (from the same full sites map) but each only creates the one VM its create_sites names. See ../../../witness/’s own README for the identical point about the witness’s placement relative to both sites.