Skip to content
Reference
Command line
qfence-orchestrator

qfence-orchestrator

Run an orchestrator: a voter of the cluster’s quorum at one site, serving the REST API and the console.

Without -raft-bind it is a single-node demo, its state in memory unless -state-dir is set: for trying qfence out, not for a deployment. With -raft-bind it is a voter of a cluster, one orchestrator per site, with a witness at a third.

qfence-orchestrator [flags]
qfence-orchestrator <command> ...

Flags

  • -advertise-api string: the URL other orchestrators use to reach this node’s API, e.g. https://10.0.0.1:8443 (default: derived from -listen, using the -raft-bind host when listening on a wildcard address). It must reach this listener itself, or through a proxy that passes TLS through: the others accept only the certificate this node serves (ADR 0011 §13)
  • -audit-retention int (default 100000): how many audit records to keep (oldest dropped first), in memory and on disk
  • -automatic-failover: run automatic failover (ADR 0002): on the raft leader, fail over an availability plan with automatic_failover: true once its owner site has had no reachable voter for its primary_unhealthy_for_seconds. Off by default: a plan’s automatic_failover setting does nothing until this is set on the orchestrators
  • -bootstrap-token-file string: if set, also write the one-time bootstrap administrator credential to this file (in addition to printing it) — for local tooling/CI that needs to authenticate without a human reading stdout, e.g. the Playwright E2E suite in web/console/tests/e2e. Discloses nothing the stdout banner doesn’t already: same credential, same one-time-only guarantee, just also on disk. Never use on a shared/production host.
  • -cluster-key-file string: the cluster keyring (ADR 0016 §1), required with -raft-bind: one “<key-id> <base64 32-byte key>” per line, the first active; mode 0600; read again when it changes. Every orchestrator in the cluster holds the same keys, and the witness never does. Keys derived from it sign every call between orchestrators and seal what they keep in raft that the witness does not read. Create a key with “qfence-orchestrator cluster-key new -id c1”; rotate by adding a new key to every orchestrator’s file, then putting it first everywhere, then removing the old one once the voter list shows every report under the new key
  • -config string: configuration file (controllers: and availability_plans:) to load at startup. In file configuration mode (ADR 0001) it is the cluster’s source of truth and must be the same on every orchestrator; controller credentials are referenced with credentials_ref (local:NAME) and resolved through this node’s encrypted secrets store
  • -config-key-file string: replicated-configuration key file (ADR 0001): one “<key-id> <base64 32-byte key>” per line, the first being the active key; mode 0600. Every orchestrator (never the witness) needs the same keys. Create one with “qfence-orchestrator config-key new -id k1”
  • -demo: demo mode: serve the built-in demo (mock controllers and two plans) when the configuration defines nothing, and allow controllers of the mock type, whose fences report success without fencing anything. Always on without -raft-bind (the single-node in-memory demo). For trying qfence out, tests and labs: never on a deployment that protects anything
  • -deployment-id string: this deployment’s identifier as known to the hosted plane (defaults to -node-id)
  • -hostedplane-ca string: PEM CA certificate(s) to verify the hosted plane’s HTTPS certificate with, instead of the system’s trusted CAs (a hosted plane run with a private CA)
  • -hostedplane-endpoint string: hosted control/observability plane’s ingest URL (e.g. https://hostedplane.example.com/v1/ingest), per §58. Empty (the default) runs with no telemetry shipping agent at all — not merely a disconnected one — since §58/Critical Acceptance Test 9 require that severing this be a no-op for local decisions, which is easiest to see is true when there is nothing here to sever in the first place
  • -hostedplane-token string: this deployment’s hosted-plane ingest credential, issued by the hosted plane for this deployment (required if -hostedplane-endpoint is set)
  • -insecure-http: serve plain HTTP instead of TLS — local development only, never production (§49 requires TLS everywhere)
  • -issue-bootstrap-token: with durable state, issue a new bootstrap administrator token even though tokens already exist
  • -listen string (default 127.0.0.1:8443): address to serve the REST API and Admin UI on
  • -master-key-file string: master key file (mode 0600) of this node’s encrypted secrets store under -state-dir, where credentials_ref: local:NAME is read: one “<key-id> <base64 32-byte key>” per line, the first active (create one with “qfence-orchestrator secret new-key -id m1”; rotate with a new key first, then “qfence-orchestrator secret rekey”). Without one, no controller credentials can be read; with one that does not open the store, every local: reference is unavailable
  • -node-id string (default orchestrator-01): this voter’s raft server ID — must be unique within the cluster and match one entry in -raft-peers
  • -oidc-audience string: the aud value an access token must carry to be accepted here (required with -oidc-issuer)
  • -oidc-authorization-endpoint string: the provider’s authorization endpoint for the console’s sign-in redirect, overriding discovery (needed with -oidc-jwks-url)
  • -oidc-client-id string: the console’s OIDC client ID (a public client using Authorization Code + PKCE; no secret). Default: -oidc-audience
  • -oidc-issuer string: OpenID Connect issuer URL, exactly as it appears in tokens’ iss claim. Setting it turns on OIDC: bearer tokens from this issuer are accepted, and the console offers a sign-in redirect. The provider’s endpoints are discovered from <issuer>/.well-known/openid-configuration
  • -oidc-jwks-refresh duration (default 15m0s): how often the provider’s signing keys are re-fetched, so a key it removed stops being trusted (a failed re-fetch keeps the cached keys)
  • -oidc-jwks-url string: the provider’s JWKS URL, overriding discovery
  • -oidc-local-tokens (default true): with OIDC on, also accept this node’s local API tokens (break-glass for an identity provider outage, and automation). false: OIDC only; no bootstrap token is issued and the token API is off
  • -oidc-role-claim string (default qfence_role): token claim holding the qfence role, as a string or a list of strings (e.g. a groups claim)
  • -oidc-role-map string: map role-claim values to roles, e.g. “qfence-admins=Administrator,qfence-ops=Operator,staff=Viewer”. Without it, claim values must be role names. A token that maps to no role, or to more than one, is refused
  • -oidc-scopes string (default openid,profile): comma-separated scopes the console requests
  • -peer-source string: the local address this orchestrator’s calls to other orchestrators’ APIs dial from (default: the system chooses)
  • -raft-bind string: the “host:port” this voter listens for raft RPCs on AND advertises to peers. Empty (the default) keeps the single-node in-memory demo mode; set this to run as a real voter in a multi-machine cluster (§6) — must be a specific, reachable address, never a wildcard like 0.0.0.0
  • -raft-bootstrap: form a brand-new cluster from -raft-peers on this voter’s first start. Set it on every founding voter, with the same -raft-peers (raft’s own bootstrap is safe to call from each). A voter that already has raft state ignores it, so it may stay set
  • -raft-data-dir string (default ./orchestrator-data): directory for this voter’s durable raft storage (only used with -raft-bind)
  • -raft-peers string: comma-separated id=host:port list of every voter in the cluster, including this one (required with -raft-bootstrap), e.g. “dc1=10.0.0.1:7000,dc2=10.0.0.2:7000,witness=10.0.0.5:7000”
  • -raft-snapshot-interval duration: how often the snapshot threshold is checked (0: raft’s default, two minutes)
  • -raft-snapshot-threshold uint: take a raft snapshot and compact the log once this many entries were committed since the last one (0: raft’s default, 8192). Old ciphertext of the cluster secrets store stays in the log until compacted
  • -raft-tls-ca string: PEM certificate of the CA every voter’s certificate must chain to
  • -raft-tls-cert string: PEM certificate for mutual-TLS raft transport (with -raft-tls-key and -raft-tls-ca; all three or none). Every voter, including the witness, must present a certificate from the same CA; “qfence-orchestrator pki” creates them
  • -raft-tls-crl string: a revocation list (PEM or DER) signed by -raft-tls-ca, e.g. the crl.pem “qfence-orchestrator pki revoke” writes; re-read on SIGHUP and every -raft-tls-crl-refresh
  • -raft-tls-crl-refresh duration (default 5m0s): how often -raft-tls-crl and -raft-tls-crl-url are re-read
  • -raft-tls-crl-url string: a URL serving the revocation list, e.g. the hosted plane’s https://…/v1/tenants/<tenant>/crl; fetched every -raft-tls-crl-refresh
  • -raft-tls-key string: PEM private key matching -raft-tls-cert
  • -secrets-config string: this orchestrator’s own secrets providers (ADR 0011 §3), a YAML file: for each, its id (the provider name references use, as in vault:secret/qfence/fgt), type (vault or aws-secrets-manager), the site it serves, and where it is. A Vault names its address and mounts, and how this orchestrator signs in there (auth: a client certificate by default, AppRole, or a static token for development), each secret by the 0600 file that holds it; AWS Secrets Manager names its region, and signs in with the EC2 instance’s role alone. local and cluster need no definition. Each orchestrator defines its own, so one reference resolves through each site’s own Vault
  • -secrets-key-file string: secrets keyring file of the opt-in cluster secrets store (ADR 0011 §2), where credentials_ref: cluster:NAME is read: one “<key-id> <base64 32-byte key>” per line, the first active; mode 0600; read again when it changes. Every orchestrator (never the witness) needs the store’s key: writes and rekeys are refused until each reports it. Create a key with “qfence-orchestrator secret new-key -id s1”; keep it apart from the configuration key and back it up apart from the data. Without it, nothing of the store reaches raft from this node, and once the store is in use its cluster: references are unavailable here
  • -site string: the site this orchestrator is at: it calls the controllers whose reach includes it (ADR 0010 §9), reports it to the cluster, and fences itself when cut off (ADR 0008 §4); empty derives it from -node-id (the site’s name, or its prefix before “-”)
  • -state-dir string: directory for this node’s durable local state: audit log, API tokens (hashes only), the encrypted secrets store and plan configuration changed through the API. Defaults to -raft-data-dir when -raft-bind is set; empty in single-node demo mode keeps everything in memory
  • -tls-cert string: path to a TLS certificate (PEM); if unset along with -tls-key, a self-signed certificate is generated automatically
  • -tls-key string: path to the TLS private key (PEM) matching -tls-cert
  • -tls-next-cert string: the certificate (PEM) that will replace -tls-cert: this node’s report carries its SHA-256 as well as the current one’s, so the other orchestrators accept it as soon as it is served (ADR 0011 §13)

qfence-orchestrator api

Make one call to an orchestrator’s API with this binary’s own client.

The API accepts only the hybrid key exchange X25519MLKEM768, which curl built against OpenSSL before 3.5 lacks: this command is for scripts and labs whose curl cannot connect. It prints the answer’s body whatever its status, as curl does.

qfence-orchestrator api [flags] METHOD PATH
  • -ca-file string: the CA certificate that verifies the API’s certificate, or that certificate itself (default: the system’s roots)
  • -data string: the request’s body, JSON
  • -data-file string: file holding the request’s body, JSON; - reads standard input
  • -fail: a status of 400 or more is an error
  • -max-time duration (default 30s): how long the call may take
  • -server string: the orchestrator’s API, e.g. https://127.0.0.1:8443
  • -status: print the HTTP status on a line of its own after the body
  • -token-file string: file holding an API token (default: none)

qfence-orchestrator conformance

Run the conformance kit against an external application controller’s service, reached as qfence reaches it.

The service should run against a double that serves the conformance hooks; without them the checks that need one are skipped and the run is incomplete. The exit status is 1 when the controller does not pass.

qfence-orchestrator conformance [flags]
  • -cert string: the client certificate qfence presents (PEM)
  • -endpoint string: the service’s address, https://host:port
  • -json: print the report as JSON
  • -key string: its key (PEM)
  • -pin string: the SHA-256 fingerprint of the service’s certificate; without it the kit shows the fingerprint the service presents, for you to check and pass here
  • -timeout duration (default 10s): the longest one call may take

qfence-orchestrator secret

Write the secrets controllers name with credentials_ref, in the stores qfence owns, and manage their keys.

With -server a command goes through a running orchestrator’s secrets API, which opens again at once the controllers that use the secret. With -state-dir it writes the local store of an orchestrator that is stopped: the store is locked while the orchestrator runs.

qfence-orchestrator secret <command> ...

qfence-orchestrator secret put

Store a secret, as its next version.

qfence-orchestrator secret put [flags]
  • -ca-file string: with -server: the CA certificate that verifies the orchestrator’s API certificate, or that certificate itself (an orchestrator without -tls-cert writes its generated one to <state-dir>/api-cert.pem); default: the system’s roots
  • -master-key-file string: with -state-dir: the orchestrator’s -master-key-file
  • -ref string: the secret: local:NAME (this orchestrator’s store) or cluster:NAME (the cluster store, with -server); a bare NAME is local:NAME. E.g. lab-redfish for credentials_ref: local:lab-redfish
  • -scope-host value: an address a controller may send the secret to, as its configuration names it: a host, which allows any port of it, e.g. bmc-dc1.example.com, or host:port, which allows that port only, e.g. bmc-dc1.example.com:443 or [fd00::1]:8443; repeat it or separate entries with commas; * allows any host (required)
  • -scope-type value: a controller type that may use the secret, e.g. redfish; repeat it or separate types with commas; * allows any type (required)
  • -server string: a running orchestrator’s API, e.g. https://127.0.0.1:8443: write through it, with it running, instead of to -state-dir
  • -state-dir string: the orchestrator’s -state-dir (or -raft-data-dir when -state-dir is not set), with it stopped
  • -token-file string: with -server: file holding an Administrator’s API token
  • -value-file string (default -): file holding the secret, a JSON object of the controller’s auth method fields such as {“username”:"…",“password”:"…"}; - reads standard input

qfence-orchestrator secret delete

Delete a secret that no controller names.

qfence-orchestrator secret delete [flags]
  • -ca-file string: with -server: the CA certificate that verifies the orchestrator’s API certificate, or that certificate itself (an orchestrator without -tls-cert writes its generated one to <state-dir>/api-cert.pem); default: the system’s roots
  • -ref string: the secret: local:NAME or cluster:NAME; a bare NAME is local:NAME
  • -server string: a running orchestrator’s API, e.g. https://127.0.0.1:8443: write through it, with it running, instead of to -state-dir
  • -token-file string: with -server: file holding an Administrator’s API token

qfence-orchestrator secret revert

Make a secret’s previous version its current one again, as its next version.

qfence-orchestrator secret revert [flags]
  • -ca-file string: with -server: the CA certificate that verifies the orchestrator’s API certificate, or that certificate itself (an orchestrator without -tls-cert writes its generated one to <state-dir>/api-cert.pem); default: the system’s roots
  • -ref string: the secret: local:NAME or cluster:NAME; a bare NAME is local:NAME
  • -server string: a running orchestrator’s API, e.g. https://127.0.0.1:8443: write through it, with it running, instead of to -state-dir
  • -token-file string: with -server: file holding an Administrator’s API token

qfence-orchestrator secret versions

List the versions a store holds of a secret, with their scopes; never a value.

qfence-orchestrator secret versions [flags]
  • -ca-file string: with -server: the CA certificate that verifies the orchestrator’s API certificate, or that certificate itself (an orchestrator without -tls-cert writes its generated one to <state-dir>/api-cert.pem); default: the system’s roots
  • -ref string: the secret: local:NAME or cluster:NAME; a bare NAME is local:NAME
  • -server string: a running orchestrator’s API, e.g. https://127.0.0.1:8443: write through it, with it running, instead of to -state-dir
  • -token-file string: with -server: file holding an Administrator’s API token

qfence-orchestrator secret rekey

Wrap a store’s data keys again under the active key of its key file; no secret value is decrypted.

qfence-orchestrator secret rekey [flags]
  • -ca-file string: with -server: the CA certificate that verifies the orchestrator’s API certificate, or that certificate itself (an orchestrator without -tls-cert writes its generated one to <state-dir>/api-cert.pem); default: the system’s roots
  • -master-key-file string: the master key file, with the new key first and every key that still wraps a secret after it
  • -server string: a running orchestrator’s API, e.g. https://127.0.0.1:8443: write through it, with it running, instead of to -state-dir
  • -state-dir string: the orchestrator’s -state-dir (or -raft-data-dir when -state-dir is not set), with it stopped
  • -store string (default local): with -server: local rewraps this orchestrator’s store under the active key of its master key file; cluster moves the cluster secrets store under the active key of its -secrets-key-file, once every orchestrator reports holding that key
  • -token-file string: with -server: file holding an Administrator’s API token

qfence-orchestrator secret new-key

Print a new key line for a master key file (-master-key-file) or a secrets key file (-secrets-key-file).

Add it to the file, for example qfence-orchestrator secret new-key -id m1 » master.key, and keep the file’s mode 0600.

qfence-orchestrator secret new-key [flags]
  • -id string: key id, e.g. m1 or 2026-10

qfence-orchestrator pki

A minimal CA for one cluster’s raft mutual TLS, and for the client certificates orchestrators sign in to Vault with.

The CA key never leaves -dir: keep the directory offline. Each voter gets its certificate, its key and ca.pem; certificates are valid for 90 days. Keep the CA that issues Vault’s client certificates apart from the raft CA, so that no voter’s certificate, the witness’s included, can sign in to Vault.

qfence-orchestrator pki <command> ...

qfence-orchestrator pki init

Create the CA in -dir: ca.pem and ca-key.pem.

qfence-orchestrator pki init [flags]
  • -dir string: directory holding the CA (ca.pem, ca-key.pem) (required)

qfence-orchestrator pki issue

Issue a voter’s certificate and key, <name>.pem and <name>-key.pem, into -dir.

qfence-orchestrator pki issue [flags]
  • -client: a certificate valid as a TLS client only, naming no host, such as one an orchestrator signs in to Vault with (ADR 0011 §11); issue it from a CA of its own, not the raft CA
  • -dir string: directory holding the CA (ca.pem, ca-key.pem) (required)
  • -hosts string: comma-separated IP addresses and host names of the voter (informational: peers are verified by CA, not by address)
  • -name string: the voter’s node ID, used as the certificate’s common name and file name (required)

qfence-orchestrator pki revoke

Revoke a certificate: add it to revoked.json in -dir, and write the signed revocation list, crl.pem.

Publish crl.pem to the cluster with POST /api/v1/cluster/crl, or serve it for -raft-tls-crl-url; raft carries it to every voter.

qfence-orchestrator pki revoke [flags]
  • -dir string: directory holding the CA (ca.pem, ca-key.pem) (required)
  • -name string: the name the certificate was issued with: revokes <name>.pem in -dir
  • -serial string: the certificate’s serial number (hex), instead of -name

qfence-orchestrator pki crl

Sign crl.pem again, for a fresh validity period.

qfence-orchestrator pki crl [flags]
  • -dir string: directory holding the CA (ca.pem, ca-key.pem) (required)

qfence-orchestrator cluster-key

The cluster keyring every orchestrator in a cluster holds, and the witness never does (-cluster-key-file, ADR 0016 §1).

qfence-orchestrator cluster-key <command> ...

qfence-orchestrator cluster-key new

Print a new cluster key line: 256 bits from the operating system’s generator.

Add it to the file, for example qfence-orchestrator cluster-key new -id c1 » cluster.keys, and keep the file’s mode 0600. To rotate, add the new line to every orchestrator’s file, then put it first everywhere, then remove the old line once the voter list shows every report under the new key.

qfence-orchestrator cluster-key new [flags]
  • -id string: key id, e.g. c1 or 2026-10

qfence-orchestrator config-key

The keyring of the replicated configuration, which every orchestrator holds and the witness never does (-config-key-file, ADR 0001).

qfence-orchestrator config-key <command> ...

qfence-orchestrator config-key new

Print a new replicated-configuration key line.

Add it to the file, for example qfence-orchestrator config-key new -id k1 » config.keys, and keep the file’s mode 0600. To rotate, put the new line first in every orchestrator’s file, restart them one at a time, run POST /api/v1/config/rotate-key, then remove the old line.

qfence-orchestrator config-key new [flags]
  • -id string: key id, e.g. k1 or 2026-09

qfence-orchestrator help

Print a command’s help, or with -json every command’s flags, as the CLI reference shows them.

qfence-orchestrator help [flags] [command ...]
  • -json: print the description of the command and every command under it, as JSON
qfence-witness