Secrets

Controllers need credentials: a BMC’s password, a FortiGate’s API token, an SSH key. None of them is ever in the configuration file, in raft in readable form, or in an API response. A controller names its credential by reference, and each orchestrator that calls the controller resolves the reference itself (ADR 0011):

auth:
  method: basic
  credentials_ref: local:bmc-dc1

    flowchart LR
  ref["A controller's<br/>credentials_ref"] --> orch["Each orchestrator<br/>at its site and reach"]
  orch -->|local| ls[("Its own store<br/>under its master key")]
  orch -->|cluster| cs[("The shared store<br/>sealed, in raft")]
  orch -->|"vault, aws"| ext[("Its site's Vault<br/>or AWS")]
  ls --> scope{"The secret's scope:<br/>this type, this address?"}
  cs --> scope
  ext --> scope
  scope -->|yes| ctrl["The controller<br/>opens with it"]
  scope -->|no| no["Refused: the controller fails"]
  

The stores

A reference starts with the store it is in. One orchestrator can use several stores at once, and asks only the one a reference names:

Vault and AWS are defined per orchestrator, in its -secrets-config file, each with the site it serves. So one reference resolves through each site’s own Vault, and a site cut off from the other keeps reading its own.

Who can read what

• The witness never can. Values in the cluster store are sealed with AES-256-GCM under a keyring only the orchestrators hold; each entry’s name is replaced by an HMAC of it. The witness stores ciphertext and learns neither names nor values, only sizes and timing. No key or value crosses a channel whose key exchange is classical only, so that ciphertext stays unreadable to a future quantum computer too.
• Every orchestrator can read every cluster secret, every site’s. Keep a credential that only one site uses, and whose exposure at the other site would matter, in that site’s local store or its own Vault.
• A secret has a scope: the controller types, and the hosts or host:ports, allowed to use it. A controller of another type, or one that would send it to another address, is refused it (ADR 0016 §6).

Writing one

qfence writes only to the stores it owns, local and cluster. While the orchestrator runs, with the value as a JSON object of its fields on standard input, such as {"username":"qfence","password":"..."}:

qfence-orchestrator secret put -server https://127.0.0.1:8443 -token-file admin.token -ca-file api-cert.pem \
  -ref local:bmc-dc1 -scope-type redfish -scope-host bmc-dc1.example.com < creds.json

stored secret local:bmc-dc1 (version 1, scope types=redfish hosts=bmc-dc1.example.com) through https://127.0.0.1:8443; refer to it with credentials_ref: local:bmc-dc1

A local secret goes into each orchestrator that calls the controller, with -server naming each one’s API in turn; -ref cluster:bmc-dc1 writes one into the shared store, once, through any orchestrator. -ca-file is the API certificate’s CA or, for the certificate an orchestrator generates without -tls-cert, the copy it writes to api-cert.pem in its state directory (/var/lib/qfence-orchestrator/ when installed with systemd). With the orchestrator stopped, -state-dir /var/lib/qfence-orchestrator -master-key-file /etc/qfence-orchestrator/master.key in place of -server writes its local store directly. The console’s Secrets page does the same, with a field that is never filled in and never read back. A write takes effect at once: the controllers that name the secret are opened again with it. Each store keeps one previous version, and secret revert puts it back. A secret a controller still names cannot be deleted.

When a store fails

What a controller keeps depends on how the store failed:

• It does not answer: the controller keeps the value it last resolved, until that value’s own expiry, and reads as degraded meanwhile.
• It answers that the secret is gone, or refused: the value is dropped at once, the controller fails, and a fence through it reads UNKNOWN.

The dry run and the plan’s attention list show, for each binding and each orchestrator, whether its secret resolves, and warn when it resolves only through a store at another site. They only report: no failover decision reads them.