Quorum, ownership and generations

Availability plans

An availability plan is one application and the sites it can run at, with what qfence does at each site: how to fence it, how to promote it, and how to tell whether it is healthy and ready to take over. At any time a plan has one owner, the site where the application runs, or none.

GET /api/v1/availability-plans lists each plan’s owner, generation and state:

{"id":"erp-db","owner":"dc2","generation":6,"state":"ACTIVE"}

Quorum

The sites’ orchestrators and the witness are the voters of one raft cluster. Raft elects a leader, and an entry is committed only once a majority of voters has stored it. Ownership changes only through committed entries, so a change needs a majority, and two sides of a partition can never both have one (spec §6–§7).

Without a majority, qfence fails closed: no site gains ownership and nothing is promoted, until the majority returns. A site cut off on its own cannot fail anything over to itself, however long it waits.

The orchestrators read ownership through the leader, with raft’s linearizable reads, before any decision: never from a cached copy, and never by trusting a clock (spec §10).

Generations

Every time a site acquires a plan, the plan’s generation goes up by one. It is a fencing token: every fence and every command qfence sends carries the generation it was issued under. Once generation 7 is committed, nothing sent under generation 6 is accepted as authoritative, so a command that was delayed in a partition and arrives late cannot undo a failover (spec §9, §40).

    sequenceDiagram
  participant L1 as Old leader, at dc1
  participant Q as dc2 and the witness
  participant H as A systemd host at dc1<br/>in dc2's reach
  Note over L1,H: The plan is ACTIVE at dc1, generation 6
  Note over L1,Q: dc1's orchestrator is cut off from the others
  Q->>Q: dc2 acquires the plan: generation 7
  Q->>H: fence: mask and stop (generation 7)
  H-->>Q: fenced, record 7
  L1->>H: restart (generation 6), delayed in the partition
  H-->>L1: refused: 6 is older than 7
  

The old leader cannot commit anything without a majority, and every orchestrator that carries out a command refuses one from a deposed leader. A target that records the generation, such as a systemd host or a PostgreSQL server, refuses a late command again on arrival.

The order of a failover

A failover always runs in this order (spec §19):

    flowchart TB
  subgraph prove["1. Prove the old owner stopped"]
    direction LR
    d["Dependencies<br/>ready"] --> q["Quorum"]
    q --> o["Ownership,<br/>new generation"]
    o --> f["Fence the<br/>other sites"]
    f --> v["Verify the<br/>fences"]
  end
  subgraph promote["2. Promote the target"]
    direction LR
    r["Check it<br/>is ready"] --> s["Raise its<br/>shields"]
    s --> u["Unfence it"]
    u --> p["Promote the<br/>application"]
    p --> svc["Move the<br/>service"]
  end
  prove --> promote
  promote --> a(["ACTIVE"])
  prove -.->|a fence not proven| b(["BLOCKED"])
  promote -.->|a step fails| b
  

Under each site’s fences, its secondary application-level fences run first, then the required ones. Unfencing the target powers its hosts on and starts its VMs; moving the service enables its virtual IP or route.

Never quorum, then promote, then fence. A plan becomes ACTIVE at a site only when that site has quorum and owns the latest generation, the old owner is verified fenced, and every check before promotion has passed (spec §12). What has to be proven, and how, is in fencing.

States

The state machine also names DEMOTING, FAILED and UNKNOWN (spec §11); qfence does not put a plan in them today.

BLOCKED is where anything unsafe or ambiguous ends: a fence that cannot be verified, a replica too stale to promote, a dependency not where it must be. Nothing leaves BLOCKED by itself. A person checks the site qfence could not, then releases the plan and fails it over (Operate). qfence prefers being unavailable to guessing.