Returning sites

Returning sites, shields and self-fencing

A failover is not the end of the story for the site that was given up. Its power comes back, its network heals, and whatever ran there may start again. If that site was fenced by an attestation, or only transiently, nothing yet proves it will stay down. This page is what qfence does about that (ADR 0008, ADR 0009).

    sequenceDiagram
  participant L as Leader, at dc2
  participant S as dc2's shield<br/>a firewall policy
  participant B as dc1's BMC
  Note over B: dc1 is dark
  L-xB: power off (generation 7): no answer, UNKNOWN
  Note over L: An operator attests that dc1 is down
  L->>S: raise
  Note over L: ACTIVE at dc2, and dc1 owes a fence
  loop Every 5 s while dc1 does not answer
    L-xB: power off
  end
  Note over B: dc1's power returns
  L->>B: power off
  B-->>L: Off, restore policy AlwaysOff: durable
  Note over L: dc1 is confirmed fenced: the debt clears
  L->>B: check, twice in a row
  B-->>L: FENCED
  L->>S: lower

Sites that owe a fence

A site owes a fence when a failover went ahead without a durable proof that it stopped: it was attested, or fenced only transiently. The debt is kept in raft, so a change of leader does not forget it.

The leader keeps trying. It fences that site’s own targets at the current generation, every 5 seconds while they do not answer, and backs off to every 60 seconds after a real attempt fails.
Only a proof clears it: the site confirmed fenced, by these attempts or a manual fence, or the site promoted again through a failover. There is no waiver.
It is visible. The plan’s fencing health is DEGRADED, and its attention list says dc1 owes a fence.

The leader only ever fences a site that does not own the plan.

Shields

The returning site’s workloads may start before its fence endpoints answer. A shield protects the survivor in that window: a fencing target at the site that took over, raised there, such as a firewall policy at dc2’s edge that drops dc1’s traffic. It needs nothing from the dark site, so it works exactly when fencing cannot.

A shield is a target with role: shield, at the site it protects. Raising it is its fence; lowering it is its unfence.
mode: unproven, the default, keeps it up while any other site of the plan is not confirmed fenced: one that owes a fence, or was last seen running, or could not be checked. It comes down only after two checks in a row confirm every other site fenced, so a flaky BMC does not make it flap.
mode: always keeps it up for as long as its site owns the plan.
A required shield that cannot be raised blocks the promotion. An attested failover is refused unless the promoted site has a shield.

A shield is not a fence. It never counts toward the old site’s verdict, and it does not stop the old site serving its own clients or its data diverging. It must also let through what the old site needs to become a standby, such as its replica connecting out to the new primary: a shield that is too broad means the plan can never move back to that site. qfence cannot see network paths, so getting that right is yours. The dry run lists the shields a failover raises.

Continuous checks

The leader reads every non-owner site’s fences, every 60 seconds by default (fencing.recheck_seconds):

FENCED: nothing to do.
UNKNOWN: the site could not be read. A dark site reads this way, so it is noted, not alarmed on.
Not fenced is drift: the plan’s fencing health goes DEGRADED, and fencing.on_drift decides the rest. With alert, the default, that is all. With fence, the leader fences the site again.

alert is the default because some fences rule out preparing a site to take a plan back: if the fence powers a site’s hosts off, re-fencing on drift would switch them straight off when an operator powers them on to catch up. An operator can mark a site being prepared for a stated time, which suppresses the drift alert, audited. Fences that keep a site useful as a standby, such as a database in its standby shape, suit fence.

Self-fencing

A site’s orchestrator is a raft voter, so it knows when it can no longer be part of a majority. With self-fencing on, a site that has lost quorum for after_seconds fences its own targets, for every plan it owns, without waiting for anyone:

fencing:
  self_fence:
    enabled: true
    after_seconds: 60      # how long without quorum before the site fences itself
    backs_failover: true   # lets the survivor fail over without an attestation

At start-up too: an orchestrator that cannot reach quorum within after_seconds fences its own site for the plans it last knew it owned. A site back from a blackout while still cut off stops its workloads, even if they started first.
Dead-man switches close the gap a dead orchestrator leaves. A BMC watchdog fences the host unless it is renewed; the owner renews it while it holds quorum, and stops when it loses quorum or dies.
A failover backed by self-fencing. With backs_failover, the survivor may fail over from an owner it cannot reach without an attestation, once the owner’s last report said self-fencing was armed and long enough has passed for it to have fenced itself. The old site then owes a fence, and the survivor’s shields stay up until it is confirmed.

The cost: if the witness is down and the link between the sites fails, both sites lose quorum, and with self-fencing the owner stops. Without it the owner keeps running, though nothing could fail over either. after_seconds trades fast backed failovers against false stops.

Back to standby

A fenced site becomes a standby again through return to standby: qfence raises the owner’s shields, releases the site’s power fence, fences it again into its standby shape, and checks its readiness. The site is then standby-ready, and a later failover can promote it. It is never unfenced as an owner: only promotion does that.

A workload can also ask before it starts in its active role, with a Viewer token:

api GET "/api/v1/availability-plans/erp-db/lease?site=dc1"

{"plan":"erp-db","site":"dc1","known":true,"active":false,"owner":"dc2","generation":6,"state":"ACTIVE","answered_by":"dc2","detail":"dc2 holds the plan, not dc1"}

One that gets no answer does not start active. That closes the window between a site’s power returning and qfence fencing it, for workloads that ask.

Fencing Controllers and bindings