Manual fence and forced operations

A forced operation acts outside the usual order of quorum, ownership, fence, promote. Each needs the force permission (EmergencyOperator or Administrator), takes a reason, and is audited as forced, with the reason, so the audit log can list them all. The console’s Audit screen filters to them in one click. In the console each is in a red frame, names the guarantee it bypasses, and asks you to type the plan’s id, with paste blocked.

Manual fence

A manual fence fences a site now, at the plan’s current generation. Without a site it fences the current owner: on an ACTIVE plan, the site serving the application stops, and nothing takes over. With site, it fences that site’s own targets, and it must not be the owner:

api -data '{"site":"dc2","reason":"dc2 was powered on by hand during maintenance"}' POST /api/v1/availability-plans/erp-db/fence

It answers with the overall result and each target’s, including whether the fence is durable. A site that owed a fence and is now confirmed fenced no longer owes it (returning sites). There is no dry run for a fence: the console asks for the reason, at least 10 characters, and the plan’s id instead.

Force release

A plan at BLOCKED stays there until a person releases it. Force release gives up the owner’s claim without fencing anything, so a failover can then run:

api -data '{"site":"dc1","reason":"dc1 checked by the facility: both hosts are off"}' POST /api/v1/availability-plans/erp-db/force-release

Release only once you have checked, by other means, that the site cannot run the application. The failover that follows still fences what it can reach; a site whose fences do not answer needs an attestation as well (runbooks). Force acquire, which makes a site the owner without a failover, is in the API too, for recovery from a broken state; no console offers it.

Attested failover and accepted data loss

A failover that carries attest_fenced or accept_data_loss is forced as well: see the dry run and fencing. With approvals: 2 in the plan, it waits for a second person, for at most 10 minutes:

api GET /api/v1/availability-plans/erp-db/failover/requests

lists the requests waiting. The second person approves one on the plan’s page, or with POST .../failover/requests/{id}/approve. Approvals compare identities, not people: one person with both an SSO login and a break-glass token counts as two.

A site being prepared

Preparing a site to take a plan back can mean powering on what qfence fenced there, so that its replica catches up. With on_drift: fence, qfence would fence it again within a check. Mark it being prepared first, for a stated time of at most 24 hours:

api -data '{"minutes":120,"reason":"powering dc2 on to resync its replica before moving the plan back"}' POST /api/v1/availability-plans/erp-db/sites/dc2/prepare

While the mark lasts, drift at that site is neither alerted on nor fenced. It never applies to a site that owes a fence. DELETE on the same path ends it early.

Return to standby

Return to standby is not forced: the site stays fenced throughout. It brings a fenced site back to a standby that a later failover can promote, as returning sites describes:

api -data '{"reason":"dc2 repaired"}' POST /api/v1/availability-plans/erp-db/sites/dc2/standby

It answers {"site":"dc2","confirmed":true,"standby_ready":true} once the site is fenced durably and its readiness passes.

It needs the Operator role, and is refused for the owner, for a site found running its workload, and while a failover runs.