Runbooks

What to do when things go wrong. Every command here was run on the demo lab, through the scenario its simulator page offers: a site failed, then blacked out, then restored. api points at the raft leader (Operate).

A site is lost

1. Check the cluster from a surviving site. The surviving orchestrator and the witness still form a majority, and one of them leads:

   api GET /api/v1/cluster
   api GET "/api/v1/availability-plans?view=table"

2. With automatic failover on, the leader fails each plan with automatic_failover: true over by itself, once the lost site has had no reachable voter for primary_unhealthy_for_seconds. If the lost site’s fences answer (its BMCs are on a management network that survived), that is the end of it.

3. Without it, fail each plan over by hand: the dry run, then the failover.

4. If the plans end BLOCKED, the lost site’s fences did not answer: see below.

If the witness was lost with the site, there is no majority, and nothing fails over until one of the two comes back (the availability arithmetic).

A failover is BLOCKED

A plan at BLOCKED has stopped because something could not be proven. Nothing changes until a person acts.

1. Find out what. The audit record of the attempt carries the error, and a dry run says what still blocks:

   api GET "/api/v1/audit?plan_id=erp-db"
   api -data '{"target_site":"dc2"}' POST "/api/v1/availability-plans/erp-db/failover?dry_run=true"

   On the lab, after a blackout of dc1: fencing did not reach FENCED (overall=UNKNOWN): bmc-dc1: UNKNOWN. Nothing could prove dc1’s hosts were off.

2. Check the site by other means: the facility, the provider’s console, someone on site. Do not go on until you know it cannot run the application.

3. Release the plan, naming the site you checked:

   api -data '{"site":"dc1","reason":"dc1 checked by the facility: both hosts are off"}' POST /api/v1/availability-plans/erp-db/force-release

4. Fail it over again. If dc1’s fences answer now, an ordinary failover fences dc1 and goes through. If they still do not, attest them, once they have been failing for min_failing_seconds (60 on the lab):

   api -data '{"target_site":"dc2","attest_fenced":{"targets":["bmc-dc1"],"statement":"dc1 lost power at 03:12; the facility confirms both hosts are off"}}' \
     POST "/api/v1/availability-plans/erp-db/failover?dry_run=true"
   api -data '{"target_site":"dc2","confirmed":true,"attest_fenced":{"targets":["bmc-dc1"],"statement":"dc1 lost power at 03:12; the facility confirms both hosts are off"}}' \
     POST /api/v1/availability-plans/erp-db/failover

   The plan is ACTIVE at dc2, audited as an attested failover, and dc1 owes a fence is on its attention list. dc2’s shield is up.

A plan BLOCKED because its target’s replica is too old needs accept_data_loss instead of, or as well as, an attestation (the dry run). One BLOCKED by a dependency needs that plan moved first.

A site comes back

When the lost site’s power and network return:

• Its debts clear by themselves. The leader fences a site that owes a fence as soon as its fences answer. On the lab, dc1’s debt cleared within seconds of its BMCs answering again, audited as plan.fence.returning-site.

• Plans left BLOCKED by their own failed attempt during the outage, like the lab’s web-portal, are released and failed over as above. Now that the site’s fences answer, an ordinary failover goes through:

  api -data '{"site":"dc1","reason":"dc1 is back; its BMCs answer again"}' POST /api/v1/availability-plans/web-portal/force-release
  api -data '{"target_site":"dc2","confirmed":true}' POST /api/v1/availability-plans/web-portal/failover

• Drift (dc1 is not fenced (drift)) is reported for a site found running while it does not own the plan, such as hosts powered on by the restore. With on_drift: fence qfence fences it again; otherwise fence it yourself, or mark it being prepared if you are preparing it to take the plan back.

• To move the plan back, return the site to standby, wait for its replica, then fail it over to the site.

The owner fenced itself

the owner dc2 is fenced: nothing serves means the owner’s own fences read fenced: it fenced itself after losing quorum, or something else stopped it. Nothing runs the application. Once you know why, promote a site through the normal sequence, the owner again or another:

api -data '{"target_site":"dc2","reason":"dc2 checked: restart it"}' POST /api/v1/availability-plans/erp-db/recover

It is allowed only while the owner verifies as fenced, and fences the other sites, checks readiness and unfences the one you chose, as a failover does. the owner dc2 fenced itself, unverified cannot be recovered this way, since nothing proves it stopped: check the site, then fail over with an attestation.

The witness, or another voter, is down

With three voters, the other two still form a majority, so nothing changes, but the next failure does: a site lost while the witness is down leaves no majority. Bring it back soon. The witness’s status shows the voters it can see; on a site, GET /api/v1/cluster/voters lists each voter and whether it is reachable:

api GET /api/v1/cluster/voters

A witness that must be replaced is a new voter: remove the old one and add the new one, from the leader (the witness).

Quorum is lost

Without a majority, no plan changes owner, nothing is promoted, and the console’s actions are refused. Plans keep running where they are; self-fencing, where a plan has it, stops the owner after after_seconds. qfence has no command that overrides a lost majority: bring voters back until a majority can talk again.

Certificates and keys