The attention list

Each plan has an attention list: what about it needs a person now, in words. The console shows it in the plan list and on the dashboard. Through the API, it is each plan’s attention in the table view, absent when there is nothing:

api GET "/api/v1/availability-plans?view=table"

From the demo lab, after erp-db failed over to dc2 and its old host was left off:

{"id": "web-portal", "owner": "dc1", "state": "ACTIVE",
 "attention": ["erp-db is ACTIVE at dc2; web-portal is at dc1: it does not follow: fail it over to dc2",
               "capability replication.lag of controller pve-dc1 is failing on dc1: its last call failed: ... node 'pve-dc1-01' is offline (powered off)"]}

The list restates state the plan already carries; nothing decides from it. It is for the person on call: each item below says what it means and what to do.

Fencing

Item	Meaning	What to do
`dc1 owes a fence`	A failover went ahead without durable proof that dc1 stopped. qfence keeps trying to fence it	Bring dc1’s fences back within reach (its BMCs, its network). A manual fence of dc1 that confirms it clears the debt (returning sites)
`dc2 is not fenced (drift)`	A site that does not own the plan was found running it, such as a host powered on by hand	Find out who powered it on and why. Fence it again, or mark it being prepared if the plan is moving back to it (forced operations)
`the owner dc1 is fenced: nothing serves`	The owner’s own fences read fenced: it fenced itself, or something stopped it	Recover the plan: promote the owner again, or another site (runbooks)
`the owner dc1 fenced itself, unverified: nothing may serve`	The owner reported self-fencing, but its fences do not answer, so nothing proves it stopped	Check the site by other means, then fail over with an attestation

Failover

Item	Meaning	What to do
`a failover waits for a second approval`	An attested failover, or one accepting data loss, needs another person within 10 minutes	Approve it, as a different person, or cancel it (forced operations)
`automatic failover suspended: its circuit breaker tripped`	Too many automatic failovers in the window: qfence stopped, so as not to flap	Find out why the plan kept failing over, then reset the breaker (failover)
`automatic failover is on, but the leader does not run it`	The plan asks for automatic failover, but the raft leader was started without `-automatic-failover`	Set `QFENCE_AUTOMATIC_FAILOVER=true` on every orchestrator: whichever leads runs it
`erp-db is ACTIVE at dc2; web-portal is at dc1: ...`	A plan that needs another at the same site was left behind when that one moved	Fail it over to the site named, or give its dependency `follow: true`

The application

Item	Meaning	What to do
`cannot see the application's health at dc1 (...)`	A health binding at the owner site does not answer	Check the controller and the host it reaches. While health cannot be read, qfence neither restarts nor fails over for it
`recovered 3 times in 1h at dc1: the application keeps failing`	Restarts keep bringing it back, and it keeps failing	Find out why: a failover may be next, if the plan allows one

Controllers and credentials

Item	Meaning	What to do
`capability power.off of controller bmc-dc1 is failing on dc1: ...`	A bound capability’s last call failed, on the orchestrators named. A fence through it would read UNKNOWN	Fix what the reason says: often a device that is off, unreachable, or refusing the credentials
`the credentials of controller ... expire at ...`	The controller’s credentials have an expiry	Renew them before then: `secret put` with the new value
`the service of controller ... presents a certificate that is not pinned`	An application controller’s service changed its certificate, so qfence sends it nothing	Check the certificate is really the service’s, then pin it on the controller’s page
`the certificate the service of controller ... presents expires at ...`	It will need renewing	Renew it, and pin the new one as the next certificate before the service changes to it
`... cannot be resolved on dc2: not found`	A binding’s credential does not resolve on that orchestrator	Write the secret there (`local`), or check the store it names (secrets)
`... did not resolve on dc2 (unavailable): the value resolved before is used ...`	The store did not answer; the last value is held until it expires	Bring the store back before then

Forced operations Runbooks