Failover and switchover

Two ways to move a plan to another site, each preceded by its dry run:

	When	What it does
Failover	The owner site is lost, or must be given up now	Fences the old site, then promotes the target
Switchover	A planned move between two healthy sites	Demotes the old owner first and waits for the target to catch up, then fences and promotes

A plan has no home site. Moving it back to a site it left is a failover or a switchover to that site, like any other (below).

Each needs the Operator role and an explicit confirmation, and is audited. Each runs on the raft leader.

Failover

In the console: the plan’s page, Failover, the target site, then the dry run, then the fence warning to acknowledge, then Execute. Through the API, the dry run, then the same request with "confirmed": true:

api -data '{"target_site":"dc2"}' POST "/api/v1/availability-plans/erp-db/failover?dry_run=true"
api -data '{"target_site":"dc2","confirmed":true,"reason":"dc1 maintenance window"}' POST /api/v1/availability-plans/erp-db/failover

It answers with the plan’s new state, {"id":"erp-db","owner":"dc2","generation":2,"state":"ACTIVE"}, once dc1 is fenced and verified, dc2’s replica is checked, dc2 is unfenced and promoted, and the service has moved. The reason is optional, and kept in the audit record. If any step cannot be proven, the answer is a 409 with the error and the state, and the plan is BLOCKED (runbooks).

A plan that depends on another at the same site can only fail over to a site where that plan is ACTIVE. The dry run’s dependents say which plans would be left behind, and whether they follow.

Switchover

A plan whose target site has a switchover binding can be switched over: the old owner’s application is demoted through its application-level fence and verified, qfence waits for the target to report zero lag, and only then are the old site’s required fences applied and the target promoted. So the target has everything the old owner wrote, as the application reports it. From the demo lab’s orders-db:

api -data '{"target_site":"dc2"}' POST "/api/v1/availability-plans/orders-db/switchover?dry_run=true"
api -data '{"target_site":"dc2","confirmed":true}' POST /api/v1/availability-plans/orders-db/switchover

The dry run’s switchover section names the strategy, the demotion, and how long it will wait for the target to catch up (readiness_timeout_seconds) before giving up at BLOCKED. A switchover that has started runs to its end, even if the caller goes away: stopping half way would leave the plan BLOCKED.

Moving a plan back

To move a plan back to a site it left, fail it over, or switch it over, to that site. First make the site ready: its hosts on again, the site back in standby, its replica caught up. A site qfence fenced durably stays off until someone powers it on. While it catches up it runs without owning the plan, which is drift; mark it being prepared if the plan fences on drift.

api -data '{"target_site":"dc1"}' POST "/api/v1/availability-plans/erp-db/failover?dry_run=true"
api -data '{"target_site":"dc1","confirmed":true,"reason":"dc1 restored, replica caught up"}' POST /api/v1/availability-plans/erp-db/failover

The dry run checks the site as it would any target: its replica’s readiness, and the fences of the site being given up.

Automatic failover

With automatic failover on, for the orchestrators (-automatic-failover) and for the plan (automatic_failover: true), the raft leader fails a plan over by itself once its owner site has had no reachable voter for primary_unhealthy_for_seconds. The plan’s detail shows what the leader’s scheduler sees, under automatic:

api GET /api/v1/availability-plans/erp-db

"automatic": {"running": true, "primary_unhealthy_for_seconds": 20, "attempts": 0, "max_attempts": 3, "window_seconds": 3600}

A circuit breaker stops it from flapping: after max_attempts automatic failovers within window_seconds, it trips, and automatic failover of that plan is suspended until a person resets it. The reason is optional, and kept in the audit record:

api -data '{"reason":"dc1 power fixed, checked by facilities"}' POST /api/v1/availability-plans/erp-db/circuit-breaker/acknowledge

Automatic failover never attests a fence and never accepts data loss. When it cannot prove a site fenced, it stops at BLOCKED like any other failover, and a person takes over.

The dry run Forced operations