Application controllers

Infrastructure controllers act on what an application runs on: its host’s power, its VM, its network. An application controller works through the application’s own interface instead (ADR 0005). It lets a plan:

watch its health, and restart it where it runs before failing it over;
check its readiness: is the standby’s copy recent enough to take over?
fence it at the application level: demote a primary, stop and mask a unit;
promote it at the new owner site;
switch it over gracefully, for a planned move.

Where it runs

Nothing of qfence’s runs on the application’s hosts. An application controller runs either:

in the site’s orchestrators, for the built-in types: systemd over SSH, PostgreSQL over SQL, Keycloak over HTTPS; or
as your own service, in any language, which the orchestrators call over the external controller protocol, with mutual TLS and your service’s certificate pinned (writing your own).

Either way, every command carries the plan’s generation, and qfence refuses a reply about an older one. A reply outside the protocol counts as UNKNOWN, never as success.

Health, restart, then failover

A plan binds a capability to each role it needs, site by site. From the demo lab’s billing plan, at dc1:

bindings:
  - {id: billing-active-dc1, controller: app-dc1, capability: unit.active, resource: billing.service, role: health}
  - {id: billing-restart-dc1, controller: app-dc1, capability: unit.restart, resource: billing.service, role: recovery,
     params: {max_attempts: 2, window: 10m, backoff: 5s, grace: 15s}}
  - {id: billing-stop-dc1, controller: app-dc1, capability: unit.stop, resource: billing.service, role: fence}
  - {id: bmc-app-dc1, controller: bmc-dc1, capability: power.off, resource: app-dc1, role: fence, required: true}

When a health binding at the owner site reads unhealthy for long enough, qfence restarts the application there through its recovery binding. When restarts do not hold, at most max_attempts within window, and the plan has failover.application_health with fail_over: true, it fails the plan over. Without that, a dead application at a live site is reported, not failed over. Automatic failover must also be on, for the orchestrators and for the plan.

Fences and promotion

An application-level fence is never what proves a site stopped. A demoted primary or a stopped unit is a secondary fence, run first so it can act while the host is still up; the compute or network fence after it is the proof. In the billing plan, unit.stop masks and stops the unit, then the BMC powers the host off. When the host is already unreachable, the application fence reads UNKNOWN, and the BMC proves the site on its own.

Promotion runs the other way: unfence the new site (power its host on), then start or promote the application there.

Failover and switchover

A failover assumes the old site may be gone: it fences, then promotes. A switchover is a planned move between two healthy sites, for maintenance. Its graceful strategy first demotes the old owner through its application-level fence and verifies it, then waits for the target to report zero lag. Only then does it apply the plan’s required fences and promote, as a failover does. A verified demotion never replaces the required fences (spec §19).

Plans that depend on each other

An application often needs another plan at the same site: Keycloak needs its database. depends_on with same_site lets the plan fail over only to a site where its dependency is ACTIVE. With follow: true, it also fails over by itself once its dependency is ACTIVE at another site (ADR 0015). A dependent left behind at a site where its dependency is no longer active is on the attention list.

Controllers and bindings Secrets