Writing an application controller

This guide is for people who write an application controller for qfence: a service that checks one application’s health and readiness, fences it at the application level, restarts it and promotes it, on qfence’s orders. It is enough to build one that passes qfence’s conformance kit, in any language, without reading qfence’s source.

The design behind it is ADR 0005. The protocol is application-controller.v1.openapi.yaml, whose messages are application-controller.v1.json; the manifest is controller-type.v1.json.

1. What an application controller is

qfence keeps one copy of a protected resource active across sites. It decides who owns an availability plan, by quorum, and fences the sites it gives up before it promotes another. Infrastructure controllers act on what the application runs on: hosts (Redfish, IPMI), VMs (Proxmox), containers (Docker), networks (FortiGate).

An application controller acts on the application itself, through its own interfaces: a database, an identity server, a service. It can:

It is not a provider of infrastructure, and it never decides anything. qfence’s leader decides; the controller does one thing when asked and reports what it observes. An application-level fence never counts as proof that a site has stopped: qfence still applies the plan’s compute or network fences, and a demotion that reports FENCED never replaces them.

2. Built in, or external

qfence builds in the application controllers it ships: systemd, over SSH (The systemd application controller); PostgreSQL, over SQL (The PostgreSQL application controller); and Keycloak, over HTTPS (The Keycloak application controller). Your own is an external controller: your service, in any language, run wherever you like, which qfence calls over the external controller protocol: HTTPS and JSON with mutual TLS. No code of yours runs inside qfence.

qfence’s orchestrators call it from the sites you allow (the controller’s site and reach). If those orchestrators are down, nothing calls it, and its fences count as UNKNOWN; that is intended.

3. The manifest

Your service describes itself with a manifest, served at GET /v1/manifest. qfence fetches it when the controller is registered, keeps a copy and records its fingerprint. The controller then advertises that manifest’s capabilities. If your service later serves a different manifest, the controller is failing until an administrator registers it again; so change a manifest on purpose, never by accident (no timestamps or build ids in it). qfence decodes the manifest and fingerprints its own encoding of it, so whitespace and key order do not matter; any change of a value does.

A complete manifest:

{
  "schema": "qfence.controller-type/v1",
  "type": "orders-db",
  "kind": "application",
  "version": "1",
  "display_name": "Orders database",
  "runs_at": ["external"],
  "connection": [],
  "auth": [{"method": "mtls"}],
  "identity": "the database's system identifier",
  "capabilities": [
    {"id": "db.health", "category": "observation", "roles": ["health"], "resource": "database",
     "description": "HEALTHY while the database accepts queries"},
    {"id": "db.lag", "category": "observation", "roles": ["readiness"], "resource": "database",
     "params": [{"name": "max_replication_lag", "type": "duration", "default": "30s"}]},
    {"id": "db.restart", "category": "actuator", "roles": ["recovery"], "resource": "database",
     "checks": ["restart.allowed"],
     "policies": [
       {"name": "max_attempts", "type": "int", "default": 3},
       {"name": "window", "type": "duration", "default": "10m"},
       {"name": "backoff", "type": "duration", "default": "30s"},
       {"name": "grace", "type": "duration", "default": "60s"},
       {"name": "flag_after_episodes", "type": "int", "default": 3},
       {"name": "flag_period", "type": "duration", "default": "24h"}
     ]},
    {"id": "db.demote", "category": "actuator", "roles": ["fence"], "level": "application",
     "resource": "database", "features": ["durable"]},
    {"id": "db.promote", "category": "actuator", "roles": ["switchover"], "resource": "database",
     "params": [{"name": "max_replication_lag", "type": "duration", "default": "30s"}],
     "strategies": [{"id": "graceful"}, {"id": "fence-then-promote"}]},
    {"id": "databases.list", "category": "discovery", "resource": "database"}
  ]
}

The rules qfence holds an external manifest to (anything else is refused at registration, with the reason):

• schema is qfence.controller-type/v1; kind is application; runs_at is exactly ["external"].
• connection is empty and auth is exactly [{"method": "mtls"}]: how qfence reaches you is the controller’s own connection, and qfence sends you no secret.
• type is a lower-case name (^[a-z][a-z0-9-]{0,63}$): your service’s name. Capability ids are <noun>.<verb>, lower case (db.demote, databases.list): the noun is yours to choose, and need not be the resource kind.
• Every capability has a category: observation (roles health, readiness), actuator (roles fence, recovery, switchover, scale), or discovery (no roles). No other role is accepted. scale is shown to operators only: it has no action and is never bound.
• A fence capability has "level": "application"; no other level.
• A recovery capability declares exactly the six policy fields above, in that order, with those types; you choose the defaults. qfence holds every binding to its own limits: max_attempts 1–10, window 1m–24h, backoff 1s–1h, grace 0–1h, flag_after_episodes 2–100, flag_period 1h–30d.
• A switchover capability lists one or both strategies, graceful and fence-then-promote, and declares no param called strategy or readiness_timeout (every switchover binding has those).
• No capability takes settings or credentials of its own (config, auth, setup other than validation; leave setup out): your service holds its own.
• params are what each binding sets; they reach you with every command, so never put a secret in one. Field types are string, int, bool, duration, list, object and pem. A duration is a string of decimal numbers, each with a unit (ns, us or µs, ms, s, m, h), such as 30s, 1h30m or 0.
• checks name the checks validate grades for the capability.
• resource is the kind of thing a binding names ("database"); a discovery capability lists things of that kind.

4. The protocol

All calls are HTTPS with mutual TLS (section 7), HTTP/1.1 or HTTP/2, TLS 1.2 or later, JSON bodies, Content-Type: application/json (qfence sends it; it does not check yours, but send it).

A registration, then a failover away from dc1, as your service sees them:

    sequenceDiagram
  participant O as Orchestrator at dc1
  participant S as Your service
  Note over O,S: Registration
  O->>S: GET /v1/manifest
  S-->>O: the manifest
  O->>S: POST /v1/actions/test
  S-->>O: identity, version
  O->>S: POST /v1/actions/validate
  S-->>O: a grade for each check
  Note over O,S: A failover away from dc1, generation 42
  O->>S: POST /v1/actions/fence, token 42
  S-->>O: state FENCED, token 42
  O->>S: POST /v1/actions/verify, token 42
  S-->>O: state FENCED, durable
  Note over O,S: A command already on its way from before the failover
  O->>S: POST /v1/actions/recover, token 41
  S-->>O: 409 stale_token
  

The command

Every POST /v1/actions/{action} carries:

{
  "command_id": "5f0c2e1a9b7d4c3e8a6f1b2c3d4e5f60",
  "token": 42,
  "deadline": "2026-10-09T10:15:30.250Z",
  "controller": "orders-dc1",
  "plan": "orders",
  "binding": "orders-demote-dc1",
  "capability": "db.demote",
  "resource": "orders-primary",
  "site": "dc1",
  "params": {"max_replication_lag": "30s"}
}

command_id, token and deadline are always present. discover carries capability; validate carries capabilities; test carries neither. Refuse unknown fields if you like; qfence sends none.

params arrive exactly as the plan’s binding sets them: qfence does not fill in your manifest’s defaults, so apply your own. A duration param arrives as a string (30s), or as a number of seconds if the plan wrote one. A recovery binding’s params also carry its recovery policy, and a switchover binding’s its strategy and readiness_timeout: those are qfence’s to act on, so ignore them.

The replies

A successful reply is HTTP 200 with a JSON object that starts with the echo of the command and carries the action’s answer. qfence decodes it strictly: a field it does not know, a value outside the enumerations, a body over 65536 bytes, trailing data, or a command_id or token that is not the command’s, and the reply counts as no reply (UNKNOWN for a fence or a verify, not ready, no health signal).

check in detail: qfence promotes a site only when ready, reachable, healthy and replica_current are all true, so ready alone never promotes.

A copy that is stopped is reachable: true, healthy: false, ready: false. A demoted copy can be ready: a standby normally is demoted, since qfence demotes the site it gives up, and a failover back to it must find it ready. A copy you cannot read is reachable: false, ready: false, with no lag_seconds. Whenever you measured the lag, report it; when you did not, leave it out and answer ready: false (a check that measures no lag at all leaves it out always). A graceful switchover waits for a measured lag of exactly 0.

Errors

Anything that is not a 200 is a refusal or a failure:

{"command_id": "5f0c2e1a…", "error": {"code": "stale_token", "message": "token 41 is older than 42, the highest seen for plan orders"}}

qfence treats every error as the call failing: UNKNOWN for a fence or a verify, not ready for readiness, no signal for health. It shows the message (at most 300 characters) to operators, so make it say what happened.

Limits qfence enforces

• Deadlines. Every call has one: the controller’s timeout (10 s unless set), shortened by the plan’s fence timeout. qfence gives up at the deadline.
• Size. 65536 bytes, request and reply.
• Concurrency. At most max_concurrent_calls (4 unless set) calls in flight to one controller; more wait within their own deadline.
• Tokens. qfence’s executors refuse a command from a deposed leader before it reaches you; your own refusal of older tokens is a second line.

Tokens

The generation token grows every time a plan changes owner (qfence’s specification calls this rule §40, and the kit’s report cites it). Keep, per plan, the highest token you have seen in a fence, unfence, recover or switchover, and refuse those actions with an older token (stale_token). The same token again is allowed: make the action idempotent for it. These four always carry a token of at least 1; refuse 0 (bad_request).

An unfence carries the generation that decided it: the new owner’s when qfence promotes your site, the plan’s current one when an operator lowers a fence by hand. So a late unfence from a leader that has since been replaced meets the newer leader’s fence token and is refused, instead of undoing that fence.

Record a token once the command is otherwise in order, before you act on it: a later refusal (not_ready, say) does not undo it. verify carries the plan’s token but is never refused for it and does not raise it. health and check carry 0 and are not refused. qfence’s own executors refuse a command from a deposed leader, and an unfence older than the plan’s generation, before it reaches you; your refusal covers what was already on its way. Keep the highest tokens durably if you can, so a restart does not forget them.

5. Implementing it

With the Go SDK

go.ivnn.dev/qfence/pkg/sdk serves the protocol from a Go interface. It checks every command before your code sees it (fields, the capability and its role, the deadline, stale tokens), echoes the command, refuses to send an answer outside the protocol, and serves the conformance hooks when you give it a double.

type Orders struct {
	sdk.Unsupported // every action you do not override answers unsupported
	db *sql.DB
}

func (o *Orders) Test(ctx context.Context) (sdk.Identity, error) {
	return sdk.Identity{Identity: "orders:" + systemID(ctx, o.db)}, nil
}

func (o *Orders) Health(ctx context.Context, cmd sdk.Command) (sdk.HealthStatus, error) {
	if err := o.db.PingContext(ctx); err != nil {
		return sdk.HealthStatus{Health: sdk.Unknown, Reason: err.Error()}, nil
	}
	// … read the application's own state, and say HEALTHY or UNHEALTHY.
}

func (o *Orders) Fence(ctx context.Context, cmd sdk.Command) (sdk.FenceStatus, error) {
	// Demote cmd.Resource, then read back whether it is demoted.
	return o.Verify(ctx, cmd)
}

handler, err := sdk.Handler(manifest, &Orders{db: db}, sdk.Options{})
tlsConfig, err := sdk.ServerTLS(certPEM, keyPEM, qfenceClientCA)
srv := &http.Server{Addr: ":9443", Handler: handler, TLSConfig: tlsConfig}
log.Fatal(srv.ListenAndServeTLS("", ""))

sdk.Errorf(sdk.CodeNotReady, "lag %d s", lag) chooses an error’s code; any other error is internal. sdk.Options.Tokens takes your own durable sdk.TokenStore; the default keeps tokens in memory.

examples/memapp-controller is a complete controller: an application in memory (go.ivnn.dev/qfence/pkg/sdk/memapp), served with mutual TLS, with its conformance hooks behind -double. Read memapp for the shape of every action.

examples/kvstore-controller-python is one in Python, standard library only, written from this guide and the schemas alone, which passes the kit.

In another language

Implement section 4 from the OpenAPI document. A checklist:

1. Serve HTTPS with a certificate qfence can pin, and require a client certificate signed by the CA you trust for qfence (section 7).
2. GET /v1/manifest returns the manifest, byte-for-byte stable.
3. POST /v1/actions/{action}: parse the command; refuse a body over 65536 bytes, a missing command_id or deadline, a passed deadline; check the capability is yours and has the role the action needs (health → health, check → readiness, fence/verify/unfence → fence, recover → recovery, switchover → switchover, discover → a discovery capability); refuse stale tokens; act; answer with the echo.
4. Answer within the deadline, and keep every reply under 65536 bytes.
5. Serve the conformance hooks while you test (section 9).

6. Safety rules

qfence’s safety rests on these. The conformance kit checks each.

• Read state, never remember it. verify and health report what you read from the application now, never what you sent. If someone undoes a demotion by hand, verify must stop saying FENCED.
• Never claim what you did not observe. No FENCED, HEALTHY or ready without having read it. When you cannot tell, answer UNKNOWN (or not ready): qfence treats UNKNOWN as unsafe, which is what it should do.
• Idempotent for one token. A fence, unfence or recover sent twice with the same token does the same thing twice, harmlessly, and answers the same. qfence may resend a command whose answer it lost. A second recover may restart again, or answer DONE without restarting while the application runs: either is harmless.
• Refuse older tokens. Section 4, “Tokens”.
• Durable means durable. durable: true on FENCED says the fenced state survives a restart of what you fenced (a demotion kept in the database’s configuration, a unit masked). When unsure, say false: qfence then treats the site as fenced only until it restarts.
• Promote only when ready. A switchover refuses (not_ready) while the copy is not ready to be primary by your own measure (stopped, unreadable, or lagging past the binding’s max_replication_lag). The same promotion serves every failover to the site, not only planned switchovers, so never require more than that: qfence’s wait for zero lag in a graceful switchover is qfence’s, done before it sends the promotion, and a failover after losing a site promotes a copy that may lag a little. When the command carries accepted_lag_seconds, an operator accepted losing that much for this failover, and qfence has checked that it covers the lag your readiness reported: promote a copy whose lag is within the larger of your limit and that. A demoted copy may be promoted: promotion ends the demotion. A copy that is already primary, and ready, answers DONE.
• Recover only what you were asked. A recover restarts the application at the binding’s resource and nothing else. qfence sends it only to the owner site of an ACTIVE plan.
• Stay within the deadline. Work that cannot finish by the deadline is better refused than left running.

An application-level fence is defence in depth. It never proves a site stopped, never confirms a site, and a plan always needs a required fence at another level for every site. Marking one required makes it one more condition for promotion; since it usually cannot answer for a site that is down, losing that site then ends a failover BLOCKED.

7. Security

• Mutual TLS. qfence presents a client certificate; require it, and trust only the CA that signed it. qfence’s controller names the certificate and key by a secret reference (credentials_ref), as every controller’s credentials are.
• Pinning. qfence pins your service’s certificate: the SHA-256 of its DER encoding, recorded at registration. A self-signed certificate is fine. A service presenting any other certificate is sent nothing. To replace the certificate without a gap, pin the new one as the next certificate first (section 8): qfence trusts both until you make it the pinned one.
• Your own credentials. qfence sends you no secret, ever: no password, no token in params. Your service holds its own credentials for the application, and is responsible for keeping them.

A lab set of certificates with OpenSSL:

# The CA that signs qfence's client certificate, and the certificate.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 365 \
  -subj "/CN=qfence client CA" -keyout client-ca-key.pem -out client-ca.pem
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -subj "/CN=qfence" -keyout client-key.pem -out client.csr
printf "extendedKeyUsage=clientAuth\n" > client.ext
openssl x509 -req -in client.csr -CA client-ca.pem -CAkey client-ca-key.pem \
  -CAcreateserial -days 365 -extfile client.ext -out client.pem

# Your service's own certificate: self-signed, since qfence pins it.
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 365 \
  -subj "/CN=orders-controller" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
  -keyout server-key.pem -out server.pem

# The fingerprint qfence pins (colons and case do not matter).
openssl x509 -in server.pem -noout -fingerprint -sha256

Your service uses server.pem, server-key.pem and trusts client-ca.pem; qfence presents client.pem and client-key.pem.

8. Registering it

qfence’s credentials for your controller are the client certificate and key, stored on every orchestrator that will call it:

printf '{"tls_cert":%s,"tls_key":%s}' "$(jq -Rs . client.pem)" "$(jq -Rs . client-key.pem)" |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref orders-ctrl \
    -scope-type external -scope-host orders-ctrl.example.com

-scope-type and -scope-host are the controller types and the addresses that may use the secret: here the external controller type, at the host of the service’s url, which allows any port of it; orders-ctrl.example.com:8443 would allow that port only (ADR 0016 §6). Any other controller that names it is refused it (ADR 0011 §8).

Run it against each of those orchestrators while it runs (writing a secret): the controllers that use the client certificate and key pick it up at once.

Then, in the console, Controllers → Application → Register, type External application controller:

1. Type and connection. An id (orders-dc1), its site (only orchestrators there call it) or the sites in its reach, and its url (https://orders-dc1.example:9443). timeout and max_concurrent_calls are optional.
2. Authenticate. Method mtls, secret reference local:orders-ctrl.
3. Test connectivity. qfence shows the certificate your service presents. Compare its fingerprint with yours, then Pin this certificate and test again. qfence fetches your manifest, checks it (section 3), and shows it with its fingerprint.
4. Capabilities. Choose them; Validate runs your validate.
5. Register. The controller, its pinned certificate and the manifest are written to the cluster’s configuration (in file mode, as the file change to deploy). You do not write the manifest into a configuration file by hand: registration fetches it.

The same flow is POST /api/v1/controllers/registrations and its steps .../connectivity, .../validate and .../register. When your service later serves a new manifest, the controller’s page says so; Register again (POST /api/v1/controllers/{id}/manifest) checks the new manifest and takes it, refusing while a plan binds a capability it drops or changes.

Renewing your service’s certificate

qfence reads when the certificate your service presents expires. 14 days before, it publishes a CertificateExpiring event (which the hosted plane alerts on) and lists it for attention in every plan that binds the controller. To renew without a gap:

1. Issue the new certificate, and take its fingerprint.
2. On the controller’s page, under Certificates, Change the pins: keep the pinned certificate and give the new one as the next certificate. Check the pins tests your service through them; Save the pins.
3. Change your service to the new certificate. qfence trusts it already; the page, and the plans’ attention lists, say your service now presents the next certificate.
4. Make it the pinned certificate, and save.

The API is POST /api/v1/controllers/{id}/certificates with {"server_certificate_sha256": "…", "next_server_certificate_sha256": "…"} (?validate_only=true to check without saving). Your service must present one of the two.

If your service presents a certificate that is pinned nowhere, the controller fails and qfence sends it nothing: every fence through it is UNKNOWN. qfence publishes a CertificateNotPinned event (alerted on), the plans list it for attention, and the controller’s page shows the fingerprint your service presents. If it is your new certificate, check it and Pin this certificate. Pinning the certificate does not take a new manifest; if your service also serves one, Register again after.

9. The conformance kit

Every application controller must pass the conformance kit. Run your service against a double of its application, a stand-in the kit can change behind your controller’s back, and serve the conformance hooks:

Never serve the hooks in production.

The kit sends reset before each role’s checks and at the end, so no check depends on the state another left.

Run the kit:

qfence-orchestrator conformance -endpoint https://127.0.0.1:9443 \
  -cert client.pem -key client-key.pem -pin 8F:C7:86:…:72:C9

Without -pin it shows the fingerprint your service presents, to check and pass. -json prints the report as JSON: {"subject", "results": [{"id", "checks", "outcome", "detail"?}], "verdict"}, where outcome is pass, fail, skipped or n/a and verdict is passed, failed or incomplete. It exits 0 when every check passed (or does not apply), 1 when one failed or could not run, 2 on a usage or connection error.

What it does, check by check. Tokens are large and grow through the run (they start from the current time in microseconds), so a service that keeps tokens between runs passes again.

A check whose role your manifest does not declare is n/a. A service without the hooks gets skipped for every check that needs them, and the run is incomplete, never passed.

10. Binding it to a plan

Bindings live in a plan’s site instances, one controller per site in the usual case (its site is the instance’s). In the plan editor, add each from its group; in a configuration file:

availability_plans:
  - id: orders
    protects: orders-db
    sites: [dc1, dc2]
    fencing:
      unfence_on_promote: true
    instances:
      - id: orders-dc1
        site: dc1
        resource: {controller: orders-dc1, resource: orders-primary}
        bindings:
          # A compute fence: what proves dc1 stopped. Required.
          - {id: bmc-dc1, controller: bmc-dc1, capability: power.off, resource: db-host-dc1, role: fence, required: true}
          # Your application fence: defence in depth, secondary by default.
          - {id: demote-dc1, controller: orders-dc1, capability: db.demote, resource: orders-primary, role: fence}
          # Readiness: a failover to dc1 needs it ready.
          - {id: lag-dc1, controller: orders-dc1, capability: db.lag, resource: orders-primary, role: readiness,
             params: {max_replication_lag: 30s}}
          # Health, and recovery: restarted before any failover.
          - {id: health-dc1, controller: orders-dc1, capability: db.health, resource: orders-primary, role: health}
          - {id: restart-dc1, controller: orders-dc1, capability: db.restart, resource: orders-primary, role: recovery,
             params: {max_attempts: 2, window: 10m, backoff: 10s}}
          # Promotion at dc1, in a failover and in a switchover.
          - {id: promote-dc1, controller: orders-dc1, capability: db.promote, resource: orders-primary, role: switchover,
             params: {strategy: graceful, readiness_timeout: 2m}}
      # … orders-dc2, the same at dc2.
    failover:
      automatic_failover: true
      application_health:
        unhealthy_for_seconds: 60
        fail_over: true

• health is checked every 10 s (application_health.interval_seconds). Without application_health it is shown and triggers nothing. With it, a health binding at the owner that reads UNHEALTHY on every check for unhealthy_for_seconds starts recovery, then a failover when recovery runs out (fail_over: true) or an attention item. UNKNOWN never counts.
• recovery restarts the application at the owner site of an ACTIVE plan only, up to max_attempts in window, with backoff between restarts (doubling) and grace before health counts again. Restarts do not count toward the automatic-failover circuit breaker; a failover they escalate to does.
• fence at level application is secondary unless you mark it required (section 6 says what that costs). Every site still needs a required fence at another level. A secondary one runs first, before its site’s required fences, for at most fencing.secondary_application_timeout_seconds (10 s by default), so it can stop or demote the application before a power fence takes the host away; one that has not finished by then is UNKNOWN.
• switchover promotes the application at its site after qfence has verified the plan’s required fences and readiness, in every failover to that site and in a planned switchover. Its strategy is one of the capability’s: graceful demotes the old owner through its application-level fence and verifies it, then waits up to readiness_timeout for this site to report a measured lag of 0 (else the plan is BLOCKED), then fences and promotes as a failover does; fence-then-promote is exactly a failover. A graceful switchover needs an application-level fence at every other site and a readiness binding at its own.

Operators start a switchover from the plan’s page (Switchover, beside Failover), after a dry run, as an Operator: POST /api/v1/availability-plans/{id}/switchover with {"target_site": "dc2", "confirmed": true, "reason": "…"}, and ?dry_run=true for the plan.

11. Troubleshooting

A capability is failing. The controller’s page says why:

• validation failed: your validate graded a check failed;
• its last call failed: the last command through it got an error or no valid reply (the Errors tab has the message);
• the service now serves a different manifest: the whole controller is failing until it is registered again (section 8);
• connectivity test failed: qfence could not fetch the manifest or run test;
• the service presents a certificate … not the pinned: its certificate was replaced and is pinned nowhere (section 8, “Renewing your service’s certificate”).

A fence or verify is UNKNOWN. qfence got no valid answer: the deadline passed, the connection broke, the reply was over 65536 bytes, malformed, had an unknown field or a value outside the protocol, or echoed another command_id or token; or your service answered an error (a refused stale token included). The plan’s last fence lists each binding’s result and why.

A switchover ends BLOCKED. The demotion did not verify, the target did not report a measured lag of 0 within readiness_timeout, a required fence did not reach FENCED, or the promotion was refused. The plan’s activity says which; an operator returns it to standby or fails it over as for any blocked plan.

Attention items.

• cannot see the application’s health at dc1: health has been UNKNOWN for unhealthy_for_seconds;
• recovered N times in P at dc1: the application keeps failing: recovery succeeds but the application keeps failing (flag_after_episodes within flag_period);
• … fail_over is off, so a person decides: recovery ran out, or there is none, and fail_over is false.