The configuration file

The configuration file’s top-level shape. Controller and plan entries use the API’s JSON field names and are decoded where they are applied (internal/controllers.ParseConfig, internal/api.ParsePlanDoc).

• controllers (object): The controllers the orchestrators may call.
• availability_plans (list of object): The availability plans the orchestrators run.

controllers

The controllers: section: one list per kind (ADR 0010 §3).

• infrastructure (list of object): Controllers of the infrastructure types: the platforms that fence and shield a site, such as its BMCs, its hypervisor and its firewall.
• application (list of object): Controllers of the application types: the applications a plan keeps available, which they check, recover, fence and switch over.

controllers.infrastructure[]

Also at controllers.application[].

One controller as configuration: what the configuration file’s controllers: section, a replicated entry and the API hold. It names its credentials with a secret reference and never holds one (ADR 0010 §5), so it may be fingerprinted, exported and shown.

• id (string): id names the controller. A binding names it by its id alone, so ids are unique across both kinds.
• type (string): Its controller type: one every orchestrator knows.
• site (string): The controller’s origin: only orchestrators at that site call it (ADR 0010 §9). reach lists further sites allowed to call it; a controller with no site must list at least one.
• reach (list of strings): site is the controller’s origin: only orchestrators at that site call it (ADR 0010 §9). reach lists further sites allowed to call it; a controller with no site must list at least one.
• network_domain (string): The network the controller reaches its target through (§15); bindings inherit it unless they override it.
• enabled (boolean): enabled, false, turns the controller off without removing it; left out, it is enabled.
• connection (map): How to reach the target system: the connection fields of its type. None of them is secret.
• auth (object): How it authenticates to the target system.
• capabilities (list of object): The capabilities of its type it enables. A binding may use only an enabled capability.
• revalidate_seconds (integer): How often the orchestrators that call the controller validate it again (ADR 0010 §6 step 6); unset means every 5 minutes, and the shortest allowed is 30 seconds.
• manifest (object, as controller-type.v1.json describes): manifest is, for a controller of a type whose services serve their own manifest (external, ADR 0005 §2), the manifest its service served at registration: the controller advertises its capabilities, and turns failing if the service later serves another. Registration fetches it; it is never written by hand.

controllers.infrastructure[].auth

Also at controllers.infrastructure[].capabilities[].auth.

The auth method a controller uses (ADR 0010 §5): the secret holding the method’s secret fields, by reference, and the method’s settings that are not secret, such as an OAuth token URL or an AWS role ARN. A secret field is never in Config.

• method (string): One of its type’s auth methods.
• credentials_ref (string): credentials_ref names the secret that holds the method’s secret fields. A method with no secret field takes none.
• config (map): The method’s fields that are not secret.

controllers.infrastructure[].auth.credentials_ref

Names a secret (ADR 0010 §5): the secrets provider that holds it, its path there, and optionally one key within it. It is configuration, never a value, so it enters the ADR 0001 fingerprint and may be shown and exported. Written in short as provider:path or provider:path#key, for example local:lab-docker.

controllers.infrastructure[].capabilities[]

One capability a controller has enabled, with its controller-level settings and, for a capability with setup auth, its own credentials (ADR 0010 §5). Written as the capability’s id alone when it has neither, or as {id, config, auth}.

• id (string): The capability’s id in its type’s manifest.
• config (map): The capability’s settings, as its type names them.
• auth (object): The capability’s own credentials, for a capability whose type asks for them.

availability_plans[]

An availability plan as configuration (ADR 0010 §8): what the configuration file’s availability_plans: entries, a replicated entry, a node-local API change and N7’s …/config endpoints hold. A plan protects one resource. Each site instance names that site’s copy and binds capabilities of controllers to resources there; the plan holds the fencing policy and the service bindings, which move with ownership.

• id (string): id names the plan; plan ids are unique in the cluster.
• display_name (string): The plan’s name for people, which the console shows.
• protects (string): protects names the logical resource the plan keeps available.
• sites (list of strings): The sites the resource can be active at, by name: each instance is at one of them.
• fencing (object): How the plan fences a site it fails over from.
• instances (list of object): The resource’s copies, one per site, each with the bindings that fence and support it there.
• service (list of object): The bindings that point clients at the active site, such as a VIP or a DNS record. They move with ownership.
• failover (object): The plan’s automatic failover.
• depends_on (list of object): The plans this one depends on: a failover of this plan waits until each is in its required state.
• readiness_override (object): readiness_override configures accepted data loss (ADR 0006); unset means the defaults: allowed, one approval, no cap.

availability_plans[].fencing

The plan’s fencing policy. Which targets there are, and which are required, is in its bindings.

• allow_network_only_fencing (boolean): allow_network_only_fencing accepts a reduced guarantee (§14): a plan with compute fences needs one of them required unless it is set, and with it a failover may rest on network or resource fencing alone. Turning it on is audited, and it needs a required shield at every site (ADR 0009 §4).
• success (string): ALL (default: every required target FENCED) or ANY (one required target FENCED per site being fenced, ADR 0007 §2).
• execution (string): SEQUENTIAL (default, in target order) or PARALLEL. Under ANY, PARALLEL fences sites in parallel, each escalating.
• escalate (boolean): escalate, under ANY (ADR 0009 §2), tries each site’s required targets in order and stops at the first that confirms the site: the gentlest first. Default true; false fences every required target.
• fence_timeout_seconds (number): fence_timeout_seconds bounds each target’s fence (and unfence), and verify_timeout_seconds its verification; unset means 5 s each.
• verify_timeout_seconds (number): fence_timeout_seconds bounds each target’s fence (and unfence), and verify_timeout_seconds its verification; unset means 5 s each.
• secondary_application_timeout_seconds (number): secondary_application_timeout_seconds bounds a fence’s secondary application-level targets, which run before the required ones (ADR 0009 §2): the required targets start once it has passed. Unset means 10 s.
• debt_alert_after_seconds (number): How long a site may owe a fence (ADR 0008 §1) before an alert; unset means an hour.
• recheck_seconds (number): How often the leader verifies that sites not owning the plan are still fenced; unset means 60. on_drift is what it does when one is not: “alert” (default) or “fence” (ADR 0008 §3).
• on_drift (string): recheck_seconds is how often the leader verifies that sites not owning the plan are still fenced; unset means 60. on_drift is what it does when one is not: “alert” (default) or “fence” (ADR 0008 §3).
• return_to_standby (string): “manual” (the default) or “automatic”: whether the leader returns a site that is fenced off to standby by itself (ADR 0009 §6).
• boots_fenced (boolean): boots_fenced declares that the plan’s workloads ask for a lease before they start in their active role (ADR 0009 §7).
• unfence_on_promote (boolean): unfence_on_promote unfences the promoted site’s own targets once every other site is verified fenced and readiness is confirmed (ADR 0002).
• attestation (object): attestation configures operator-attested fencing (ADR 0003); unset means the defaults: allowed, one approval, 60 s.
• self_fence (object): self_fence makes each site’s orchestrator fence its own site’s targets when it has been cut off from quorum for after_seconds (ADR 0008 §4); its dead-man switches are the instances’ dead-man bindings.

availability_plans[].fencing.attestation

fencing.attestation (ADR 0003 §6). Unset fields take their defaults.

• requires_shield (boolean): Always true (ADR 0009 §4): an attested failover needs a raised required shield at the promoted site. false is rejected.
• allowed (boolean): allowed, false, refuses every attestation for the plan.
• approvals (integer): How many operators approve an attestation, 1 or 2.
• min_failing_seconds (number): How long the leader must have seen a target’s fence failing before the target may be attested; 0 turns the check off.

availability_plans[].fencing.self_fence

fencing.self_fence (ADR 0008 §4).

• enabled (boolean): enabled turns self-fencing on for the plan.
• after_seconds (number): How long a site may be cut off from quorum before it fences itself; unset means 60.
• backs_failover (boolean): backs_failover lets the other site’s automatic failover treat a cut-off owner as self-fenced, without an attestation, once the timeout has certainly passed.
• auto_recover (boolean): auto_recover promotes a site automatically when quorum returns and the owner is found self-fenced.

availability_plans[].instances[]

One site’s copy of the protected resource and the bindings that fence and support it there. Its bindings’ order is the order targets are tried (ADR 0009 §2) and unfenced.

• id (string): id names the instance; instance ids are unique in the plan.
• site (string): The site it is at: one of the plan’s sites.
• resource (object): The site’s copy of the protected resource.
• bindings (list of object): bindings fence, shield, check and recover the copy at its site.

availability_plans[].instances[].resource

Names a resource on one controller.

• controller (string): The controller’s id.
• resource (string): The resource’s id on that controller, as its discovery capabilities list it.

availability_plans[].instances[].bindings[]

Also at availability_plans[].service[].

One binding: a capability of a controller, the resource it acts on or observes, its parameters, and its role (ADR 0010 §8). Its id names it wherever the engine and the API name a target.

• id (string): id names the binding; binding ids are unique in the plan.
• controller (string): The id of the controller whose capability it binds.
• capability (string): The capability’s id: one the controller enables.
• resource (string): The resource the capability acts on or observes, on that controller.
• role (string): Fence, shield, readiness, health, recovery, switchover or dead-man in a site instance, and service (the default there) in the plan’s service list.
• required (boolean): required, for a fence or a shield: what must be fenced (or raised) for the failover to proceed. A fence that is not is secondary.
• mode (string): mode, for a shield: “unproven” (the default) keeps it up while another site cannot be confirmed fenced; “always” while its site owns.
• network_domain (string): network_domain overrides the controller’s (§15).
• label (string): What the API and console show for a readiness or service binding; its id by default.
• params (map): The capability’s parameters for this binding, as its type names them.

availability_plans[].failover

The plan’s automatic failover (§41, ADR 0002 §5), behind a circuit breaker that stops it after too many.

• automatic_failover (boolean): automatic_failover fails the plan over without an operator once its owner site has had no reachable voter for primary_unhealthy_for_seconds. It does nothing unless the orchestrators run with -automatic-failover.
• max_automatic_failovers (integer): How many automatic failovers within window_seconds trip the circuit breaker, which stops automatic failover for the plan until an operator re-arms it; unset means 3.
• window_seconds (number): The breaker’s window; unset means an hour.
• cooldown_after_failover_seconds (number): How long after an automatic failover the next may start; unset means 10 minutes.
• primary_unhealthy_for_seconds (number): How long the owner site must be unreachable before an automatic failover (§41, ADR 0002 §5). Unset means the default, 30 seconds.
• application_health (object): application_health makes the plan’s health bindings a trigger, behind local recovery (ADR 0005 §5, §6).

availability_plans[].failover.application_health

failover.application_health (ADR 0005 §5): the plan’s health bindings become a trigger. Without it, health bindings are shown and trigger nothing, and a plan cannot have recovery bindings.

• unhealthy_for_seconds (number): How long a health binding at the owner site must report unhealthy on every check before recovery, or failover, follows; unset means 60.
• fail_over (boolean): fail_over fails the plan over once recovery has run out, or at once when the owner site has no recovery binding. false raises an attention item instead. It needs failover.automatic_failover.
• interval_seconds (number): How often health bindings are checked; unset means 10.

availability_plans[].depends_on[]

One plan an availability plan depends on (§3.2).

• id (string): The id of the plan depended on.
• require_state (string): The state it must be in before this plan fails over, such as ACTIVE.
• same_site (boolean): same_site requires the dependency to be ACTIVE at the site being promoted, not merely ACTIVE somewhere (ADR 0002).
• follow (boolean): follow fails the plan over after the dependency once it is ACTIVE at another site, as an automatic failover (ADR 0015 §1). Only with same_site.

availability_plans[].readiness_override

readiness_override (ADR 0006 §4): whether an operator may fail the plan over when readiness fails, accepting the data its replica has not received.

• allowed (boolean): allowed, false, refuses every readiness override; unset means allowed.
• approvals (integer): How many operators approve an override, 1 or 2; unset means 1.
• max_lag_seconds (number): max_lag_seconds caps the replication lag an operator may accept, whatever bound they give; unset means no cap.