Docker

The Docker infrastructure controller

The docker controller type works through the Docker Engine API. It stops and starts containers, runs a swarm service in a fenced or an active shape, drains swarm nodes, cuts containers and services off the network, restarts them in place, and reads their health and replication lag. Nothing is installed beyond the Engine.

One controller is one Engine: a host’s Engine for its containers, or a swarm manager for the swarm’s services and nodes. Use it to:

stop a site’s containers, or scale its service down, when the site is given up, and start the new owner’s;
run a service as a standby while its site is fenced (ROLE=standby) and as the primary once promoted;
restart a container or service in place before failing it over;
check a container’s health, or the lag its own command reports, before promoting it.

1. What it can do

Capability	Roles	What it does
`container.stop`	`fence`, level `compute`	Stops the container (Docker kills it after `stop_timeout`) and waits until it is not running. FENCED while it exists and is not running; a container that does not exist is UNKNOWN, so a wrong name never reads as proof. Durable while its restart policy is `no` or `unless-stopped`. The unfence starts it.
`service.shape`	`fence`, level `compute`	Makes the swarm service run its fenced shape, a replica count and environment entries, and waits until every task runs it. FENCED, and durable, while the service asks for that shape and no task of another shape runs. The unfence makes it run its active shape.
`node.drain`	`fence`, level `compute`	Drains the swarm node, by hostname, and waits until it runs no task. FENCED, and durable, while it is drained and runs none. The unfence restores the availability it had.
`container.disconnect`	`fence` or `shield`, level `network`	Disconnects the container from the binding’s `networks`. FENCED, and durable, while it is on none of them. The unfence connects it again.
`service.unpublish`	`fence` or `shield`, level `network`	Removes the service’s published ports, keeping them in a service label. FENCED, and durable, while it publishes none. The unfence restores exactly those ports.
`container.restart`, `service.restart`	`recovery`	Restarts the container, or starts it if it stopped; replaces the service’s tasks with a forced update, as `docker service update --force` does.
`engine.ping`	`readiness`, `health`	The Engine, and its swarm for a manager, is up: enough to promote a stateless workload.
`container.health`, `service.health`	`readiness`, `health`	The container, or the service’s task on this Engine, runs and its health check, if it has one, passes.
`container.lag`, `service.lag`	`readiness`	As the health capabilities, and the replication lag in seconds that `command` prints, run in the workload, is within `max_replication_lag`. A command that fails is not ready: the lag is unknown, never current.
`containers.list`, `services.list`, `nodes.list`, `networks.list`	discovery	For the plan editor’s resource picker.

Shapes. A service’s fenced shape is what it runs while its site does not own the plan: fenced_replicas: 0 stops it, and fenced_env: [ROLE=standby] keeps it running as a standby that your entrypoint reads. Its active shape is what it runs once promoted. A shape gives a replica count, environment entries, or both. Tasks are told apart by their environment: a fenced shape without environment entries is FENCED only once no task runs, so one that keeps replicas running needs an entry that marks them.

Draining moves work, it does not stop it. The swarm runs a drained node’s tasks on its other nodes. A site fence must therefore drain every node of the site’s swarm, and only fits a swarm that does not span sites. Containers the node runs outside the swarm are not touched.

2. What the Engine needs

Docker Engine with API version 1.43 or later (Docker 24), which qfence pins its requests to (api_version).
For swarm services, nodes and their restarts and probes: the controller’s Engine is an active manager of the swarm.
A way in for the orchestrators of the controller’s site and reach:
- a Unix socket, unix:///var/run/docker.sock, when the orchestrator runs on the Engine’s host, with auth none;
- TCP with mutual TLS, https://host:2376, from anywhere else, with a client certificate (§3).

Access to the Engine API is root on its host. The Engine has no permission model: whoever can call it can run any container with the host’s file system mounted. qfence’s client certificate, or its access to the socket, is therefore a root credential for that host. Never expose the API over plain TCP (tcp://host:2375) beyond a test.

3. Set up mutual TLS

Give the Engine a server certificate and qfence a client certificate, both from a CA the Engine trusts for clients (Docker’s guide to protecting the daemon socket). Then, in /etc/docker/daemon.json:

{
  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
  "tlsverify": true,
  "tlscacert": "/etc/docker/pki/ca.pem",
  "tlscert": "/etc/docker/pki/server-cert.pem",
  "tlskey": "/etc/docker/pki/server-key.pem"
}

On a host where systemd starts the Engine with -H fd://, the two disagree and the Engine does not start: add a drop-in that clears ExecStart and starts dockerd without -H, then restart it.

The Engine accepts any client certificate its CA signed, so give that CA no other use. tls_ca_file on the controller is the CA that signed the Engine’s server certificate, which qfence checks it against.

4. Store the client certificate on the orchestrators

Put the certificate and its key in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of the mtls method’s fields:

jq -n --rawfile c qfence-cert.pem --rawfile k qfence-key.pem '{tls_cert: $c, tls_key: $k}' |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref docker-dc1 \
    -scope-type docker -scope-host mgr1.dc1.example.com

-scope-host is the host of the controller’s endpoint (ADR 0016 §6). Run it against each of those orchestrators while it runs (writing a secret). A controller on the Unix socket needs no secret.

5. Register the controller

In the console: go to Controllers, then Infrastructure, then Register. Choose Docker Engine, its site and reach, enter the endpoint, and pick the auth method and the secret reference. The connectivity test pings the Engine and reads its version and swarm state. The identity is swarm:<cluster id> for a manager, so two controllers for one swarm are warned of, and engine:<engine id> otherwise.

In a configuration file (ADR 0001), a swarm manager over mutual TLS, from the swarm lab:

controllers:
  infrastructure:
    - id: docker-dc1
      type: docker
      site: dc1
      network_domain: docker-mgmt
      connection: {endpoint: "https://mgr1.dc1.example.com:2376", tls_ca_file: /etc/qfence/docker-ca.pem}
      auth: {method: mtls, credentials_ref: "local:docker-dc1"}
      capabilities: [service.shape, service.unpublish, service.restart, service.lag, service.health, engine.ping, services.list]

and an Engine on the orchestrator’s own host:

    - id: docker-local
      type: docker
      site: dc1
      connection: {endpoint: "unix:///var/run/docker.sock"}
      auth: {method: none}
      capabilities: [container.stop, container.restart, container.health, containers.list, networks.list]

Every field is in the reference.

Validation. The capabilities on swarm services and nodes check swarm.manager: the Engine is an active manager. It passes as inferred, since the Engine API has no permissions to prove. The others have nothing to check beyond the connectivity test.

6. Bind it to a plan

A site’s Engine is lost with the site, so its fences cannot prove a lost site stopped. Pair them with the hosts’ BMCs, required, under success: ANY: the shape fence is tried first, and when the site is dark, its BMC proves the manager off. This is the swarm lab’s ERP database at dc1:

availability_plans:
  - id: erp-db
    sites: [dc1, dc2]
    fencing: {success: ANY, unfence_on_promote: true}
    instances:
      - id: erp-db-dc1
        site: dc1
        resource: {controller: docker-dc1, resource: erp-db}
        bindings:
          - {id: erp-db-dc1, controller: docker-dc1, capability: service.shape, resource: erp-db, role: fence, required: true,
             params: {fenced_env: [ROLE=standby], active_env: [ROLE=primary]}}
          - {id: bmc-dc1, controller: bmc-dc1, capability: power.evidence, resource: dc1-mgr, role: fence, required: true}
          - {id: erp-db-net-dc1, controller: docker-dc1, capability: service.unpublish, resource: erp-db, role: fence}
          - {id: erp-app-dc1, controller: docker-dc1, capability: service.shape, resource: erp-app, role: fence,
             params: {fenced_replicas: 0, active_replicas: 1}}
      # erp-db-dc2: the same, at dc2.

While dc1 is fenced, its database runs as a standby (ROLE=standby), its writer is scaled to zero, and its port is not published, which takes it out of the VIP’s rotation. A failover to dc1 runs the active shapes and publishes the port again.

Recovery and health. A service’s health binding at the owner site, with a service.restart recovery binding, restarts it before qfence fails it over (application controllers):

          - {id: orders-health-dc1, controller: docker-dc1, capability: service.health, resource: orders-api, role: health}
          - {id: orders-restart-dc1, controller: docker-dc1, capability: service.restart, resource: orders-api, role: recovery,
             params: {max_attempts: 2, window: 10m}}

Readiness on a standby that is stopped. container.health and container.lag read the standby’s own container. When container.stop keeps that container stopped while its site is fenced, they are never ready there, and every failover to that site fences the owner, then blocks at readiness, leaving both down. Bind engine.ping as readiness for a stateless workload, or keep the standby running in a fenced shape, such as a service with ROLE=standby, and read its lag. The dry run shows readiness at the target before anything is fenced.

Lag. service.lag and container.lag run your command in the workload, such as a script that asks the database for its replay lag and prints it in seconds. A standby that prints a lag beyond max_replication_lag, or whose command fails, is not ready.

7. Troubleshooting

What you see	Why, and what to do
The connectivity test fails with `remote error: tls: bad certificate`	The Engine does not accept the client certificate: it was not signed by the Engine’s `tlscacert`.
The connectivity test fails with `x509: certificate signed by unknown authority`	`tls_ca_file` is not the CA that signed the Engine’s server certificate, or the certificate does not name the host in `endpoint`.
The connectivity test fails with `client version 1.43 is too new`	The Engine is older than Docker 24. Upgrade it.
`swarm.manager` failed	The Engine is a worker, or not in a swarm. Point the controller at a manager.
A `container.stop` fence is UNKNOWN: `no container … on this Engine`	The name is wrong, or the container was removed. A removed container proves nothing: recreate it stopped, or bind the one that runs.
The site owes a fence after a failover	A stopped container’s restart policy is `always` or `on-failure`, so the Engine starts it when the daemon restarts. Use `unless-stopped`.
A `service.shape` fence is UNKNOWN: tasks on nodes the swarm cannot see	A node is down with a task of another shape on it, and the swarm cannot say whether it still runs. Fence that node another way, such as its BMC.
A failover blocks at readiness, `Healthy:false`, after fencing the owner	The target’s readiness reads a container its own fence keeps stopped (§6). Release the plan, start the target’s container, and fail over again; then bind a readiness that a fenced standby can pass.
A `service.unpublish` unfence fails: `none were recorded to restore`	The service’s ports were removed by something other than qfence. Publish them by hand.

8. Limits

Tested against a real Docker Engine. The type’s integration tests run against Docker-in-Docker swarms (QFENCE_DOCKER_IT=1 go test ./internal/controllers/infrastructure/docker/), and the swarm lab runs two real swarms. This guide’s container capabilities were followed against a local Engine: a stop and a disconnect, their unfence, a transient fence and its debt, and the readiness trap in §6.
No generation on the Engine. Docker has nowhere to record a plan’s generation. The orchestrator that carries out a restart or an unfence refuses one for an older generation than its own record of the plan, but the Engine cannot check it again on arrival, as a systemd host or a PostgreSQL server does.
Docker Compose is not a unit of its own. Bind each container of a Compose project by its name.

FortiGate