The Keycloak application controller

The keycloak controller type tells qfence whether Keycloak at a site can sign users in. qfence’s orchestrators at Keycloak’s site reach it over HTTPS, with its certificate pinned: its base URL, and its management interface on the same host. Nothing is installed in Keycloak apart from a probe user and its client, and, if you want realm discovery, a read-only admin account. ADR 0014 sets out the design.

One controller is one Keycloak server. A plan has one controller per site: Keycloak at the owner site, serving, and a stopped Keycloak at each other site.

The type observes and changes nothing. Keycloak has no API that restarts, stops or starts it, and every setting it has, a disabled realm or client included, is a row in its database, which replicates to the other site. So:

• restarting Keycloak is the job of the controller that runs it: Docker’s service.restart or container.restart, systemd’s unit.restart;
• fencing it, and keeping a standby site’s copy stopped, is that controller’s too: Docker’s service.shape (no replicas) or systemd’s unit.stop;
• its database is its own plan, usually a PostgreSQL plan (the PostgreSQL controller’s guide), which the Keycloak plan depends on at the same site.

1. What it can do

A binding names a realm. realm.health signs in there.

Why the probe user. /health/ready checks that Keycloak can reach its database, not that the database takes writes. Measured on Keycloak 26.7.4: while its database refused writes, readiness stayed UP, and so did a client-credentials token and the realm’s discovery document. A user’s sign-in, which writes the user’s session, failed with HTTP 500 after 10 s. Without a probe user, realm.health misses a database that refuses writes, as after a database switchover that did not take Keycloak with it (§7).

Refused probe credentials are UNKNOWN, never UNHEALTHY. A wrong or rotated password, a missing, disabled or locked-out probe user, and a missing client, or one without direct sign-in, all say nothing about Keycloak. A plan must never fail over because a password was rotated. After unhealthy_for_seconds of UNKNOWN, the plan raises an attention item instead. A missing user answers exactly as a wrong password does, so it is UNKNOWN too.

There is no readiness, fence, recovery or switchover capability. A standby site’s Keycloak cannot run before its database is promoted, so nothing there could report it ready: the database plan’s readiness, through depends_on, gates a failover instead.

2. What Keycloak needs

• Keycloak 25 or later, which has the management interface; tested with 26.7.4.
• HTTPS, with a certificate. qfence pins it by its SHA-256 fingerprint, so a self-signed one is fine. The management interface uses Keycloak’s certificate unless it has one of its own (https-management-certificate-file), which qfence then pins on its own.
• health-enabled=true, so that the management interface serves /health/ready.
• metrics-enabled=true. Without it, readiness does not check the database, and realm.health misses a database that is down (the registration check ready.database warns).
• The base URL’s port and the management port (9000 by default) reachable from the site’s orchestrators.
• Recommended: a probe user in each realm you bind, and its client (§3).
• Optional: an admin account, for realm discovery (§3).

3. Set up Keycloak

Run these with kcadm.sh, signed in as an administrator. Realms, clients and users are rows in Keycloak’s database, so they reach every site.

The probe user signs in through a public client that allows direct sign-in and nothing else, and holds only the realm’s default roles. Give it an email, a first and a last name, so that its profile is complete and Keycloak asks it for nothing at sign-in:

kcadm.sh create clients -r erp -s clientId=qfence-probe -s publicClient=true \
  -s directAccessGrantsEnabled=true -s standardFlowEnabled=false -s implicitFlowEnabled=false \
  -s serviceAccountsEnabled=false
kcadm.sh create users -r erp -s username=qfence-probe -s enabled=true -s emailVerified=true \
  -s firstName=qfence -s lastName=probe -s email=qfence-probe@example.com
kcadm.sh set-password -r erp --username qfence-probe --new-password "$PROBE_PASSWORD"

Do the same in every realm a binding names. Keep the client’s default scopes, roles among them, and its full scope allowed: qfence reads the probe user’s roles from the access token its sign-in returns, and refuses a token that names none.

The admin account, if you want qfence to list the realms, is a confidential client in the master realm whose service account may view each realm and change nothing:

kcadm.sh create clients -r master -s clientId=qfence-admin -s publicClient=false \
  -s serviceAccountsEnabled=true -s standardFlowEnabled=false -s directAccessGrantsEnabled=false \
  -s secret="$ADMIN_SECRET"
kcadm.sh add-roles -r master --uusername service-account-qfence-admin \
  --cclientid erp-realm --rolename view-realm --rolename query-realms

Each realm has a client in the master realm named after it (erp-realm); repeat the last command for each realm to list. Registration refuses an admin account with any role that may change something (§5).

4. Store the credentials on the orchestrators

The probe user’s password and the admin account’s secret are secrets, so configuration only names them. Put each in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of its auth method’s secret fields:

jq -n --arg p "$PROBE_PASSWORD" '{password: $p}' |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref kc-probe \
    -scope-type keycloak -scope-host sso-dc1.example.com
jq -n --arg s "$ADMIN_SECRET" '{client_secret: $s}' |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref kc-admin \
    -scope-type keycloak -scope-host sso-dc1.example.com

-scope-type and -scope-host are the controller types and the addresses that may use the secret: the host of the controller’s url, which allows any port of it, or that host with its port, sso-dc1.example.com:8443, which allows that port only (ADR 0016 §6). Keycloak’s management interface, on management_port, is sent no credentials, so the scope need not name its port. Any other controller that names it is refused it (ADR 0011 §8).

Run it against each of those orchestrators while it runs (writing a secret): the controllers that use the credentials pick it up at once.

The probe user’s name, its client and the admin client’s id are not secret. They go in configuration.

5. Register the controller

Keycloak must be running to be registered: registration reads and pins its certificate on both endpoints, and runs its checks.

In the console: go to Controllers, then Application, then Register. Choose Keycloak and enter its base URL, with its http-relative-path if it has one, and the management port. At step 2, the method is basic for a probe user, with its name, its client and the secret reference, or none without one. Below it, Credentials for realms.list (optional) takes the admin account: its client id and secret reference. Without them realms.list is left out, and each binding’s realm is typed. The first connectivity test is refused, and shows the certificate Keycloak presents. Compare it with Keycloak’s own:

openssl x509 -noout -fingerprint -sha256 -in /opt/keycloak/conf/server.crt

Then choose Pin this certificate and test again. If the management interface presents a certificate of its own, the next test shows it, for its own pin. The identity is keycloak@<host>:<port>. The version is not shown: Keycloak 26 shows it only to its master realm’s admin role, which qfence’s admin account must not hold.

In a configuration file (ADR 0001):

controllers:
  application:
    - id: kc-dc1
      type: keycloak
      site: dc1
      connection:
        url: https://sso.dc1.example.com:8443
        management_port: 9000
        server_certificate_sha256: "d677a0…"
      auth: {method: basic, credentials_ref: "local:kc-probe", config: {username: qfence-probe, probe_client: qfence-probe}}
      capabilities:
        - realm.health
        - {id: realms.list, auth: {method: oauth2_client_credentials, credentials_ref: "local:kc-admin", config: {client_id: qfence-admin}}}

qfence never uses a proxy from the orchestrator’s environment to reach Keycloak, and never follows a redirect: a base URL that redirects is refused, with where it redirects to.

Renewing the certificate. The controller’s Overview shows its pins and the certificate each orchestrator is presented, and warns 14 days before that one expires. Pin the renewed certificate as the next one before Keycloak changes to it, then make it the pinned one. POST /api/v1/controllers/{id}/certificates takes the management interface’s pins too (management_certificate_sha256, next_management_certificate_sha256).

Validation. At registration:

For each binding, probe.signin proves that the realm exists, from its public document, and, with a probe user, that the user signs in there and out again holding only the realm’s default roles: default-roles-<realm>, offline_access, uma_authorization, and the account client’s view-profile, manage-account and manage-account-links. A probe user with any other role is refused: anyone holding its password could use it.

6. Bind it to a plan

A Keycloak plan combines three controllers’ bindings: this type’s health, the runner’s restart and fence, and the database’s plan through depends_on. On Docker Swarm, as the swarm lab’s sso plan does:

availability_plans:
  - id: sso
    sites: [dc1, dc2]
    fencing: {fence_timeout_seconds: 60, unfence_on_promote: true}
    instances:
      - id: sso-dc1
        site: dc1
        resource: {controller: docker-dc1, resource: sso}
        bindings:
          # Fenced, Keycloak runs no replica: it could not start against a
          # standby database. Promotion unfences it.
          - {id: sso-dc1, controller: docker-dc1, capability: service.shape, resource: sso, role: fence, required: true,
             params: {fenced_replicas: 0, active_replicas: 1}}
          - {id: sso-restart-dc1, controller: docker-dc1, capability: service.restart, resource: sso, role: recovery}
          - {id: sso-health-dc1, controller: kc-dc1, capability: realm.health, resource: erp, role: health}
          - {id: sso-swarm-dc1, controller: docker-dc1, capability: engine.ping, role: readiness}
      # sso-dc2: the same, with docker-dc2 and kc-dc2.
    failover:
      automatic_failover: true
      application_health: {fail_over: true}
    # Fail over after the database, once it is ACTIVE at another site (§7).
    depends_on: [{id: erp-db, require_state: ACTIVE, same_site: true, follow: true}]

With systemd, the fence is unit.stop (it masks the unit, so nothing starts it), and recovery is unit.restart (the systemd controller’s guide). Readiness is checked at the site being promoted, where Keycloak is stopped, so it must not be unit.active of Keycloak’s own unit: use a command.check that the host answers, as engine.ping does above for the swarm.

• Health and recovery follow ADR 0005: only UNHEALTHY for unhealthy_for_seconds (60 s by default) counts. Recovery then restarts Keycloak at the owner, up to max_attempts (3) within window (10 minutes), with backoff (30 s, doubling) and grace (60 s) after each, and escalates to a failover once they are used up. A restart restarts every realm: a probe failing in one realm restarts them all.
• A standby site’s health is shown and triggers nothing. There Keycloak is stopped, so it reads UNHEALTHY, “not running” (§8).
• depends_on with same_site lets the plan fail over only to a site where its database’s plan is ACTIVE. follow: true makes it fail over there by itself once the database’s plan is ACTIVE at another site (§7, ADR 0015).
• probe_timeout (default 15s, at most 20s, a health check’s own deadline) bounds the sign-in and sign-out. It must outlast the 10 s Keycloak takes to fail a sign-in on a database that refuses writes, so that reads UNHEALTHY rather than UNKNOWN.
• A planned move. The type has no switchover capability, so a Keycloak plan moves with a failover by hand: the plan page’s Fail over, or POST .../failover. That is the sequence a fence-then-promote switchover would run: fence the old site’s Keycloak, then start the new site’s. With follow, moving the database’s plan moves it too (§7); without, move the database’s plan first.

7. When its database moves

With follow: true on its database’s dependency (§6), the Keycloak plan follows the database. Once the database’s plan is ACTIVE at another site, the raft leader fails the Keycloak plan over there as an automatic failover (ADR 0015): it checks the new site’s readiness first, then fences the old site’s Keycloak and starts the new site’s. It needs automatic_failover: true on the plan and -automatic-failover on the orchestrators; without them, follow does nothing, and the plan page says so.

Measured on the swarm lab with follow, after a graceful switchover of its database from dc2 to dc1 (lab.sh switchover erp-db dc1), all times from the switchover’s start:

Sign-ins were down for 59 s, and no site signed users in meanwhile. The other way, from dc1 to dc2, they were down for 64 s, Keycloak taking 37 s to start. Nothing was restarted at the old site: the plan had moved long before unhealthy_for_seconds ran out.

Without follow, the plan stays at its old site, where Keycloak is left on a database that is read-only, then a standby. Its sign-ins fail and realm.health reads UNHEALTHY. Recovery restarts it, which cannot help, since it cannot start against a standby, and the plan escalates to a failover once recovery has run out. With the defaults, that measured 4 min 43 s without sign-ins (ADR 0014, build notes):

Keycloak’s start is a production image’s, built ahead (kc.sh build, then start --optimized): one that builds itself at each start adds about a minute to it.

A follow is an automatic failover like any other:

• It counts toward the plan’s circuit breaker, and waits out its cooldown. The defaults are three automatic failovers within an hour and 10 minutes’ cooldown (max_automatic_failovers, window_seconds, cooldown_after_failover_seconds; the swarm lab sets the cooldown to 60 s). So a database moved and moved back within 10 minutes leaves Keycloak at the wrong site until the cooldown ends, and three moves within an hour use the Keycloak plan’s whole budget. The plan’s attention item says why it waits, and until when.
• Readiness at the new site is checked before anything is fenced. If it refuses, the old site’s Keycloak keeps running, and the follow is tried again 15 s later.
• The dry run of a failover or switchover of the database lists the Keycloak plan, and says whether it will follow.

8. A Keycloak stopped by design

A standby site’s Keycloak is stopped whenever the plan is healthy. Once Keycloak is registered, a connection its pinned host refuses means Keycloak is not running:

• realm.health reads UNHEALTHY, “not running”: shown, and at a standby site triggering nothing;
• the controller stays ready, with Not running beside its state in the console. Its connectivity test passes with that detail, its checks are unverifiable until Keycloak runs, and its bindings can still be made;
• an operator’s test, the registration steps, enabling or validating a capability, new pins or new credentials, needs Keycloak running, and fails while it is not.

A Keycloak that does not answer at all (frozen, or a firewall that drops the connection) is UNKNOWN. A certificate that matches no pin fails the controller, as for every type that pins one.

9. Troubleshooting

10. Limits

• About a minute without sign-ins after the database moves, with follow (§7: 59 s and 64 s measured): the database’s switchover, then the Keycloak plan’s failover and Keycloak’s start. Without follow, about 5 minutes (4 min 43 s with the defaults), until health fails the plan over.
• Healthy means healthy as the orchestrator sees it. Users may still fail to reach Keycloak through their load balancer or proxy, and realms other than the bound one are not checked.
• No same-cluster check. The standby never runs beside the owner, so they cannot be compared. A Keycloak promoted on the wrong database is caught only when the probe’s realm is missing there (UNHEALTHY) or its user is (UNKNOWN, then an attention item).
• The probe user is a real account. Anyone holding its password can sign in as a user who can do nothing.
• Every health check writes and removes a session at the owner: two requests per check interval.
• Direct sign-in is not in OAuth 2.1. Keycloak 26 allows it per client. If a later version drops it, realm.health falls back to readiness alone, and no longer catches a database that refuses writes.
• The server’s version is not read (§5).
• Tested against Keycloak 26.7.4 on PostgreSQL 18 in containers, by the conformance kit, registration and the chaos slice, on every push in CI (QFENCE_KEYCLOAK_APP_IT=1 go test ./internal/controllers/application/keycloak/; the local Docker daemon, or the one DOCKER_HOST names), and on the swarm lab’s sso plan.