The FortiGate infrastructure controller

The fortigate controller type works through a FortiGate’s REST API (FortiOS). It disables firewall policies and takes interfaces and VLANs down, as network fences and shields; it enables the virtual IP or static route that points clients at the site that owns a plan; and it reads the firewall’s link monitors and load-balancing health checks as probes. Nothing is installed on the firewall beyond an API administrator.

One controller is one FortiGate, or one HA cluster through its management address. Use it to:

• cut a site off the network when it is given up, as a fence or as a shield at the site that takes over;
• move a service’s virtual IP or route to the new owner, once its fences are verified;
• check a site’s links and real servers before promoting it.

1. What it can do

A network fence is not a compute fence. A site cut off the network may still run, and write to what it can still reach. A plan with a compute fence at a site needs one of them required, unless fencing.allow_network_only_fencing accepts the weaker guarantee (fencing levels).

Moving a VIP is not fencing. vip.own and route.own change where clients are sent through this firewall; they never stop the old owner. qfence enables them only after the old site’s fences are verified (spec §23). They do not choose a site either: the VIP’s real servers and health check do. The old site’s real server fails its check because its fence took it down.

2. What the FortiGate needs

• FortiOS with the REST API, over HTTPS, reachable from the orchestrators of the controller’s site and reach.
• A certificate on its administrative HTTPS that the orchestrator hosts trust. The factory certificate is self-signed: install one from your CA, or add the firewall’s CA to the hosts’ trust store. There is no option to skip the check.
• A REST API administrator for qfence, with an admin profile that grants the groups below, and trusted hosts that cover the orchestrators.

3. Set up the API administrator

In the FortiOS CLI, an admin profile with only what qfence needs, then the API user, then its token:

config system accprofile
    edit "qfence-fencing"
        set fwgrp read-write
        set netgrp read-write
    next
end
config system api-user
    edit "qfence"
        set accprofile "qfence-fencing"
        set vdom "root"
        config trusthost
            edit 1
                set ipv4-trusthost 10.20.0.0 255.255.255.0
            next
        end
    next
end
execute api-user generate-key qfence

The last command prints the token once: keep it for §4. ipv4-trusthost is the network the orchestrators connect from; list each site’s. A token used from any other address is refused.

4. Store the token on the orchestrators

Put it in the encrypted secrets store of every orchestrator at the controller’s site and reach, as a JSON object of the api_key method’s fields:

jq -n --arg t "$TOKEN" '{token: $t}' |
  qfence-orchestrator secret put -server https://orchestrator-dc1:8443 -token-file admin.token -ca-file ca.pem -ref fgt-edge \
    -scope-type fortigate -scope-host fw.example.com

-scope-host is the host of the controller’s endpoint (ADR 0016 §6). Run it against each of those orchestrators while it runs (writing a secret).

5. Register the controller

In the console: go to Controllers, then Infrastructure, then Register. Choose FortiGate, its site and reach, enter the endpoint and the API user’s name, and pick the secret reference. The connectivity test reads the system status: FortiOS version and build, and host name. The identity is serial:<the unit's serial number>.

In a configuration file (ADR 0001):

controllers:
  infrastructure:
    - id: fgt-edge
      type: fortigate
      reach: [dc1, dc2]
      network_domain: fortigate-mgmt
      connection: {endpoint: https://fw.example.com, api_user: qfence}
      auth: {method: api_key, credentials_ref: "local:fgt-edge"}
      capabilities: [policy.disable, vip.own, policies.list, vips.list]

A firewall in front of both sites has no site of its own: its reach lists both. Every field is in the reference.

Validation. With api_user set, qfence reads that administrator’s admin profile and grades each capability’s group. A token does not say whose it is, so a profile that grants the group passes as inferred. A profile that grants less fails. Without api_user, with a profile set per object type (custom), or when FortiOS refuses the token the administrator’s settings (system/api-user and system/accprofile, which a profile without sysgrp read may not see), the checks are unverifiable.

Interfaces on qfence’s own path. An interface fence that cut qfence off from the firewall could not be checked or undone, and may take the orchestrators’ own links with it. So interface.down and vlan.down refuse an interface, and the VLANs on it, and an aggregate or switch it belongs to, when it:

• carries the address qfence reaches the firewall at;
• is dedicated to management, or is an HA management interface;
• is where the firewall’s route back to the orchestrator leaves.

This is checked when a binding names the interface, and again before every fence. Behind NAT, the firewall sees another address than the orchestrator’s own, so the route part is inferred; a firewall that cannot look the route up leaves the check unverifiable.

6. Bind it to a plan

This is the demo lab’s ERP database, where one FortiGate fronts both sites. Each site has a shield, the policy that lets the VIP reach the other site, raised while that site is not confirmed fenced; the VIP moves with the plan:

instances:
  - id: erp-db-dc1
    site: dc1
    resource: {controller: pve-dc1, resource: "101"}
    bindings:
      - {id: bmc-dc1, controller: bmc-dc1, capability: power.off, resource: pve-dc1-01, role: fence, required: true}
      - {id: shield-dc1, controller: fgt-edge, capability: policy.disable, resource: "11", role: shield, required: true}
  - id: erp-db-dc2
    site: dc2
    resource: {controller: pve-dc2, resource: "101"}
    bindings:
      - {id: pve-dc2-erp, controller: pve-dc2, capability: vm.stop, resource: "101", role: fence, required: true, params: {node: pve-dc2-01}}
      - {id: shield-dc2, controller: fgt-edge, capability: policy.disable, resource: "12", role: shield, required: true}
service: [{id: vip-erp-db, controller: fgt-edge, capability: vip.own, resource: vip-erp-db}]

Policy 11 lets erp-db’s VIP reach dc2, and 12 lets it reach dc1. When dc2 takes over, it raises its shield, so the VIP cannot send clients to dc1 while dc1 may still be running, and the old primary cannot be reached through this firewall. A shield must still let through what the old site needs to become a standby, such as its replica connecting to the new primary (returning sites).

One firewall in front of both sites is a single point: when it is down, no site’s network fence or shield can be raised, and no VIP moves. A FortiGate HA cluster, reached at its cluster address, is one controller.

7. Troubleshooting

8. Limits

• Not tested against a real FortiGate. The type is tested against a double of the FortiOS REST API built from its published request and response shapes, and against the demo lab’s simulator (limitations). The simulator has policies and VIPs only, so this guide was followed there for policy.disable and vip.own; the interface, VLAN, route and probe capabilities are tested against the double alone. Not confirmed on real firmware: the comment field that holds the generation on each object type, the route lookup (monitor/router/lookup), the HA management interfaces, and the link-monitor and load-balancing monitors’ answers.
• One VDOM. The API user acts in its own VDOM; a controller does not choose one.
• Its comment is qfence’s. vip.own and route.own overwrite the comment of the VIP or route they manage.