Clustering & High Availability

ForgeVis supports a distributed High Availability (HA) mode based on the Raft consensus algorithm. This allows you to combine multiple servers into a single cluster that provides:

• Fault Tolerance: If a node fails, cameras are automatically migrated to healthy nodes.
• Load Balancing: Cameras are distributed across the cluster.
• Centralized Management: API requests to any node are forwarded to the Leader.

Architecture

We use the Raft consensus algorithm to ensure data consistency across the cluster.

Configuration

To enable clustering, add the cluster section to your config.yaml:

cluster:
  enabled: true
  node_id: 1                     # Unique node ID (1, 2, 3...)
  rpc_addr: "192.168.1.10:9091"  # Raft RPC address of THIS node
  dataDir: "/var/lib/forgevis/data"  # Where this node keeps its Raft state

The config only describes the node itself. Cluster membership is managed entirely through the API — a node never forms or joins a cluster on its own, so starting a node with cluster mode enabled is always safe.

rpc_addr must be an IP address with a port: it is both the bind address and the address other nodes use to reach this one.

dataDir holds everything that makes this node itself: the Raft log, the vote it cast, the snapshot, and — under node_<node_id>/tls — the certificate the Control Plane issued it. Keep it absolute, and keep it on storage that survives a restart. A node that comes up to an empty directory starts as though it had never joined: it votes as if for the first time and asks to join a cluster it is already part of. It says so in the log when that happens.

Leaving a cluster erases the Raft log, the vote and the snapshot at once, with no restart, and keeps the key and the certificate: a node that left is safe to start again.

A node's two channels

A node has two independent channels, protected in different ways.

	Node-to-node channel	Management API
Who talks	node → node	operator or platform → node
Port	cluster.rpc_addr, usually 9091	api.address, usually 9997
Protected by	mutual TLS, always	HTTP Basic from security.users

The split is not cosmetic: nodes prove cluster membership to each other with a certificate, not with the operator's password. That is why you can enable and change the management API password without disturbing the cluster.

Certificates

Nodes authenticate each other with certificates, and that cannot be switched off. Anyone who reaches the consensus port of an unprotected cluster could vote, append entries and install snapshots.

Hence the central property of startup: a node without a certificate does not open its consensus port, and says so in the log. For a freshly installed node this is a normal state, not a fault.

A certificate arrives in one of two ways.

Your own certificate authority

The path for a cluster raised by hand. The node takes no part in issuance: you produce the files, and the config says where to find them.

cluster:
  tls:
    ca_cert: /etc/forgevis/tls/ca.pem
    cert:    /etc/forgevis/tls/node-1.pem
    key:     /etc/forgevis/tls/node-1.key

All three are set together — a partial configuration is rejected at startup.

Issuing with openssl. The authority is created once; the node block is repeated for each node, with its own name and its own address from rpc_addr:

# The cluster's certificate authority
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
    -keyout ca.key -out ca.pem -subj "/CN=forgevis-ca-north"

# A node's certificate
openssl req -newkey rsa:2048 -nodes \
    -keyout node-1.key -out node-1.csr -subj "/CN=S1"

openssl x509 -req -in node-1.csr \
    -CA ca.pem -CAkey ca.key -CAcreateserial \
    -days 825 -out node-1.pem -extfile - <<'EXT'
subjectAltName = IP:192.168.1.10
extendedKeyUsage = serverAuth, clientAuth
keyUsage = digitalSignature, keyEncipherment
EXT

chmod 600 node-1.key

Each node gets ca.pem plus its own node-N.pem and node-N.key. The authority's key, ca.key, never goes onto a node — it belongs only where you issue certificates.

Three rules, each of which produces a confusing error when broken.

• serverAuth and clientAuth in the same certificate. A node is a server to the peer dialling it and a client to the peer it dials.
• The SAN carries the same host as the target node's rpc_addr. For an address literal that is an IP: entry, not a DNS: one.
• One authority per cluster. A shared root would let a node of one cluster speak consensus to another.

A node reads these files at startup, so replacing a certificate means laying down new files and restarting. Watch the expiry date: with an expired certificate a node drops out of the cluster and the log fills with handshake errors.

Issued by the platform

Under a platform, issuance is automatic: the node creates a private key and a signing request, the platform signs it with the cluster's authority and hands back the result. The private key never leaves the node, no restart is needed, and cluster.tls stays empty. See Nodes and clusters.

Building a cluster

Forming a cluster

cluster_id is required and must be a UUID. It is what a camera row points at when it names the cluster that owns it — cameras.cluster_id — so the cluster reads exactly the rows carrying this value. A cluster cannot be initialised without one: without an identity it would read every enabled camera in the database as its own, and it cannot be re-initialised to acquire one later.

Under a Control Plane the identity is the one it assigned. For a cluster raised by hand, generate a UUID and put the same value in the camera rows.

# 1. Initialise the cluster on the first node. It registers itself using its
#    own config — the request carries no addresses, only the cluster's identity.
curl -X POST http://192.168.1.10:9997/cluster/init \
     -H 'Content-Type: application/json' \
     -d '{"cluster_id": "3f8c1d2e-7a45-4b91-9c30-5e6f8a1b2c3d", "cluster_name": "north"}'

# 2. Add the other nodes as learners (they catch up on the log first)
curl -X POST http://192.168.1.10:9997/cluster/add-learner \
     -H 'Content-Type: application/json' \
     -d '{"node_id": 2, "node_name": "S2",
          "rpc_addr": "192.168.1.11:9091", "api_addr": "192.168.1.11:9997"}'

# 3. Promote them to voters
curl -X POST http://192.168.1.10:9997/cluster/change-membership \
     -H 'Content-Type: application/json' -d '{"members": [1, 2, 3]}'

Removing a node

Remove it from the cluster, then tell the node itself to drop its local cluster state — otherwise it keeps acting on the state it still holds and cannot join another cluster later:

curl -X POST http://192.168.1.10:9997/cluster/remove-node \
     -H 'Content-Type: application/json' -d '{"node_id": 3}'
curl -X POST http://192.168.1.12:9997/cluster/leave \
     -H 'Content-Type: application/json' -d '{"force": true}'

The node stops its cameras immediately and erases its local Raft state, with no restart, after which it can be added to a cluster again.

remove-node for a node that is no longer in the membership answers with success: the requested state has been reached. That lets you take a cluster apart to the end without parsing error messages.

Removing the last node is not a membership change but the dissolution of the cluster. Raft cannot remove its only voter, and there is nothing left to update: all that remains is telling that node to leave.

Requests go to the leader

Only the leader accepts changes. A node that receives one and is not the leader answers 307 Temporary Redirect with the leader's address in Location — it does not fetch the answer itself, which is why nodes need no credentials for one another.

Follow that redirect explicitly. httpx and requests drop the Authorization header when following across hosts, so automatic following yields 401: repeat the request yourself, with the credentials of the node you were sent to.

Node Roles

Cluster nodes support two runtime roles:

• recorder — node can own camera assignments and run recording pipelines.
• streamer — node can serve HLS/RTSP streams in cluster mode.

By default, both roles are enabled for a node. Roles can be updated at runtime via cluster API.

Cluster-aware Stream Policies

The scheduler and serving layer apply per-camera policies in cluster mode:

• hls — enable/disable HLS for a specific camera.
• rtsp — enable/disable RTSP serving for a specific camera.
• alwaysRemux — force always-on HLS muxing for selected cameras.
• node_id — preferred node placement for a camera.

This allows separating recording and serving responsibilities across nodes while preserving failover behavior.

Failure Recovery

When a worker node fails (no heartbeat for > 15s):

1. The Leader detects the timeout.
2. The node is marked as failed.
3. Cameras assigned to that node are automatically reassigned to other healthy nodes (Failover).

API

Endpoint	Description
GET /cluster/metrics	Current cluster health, node roles, and storage state
POST /cluster/node/roles	Set runtime roles (recorder / streamer) for a node
POST /cluster/init	Initialise a cluster on this node (single voter)
POST /cluster/add-learner	Add a node to the cluster as a learner
POST /cluster/change-membership	Promote Learners to Voters
POST /cluster/remove-node	Remove a node from cluster membership
POST /cluster/leave	Tell a node to drop its local cluster state (force to skip the membership update)

Diagnostics

• GET /cluster/metrics — leader, term, membership.
• GET /api/metrics — CPU, memory and disk for one node. Resources come from the node itself: only storage state is replicated through Raft, because camera placement is based on it.

Symptom	Cause
Consensus port closed, certificate warning in the log	The node is not certified yet — expected until one is issued
tls-name-mismatch, NotValidForName	The host from rpc_addr is missing from the target node's certificate SAN
h2 protocol error: FRAME_SIZE_ERROR, GoAway	One side is speaking plaintext to a TLS port: the first handshake bytes are read as an HTTP/2 frame header
Nodes read as offline while their processes are alive	Heartbeats are not reaching the leader — check connectivity over rpc_addr