Operations

Operating Soul Stack after the first launch: monitoring, scaling, upgrades, state migrations, backup and restore, and common troubleshooting.

The basic model is simple: Keeper instances are stateless and sit behind an L4 load balancer, all authoritative state lives in PostgreSQL, and the hot layer and coordination live in Redis. Most operational tasks come down to manipulating Keeper instances and maintaining these two external stores.

Monitoring

Telemetry flows over three channels:

Channel	What it carries	How to collect
Prometheus /metrics	counters, gauges, and histograms from Keeper and agents	scrape (pull): Keeper on port :9090, agent on metrics.listen (loopback 127.0.0.1:9091 by default)
OpenTelemetry	end-to-end traces (operator → Keeper → agent)	push: Keeper and the agent send OTLP over gRPC to otel.endpoint themselves
Logs	structured JSON	stdout/file (systemd journal), built-in rotation

An important asymmetry: metrics work on a pull model (Prometheus scrapes /metrics itself), while OpenTelemetry works on a push model (Keeper sends traces to the collector itself; it has no inbound OTel port).

Protecting /metrics

• Keeper — /metrics is served on a separate port without the Operator API auth chain. Optionally, HTTP Basic auth (metrics.auth.basic, password from Vault); see keeper.yml → metrics.
• Agent — binds to loopback by default and is not exposed externally. For external scraping, bind to an interface and add Basic auth with a password from a file; see soul.yml → metrics.

Key signals

Keeper metrics are prefixed keeper_, agent metrics soul_. What to watch first:

Signal	Metric (reference)	What it tells you
Instance availability	up{job="keeper"}	instance not responding — investigate via logs
Connected agents	keeper_grpc_streams_active	a drop below the expected count — some souls have dropped off
Background cleanup	Reaper leadership gauge (sum across the cluster = 1)	zero — DB cleanup has stopped and the tables will grow; more than one — a coordination anomaly (restart Redis)
Recurring-run scheduler	scheduler leadership gauge (sum = 1 when enabled)	zero while the scheduler is enabled — recurring runs are not being spawned
Failed-run ratio	rate of failed / all scenario runs	rising above normal — investigate the scenario or the hosts
Vault latency	Vault read histogram	rising — Vault is responding slowly and new operations slow down

Domain identifiers (run id, sid, scenario name) live in traces, not in metric labels — this is deliberate, to avoid blowing up Prometheus cardinality. Track down the specific culprit through traces and logs (by correlation id and run id).

Dashboards and alerts

Metrics and traces are ready to be collected by standard Prometheus / OpenTelemetry stacks. The exact dashboards and alert thresholds are tuned to your installation (number of agents, run frequency, SLOs).

Scaling

Keeper instances are stateless: all authoritative state lives in the shared PostgreSQL, presence and coordination in the shared Redis. So scaling is horizontal:

flowchart TB
    A["agents"] --> LB["L4 load balancer"]
    LB --> K1["keeper-1"]
    LB --> K2["keeper-2"]
    LB --> K3["keeper-N"]
    K1 --> D[("PostgreSQL + Redis — shared")]
    K2 --> D
    K3 --> D

• Any instance serves any operator request and any agent stream.
• Adding capacity = starting another instance against the same PostgreSQL + Redis behind the same load balancer. There is no state replication between instances.
• For a health probe, a TCP check of the EventStream port is enough for the load balancer; Keeper also exposes an HTTP /readyz.
• Each instance has a kid that is unique within the cluster.

Multiple instances — enable the worker pool

For HA across multiple Keepers, set acolytes > 0 in keeper.yml. A run then goes into a shared queue in PostgreSQL, and any instance can carry it to completion — even if another instance is holding the agent’s stream at that moment. This is the standard horizontal-scaling mode.

If instead you leave acolytes: 0 with several live instances (worker pool off, no shared queue), Keeper by default refuses to start and tells you what to enable — this guards against a run started by one instance that another cannot finish. A single instance with acolytes: 0 is a valid mode and starts normally.

How to choose the number of workers — see Sizing acolytes.

Storage requirements. The total connection load on PostgreSQL is roughly postgres.pool.max × number of instances — budget for this in PostgreSQL’s max_connections. Redis must sustain presence/heartbeat for all souls; for large installations, use a Redis cluster topology.

Zero-downtime upgrades

A multi-instance cluster is upgraded rolling-style, one instance at a time:

1. Take a PostgreSQL backup (see below) and check the changelog for the version you are upgrading to.
2. Remove the first instance from the load balancer’s active backend.
3. Install the new version and restart the instance. On shutdown the instance closes streams gracefully — agents move to the remaining instances via their fallback list. On startup, the new version applies DB schema migrations (if any).
4. Wait for readiness (/readyz → 200, the number of active streams grows as agents return), then put the instance back into the load balancer.
5. Pause to let agents rebalance, then move on to the next instance.

Compatibility. The Keeper↔agent contract is forward-compatible (fields are only added, never removed or reused). A new Keeper understands an old agent and vice versa, so within the upgrade window Keeper and agent versions can coexist. Breaking changes are moved to a new contract version.

Rollback. If the new version won’t start, the remaining instances keep serving. Roll the package back to the previous version and restart the instance, then investigate the problematic version separately. Because state migrations are forward-only, once a schema migration has been applied a version rollback may require restoring from backup — hence the “backup before upgrade” rule.

Agent upgrades proceed independently of Keeper and don’t need to be simultaneous (contract forward-compat). Roll out the new soul version with your package delivery system; in pull mode the agent reconnects itself after the service restarts.

Incarnation upgrade (state migrations)

An incarnation’s state (the runtime instance of a service) is versioned. Moving to a new schema version is an explicit operator step, not an automatic “lazy” upgrade:

• The operator initiates an incarnation upgrade to the target version through the Operator API.
• The migration chain is applied atomically, in a single PostgreSQL transaction: either the state moves fully to the new version, or it stays unchanged on error.
• Before the change, a snapshot of the previous state is saved to history — this is the rollback path (migrations are forward-only; there is no reverse migration).

A migration is a pure function “old state → new state” with no side effects on hosts. The DSL grammar, the layout of migrations/ files, and migration tests are in the reference DSL → State migrations.

Backup and restore

PostgreSQL is the only cold store for cluster state: the agent and operator registries, the service catalog, incarnation states, and logs. Therefore:

• Backup = PostgreSQL backup. Use your PostgreSQL’s standard backup tooling (a logical dump or continuous WAL archiving for point-in-time recovery). Take a backup before every upgrade.
• Redis does not need backing up to restore state — it holds only hot, ephemeral data (presence, leases, cache). After Redis is lost, the instances refill it themselves; while Redis is unavailable the functions that rely on coordination (leader election, presence) degrade, but authoritative state is not lost.
• Vault is maintained and backed up according to your own Vault installation’s procedures.

Restore. Bring PostgreSQL up from backup, make sure Redis and Vault are reachable, and start the Keeper instances — they pick up state from the DB. Agents reconnect via their fallback lists. If a run got “stuck” because its owner instance was suddenly lost, there is a built-in mechanism to reclaim stuck runs — see troubleshooting below.

RBAC and operators

The role catalog and the operator registry live in PostgreSQL and are managed through the Operator API (not through a config file):

• the first operator is created by the administrative keeper init subcommand during initial cluster setup;
• further operators and roles are created through the API by an authorized operator;
• revoking an operator takes effect almost instantly: on revocation every instance in the cluster rebuilds the RBAC snapshot, and the revoked operator’s very next request is rejected (401) — single-digit milliseconds with a healthy Redis, and no longer than the snapshot refresh interval (~10 seconds by default) if the signal is lost. The token’s limited TTL (auth.jwt.ttl_default) remains an additional line of defense, not the primary one.

More on bootstrapping the first operator — Installing from packages.

Troubleshooting

A detailed walkthrough of common situations, framed as “symptom → cause → check/fix”, is in the separate Troubleshooting section: an agent that won’t reach connected, an incarnation in error_locked, an expired bootstrap token, render/template errors, an unreachable Vault/PKI, mTLS that won’t line up, and access denials.

A quick “where to look” cheat sheet:

Symptom	Likely cause	What to check
Agent not in connected status	connection failure / expired identity / crashed service	agent logs (bootstrap, mTLS, verify); network reachability of Keeper’s bootstrap and EventStream ports; status of the soul systemd service
Run stuck in applying for a long time	the run’s owner instance is unavailable (relevant with several instances and the worker pool off)	whether acolytes > 0 is enabled; the number of live instances; the stuck-run reclaim mechanism
Incarnation in error_locked status	the run failed and changes weren’t committed to the DB	the run report and logs; fix the cause and re-run (unlock + retry are Operator API operations)
Access denied (403)	the operator lacks the required permission / the token expired	the operator’s role and permissions; the token’s validity period
Secret resolution fails at startup	Vault is unreachable or the path/key is wrong	Vault availability; the correctness of vault: refs in keeper.yml; the presence of the required fields in Vault KV
Hot reload didn’t apply	the new config failed validation / the field requires a restart	logs for a failed-reload event; the “reload-able / restart-required” table in keeper.yml → Hot reload

Logs are structured (JSON by default) and carry kid/sid, the run id, and the trace id — enough to tie a log entry to a trace and a network event.

See also

• Troubleshooting — a detailed walkthrough framed as “symptom → cause → fix”.
• Configuration → keeper.yml — Keeper parameters, including the HA invariant and hot reload.
• Configuration → soul.yml — agent parameters.
• Components → Keeper — ports, HA, the operator’s primary interface.
• Installation — external infrastructure (PostgreSQL/Redis/Vault) and installation methods.