Troubleshooting

A practical walkthrough of common “something isn’t working right” situations: symptom → cause → check and fix. This section is about operating an already-running installation; the first stumbles when bringing up a demo setup are in Quick Start, and install-time onboarding problems (ports, CA, the first agent) are in the tables under Installing from packages → Troubleshooting and Souls onboarding → Troubleshooting.

Before diving into a specific situation, it helps to keep in mind two invariants that explain most of the “weirdness”:

• Authoritative state lives in PostgreSQL; presence and coordination live in Redis. An agent’s status in the registry (connected/disconnected) is a snapshot that Keeper reconciles with reality in the background; the snapshot lags reality by a few seconds. The minimum required cluster core is PostgreSQL + Redis; Vault is required for PKI/secrets.
• Identity is proven, not asserted. An agent’s connection to Keeper rests on a shared PKI root (mTLS), and an operator’s access on a short-lived token. Most “not trusted” and “access denied” failures come down to these two mechanisms.

Summary table

Symptom	Likely cause	Where to look
Agent won’t reach connected	service not running / mTLS won’t line up / port unreachable / SID ≠ FQDN / snapshot lagging	§ Agent won’t reach connected
Agent doesn’t trust the Keeper server	Keeper’s server cert and the SoulSeed were issued from different PKI roots	§ TLS/mTLS won’t line up
Incarnation in error_locked	the run failed and the state is locked as a safeguard	§ Incarnation in error_locked
Bootstrap token expired or lost	TTL elapsed / the token is shown only once	§ Bootstrap token expired or lost
Run fails at template rendering	undeclared variable / wrong interpolation marker / strict mode	§ Render and template errors
Operations fail at secret resolution	Vault is unreachable or the KV path/field is wrong	§ Vault unreachable and PKI
Access denied (401/403)	token expired / the operator lacks a permission	§ Access denied

Agent won’t reach connected

Symptom. The host is registered, soul init completed, but in the registry (GET /v1/souls) it isn’t in connected status — it stays pending or drops to disconnected.

Work down the checklist top to bottom — it’s ordered from the most common cause to the rarest.

1. Is the working daemon running

soul init only obtains the identity (SoulSeed) and exits — it does not hold a connection. The persistent stream to Keeper is held by a separate command, soul run (in production, a systemd service).

• Check that the agent service is active (systemctl status soul) and that there are no startup errors in the logs.
• If init completed but the daemon isn’t running, the host will simply stay not connected.

2. Wait a couple of seconds — the status is a snapshot

The status in the registry (souls.status) is a lazy snapshot for the Operator API, not the source of truth about presence. The authority for “agent online” is the live stream (a lease in Redis); Keeper reconciles the snapshot with reality in the background and catches up with a lag of a few seconds.

• Right after soul run, poll GET /v1/souls/{sid} again a few seconds later — the pending → connected transition isn’t instantaneous.
• This is by design, not a hang. If there’s still no status after a minute, move on to the next checks.

3. mTLS: shared PKI root

The working stream (soul run) runs over mTLS to the EventStream port. The stream will establish only if Keeper’s server certificate and the agent’s SoulSeed were issued from the same PKI root — otherwise the agent doesn’t trust the server (and vice versa). Detailed breakdown — § TLS/mTLS won’t line up.

4. Network reachability of the EventStream port

The agent establishes the outbound connection to Keeper itself (there are no inbound ports on the managed host). The stream won’t come up if the EventStream port is unreachable:

• From the agent’s host, endpoints[].host must be reachable on endpoints[].event_stream_port (the mTLS phase of soul run) — a separate port from bootstrap.
• Check the firewall/NAT between the host and Keeper (or its load balancer) on that port.
• host/event_stream_port in soul.yml must point at a real Keeper listener — the layout of the keeper.endpoints block is described in Configuration → soul.yml.

5. SID = FQDN matches

The agent’s SID is the host’s FQDN, and it’s baked into the SoulSeed certificate. If the SID the host is registered under doesn’t match the FQDN in the certificate, Keeper won’t match the stream to the registry record.

• The SID in soul.yml and the SID at registration (POST /v1/souls) must match the real FQDN.
• The PKI role issues a SoulSeed only for names within its allowed domain — use an FQDN from that domain, not a short hostname.

Agent logs are the primary diagnostic channel

Logs are structured (JSON by default) and carry stream and host identifiers. Connection errors show up in the soul run logs: verify/certificate point to mTLS (check 3), connection refused/timeout to port reachability (check 4).

TLS/mTLS won’t line up

Symptom. The agent won’t connect; the logs show a certificate verification error (“the agent doesn’t trust the server”, certificate validation failed, unknown authority).

Cause. For mTLS, both sides must trust the same certificate authority. If Keeper’s server certificate is signed by a different root than the hosts’ SoulSeed certificates, mutual trust breaks and the stream doesn’t establish.

Check and fix.

• Keeper’s server certificate and the agents’ certificates must originate from the same PKI root (Vault PKI). You can’t grab Keeper’s server cert from just anywhere (for example, a public ACME certificate for a web domain) — it must come from the same root as the hosts’ SoulSeeds.
• On the agent’s host, the trusted CA is set explicitly (the CA field in the keeper.tls block in soul.yml) and must point at the root of the same PKI.
• Keeper’s FQDN from endpoints[].host must be in the server certificate’s SAN — otherwise name verification fails even with the right root.

The full transport and PKI-root model — Security → Transport; how this root is issued and where the CA’s private key lives — Security → Secrets.

Incarnation in error_locked

Symptom. GET /v1/incarnations/{name} shows status: error_locked. A new run on this incarnation won’t start.

What it means. A scenario run failed, and the changes were not committed to the incarnation’s state. The state is locked deliberately: it’s a safeguard against continuing to work with partially applied or unknown state. The DB holds the previous (consistent) state, and the incarnation waits for operator intervention.

How to find the cause.

• Open the run report and the history — GET /v1/incarnations/{name}/history (state snapshots for each change). It shows which step the run failed at and why.
• Keeper’s and the agent’s logs for this run are tied together by shared identifiers (run id, trace id) — enough to get to the specific failed step.

How to unlock.

1. Fix the cause of the failure — it may be an error in the scenario/Destiny, an unreachable host, a wrong input, or a failed external step.
2. Re-run after the fix. Clearing the lock and re-running are Operator API operations on the incarnation.
3. The state moves to ready only on a successful run; until then the previous state stays untouched.

The request returned 409 incarnation already applying

This isn’t error_locked but an atomicity safeguard: you can’t start a new run while the previous one is still in progress. Wait for the current run to finish (status: applying → ready/error_locked) and retry.

Bootstrap token expired or lost

Symptom. soul init fails with bootstrap token invalid / expired / used, or the token is lost and there’s nowhere to get it.

Cause. The bootstrap token is:

• single-use — burned on the first successful onboarding;
• short-lived — TTL is 24 hours by default;
• shown once — Keeper stores only its hash; the plain token can’t be recovered after it’s issued.

Fix — reissue the token. For an already-registered host that hasn’t been onboarded yet:

curl -s -X POST https://keeper.example.com/v1/souls/host-01.example.com/issue-token \
  -H "Authorization: Bearer $TOKEN"

• The new plain token is returned in the response once — save it immediately for delivery to the host.
• The “at most one active token per host” invariant applies: if the previous token is still active, a request without the flag returns 409 (protection against a proliferation of valid tokens). To replace the active token with a new one, use the force flag (CLI/wrapper) or force: true (MCP) — the old one is marked used and a new one is issued.
• Deliver the new token to the host by whatever means is convenient and re-run soul init.

If the host has already onboarded successfully (the SoulSeed is on disk) and the token is “lost” — it’s no longer needed: onboarding is done and the identity is on the host. Reissuing is only needed for re-onboarding.

The full onboarding and token-delivery flow — Souls onboarding. For an agent in push mode (over SSH), the bootstrap token isn’t used at all.

Render and template errors

Symptom. The run fails at the render phase — before anything is applied to the host. The error points to an undeclared variable, an unknown function, or an expression syntax error.

Cause. Soul Stack has two templating engines with a strict boundary by file, and both run in strict mode (no silent “empty string instead of the unknown” — a missing name is an error, not an empty value):

• CEL — all expressions in YAML. Top-level expression keys (when:, where:, changed_when:, failed_when:, until:) — the entire string is a CEL expression, with no wrapper. Interpolation inside strings (params:, vars:, literals) — via the ${ … } marker.
• Go text/template — rendering of templates/<path>.tmpl template files. Invoked by the single core.file.rendered step, also in strict mode.

Common errors and fixes.

What’s in the error	Cause	Fix
undeclared reference / unknown name	the variable isn’t in context (typo, not declared in input:/vars:, wrong accessor)	declare the input in Destiny’s input: or the value in vars:; double-check the name; host facts are available as soulprint.self.<path> (bare soulprint.<path> is an error)
${...} ended up in the file verbatim	the ${ … } interpolation marker works in YAML expressions, not inside .tmpl files	in .tmpl, use Go text/template syntax ({{ .var }}), not ${ … }
a string wasn’t recognized as an expression	the contexts got mixed up: wrapped in ${ } inside an expression key, or forgot the marker in a string context	in when:/where:/… write bare CEL with no wrapper; in strings, only via ${ … }
an unknown function in a file template	the function is outside the allowed set (excluded: those that read the FS/network/environment, run commands, or generate crypto)	use a function from the allowed set, or move the computation into the CEL phase

Catch errors offline — soul-lint before applying. The soul-lint linter checks artifacts (Destiny/Scenario/templates) offline, with no access to Keeper or hosts — the right place to catch a render error before a run. Run it locally and in CI as a required step. Subcommands, exit codes, and an example CI step — Components → soul-lint. The Destiny syntax itself and file rendering — DSL → Destiny.

Vault unreachable and PKI

Symptom. A new Keeper instance fails to start, or runs start failing at the render phase with a secret-read error; onboarding of new agents doesn’t go through.

Cause. Vault serves two critical functions: secrets (the PostgreSQL DSN, the operator-token signing key, the values for vault(...) in expressions and vault: refs in the config) and PKI (issuing SoulSeed certificates, the shared trust root for mTLS).

The minimum required cluster core is PostgreSQL + Redis; Vault is required specifically for PKI and secrets. Without Vault, the operations that need a secret or a new certificate suffer.

What keeps working when Vault is unavailable:

• already-established agent streams (the connection is up, resolved secrets are cached);
• Keeper’s background cleanup and coordination;
• Operator API operations that don’t need secrets from Vault.

What fails:

• starting a new Keeper instance — it resolves the PostgreSQL DSN from Vault at startup (fail-fast);
• any run with a secret — a step pulling vault(...) in CEL or a vault: ref in a template won’t render;
• onboarding a new agent — PKI is unavailable, so a SoulSeed can’t be issued.

Check and fix.

• Check that Vault is reachable and its status (unsealed). A “cannot read postgres dsn” symptom at Keeper startup is almost always about Vault: either it’s unreachable or the required KV path/field is missing.
• Verify the vault: refs in Keeper’s config: the path and the field name in KV must exist.
• After Vault recovers, existing sessions survive the outage — Keeper re-establishes its token; if Vault comes back quickly, operators may not even notice the disruption.
• How Vault is integrated, how secrets are resolved, and where the PKI root lives — Security → Secrets.

Access denied

Symptom. An Operator API call returns 401 or 403.

Cause and fix. These are two different cases:

• 401 — the token wasn’t accepted. The operator token (JWT) is short-lived, and there’s no immediate revocation by token contents: an expired token simply stops being accepted (this is the main protection — a revoked operator stops working once the TTL elapses). The fix is to get a fresh token. Reissuing an operator token is an Operator API operation; a breakdown of the operator and role registries — Operator management.
• 403 — the token was accepted, but there are no rights. The operator lacks the required permission for the operation (the default-deny model). Check the operator’s role and its permissions. The grammar of permissions and scope — Operators → Roles and access.

See also

• Operations → Overview — monitoring, scaling, upgrades, backup; a short triage table at the end.
• Installing from packages → Troubleshooting — install-time problems (ports, CA, first launch).
• Souls onboarding — the full flow of registering and connecting agents.
• Security → Transport — mTLS, the shared PKI root, connection direction.
• Configuration → soul.yml and keeper.yml — parameters, connection, hot reload.
• Components → soul-lint — offline artifact checking before applying.