Architecture and diagrams

A technical overview: how Keeper, the agents, PostgreSQL, Redis, and Vault fit together, and how a run travels from a description of the desired state to being applied on a host.

This section assumes you are familiar with the vocabulary from Concepts (Keeper, Souls, Destiny, Soulprint, Essence, Coven, Incarnation).

Topology

At the center is the Keeper cluster: one or several identical stateless instances over shared PostgreSQL (cold state) and Redis (presence, coordination). Any instance serves any request. On the managed hosts are the Soul agents, which initiate the connection to Keeper themselves.

flowchart TB
    Op["Operator / CI<br/>OpenAPI · MCP · soulctl"] -->|HTTP / mTLS| KC
    subgraph KC["Keeper cluster — HA, stateless"]
        direction TB
        KK["keeper × N instances"]
        R[("Redis<br/>presence · lease · pub/sub")]
        PG[("PostgreSQL<br/>registries · catalog · logs")]
        KK --- R
        KK --- PG
    end
    KC -->|"pull: Soul initiates the stream<br/>gRPC bidi-stream over mTLS"| SD["soul<br/>daemon, agent"]
    KC -->|"push: Keeper reaches out<br/>over SSH, one-shot"| HO["host<br/>no agent"]
    SL["soul-lint · CI / dev<br/>offline validation of Destiny / scenarios / Essence — no Keeper, no network"]

The external infrastructure (PostgreSQL, Redis, Vault) is components you stand up and operate yourself; Soul Stack does not bundle them. The required set is PostgreSQL + Redis + Vault (all three; Keeper checks each one at startup and will not start without any of them).

Transport: Soul initiates the stream

The Keeper ↔ Soul link is a long-lived gRPC bidirectional stream over mTLS. The key point: the agent initiates the connection, not the server.

• On managed hosts no inbound ports need to be opened — the agent dials out to Keeper itself and holds the stream.
• The same stream carries both directions: apply jobs (Keeper → Soul) and run reports, task events, the Soulprint report (Soul → Keeper).
• There is no separate heartbeat message: the gRPC keepalive and any application message on the stream update the agent’s “last activity time”.

Authentication on the stream is mutual (mTLS): Keeper verifies the agent’s client certificate, the agent verifies Keeper’s server certificate. Both certificates are issued from the same PKI root.

HA and horizontal scaling

Keeper instances are stateless and interchangeable. Separating the stores is the key to scaling:

• PostgreSQL — cold storage. The registries of agents and operators, the service catalog, runtime instances (incarnations), run logs, and audit. The single source of truth for the cluster’s state.
• Redis — the hot layer. Presence (who is currently connected), leases on agent identifiers, the heartbeat cache, pub/sub coordination between Keeper instances. Volatile data is not written synchronously to PostgreSQL — it lives in Redis.

Background cleanup is done by the Reaper — a task inside Keeper: it clears expired onboarding tokens, zombie records, and stale agent TLS material. In a cluster the Reaper runs on only one instance at a time — the leader is elected via a Redis lease.

Two apply modes: pull and push

The same soul binary and the same set of modules work in two delivery modes:

Mode	Who initiates	How it works
pull	the agent	soul runs as a daemon, holds a long-lived gRPC stream to Keeper, waits for and applies jobs. For continuously managed hosts.
push	Keeper	Keeper reaches the host over SSH, where the agent is not permanently installed, performs the apply once (one-shot), and leaves nothing permanent behind beyond the changes themselves.

In both modes the modules are applied identically — the mode determines only the delivery method, not the apply logic. In the registry a push host and a pull host are records of the same type, differing by a transport field; switching between modes is a change of a single field with no loss of history.

The apply flow: from description to report

A defining feature of Soul Stack: rendering happens on the Keeper side, not on the host. The agent carries no templating engine, resolves no secrets, and has no access to Vault — it receives tasks that are already ready to apply.

flowchart TB
    Op["Operator — creates / updates an incarnation<br/>(via OpenAPI / soulctl)"] --> K1
    K1["Keeper<br/>1. Resolve the service — git source + ref<br/>2. Resolve the hosts — coven / where<br/>3. Render Destiny — vault-resolve → input-validation → CEL → text/template"]
    K1 -->|"ApplyRequest — ready tasks<br/>over the gRPC stream on mTLS"| S1
    S1["Soul<br/>4. Applies modules one by one, idempotently<br/>→ task events and the run report"]
    S1 -->|"run report (RunResult)"| K2
    K2["Keeper<br/>5. All hosts succeeded → state commit to PostgreSQL<br/>Any host failed → incarnation: error_locked (state not committed)"]

Render phases (on the Keeper side)

All YAML sources go through the same pipeline. The order is fixed; the phases do not interleave:

1. vault-resolve — secret references in parameters are replaced with values; done before expressions are evaluated.
2. input-validation — the operator’s effective input is checked against the declared contract (types, required-ness, pattern/enum).
3. CEL-render — the expressions in the YAML are evaluated: targeting conditions, value interpolation (the ${ … } marker), reading host facts.
4. text/template-render — rendering files from templates (.tmpl) with explicitly surfaced values.

Why rendering happens on the server, not on the host:

• Security. Secrets are resolved centrally; only the needed values travel to the host, and the agent has no Vault tokens and no access to the secret store.
• A lightweight agent. The agent carries no expression or templating engine — it applies ready tasks.
• A single point of truth. The render context (host facts, parameters, secrets) is assembled where it is fully available.

More on the expression model and the boundary between engines is in DSL → Templating.

The apply model and atomicity

The incarnation’s state in the database is updated only after all target hosts have finished successfully — this is a cross-host barrier:

• the run waits for all tasks on all hosts to complete;
• only after the barrier is the scenario’s result committed to incarnation.state (PostgreSQL);
• if even a single task on even a single host failed, the state is not committed, the incarnation moves to error_locked, and the next operation is rejected until the operator explicitly clears it.

This is a deliberate fail-fast: partial applies do not slip through silently, and the recorded state always matches the reality on the hosts.

Agent identity

Soul’s identity is built on two entities:

• SID — the agent’s identifier, equal to the host’s FQDN. This automatically gives deduplication when the agent is reinstalled on the same host. The flip side: renaming the FQDN is a migration (in-place rename is not supported).
• SoulSeed — the mTLS pair (certificate + private key) the agent authenticates with on the stream. Issued via a CSR: the private key is generated on the host and never leaves it. Only the certificate fingerprint is stored in the database — no PEM and no private keys. The SoulSeed is rotated regularly over the live stream.

Onboarding happens in two moves: the operator registers the host and gets a one-time bootstrap token, then the agent on the host exchanges the token for a SoulSeed. Step-by-step onboarding is in Quick Start.

Where things live: git vs the database

Artifact	Where it lives	Why
Service / Destiny / Module	git	This is code: review, versioning through a git ref, change history.
Incarnation / Coven	PostgreSQL	This is runtime state: a full operator flow through OpenAPI/MCP, changed during operation.
Provider / Profile	PostgreSQL	The registry of cloud providers and profiles: a full operator flow through REST (/v1/providers, /v1/profiles) / MCP / UI under RBAC provider.* / profile.*. On the roadmap — ready-made soul-cloud-* drivers (not in the base distribution yet). See core.cloud.

The version of a Service / Destiny / Module is a git ref (a tag or a branch), not a field in the manifest: dependencies are pinned to an exact ref, with no semver ranges.

What’s next

• Components — a detailed breakdown of Keeper, Soul, soulctl, soul-lint.
• DSL — how to describe Destiny, scenarios, and Essence.
• Configuration — configuring Keeper and the agent.
• Operations — ongoing operation: updates, monitoring, recovery.