Infrastructure prerequisites for Keeper

Keeper doesn’t package its external dependencies — it consumes them. Before the first start you stand up and configure the three required components yourself: PostgreSQL, Redis and Vault. The full stack is required: Keeper checks each on startup and won’t start if any one is unreachable or, in Vault’s case, sealed.

This page is requirements-first: first what’s mandatory, then tuning and scaling. The config field contract isn’t duplicated here — for that, go to keeper.yml. A minimal end-to-end demo scenario for standing up the infrastructure is in Quick Start → Step 1; production Vault setup (AppRole, persistent backend, auto-unseal) lives in the core docs, links at the end.

dev vs prod

Throughout the sections below, we call out separately what’s fine only for local exploration and what’s mandatory in production. A summary of the differences is at the end of the page: dev vs prod.

PostgreSQL

The cluster’s only cold store: registries of agents and operators, the service catalog, incarnation states, audit logs. Everything that survives a restart lives here.

What’s mandatory

Requirement	Value
Version	14+ (16 recommended — a longer LTS window). The real code floor is 13 (the built-in gen_random_uuid() is used).
Extensions	none. Neither uuid-ossp nor pgcrypto — UUID generation is built into PostgreSQL 13+.
Driver	pgx/v5 (inside Keeper, nothing to install).
Config	the postgres block with dsn_ref is required.

The DSN is passed through a Vault reference: the production convention is dsn_ref: vault:secret/keeper/postgres (Keeper reads the dsn field from KV). The parser technically accepts a plaintext DSN in keeper.yml, but the recommendation is always to go through Vault so the secret doesn’t sit in the config.

Migrations are applied automatically

You don’t need to apply the DB schema by hand. Keeper applies migrations itself on startup (keeper init and when the daemon starts) under pg_advisory_lock — this is safe even when several instances start at once: one takes the lock, the rest wait. It’s enough to create an empty database and grant Keeper rights on it.

Tuning

• Connection pool — postgres.pool.min / postgres.pool.max (pgx defaults to max=4; set min ≥ 1, max ≥ min).

• PostgreSQL max_connections — plan it from the total load: pool.max × number of Keeper instances, with ×1.5 headroom for backup tools and third-party clients.

  Example: 3 instances × pool.max: 50 = 150 connections → max_connections ≥ 225. The PostgreSQL default (100) is too small for this layout.

• Channel encryption — sslmode=verify-full in production (a server cert from your internal PKI), sslmode=disable is acceptable only in dev.

The pool fields and their relationship to the execution worker pool (acolytes) are in keeper.yml → postgres and Sizing acolytes.

Supported modes and what’s absent

Keeper works with a single logical PostgreSQL over a single DSN.

Read replicas and sharding aren’t supported on the Keeper side

Keeper has no read/write split (r/w routing), no second read DSN for a replica, and no sharding. A single DSN is the only entry point to the database.

This is not a ban on HA — it means HA is done at the infrastructure level, transparently to Keeper:

• Patroni (streaming replication + automatic failover) or managed PostgreSQL (RDS Multi-AZ, Cloud SQL, Yandex Managed PG, Azure Database).
• Promoting a replica when the primary fails is done by Patroni / the managed service. Keeper knows nothing about replicas — it simply reconnects to the primary’s address (the pool is recreated). The DSN points at an endpoint that fronts the current primary (a VIP / DNS / cluster proxy).

When to scale

The strategy is PostgreSQL vertical (more CPU/RAM/disk) + Keeper horizontal (stateless instances behind a load balancer) + infra HA (Patroni/managed). Sharding PG as a way to scale Keeper is not available (see above).

With many souls the audit_log table is the first to hit its limit — by INSERT rate and by volume (the target reference is the “~100k VM” invariant). Mitigations (partitioning by date / hot-cold tiering / write batching) are in the backlog, not in the current release.

Disk sizing (ballpark):

Number of souls	DB size
up to 10k VM	20–50 GB
~100k VM	100–200 GB (with headroom for a year of audit_log)

Operational details — backup/restore, retention, housekeeping — are in the core docs infra.md → Postgres.

Redis

The hot layer and the cluster’s coordination bus. It stores only ephemeral data: the SID lease (which instance holds an agent’s stream), Keeper-instance presence (Conclave), the leader lease for background tasks (Reaper), pub/sub channels (summons / applybus / invalidate / cancel).

What’s mandatory

Requirement	Value
Version	6.2+ (7.x recommended — improved ACL).
Config	the redis block with fields for the chosen mode (addr for standalone; master_name + sentinels[] for sentinel; nodes[] for cluster).
Backup	not needed. All keys are naturally recoverable: a lease is recreated on an agent’s reconnect, presence on instance startup, leadership by re-election on TTL, and pub/sub is ephemeral.

Password

The Redis password is set in redis.password_ref and supports both forms:

• vault-ref vault:secret/keeper/redis[#field] — Keeper resolves it from Vault on startup, the same way as the PG DSN and the JWT signing key. The default field of the KV secret is password; a different field is selected with the #field suffix (for example vault:secret/keeper/redis#sentinel for a separate password for the sentinel nodes). This is the recommended production form.
• plaintext string — the password directly in keeper.yml (dev / tests without Vault).
• empty — Redis with no password (dev only).

If the field in KV is missing or empty, Keeper fails fast on startup with a clear error (password field missing or empty).

TLS

There are no separate TLS fields for Redis in keeper.yml. Only what the go-redis client derives from the address itself works (the rediss:// scheme and the system trust store). Terminate TLS on the Redis / proxy side if you need an encrypted channel.

Supported modes

The Keeper client supports three topologies natively — selected by the redis.mode field (slot routing for cluster and master discovery for sentinel are handled by the client itself, with no external proxy). An empty/omitted mode = standalone (forward-compat for older configs).

mode	Required fields	When
standalone (default)	addr (host:port)	dev / staging, a single node.
sentinel	master_name + sentinels[] (addresses of the sentinel nodes)	The recommended HA path for a typical on-premise setup. Single-master with automatic failover — simpler and safer than cluster. Optional sentinel_password_ref — a separate password for the sentinel nodes themselves.
cluster	nodes[] (node addresses for bootstrap discovery)	Horizontal scaling for a large key volume. See the pub/sub caveat below.

Cluster pub/sub: broadcast with very many souls

In cluster mode, Redis pub/sub is classic broadcast (a message reaches every node). This is correct, but with very many souls broadcast doesn’t remove the known pub/sub-load bottleneck. Sharded pub/sub is planned for GA. For a typical scale and for on-premise HA via sentinel this caveat doesn’t apply — which is why sentinel is the recommended HA path.

The full field contract for the redis block is in keeper.yml → redis.

Tuning

maxmemory 1gb
maxmemory-policy noeviction

• maxmemory — for the target of 100k VM ~1gb is enough: the total size of all Soul Stack keys is on the order of 10 MB (soul:<sid>:lock × 100k ≈ 10 MB, presence is a few KB, pub/sub isn’t kept in memory). 1gb leaves a large margin.
• maxmemory-policy noeviction — strictly noeviction, not allkeys-lru. Every key has its own TTL and a meaningful fallback in code; evicting a lease mid-life via LRU would cause split-brain (a double handler for one SID, two Reaper leaders). noeviction plus a memory margin prevents that.

When to move to sentinel / cluster

By data volume, standalone Redis comfortably handles the target 100k VM — moving to an HA topology is dictated by Redis fault-tolerance requirements, not by data size.

• For HA in a typical on-premise setup, choose sentinel (mode: sentinel): automatic failover, single-master, no sharding. This is the recommended path.
• Choose cluster (mode: cluster) deliberately — when the key volume has outgrown a single master and you need horizontal sharding. Keep in mind the caveat about cluster pub/sub broadcast (see modes).

Both topologies are supported by the client natively — no external cluster-aware proxy is required.

Operational details (what’s stored in Redis by key, the restore procedure) are in the core docs infra.md → Redis.

Vault

The store for all installation secrets and the PKI for agents’ mTLS identity. Mandatory and hard-required: a sealed Vault → Keeper runs in fail-closed mode, and there’s no “run without Vault” flag, nor is one planned.

Engines and requirements

Engine	Required	Why
KV (v1 or v2)	required	Keeper secrets (DSN, JWT key, …) and services’ essence secrets. The version is detected automatically; v2 is recommended (versioning + metadata; Sigil multi-anchor requires v2).
PKI (mount + role)	required	Signing CSRs during agent onboarding — without PKI a new agent won’t get its mTLS identity (SoulSeed).
SSH	optional	Only for push mode keeper.push via the Vault SSH provider.

The Vault version in dev/CI is 1.18.

Mandatory config parameters

Parameter	Required	Note
vault.addr	required	Empty → fail-fast on startup.
vault.auth.method	optional	token (default) or approle.
vault.kv_mount	optional	Default secret.
vault.kv_version	optional	Empty = auto-probe; set explicitly only if ACL blocks the probe (see below).
vault.pki_mount / vault.pki_role	needed for agent onboarding	CSRs are signed through them.

The full field contract is in keeper.yml → vault.

AppRole in production

In production Keeper authenticates to Vault via AppRole (not a static token):

vault:
  addr: "https://vault.internal:8200"
  auth:
    method: approle
    role_id: keeper-prod                          # NOT a secret — inline is fine
    secret_id_file: /etc/keeper/vault-secret-id   # mode 0400, absolute path
  pki_mount: "pki/soulstack"
  pki_role: "soul-seed"

• role_id — not a secret, the role identifier, placed inline in keeper.yml.
• secret_id — a secret; set by exactly one of: secret_id_file (a mode 0400 file, absolute path) or secret_id_env (the name of an env variable).

AppRole secret_id isn’t read from Vault — that’s chicken-and-egg

secret_id is deliberately not resolved through a vault: ref: these are the very credentials Keeper uses to log in to Vault so it can then read everything else. The source of secret_id is therefore always local — a file or env — before the Vault client comes up.

In dev a static root token (token: "root") with Vault in dev mode is acceptable. dev mode is unusable for production — it keeps secrets in RAM and loses them on every restart (losing the JWT key = invalidation of all operator tokens, loss of the PKI root).

Minimal least-privilege policy

The reference is examples/keeper/vault-policy.hcl in the core repository. Exactly 4 paths, each granting the minimum:

Path	Capabilities	Why
secret/data/keeper/*	read	Reading Keeper’s KV secrets. Read only — Keeper doesn’t write them. For KV v1 the path has no /data/: secret/keeper/*.
pki/issue/<pki_role>	update	Signing a CSR during agent onboarding (the issue endpoint needs only POST).
secret/metadata/keeper/sigil-keys/*	list, read	Reaper orphan-reconcile of Sigil keys — report-only, names + metadata only. KV v2 only.
auth/token/renew-self	update	Renewing Keeper’s own client token.

Additionally:

• push via Vault SSH — add a path for SSH-CA signing (ssh/sign/<role>).
• the KV version probe requires access to sys/internal/ui/mounts/<mount>. If ACL blocks it, set kv_version in the config explicitly, and the probe won’t run.

What to seed into KV before starting Keeper

Path	Field	Purpose	Required
secret/keeper/jwt-signing-key	signing_key	HS256 key for signing operator JWTs	always mandatory (without it there’s no Archon auth)
secret/keeper/postgres	dsn	PostgreSQL DSN	required if dsn_ref = vault-ref
secret/keeper/sigil-signing-key, secret/keeper/sigil-keys/<key_id>	—	Sigil signing	optional (without it — fail-closed degradation)
secret/keeper/providers/*	—	cloud-driver credentials	optional (if cloud is used)
secret/keeper/toll-webhook-url	url	Webhook notification URL	optional
secret/keeper/metrics-password	password	Basic auth on /metrics	optional
secret/keeper/ssh-host-ca	—	SSH host CA for push	optional
secret/keeper/* (essence secrets)	—	resolving ${ vault(...) } in CEL	optional

PKI (not a KV secret, but an engine):

vault secrets enable -path=pki/soulstack pki
vault secrets tune -max-lease-ttl=87600h pki/soulstack
vault write -field=certificate pki/soulstack/root/generate/internal \
    common_name="Soul Stack SoulSeed CA" ttl=87600h > /tmp/ca.crt
vault write pki/soulstack/roles/soul-seed \
    allowed_domains="example.com,internal" \
    allow_subdomains=true \
    max_ttl=720h

Mount (pki/ or pki/soulstack/) + root cert + role (soul-seed) with allowed_domains / allow_subdomains / max_ttl. The path names in the policy and in keeper.yml::vault.pki_mount / pki_role must match.

TLS to Vault

A custom CA bundle in the config isn’t parameterized yet (post-MVP)

HTTPS to Vault works if Vault’s server certificate chains to the system trust store of the Keeper host (or is passed to the Vault SDK via standard env like VAULT_CACERT). Setting a custom CA bundle for Vault through keeper.yml is currently not possible — this is a current limitation, and a separate TLS field for Vault in the config will land post-MVP.

Modes

Aspect	dev	prod
Auth	token: "root" (dev mode)	AppRole + secret_id from a file/env
Backend	in-memory (lost on restart)	persistent (raft / consul)
Unseal	automatic (dev mode unseals itself)	auto-unseal via KMS/HSM
Transport	HTTP	HTTPS

A sealed Vault → Keeper fail-closed (it resolves nothing, a new agent won’t onboard). Detailed production Vault setup is in the core docs prod-setup.md.

dev vs prod

A summary of the key differences across all three components:

Component	dev	prod
PostgreSQL sslmode	disable	verify-full
PostgreSQL topology	single instance	Patroni / managed + HA (transparent to Keeper)
Redis password	password_ref: "" (no password)	password_ref: vault:secret/keeper/redis (resolved from Vault)
Redis topology	mode: standalone (single instance)	mode: sentinel (recommended HA) or mode: cluster — native, no proxy
Vault auth	token: "root"	AppRole (secret_id from a file/env)
Vault backend	dev mode, in-memory	persistent (raft/consul) + auto-unseal
Vault transport	HTTP	HTTPS (cert via the system trust store)

For details of production Vault setup (AppRole, persistent backend, auto-unseal, JWT signing-key rotation), see docs/keeper/prod-setup.md in the core repository.

See also

• Quick Start → Step 1 — stand up the infrastructure in the demo scenario.
• keeper.yml — the normative config field contract (the postgres / redis / vault / auth blocks).
• Installation — ways to ship the binaries; external infrastructure is on you.
• Operations — scaling, monitoring, backup/restore, upgrades.
• The core repository (not on this site): docs/operations/infra.md — the operational part of the infrastructure (backup/restore, retention); docs/keeper/prod-setup.md — production Vault (AppRole, persistent, auto-unseal); examples/keeper/vault-policy.hcl — the reference least-privilege policy.