Installing from deb/rpm packages

The recommended method for production: packages install the binary, a systemd unit, an env file and an example config, and behave correctly on upgrade (your working config isn’t overwritten). This guide walks the path from scratch to a connected agent — installing the packages, provisioning Vault, issuing the Keeper’s server TLS material, filling in the configs, bootstrapping the first operator and onboarding the first agent.

External infrastructure is already up

The guide assumes PostgreSQL, Redis and Vault are already reachable over the network from the keeper host. The required stack is PostgreSQL + Redis + Vault (all three; Keeper checks each on startup and won’t start without any one of them). Standing up and operating these components is on your side.

All examples use placeholder names: keeper.example.com is the Keeper’s FQDN, host-01.example.com is a managed host, archon-alice is the first operator.

Ports

Keeper listens on several listeners. The values below are the production defaults from the example config (you can change them):

Port	Listener	Protocol	Who connects
9442	listen.grpc.bootstrap	server-only TLS	the agent in the soul init phase (bootstrap token + CSR)
8443	listen.grpc.event_stream	mTLS	the agent in the soul run phase (a long-lived EventStream)
8080	listen.openapi	HTTP	operators (Operator API), the /readyz health check, the /ui web UI, the /docs spec viewer
8081	listen.mcp	HTTP	MCP clients
9090	listen.metrics	HTTP	Prometheus scrape (/metrics)

On managed hosts you don’t need to open any inbound ports — the agent initiates the connection to Keeper itself. Locally the agent listens only on the metrics listener.

The production EventStream port is 8443; Quick Start used 9443

Quick Start uses the dev port 9443 for EventStream for local exploration. The production default is 8443 (as in this guide and in the keeper.yml example config). The difference is purely configuration: the port is set in listen.grpc.event_stream.addr on the Keeper side and in keeper.endpoints[].event_stream_port on the agent side — both values must match. The bootstrap port 9442 is the same in dev and production. For an L4 load balancer a TCP probe of port 8443 is enough.

Firewall rules:

• On keeper hosts — open inbound 9442 and 8443 for the managed-host subnet; 8080 / 8081 for the operator/MCP network; 9090 for Prometheus.
• Outbound from keeper hosts — access to PostgreSQL / Redis / Vault and to git hosting (resolving services and plugins).
• Outbound from agent hosts — access to the keepers on 9442 and 8443.

1. Installing the packages

Three packages:

Package	Where to install	What it carries
soul-stack-keeper	central node (1+ instance)	keeper + systemd unit + env + example config
soul-stack-soul	every managed host	soul + systemd unit + env + example config
soul-stack-soul-lint	operator workstation / CI	just soul-lint (a CLI, no daemon or config)

soulctl is separate

The operator CLI soulctl isn’t shipped as a deb/rpm package. Get it from the release binary or build it (make build). See From binary releases and Components → soulctl.

Keeper (on the central node)

sudo dpkg -i soul-stack-keeper_<version>_amd64.deb   # Debian/Ubuntu
sudo rpm -i  soul-stack-keeper-<version>.x86_64.rpm   # RHEL family

The package lays down:

Path	What	Note
/usr/local/bin/keeper	the binary, 0755	—
/etc/systemd/system/keeper.service	systemd unit	Type=exec, User=soul-stack, hardening (ProtectSystem=strict, the only writable path /var/lib/keeper)
/etc/keeper/keeper.env	env file, `config	noreplace`
/etc/keeper/keeper.yml.example	example config, 0640	the operator creates the working config by copying (step 5)

Why .yml.example and not .yml

The package deliberately installs the example under a separate name so that a package upgrade never touches your working /etc/keeper/keeper.yml. You make the initial copy by hand.

Soul (on every managed host)

sudo dpkg -i soul-stack-soul_<version>_amd64.deb

The layout mirrors the Keeper’s: /usr/local/bin/soul, /etc/systemd/system/soul.service, /etc/soul/soul.env (SOUL_CONFIG=/etc/soul/soul.yml), /etc/soul/soul.yml.example.

The soul unit’s hardening is softer than the keeper’s

Soul applies Destiny (installs packages, edits files, manages services) — this needs real privileges on the host, so the strict ProtectSystem=strict / MemoryDenyWriteExecute aren’t set for it. The only writable path is /var/lib/soul-stack (a SHA-256-keyed module cache + the agent’s mTLS identity).

System user and directories

Both units run under the soul-stack system user. Creating the user and the state directories is the operator’s job (the units expect them ready). Once per host:

On the keeper host:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin soul-stack
sudo install -d -o soul-stack -g soul-stack /etc/keeper /var/lib/keeper

On the soul host:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin soul-stack
sudo install -d -o soul-stack -g soul-stack /etc/soul /var/lib/soul-stack

soul-stack-soul-lint is just a CLI with no daemon, installed with a single dpkg -i / rpm -i and needs no setup.

2. Provisioning Vault

The operator runs these steps against their production Vault (under a token/policy with rights on the mounts). Production uses a persistent backend, auto-unseal and a least-privilege policy for Keeper itself; the provisioning below is a one-time admin operation, separate from Keeper’s runtime access.

2.1. KV secrets

Keeper resolves the PostgreSQL DSN, the Redis password and the operator-JWT signing key through vault: refs from the config. Write them into KV (mount secret/). Both KV v2 (recommended — versioning and metadata) and KV v1 work — Keeper detects the mount version automatically, so there’s nothing extra to configure:

# DSN of the external PostgreSQL (the `dsn` field)
vault kv put secret/keeper/postgres \
  dsn="postgres://keeper:<password>@postgres.example.com:5432/keeper?sslmode=require"

# Password of the external Redis (the field that redis.password_ref points to)
vault kv put secret/keeper/redis \
  password="<redis-password>"

# Operator-JWT signing key — 32 random bytes, base64.
# Generate ONCE and fix it: changing the key invalidates all live JWTs.
vault kv put secret/keeper/jwt-signing-key \
  signing_key="$(openssl rand -base64 32)"

Don’t regenerate the JWT signing key

If the key is already written, don’t touch it. Any change invalidates all previously issued operator JWTs (the signature won’t match). Rotation is a separate, planned procedure.

2.2. PKI: engine + root + issue role

PKI issues agents’ mTLS certificates. Enable the engine, generate a root and set up a role (the mount/role names are examples, substitute your own):

# 1. Enable the PKI engine and raise max-lease-ttl
vault secrets enable -path=pki pki
vault secrets tune -max-lease-ttl=87600h pki

# 2. Generate the root certificate — the shared trust anchor for all agents
#    and for the Keeper's server cert (see step 3).
vault write pki/root/generate/internal \
  common_name="soul-stack" ttl=87600h

# 3. The soul-seed role: domains/SAN allowed for the issued certificates.
#    allowed_domains — for your hosts' FQDN scheme.
vault write pki/roles/soul-seed \
  allowed_domains="example.com" \
  allow_subdomains=true \
  max_ttl=720h

2.3. AppRole for Keeper’s runtime access

In production Keeper authenticates to Vault via AppRole (not a root token). Create a role bound to the least-privilege policy:

vault policy write keeper-prod keeper-vault-policy.hcl

vault write auth/approle/role/keeper-prod \
  token_policies=keeper-prod \
  secret_id_ttl=720h token_ttl=1h token_max_ttl=24h

# role_id — NOT a secret, goes into keeper.yml::vault.auth.role_id
vault read auth/approle/role/keeper-prod/role-id

# secret_id — a SECRET, put it in a mode 0400 file (step 5)
vault write -f auth/approle/role/keeper-prod/secret-id

role_id is the role identifier, not a secret (stored in the clear in keeper.yml). secret_id is a secret, not stored as plaintext in the config: its source is a local file secret_id_file (mode 0400) or the env secret_id_env. AppRole credentials are deliberately not read from Vault (chicken-and-egg: these are exactly what Keeper uses to log in to Vault).

3. Keeper’s TLS material

This is the most delicate part of onboarding — two trust chains meet here.

What to put in place

Keeper serves both gRPC listeners (bootstrap 9442 and event_stream 8443) with a server certificate:

File	Role
/etc/keeper/tls/server.crt	Keeper’s server leaf cert (presented on bootstrap + event_stream)
/etc/keeper/tls/server.key	the leaf’s private key
/etc/keeper/tls/ca.crt	the CA for validating agents’ client certificates on the mTLS event_stream

One PKI root for everything

The Keeper’s server cert must chain to the same PKI root as the agents’ certificates. Otherwise, on the mTLS stream, the agent won’t trust the Keeper’s server cert, and Keeper won’t trust the agent’s client cert. So the Keeper’s server leaf is issued from the same pki/issue/soul-seed role as the agents’ certificates.

Issue procedure

vault write -format=json pki/issue/soul-seed \
  common_name="keeper.example.com" \
  alt_names="keeper.example.com" \
  ttl=720h > keeper-issue.json

From the JSON response, split three fields into files (certificate → server.crt, private_key → server.key, issuing_ca → ca.crt) and set permissions:

sudo install -d -o soul-stack -g soul-stack -m 0750 /etc/keeper/tls
sudo install -o soul-stack -g soul-stack -m 0640 server.crt /etc/keeper/tls/server.crt
sudo install -o soul-stack -g soul-stack -m 0600 server.key /etc/keeper/tls/server.key
sudo install -o soul-stack -g soul-stack -m 0640 ca.crt     /etc/keeper/tls/ca.crt

SAN is required

The server cert must contain in its SAN the FQDN (or IP) that agents put in keeper.endpoints[].host — the agent checks the hostname of the server cert during the bootstrap phase. A mismatch → certificate validation failed.

Rotating the leaf is a repeat of this procedure + a Keeper restart; the CA root doesn’t change, so already-onboarded agents aren’t affected.

4. The keeper.yml config

Copy the example to the working path and fill it in:

sudo cp /etc/keeper/keeper.yml.example /etc/keeper/keeper.yml
sudo chown soul-stack:soul-stack /etc/keeper/keeper.yml
sudo chmod 0640 /etc/keeper/keeper.yml

Blocks you must review/edit:

# Instance identity — unique in the cluster (several keepers = different kid)
kid: keeper-01

listen:
  grpc:
    bootstrap:
      addr: "0.0.0.0:9442"
      tls: { cert: /etc/keeper/tls/server.crt, key: /etc/keeper/tls/server.key }
    event_stream:
      addr: "0.0.0.0:8443"   # production EventStream port (9443 in dev)
      tls: { cert: /etc/keeper/tls/server.crt, key: /etc/keeper/tls/server.key, ca: /etc/keeper/tls/ca.crt }
  openapi: { addr: "0.0.0.0:8080" }
  mcp:     { addr: "0.0.0.0:8081" }
  metrics: { addr: "0.0.0.0:9090" }

# External stores — via vault: refs (values written in step 2.1)
postgres:
  dsn_ref: vault:secret/keeper/postgres
redis:
  addr: "redis.example.com:6379"
  password_ref: vault:secret/keeper/redis

# Vault — AppRole (step 2.3) + PKI mount (step 2.2)
vault:
  addr: "https://vault.example.com:8200"
  auth:
    method: approle
    role_id: keeper-prod                        # role_id from step 2.3 (not a secret)
    secret_id_file: /etc/keeper/vault-secret-id # secret_id, a mode 0400 file
  pki_mount: "pki"

# Operator JWTs (signing key written in step 2.1)
auth:
  jwt:
    signing_key_ref: vault:secret/keeper/jwt-signing-key
    issuer: keeper-01
    ttl_default: 24h
    ttl_bootstrap: 720h     # 30 days

Put the secret_id (from step 2.3) into the file that secret_id_file points to:

echo -n "<secret_id>" | sudo tee /etc/keeper/vault-secret-id >/dev/null
sudo chown soul-stack:soul-stack /etc/keeper/vault-secret-id
sudo chmod 0400 /etc/keeper/vault-secret-id

The service registry and RBAC aren’t configured in the config

They live in PostgreSQL and are managed through the Operator API / MCP after startup. The appearance of services: / rbac: keys in the config is rejected as unknown_key.

5. Starting Keeper

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now keeper

Check:

systemctl status keeper
journalctl -u keeper -n 100 --no-pager
curl -fsS http://127.0.0.1:8080/readyz && echo OK

/readyz returns 200 when the instance is ready to take traffic (PostgreSQL and Redis are reachable).

The web UI works out of the box

The operator web UI is built into the Keeper binary and served on /ui (the same port 8080) — no separate package/process/deployment needed. Open http://<keeper>:8080/ui in a browser and log in with the issued JWT (step 6). The UI is on by default; to turn it off, set web_ui_enabled: false in keeper.yml (this doesn’t affect the API or /docs).

Keeper refuses to start without the first operator

If the operator registry is empty and --initialize isn’t passed, Keeper won’t start: operators registry is empty; run 'keeper init …'. This is normal on the very first start — do step 6.

6. Bootstrapping the first operator (Archon)

The operator registry in a fresh DB is empty — every API returns 403 until the first operator is created. Bootstrap is an administrative subcommand of the keeper binary itself (not a separate mode), run once on one instance:

sudo -u soul-stack keeper init \
  --archon=archon-alice \
  --config=/etc/keeper/keeper.yml \
  --credential-out=/etc/keeper/archon-alice.jwt

Under a PG advisory lock it checks that the operator registry is empty; it creates the first operator with the cluster-admin role (permissions ["*"]); issues a JWT (TTL = auth.jwt.ttl_bootstrap, 30 days by default) and writes it to --credential-out with mode 0400.

The bootstrap JWT is an admin credential

This token grants * rights over the whole cluster. Move it into a password manager or Vault right away and don’t leave it in /etc/keeper/ for long. Create further operators through the Operator API.

Save the token in a variable for the next steps:

TOKEN=$(sudo cat /etc/keeper/archon-alice.jwt)

7. Onboarding an agent (Soul)

Onboarding is two-sided: the operator registers the host and gets a one-time bootstrap token; on the host soul init exchanges the token + CSR for an mTLS identity. The agent’s identifier (SID) equals the host’s FQDN; the private key is generated on the host and never leaves it.

7.1. Register the host

On the Keeper side (through the Operator API; SID = the future host’s FQDN):

curl -s -X POST http://keeper.example.com:8080/v1/souls \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sid": "host-01.example.com", "transport": "agent", "covens": ["demo"]}'

The response contains bootstrap_token (returned once, TTL 24 hours by default). The agent record appears in pending status. The covens: ["demo"] label is a stable host tag for targeting scenarios. A lost token can’t be recovered — only reissued via POST /v1/souls/{sid}/issue-token.

transport is required

The transport field is agent (the daemon connects to Keeper itself, pull mode) or ssh (agentless delivery over SSH, push mode). If absent — 422 Unprocessable Entity.

7.2. Trust material on the host

Before soul init the agent has no mTLS identity yet, but the bootstrap phase (9442) runs over server-only TLS — the agent must verify the Keeper’s server cert against the CA file from soul.yml::keeper.tls.ca. Trust is established not by TOFU but by explicitly pre-loading the CA. Put the PKI root on the host (the same CA as the Keeper’s server cert, step 3):

sudo install -d -o soul-stack -g soul-stack -m 0750 /var/lib/soul-stack/seed
sudo install -o soul-stack -g soul-stack -m 0644 ca.crt /var/lib/soul-stack/seed/ca.crt

After a successful soul init Keeper returns the PKI chain, which the agent stores and uses to verify the server during the EventStream phase. The pre-loaded ca.crt is only needed until the first soul init.

7.3. The soul.yml config

sudo cp /etc/soul/soul.yml.example /etc/soul/soul.yml
sudo chown soul-stack:soul-stack /etc/soul/soul.yml
sudo chmod 0640 /etc/soul/soul.yml

The minimum to edit:

keeper:
  endpoints:
    - host: keeper.example.com   # FQDN from the server cert's SAN (step 3)
      event_stream_port: 8443    # production EventStream port (9443 in dev)
      bootstrap_port: 9442       # server-only TLS, the `soul init` phase
      priority: 1
  tls:
    ca: /var/lib/soul-stack/seed/ca.crt   # pre-loaded in step 7.2

paths:
  modules: /var/lib/soul-stack/modules
  seed:    /var/lib/soul-stack/seed       # soul init writes the identity here

The EventStream port must match the Keeper

The agent’s event_stream_port must equal the listen.grpc.event_stream.addr port on the Keeper. In production that’s 8443 (see the note in the Ports section above); in the Quick Start dev install it’s 9443. event_stream_port and bootstrap_port are both required and both explicit — there’s no silent fallback of bootstrap to the event-stream port.

7.4. soul init — exchanging the token for an identity

The bootstrap token is passed through an env variable (preferred — it doesn’t show up in ps/history) or from stdin:

SOUL_BOOTSTRAP_TOKEN='<bootstrap_token from 7.1>' \
  sudo -u soul-stack -E soul init --config=/etc/soul/soul.yml

The command determines the SID (= FQDN), generates a private key + CSR (the key never leaves the host), connects to the Keeper’s bootstrap listener, presents the token + CSR, receives the signed mTLS identity and lays it out in paths.seed. If an identity already exists, init fails (a guard against accidental reissue).

7.5. Starting the daemon and verifying

sudo systemctl daemon-reload
sudo systemctl enable --now soul
systemctl status soul
journalctl -u soul -n 100 --no-pager

The agent initiates the EventStream stream to Keeper (mTLS, port 8443). Check that the host has moved to connected from the Keeper side:

curl -s http://keeper.example.com:8080/v1/souls/host-01.example.com \
  -H "Authorization: Bearer $TOKEN"
# the response has status: connected

The move to connected isn’t instant

The status in the registry is a snapshot that Keeper reconciles with the fact of a live stream in the background. After soul run, wait a few seconds and query again.

8. Upgrading

Packages are upgraded with the usual dpkg -i / rpm -U of the new version. What matters:

• Working configs aren’t overwritten. *.yml.example comes from the package, but your /etc/keeper/keeper.yml and /etc/soul/soul.yml were created by you — the upgrade doesn’t touch them. Env files are marked config|noreplace. After an upgrade, compare your config against the new *.yml.example for new keys.
• DDL migrations of the Keeper DB schema are applied idempotently on Keeper startup. Before an upgrade, back up PostgreSQL.
• Incarnation state migrations (state_schema) — a separate operator-initiated operation through the Operator API (POST /v1/incarnations/{name}/upgrade), forward-only, not triggered automatically on a Keeper restart.

Troubleshooting

Symptom	Likely cause	What to check
soul init: connection refused	Keeper isn’t listening on the bootstrap port / firewall blocks 9442	Keeper is running; inbound 9442 is open from the soul host; host/bootstrap_port in soul.yml are correct
soul init: certificate validation failed	the pre-loaded CA isn’t from the right PKI root; the keeper’s FQDN isn’t in the server cert’s SAN	keeper.tls.ca = the PKI issuing_ca (step 7.2); the FQDN from endpoints[].host is in the SAN (step 3)
soul init: keeper.tls.ca is empty	the CA file isn’t specified/pre-loaded	fill in keeper.tls.ca and place the file (step 7.2)
soul init: bootstrap token invalid / expired / used	the token is spent, expired (TTL 24h), or the SID didn’t match	reissue the token POST /v1/souls/{sid}/issue-token; verify SID = FQDN
Keeper: operators registry is empty; refusing to start	first start without bootstrap	run keeper init (step 6)
The agent starts but doesn’t move to connected	the EventStream phase doesn’t complete (mTLS on 8443)	inbound 8443 is open; the port matches keeper.endpoints[].event_stream_port; the server cert and the agent’s identity are from the same PKI root (step 3)