Souls onboarding

This guide takes you from an empty registry to several hosts in the connected state — ready to receive Destiny. Quick Start connected a single host; here the same mechanics scale up to many souls, and covens — the labels you’ll later use to target scenarios — are laid out in detail.

This assumes Keeper is already up, the first Archon is created, and its JWT is in the TOKEN variable. If not, work through Quick Start (steps 1–3) or Installation from packages.

The big picture

Onboarding each host is a two-way exchange:

1. The operator registers the host with Keeper (POST /v1/souls) and gets a one-time bootstrap token.
2. On the host, soul init exchanges the token + a CSR for a permanent mTLS identity (SoulSeed). The private key is generated on the host and never leaves it.
3. On the host, soul run starts a daemon that holds a long-lived stream to Keeper — the host moves to connected.

sequenceDiagram
    participant Op as Operator
    participant K as Keeper
    participant H as Host (Soul)
    Op->>K: POST /v1/souls
    K-->>Op: bootstrap_token
    Note over Op,H: deliver the token to the host
    H->>K: soul init (CSR + token)
    K-->>H: SoulSeed
    H->>K: soul run (mTLS stream)
    K-->>H: status: connected

A host’s identifier (SID) is its FQDN. Throughout, the examples use host-01.example.com … host-NN.example.com.

Step 1. Plan your covens

A Coven is a stable logical label on a host: cluster, project, environment, data center, hardware type. You’ll later target scenarios by covens (on: [web, prod]). The label is assigned at registration and stays stable — so it’s worth designing your coven scheme before onboarding.

The key rule: a Coven holds stable tags only. A host’s volatile role (which host is master right now, which is replica) is not a Coven; it’s determined by a live check during a run (Orchestration → probe role). Put in a Coven only what doesn’t change from run to run.

An example scheme for a few souls:

Host	Covens	Meaning
host-01.example.com	[web, prod, eu]	web node, prod, EU region
host-02.example.com	[web, prod, eu]	web node, prod, EU region
host-03.example.com	[db, prod, eu]	DB node, prod, EU region

A single host can carry several covens. Targeting by multiple labels takes their intersection (on: [web, prod] → only hosts that have both labels).

Step 2. Register hosts and issue tokens

Registration is POST /v1/souls on Keeper’s side. For each host:

curl -s -X POST http://keeper.example.com:8080/v1/souls \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sid": "host-01.example.com", "transport": "agent", "covens": ["web", "prod", "eu"]}'

The response contains bootstrap_token (returned exactly once, default TTL 24 hours) and expires_at. The host record appears in the pending state.

transport is required

The transport field is either agent (the daemon connects to Keeper itself, pull mode) or ssh (agentless delivery over SSH, push mode). Without it you get 422 Unprocessable Entity. This guide uses agent throughout.

The token is returned once

A lost bootstrap_token can’t be recovered — only reissued via POST /v1/souls/{sid}/issue-token (with the force flag if the previous one is still active). Save it to a safe place right away for delivery to the host.

Registering souls with a script

Registering dozens of hosts by hand is awkward. A common pattern is a loop over a list of FQDNs that stores the issued tokens in a protected file (mode 0600):

# hosts.txt: one FQDN per line
while read -r sid; do
  token=$(curl -s -X POST http://keeper.example.com:8080/v1/souls \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d "{\"sid\": \"$sid\", \"transport\": \"agent\", \"covens\": [\"web\", \"prod\"]}" \
    | jq -r '.bootstrap_token')
  printf '%s\t%s\n' "$sid" "$token" >> souls-tokens.tsv
done < hosts.txt
chmod 0600 souls-tokens.tsv

Each token is then delivered to its host (see step 4). The delivery method is the operator’s choice: SSH/SCP, cloud-init, a CI/CD pipeline, config management.

Step 3. Prepare trust material on the hosts

The bootstrap phase (soul init) runs over server-only TLS: the host must verify Keeper’s server certificate before it receives its own identity. Trust is established by explicitly preloading the PKI root — the same CA that signed Keeper’s server certificate.

Place this CA file on each host ahead of time, at the path you’ll set in soul.yml (keeper.tls.ca):

# ca.crt — the root/issuing CA of your PKI (the same one behind Keeper's server cert)
sudo install -d -m 0750 /var/lib/soul-stack/seed
sudo install -m 0644 ca.crt /var/lib/soul-stack/seed/ca.crt

Where does ca.crt come from? It’s the PKI issuing CA captured when Keeper’s server certificate was issued — for a local trial grab it with the quick vault commands in Quick Start → local trial; for production see Installing from packages.

If the file is missing, soul init stops with keeper.tls.ca is empty. If you place a CA from a different root, you get certificate validation failed.

Step 4. Fill in soul.yml

The minimal agent config is Keeper’s address, two ports (bootstrap and event-stream), and the path to the trusted CA:

# /etc/soul/soul.yml on host-01.example.com
sid: host-01.example.com          # = FQDN; defaults to the hostname
paths:
  modules: /var/lib/soul-stack/modules
  seed:    /var/lib/soul-stack/seed   # soul init writes the SoulSeed here
keeper:
  endpoints:
    - host: keeper.example.com
      bootstrap_port: 9442      # server-only TLS, soul init phase
      event_stream_port: 9443   # mTLS, soul run phase
      priority: 1
  tls:
    ca: /var/lib/soul-stack/seed/ca.crt   # preloaded in step 3

Both ports are required and explicit — bootstrap never silently falls through to the event-stream port. Multiple Keepers are listed as multiple endpoints[] entries with different priority (the agent uses them as a fallback list).

The FQDN must resolve and be in the server cert’s SAN

SID = FQDN. The hostname must resolve (DNS / /etc/hosts), and Keeper’s FQDN from endpoints[].host must be in the SAN of Keeper’s server certificate. Otherwise bootstrap won’t pass the TLS check. The full soul.yml contract is in Configuration → soul.yml.

Step 5. soul init — exchange the token for an identity

soul init determines the SID, generates a private key + CSR, connects to Keeper’s bootstrap listener, presents the token + CSR, and atomically writes the resulting SoulSeed into paths.seed.

The bootstrap token from step 2 is passed in one of two ways:

Environment variable (recommended)

SOUL_BOOTSTRAP_TOKEN='<host's bootstrap_token>' soul init --config /etc/soul/soul.yml

Safer: the token isn’t visible in ps output and doesn’t land in shell history.

Flag (for one-off manual debugging)

soul init --token='<host's bootstrap_token>' --config /etc/soul/soul.yml

The flag takes precedence over the environment variable.

Prefer the environment variable in production

A token passed via --token is visible in ps output and lands in shell history, so in production pass it through the SOUL_BOOTSTRAP_TOKEN environment variable instead.

Even better, on systemd 250+ deliver it via LoadCredential=, which stages the secret in a private tmpfs directory that never touches the service’s disk.

On success the SoulSeed (cert/key/ca) sits in paths.seed, and the host record moves out of pending. If a SoulSeed already exists on the host, init stops — a guard against accidentally reissuing the identity.

Step 6. soul run — start the daemon

soul run --config /etc/soul/soul.yml

The daemon opens a long-lived EventStream to Keeper over mTLS (port 9443) — after that the host becomes connected. In production soul runs as a systemd service (systemctl enable --now soul); the manual start above is handy for a first check.

Step 7. Verify the souls are connected

Query a specific host:

curl -s http://keeper.example.com:8080/v1/souls/host-01.example.com \
  -H "Authorization: Bearer $TOKEN"
# response shows status: connected

Query the whole registry and filter by covens (for example, to see all prod hosts):

curl -s 'http://keeper.example.com:8080/v1/souls?coven=prod' \
  -H "Authorization: Bearer $TOKEN"

The move to connected is not instant

The status in the registry is a snapshot that Keeper reconciles against the live stream in the background. After soul run, wait a few seconds and query again. The authority for “host online” is the live stream; the snapshot catches up with a small lag, by design.

The same is available in the web UI at /ui (the host registry tab) and through the soulctl CLI — see soulctl.

Common pitfalls

Symptom	Likely cause	What to check
soul init: connection refused	Keeper isn’t listening on the bootstrap port / a firewall blocks 9442	Keeper is running; the inbound bootstrap port is open from the host; host/bootstrap_port in soul.yml are correct
soul init: certificate validation failed	the preloaded CA is from the wrong PKI root; or Keeper’s FQDN isn’t in the server cert’s SAN	keeper.tls.ca is the same CA as the server cert; the FQDN from endpoints[].host is in the SAN
soul init: keeper.tls.ca is empty	the CA file isn’t specified / isn’t in place	set keeper.tls.ca and place the file (step 3)
soul init: bootstrap token invalid / expired / used	the token is already used, expired (TTL 24h), or the SID didn’t match	reissue the token (POST /v1/souls/{sid}/issue-token, force if one is active); check SID = FQDN
soul init: invalid sid	the FQDN doesn’t match ^[a-z0-9][a-z0-9.-]{0,253}$	bring the hostname to a valid lower-case FQDN, or set sid: explicitly in soul.yml
the host stays pending, not connected	the EventStream phase (mTLS on the event-stream port) doesn’t complete	the inbound event-stream port is open; the SoulSeed is written to paths.seed; the server cert and SoulSeed are from the same PKI root

Next steps

• Your first souls service — roll state out to the connected hosts by their covens.
• Scenario orchestration — control the rollout order across all souls.
• Installation from packages — production onboarding from deb/rpm with systemd.
• Security → Identity — SID, SoulSeed, rotation, the trust model.