Monitoring out of the box

Observability in Soul Stack is built into both binaries: Keeper and the soul agent publish metrics and traces without separate sidecars or exporter processes. This guide shows how to collect the telemetry, what to look at first, and how to deploy a Prometheus exporter onto the souls themselves using Soul Stack.

The telemetry config reference is keeper.yml → metrics/otel and soul.yml → metrics/otel; the operational overview is Operations → Monitoring.

Two channels, two models

Soul Stack telemetry travels over two different mechanisms — and this asymmetry is the key thing to grasp up front:

Channel	Model	Who initiates	What it carries
Prometheus /metrics	pull (scrape)	your Prometheus comes to Soul Stack	counters, gauges, histograms
OpenTelemetry	push (OTLP)	Soul Stack sends to a collector itself	end-to-end traces (operator → Keeper → agent)

• Metrics are pull. Soul Stack listens on /metrics; Prometheus scrapes that endpoint periodically. On Keeper this is port :9090.
• OpenTelemetry is push. Soul Stack has no inbound OTel port. With OTel enabled, Keeper and the agent send OTLP traces to the collector you specify (otel.endpoint), listening for nothing.

flowchart LR
    subgraph pull["Pull — Prometheus scrapes us"]
        direction LR
        P["Prometheus"] -->|scrape| K1["Keeper :9090/metrics"]
        P -->|scrape| S1["soul — metrics.listen"]
    end
    subgraph push["Push — we send OTLP outward"]
        direction LR
        K2["Keeper"] -->|OTLP| O["your OTel collector"]
        S2["soul"] -->|OTLP| O
    end

Logs are a third channel: structured JSON to stdout / the systemd journal, with built-in rotation. A log record carries kid/sid, the run id, and a trace-id — enough to tie a log to a trace and a metric.

Metrics: scrape /metrics

Keeper

Keeper serves /metrics on a separate listener (port :9090 in the examples) — without the Operator API auth chain. Check it by hand:

curl -s http://keeper.example.com:9090/metrics | head -40

The Prometheus scrape config:

prometheus.yml

scrape_configs:
  - job_name: keeper
    static_configs:
      - targets:
          - keeper.example.com:9090

Keeper’s metrics are prefixed keeper_.

Protecting Keeper’s /metrics

The endpoint sits on a separate port with no operator authorization. If needed, enable HTTP Basic auth (metrics.auth.basic, password from Vault) — see keeper.yml → metrics. Open port :9090 only to the monitoring network.

The soul agent

The agent also publishes /metrics (prefix soul_), but listens on loopback by default (127.0.0.1) — nothing exposed outward. This is deliberate: a managed host shouldn’t open extra inbound ports.

To scrape agents from outside, change the bind in soul.yml to an external interface and enable Basic auth:

soul.yml

metrics:
  listen: "0.0.0.0:9091"
  auth:
    basic:
      username: prometheus
      password_file: /etc/soul/metrics-password   # mode 0400

Otherwise the agent stays on loopback — correct for most installations, where a local node agent collects host metrics. Details in soul.yml → metrics.

What to look at first

Domain identifiers (run id, sid, scenario name) live in traces, not in metric labels — deliberately, to avoid blowing up Prometheus cardinality. So metrics answer “is something wrong?”, while you find the specific culprit through traces and logs.

Signal	Metric (guideline)	What it tells you
Instance availability	up{job="keeper"}	the instance isn’t responding — check the logs
Connected agents	keeper_grpc_streams_active	a drop below expected — some souls dropped off
Background cleanup	Reaper leadership gauge (sum across the cluster = 1)	zero — DB cleanup has stopped and tables are growing; >1 — a coordination anomaly
Schedule planner	planner leadership gauge (sum = 1 when enabled)	zero while enabled — recurring runs aren’t being generated
Failed-run ratio	rate of failed / all scenario runs	a rise above normal — investigate the scenario or hosts
Vault latency	Vault read histogram	rising — Vault is slow to respond and operations slow down

Dashboards and alerts

Metrics and traces are ready to be collected by standard Prometheus / OpenTelemetry stacks. The set of dashboards and alert thresholds are chosen to fit the installation (number of agents, run frequency, SLOs).

Traces: OpenTelemetry push

OTel is turned on in the config; with otel.enabled: true the binary pushes OTLP over gRPC to otel.endpoint itself:

# keeper.yml (symmetric in soul.yml)
otel:
  enabled: true
  endpoint: "otel-collector.example.com:4317"

Traces are end-to-end: a single operator operation (a scenario run, say) is traced through Keeper down to a specific agent and back. Traces (by correlation-id and run id) are how you find where something slowed down or failed — metrics give the aggregate, a trace gives the concrete path.

Details on the keys are in keeper.yml → otel and soul.yml → otel.

Deploy an exporter to the souls with Soul Stack

Soul Stack’s own metrics are one thing; host metrics (CPU, memory, disk, network) are collected by a Prometheus exporter on each host. Rolling it out is an ordinary job for Soul Stack: the same kind of scenario as any other service.

An exporter is packaged as its own service in git — a Destiny that downloads the release (with a checksum check), installs the binary under a dedicated system user, lays down a systemd unit, and starts it — and applied to the souls exactly like any other service (Your first souls service, Orchestration). It leans on ordinary core modules: core.url.fetched (with checksum:) → core.archive.extracted → core.cmd.shell (install) → core.file.rendered (the unit) → core.service.running; take the host architecture from soulprint.self.os.arch so a single run covers mixed amd64/arm64 souls.

There is no ready-made public exporter service yet — build your own from the first-souls-service template, or port an existing node_exporter playbook onto those modules. The result is :9100/metrics on every host for your Prometheus to scrape, right alongside Soul Stack’s own metrics.

A ready-made exporter service is on the way

A public node_exporter service you can pull straight from git with a single registration is planned. Until then, deploy the exporter with your own service as described above.

Next steps

• Operations → Monitoring — an overview of the channels, protecting /metrics, the key signals.
• Configuration → keeper.yml — Keeper’s metrics and otel keys.
• Configuration → soul.yml — the agent’s metrics and otel keys.
• Scenario orchestration — roll the exporter out in waves across all souls.