Skip to content
Soul Stack

Scenario orchestration

This guide is about controlling exactly how state rolls out across all souls: in what order, with how much parallelism, on which hosts. The initial rollout (Your first souls service) applied state to all hosts at once; here we add the orchestration keys that turn “apply everywhere” into “apply in waves, with a health check between them”.

The reference grammar for orchestration is DSL → Scenario; this page has practical recipes built on it.

The orchestration delta in a nutshell

Section titled “The orchestration delta in a nutshell”

A Scenario receives the full set of Destiny task blocks (module:, params:, when:, register:, onchanges:, …) plus an orchestration delta — keys that Destiny doesn’t have:

KeyPurpose
on:Stable target: which hosts to run on (by covens — real stable labels). Omitted → all members of the incarnation.
where:Volatile filter: select hosts by a live check or a stable fact.
serial:Wave-based (rolling) execution: N hosts at a time.
run_once:Run the step on exactly one host of the target.
apply:Invoke an isolated Destiny instead of an inline module:.

The resolution order is strict: first on: narrows the set from the database (stable), then where: filters the result on live data.

Omit on: and the step runs on all members of the incarnation (a host’s membership in an incarnation is a standalone relation, not a coven). on: is only there to narrow the target: on: keeper, or down to a real sub-coven — a stable label assigned at onboarding (on: [baremetal]). Covens are real labels (cluster, project, environment, datacenter); the incarnation name is not one of them.

Targeting by stable facts (where:)

Section titled “Targeting by stable facts (where:)”

A host’s stable facts are available as soulprint.self.<path> and are known from the registry — you can use them in where: without a probe:

# Tune sysctl only on the Debian family
- name: Tune kernel on Debian-family hosts
  module: core.sysctl.present
  where: soulprint.self.os.family == "debian"
  params:
    name: net.core.somaxconn
    value: "1024"

Available stable facts include soulprint.self.os.family, soulprint.self.os.arch, soulprint.self.sid, soulprint.self.network.primary_ip, soulprint.self.covens, and so on (the full set is in DSL → Scenario).

probe role: volatile role via a live check

Section titled “probe role: volatile role via a live check”

Stable covens deliberately don’t carry a host’s volatile role (which host is master right now, which is replica) — the role changes from run to run, and a Coven has to stay stable. To target by the live role, you learn it with a probe step right before the targeting.

A probe is an ordinary read-only step (core.exec.run or core.cmd.shell) with register: and changed_when: false. To safely target a destructive operation on a probe’s result, the probe must assert completeness of the response via failed_when::

# probe: detect each host's live role (on: omitted — whole incarnation)
- name: Detect actual replication role per host
  module: core.cmd.shell
  register: node_role
  changed_when: false
  failed_when: size(register.node_role) < incarnation.host_count
  params:
    cmd: "the-service role | head -1"

failed_when: size(register.node_role) < incarnation.host_count guarantees that all hosts answered. Without this check, a partial probe (not every host answered) would silently apply the destructive operation to only the part that did answer. With the check, an incomplete probe marks the step failed, the run stops, and state isn’t committed.

Then we target by the probe’s result:

# apply only to the current replicas
- name: Act only on the current replicas
  module: core.service.restarted
  where: register.node_role.stdout == "replica"
  params:
    name: the-service

Rolling rollout (serial:)

Section titled “Rolling rollout (serial:)”

serial: splits the target’s hosts into sequential waves: parallel within a wave, waves strictly one after another. A failure inside a wave stops the rollout (later waves don’t start).

# 2 hosts at a time
serial: 2


# 25% of the target per wave
serial: "25%"

serial: can go on a block: — then the whole block (several tasks) runs over one set of hosts in full before the next wave starts. That’s the idiom “a wave = {change, check health}”.

run_once: — one host per cluster

Section titled “run_once: — one host per cluster”

run_once: true runs the step on exactly one host of the target (the first by SID) — for operations that only need to happen once per cluster (initialization, failover, data migration):

- name: Trigger failover once, on the current master
  module: core.cmd.shell
  where: register.node_role.stdout == "master"
  run_once: true
  params:
    cmd: "the-service failover"

serial: and run_once: are mutually exclusive.

Recipe: rolling restart via onchanges:

Section titled “Recipe: rolling restart via onchanges:”

Let’s put it all together — the common operation “update the config and restart the service in waves, only where the config actually changed, with a health check between hosts”. This brings several things together at once: the probe role, where:, serial: on a block, onchanges:, and retry/until.

scenario/rolling-restart/main.yml
input:
  max_memory:
    type: string
    default: "512mb"


state_changes:
  sets:
    max_memory: "${ input.max_memory }"   # persist the applied config value — it's desired state, not just a restart knob


tasks:
  # 1. probe: live role of each host (on: omitted — whole incarnation)
  - name: Detect actual replication role per host
    module: core.cmd.shell
    register: node_role
    changed_when: false
    failed_when: size(register.node_role) < incarnation.host_count
    params:
      cmd: "the-service role | head -1"


  # 2. rolling one host at a time: re-render the config, restart ONLY on a
  #    config change, wait for health — and only then the next host
  - name: Rolling config update on replicas, one host at a time
    where: register.node_role.stdout == "replica"
    serial: 1
    block:
      - name: Render service config
        module: core.file.rendered
        register: svc_conf
        params:
          path: /etc/the-service/the-service.conf
          template: templates/the-service.conf.tmpl
          vars:
            max_memory: "${ input.max_memory }"
          mode: "0640"


      # restart fires ONLY if the config render reported changed=true
      - name: Restart the service because config changed
        module: core.service.restarted
        onchanges: [svc_conf]
        timeout: 30s
        params:
          name: the-service


      # health gate before the next wave: time budget = count × delay
      - name: Wait until the host is healthy again
        module: core.exec.run
        changed_when: false
        retry:
          count: 12
          delay: 5s
          until: contains(register.self.stdout, "status:ok")
        failed_when: '!contains(register.self.stdout, "status:ok")'
        params:
          cmd: the-service
          args: ["healthcheck"]


  # 3. master LAST: runs only after every replica above is done. run_once pins
  #    the single current master (a probe may report more than one mid-failover).
  #    In production, trigger a failover to promote a fresh replica first, then
  #    restart the former master; here it's restarted in place as the final step.
  - name: Update and restart the master last, on its own
    where: register.node_role.stdout == "master"
    run_once: true
    block:
      - name: Render service config on the master
        module: core.file.rendered
        register: master_conf
        params:
          path: /etc/the-service/the-service.conf
          template: templates/the-service.conf.tmpl
          vars:
            max_memory: "${ input.max_memory }"
          mode: "0640"


      - name: Restart the master because config changed
        module: core.service.restarted
        onchanges: [master_conf]
        timeout: 30s
        params:
          name: the-service

What happens here, step by step:

  1. The probe with its failed_when completeness check learns the live role of all hosts — without it, the rolling update could silently skip an invisible subset of souls.
  2. serial: 1 on the block rolls the whole block (render → restart → health) over one host at a time; the next host starts only once the previous one has passed the health gate.
  3. onchanges: [svc_conf] restarts the service only where the config actually changed: if the render returned changed=false, the restart is skipped. Idempotence holds — a no-op run doesn’t disturb a live service.
  4. retry + until holds the health gate: up to 12 attempts at 5s intervals (a one-minute budget); if the host doesn’t recover, the step fails and the rolling update stops without taking down the remaining hosts.
  5. The master goes last. Every replica is updated before the final task touches the master — run_once: pins the one current master (found by the same probe). You never restart the master mid-rollout: the replicas still syncing from it would stall. In production the clean move is a failover first — promote a healthy replica to master, then restart the former master (now a replica); the recipe restarts it in place to keep the shape clear.

Writing state — a cross-host barrier

Section titled “Writing state — a cross-host barrier”

What a scenario writes to incarnation.state is declared in state_changes. The state commit is a barrier: the scenario waits for all tasks to finish on all hosts in the run, and only then are state_changes committed. If a single task fails on a single host, state isn’t committed and the incarnation moves to error_locked. This is an invariant: committed state always matches the facts on the hosts.

Next steps

Section titled “Next steps”