Scenario

Scenario is the orchestration layer: “how to run one operation over a whole cluster.” create, add_user, restart, add_replica — each such operation on an Incarnation is described by its own scenario.

A scenario receives the full set of Destiny task blocks (module:, params:, vars:, register:, when:, onchanges:, loop:, …) — it inherits that whole core — plus the orchestration delta: which hosts to run on, in what order, with what degree of parallelism. So what follows covers only what Destiny lacks; everything else in a scenario task works exactly as it does in Destiny.

Layout

A scenario lives in the service repository:

scenario/<name>/
├── main.yml            # entry point: input, state_changes, tasks
├── <sub>.yml           # includable neighbors (include)
├── templates/          # templates used by this scenario's steps
├── vars.yml            # the scenario's local values
└── tests/              # scenario tests

The orchestration delta

On top of an ordinary task, a scenario adds these keys:

Key	Purpose
on:	The step’s stable target: which hosts to run on (by group).
where:	A volatile predicate: filter hosts by a live probe.
apply:	Call a Destiny (instead of an inline module:).
serial:	Wave (rolling) execution: N hosts at a time.
run_once:	Run on exactly one host of the target.

on: — the stable target

on: defines which hosts to run the step on, based on stable tags (resolved against the database). Three forms:

Form	Semantics
omitted	the whole incarnation — all its member hosts, resolved via the incarnation’s membership relation (not a coven).
on: keeper	the task runs on the Keeper itself (for example, provisioning a cloud instance).
on: [coven-a, coven-b]	the intersection of the listed groups — the result is always within the current incarnation’s hosts.

on: is only there to narrow the target: on: keeper, or an intersection of stable covens (on: [baremetal] — the incarnation scope is already implicit, since the roster is scoped to the incarnation’s members). Note that incarnation.name is not a Coven: on: ["${ incarnation.name }"] is a validation error — to target the whole incarnation, just omit on:.

# All members of the incarnation (on: omitted)
- name: Apply base config everywhere
  apply: { destiny: redis-base, input: { ... } }

# Keeper-side only (cloud provisioning; the provider/profile are
# registered via the Operator API, see docs/modules/cloud.md)
- name: Provision VMs
  on: keeper
  module: core.cloud.created
  params: { ... }

# Narrow to a stable coven (the incarnation scope is already implicit)
- name: Tune kernel on bare-metal hosts of this cluster
  on: [baremetal]
  apply: { destiny: kernel-tuning, input: { ... } }

Targeting across different incarnations is forbidden by the grammar — the on: resolver cannot return a host outside the current incarnation for any set of groups.

where: — the volatile filter

where: selects hosts per host within the set already chosen by on:. It relies on the results of a previous live probe and/or on the host’s stable facts (soulprint.self.*):

# probe: find each host's live role (on: omitted — the whole incarnation)
- name: Detect actual redis role per host
  module: core.exec.run
  register: redis_role
  changed_when: false
  failed_when: size(register.redis_role) < incarnation.host_count
  params:
    command: "redis-cli role | head -1"

# target by the probe result
- name: Restart only the current replicas
  where: register.redis_role.stdout == 'slave'
  module: core.service.restarted
  params:
    name: redis-server

The resolution order is strict: first on: narrows the set against the database (stably), then where: filters the resulting set by live data.

Stable tags versus volatile role

This is a key architectural separation:

• Coven (a group in on:) — stable tags only: cluster, project, environment, hardware type. A host’s role (master/replica) is never a Coven.
• The volatile role (who is actually master right now) is determined only by a live probe during the run — a probe step and a where: filter on its result.

A host’s stable facts are available as soulprint.self.<path> (soulprint.self.os.family, soulprint.self.sid, soulprint.self.network.primary_ip, soulprint.self.covens). You can use them in where: without a probe — they are known from the registry.

apply: — calling a Destiny

A scenario delegates work to an isolated Destiny through apply:, passing it input:

- name: Install redis on all cluster hosts
  apply:
    destiny: redis
    input:
      version:  "${ essence.redis_version }"
      password: "${ input.redis_password }"

apply: and module: are mutually exclusive within one task. To read the result of the called Destiny (its declared output:), put a register: on the applier task.

serial: and run_once:

• serial: — wave execution: the target’s hosts are split into sequential waves of size ≤N (serial: 2) or a percentage (serial: "25%"). Within a wave — in parallel; the waves — strictly one after another. A failure in a wave stops the rollout (later waves don’t start). The typical idiom is a rolling restart with a health check between waves.
• run_once: — run the step on exactly one host of the target (the first by SID). For operations that only need to happen once per cluster.

serial: and run_once: are mutually exclusive.

Passing data between hosts

A scenario (unlike Destiny) sees the whole run topology:

• soulprint.hosts — the list of all hosts in the run with each one’s stable facts; soulprint.hosts.where("<predicate>") filters the list by any stable attribute.
• soulprint.where("<predicate>") — host data by stable group (a cross-host lookup).
• incarnation.host_count — the number of hosts in the target; used in the probe-completeness idiom (see below).

Don’t confuse the two positions:

• where: — a step key: “which hosts to run on.”
• soulprint.where(...) / soulprint.hosts — a function in an expression: “where to get the data from.”

Destiny does not see these accessors directly — it receives the topology only through an explicit hand-off via apply: input:.

Probe and error handling

A probe is an ordinary step, not a special construct: a read-only module (core.exec.run) + register: + changed_when: false. To safely target destructive operations by a probe’s result, the probe must assert the completeness of the response through failed_when::

failed_when: size(register.redis_role) < incarnation.host_count

Without this check, a partial probe (not every host answered) would silently apply the destructive operation to only the “answering” subset. With the check, a partial probe makes the step fail, the run stops, and no state is committed.

All error handling in a scenario uses the same mechanisms as in Destiny: retry:, onfail:, failed_when:.

State writes and atomicity

What a scenario writes to incarnation.state is declared in the state_changes block:

state_changes:
  sets:
    redis_version: "${ input.version }"
    leader_host:   "${ register.elect.stdout }"

The state commit is a cross-host barrier:

1. the scenario waits for all tasks on all hosts of the run to finish;
2. only after the barrier are the state_changes committed to the database;
3. if even one task on even one host failed — the state is not committed, and the incarnation moves to error_locked.

This is an invariant, not an option: committed state always matches the fact on the hosts.

Full example

scenario/restart/main.yml

input:
  drain_timeout:
    type: string

state_changes: {}

tasks:
  - name: Detect actual redis role per host
    module: core.exec.run
    register: redis_role
    changed_when: false
    failed_when: size(register.redis_role) < incarnation.host_count
    params:
      command: "redis-cli role | head -1"

  - name: Rolling restart of replicas, two at a time
    module: core.service.restarted
    where: register.redis_role.stdout == 'slave'
    serial: 2
    params:
      name: redis-server

What’s next

• Destiny — the task core that a scenario inherits.
• Essence — the parameters available to a scenario directly.
• Core module catalog — modules, including Keeper-side ones (on: keeper).