State migrations

An incarnation’s state (incarnation.state) is a service’s runtime data: the list of created users, current limits, chosen nodes, and so on. The shape of this data is versioned by the state_schema_version field in service.yml. When the service author changes the shape of the state (renames a field, turns an array into a map, introduces a new limit), they bump the schema version and describe a migration: transforming the state from the old version to the new one.

A migration is a pure function “old state → new state”. It runs on the Keeper side, does not reach out to hosts, has no side effects, and depends only on the old state.

Why it’s needed

A service is long-lived: its repository gets updated, and incarnations accumulate state. If you simply ship a new service version with a different state shape, old incarnations break — their state is written to the previous schema. A migration brings existing state to the new shape during an upgrade, without requiring the incarnation to be recreated.

Key properties:

Forward-only. Migrations go only forward, from version N to N+1. There is no reverse migration — the rollback path goes through the state history (see below).
Atomicity. The migration chain is applied in a single PostgreSQL transaction: either the state moves fully to the target version, or it stays as it was on any error.
Purity. A migration reads no secrets, contacts no hosts, and does not depend on the current time or on operator parameters. The same input always yields the same output — which makes migrations reproducible and testable.

File layout

Migrations live in the service repository, in the migrations/ directory. One file is one migration step, and the file name encodes the transition between versions:

my-service/
├── service.yml                       # state_schema_version: 3
├── migrations/
│   ├── 001_to_002.yml                # step 1 → 2
│   ├── 001_to_002/
│   │   └── tests/
│   │       ├── users-array-to-map.yml
│   │       └── empty-users.yml
│   ├── 002_to_003.yml                # step 2 → 3
│   └── 002_to_003/
│       └── tests/...
└── ...

The file name is NNN_to_MMM.yml: three zero-padded digits, separated by _to_. The chain 001 → 002 → 003 → … is run by Keeper sequentially during an upgrade. Next to each migration file sits a directory of the same name holding the tests (NNN_to_MMM/tests/).

Migration DSL

A migration file is a pair of versions plus a list of transform: operations applied in order. Each operation sees the state as already mutated by the previous operations in the same migration.

from_version: 1
to_version: 2

description: >
  Migrate from a users[] array to a users{name: {acl, enabled}} map
  to support per-user ACL and an enabled/disabled flag.

transform:
  # Rename the field to free up the name for the new shape.
  - rename: { from: state.users, to: state.users_legacy }

  # CEL expression in the value: recompute the limit from MB to bytes.
  - set:
      path: state.maxmemory_bytes
      value: "${ int(state.maxmemory_mb) * 1048576 }"

  - delete: { path: state.maxmemory_mb }

  # Iterate over the old list: for each name, create an entry in the map.
  - foreach: "${ state.users_legacy }"
    as: user_name
    do:
      - set:
          path: "state.users.${ user_name }"
          value:
            acl: "off ~* &* +@all"
            enabled: false

  - delete: { path: state.users_legacy }

`transform:` operations

Operation	Parameters	Semantics
`rename`	`from: <path>`, `to: <path>`	Move the value from `from` to `to`. If `to` already exists — error (do a `delete` before the `rename`).
`set`	`path: <path>`, `value: <yaml>` or `<CEL>`	Write `value` to `path`. An existing key is overwritten. `value` is a YAML literal (scalar/list/map), a CEL expression via `${ … }`, or a nested structure with embedded `${ … }` interpolations.
`delete`	`path: <path>`	Delete the value at `path`. If there is no value, it’s not an error — the operation does nothing.
`move`	`from: <path>`, `to: <path>`	Same as `rename`.
`foreach`	`in: <CEL>` (or the short form `foreach: <CEL>`), `as: <var>`, `do: [<operation>, …]`	A structural loop over a list or a map’s values. At each step `<var>` binds to the current element, and `do:` is a nested list of operations. Inside `do:`, both `<var>` and the whole current `state` are available.

Addressing — `path:`

Dotted notation from the state root:

The state. prefix is required — it names the scope explicitly.
Nesting is written with dots: state.limits.maxmemory.
A path segment can be an interpolation: state.users.${ user_name }.acl — the segment name is computed by a CEL expression.

CEL in migrations

Any value in set.value, in foreach.in, and in path: segments can be a CEL expression via the ${ … } marker. The standard CEL operators and built-in functions are available (int, string, bool, size, has, keys, values, the comprehensions map/filter/all/exists).

The expression context is deliberately narrow — a migration must be a pure function of the old state:

Available	What it is
`state`	The current state, mutated as the operations run. Its root is `incarnation.state`.
`<as>` inside `foreach.do`	The current iteration element: a map value or a list element.

Forbidden	Why
`vault(…)`	A migration doesn’t pull secrets.
`now()`	Reproducibility: one input → one output, otherwise tests aren’t deterministic.
`register.*`	No host context — a migration runs on the Keeper, not on a host.
`soulprint.*`	Same: there are no host facts during a migration.
`essence.*`	A migration depends only on the old state, not on the service’s current parameters.
`input.*`	A migration takes no operator parameters.

This sandbox is part of the design, not a temporary limitation: thanks to it, a migration’s result depends on exactly one input — the old state.

Incarnation upgrade

Moving an incarnation to a new schema version is an explicit operator step, not an automatic “lazy” update on first access. The operator initiates an upgrade to a target version (keeper.incarnation.upgrade name=<incarnation> to_version=<N>), and Keeper runs the migration chain in a single PostgreSQL transaction:

BEGIN.
Read the incarnation’s current state and state_schema_version under a row lock.
Apply the migrations sequentially in memory: state_v1 → state_v2 → state_v3 → … up to the target version.
At each step, write a snapshot to the state history (state_history): what it was before and what it became after, marked as a migration.
Update the incarnation’s state, state_schema_version, and service version.
COMMIT.

If any step fails — ROLLBACK: the state stays as it was, the incarnation is flagged with a migration-error status, and the upgrade does not count as done. All or nothing.

Since migrations are forward-only, the recovery path after a failed or unwanted upgrade is the state history (state_history), which holds a snapshot of the previous state before every change.

Migration tests

Every migration has tests — they live in migrations/<NNN_to_MMM>/tests/<case>.yml. A test sets the state before the migration and the expected state after; the runner applies the migration and checks the result for an exact match.

name: users-array-to-map
description: >
  Base case: an array of names becomes a map with per-user ACL.

state_before:
  users: ["app", "monitor"]
  maxmemory_mb: 512

state_after:
  users:
    app:     { acl: "off ~* &* +@all", enabled: false }
    monitor: { acl: "off ~* &* +@all", enabled: false }
  maxmemory_bytes: 536870912

What the runner does:

loads state_before as state;
applies the transform: operations from the migration file;
compares the resulting state against state_after (a full, deep comparison).

It’s worth covering not just the “normal” case with tests but the edge cases too: an empty array, a missing optional field, an already partially migrated state. Tests run offline, applying nothing on hosts.

Examples

Renaming a field

The field max_clients is renamed to max_connections without changing the value:

from_version: 3
to_version: 4

description: Rename max_clients → max_connections.

transform:
  - rename: { from: state.max_clients, to: state.max_connections }

Test:

name: rename-max-clients
state_before:
  max_clients: 1000
state_after:
  max_connections: 1000

`foreach` over an array

The old state stored nodes as a list of name strings; the new schema is a list of maps with an added weight field defaulted:

from_version: 5
to_version: 6

description: nodes[] from a list of names → a list of {name, weight} objects.

transform:
  - rename: { from: state.nodes, to: state.nodes_legacy }
  - set: { path: state.nodes, value: [] }

  - foreach: "${ state.nodes_legacy }"
    as: node_name
    do:
      - set:
          path: "state.nodes_by_name.${ node_name }"
          value:
            name: "${ node_name }"
            weight: 100

  - delete: { path: state.nodes_legacy }