Replay and Recovery

step:pending goes in before a durable step, step:done after it succeeds. Recovery uses the pair to find the first unfinished step.

After a crash

When a worker dies or loses its lock, another worker claims the run, replays the stream and resumes at the first step with no step:done. Finished steps return their recorded results.

@workflow(name="fulfill_order")
async def fulfill_order(ctx, order_id: str) -> str:
    charge_id = await ctx.step("charge", lambda: charge_card(order_id))
    await ctx.step("email_receipt", lambda: send_receipt(order_id, charge_id))
    return charge_id

run:init
step:pending   charge
step:done      charge          result="ch_1a2b"     # worker crashes here
--- another worker claims the run and replays the stream ---
step:done      charge          # from the stream; the card is NOT re-charged
step:pending   email_receipt
step:done      email_receipt
run:complete   result="ch_1a2b"

The window where a step can run twice

The crash that matters falls between charge_card returning and step:done reaching the store:

run:init
step:pending   charge
                               # charge_card() has charged the card...
                               # ...worker dies here, before step:done
--- another worker claims the run and replays the stream ---
step:pending   charge          # no step:done, so charge runs AGAIN
step:done      charge

Side effects are at-least-once. The window is the round trip that writes step:done; with step_boundary_durability=False it widens to every step in the unflushed batch (Journal durability).

Close it with an idempotency key the downstream dedupes on. run_id and the step name are stable across replays:

charge_id = await ctx.step(
    "charge",
    charge_card,
    order_id,
    kwargs={"idempotency_key": f"{ctx.run_id}:charge"},
)

ctx.once

ctx.once(key, fn) stores fn’s result under your key and returns it on any later attempt, across runs. It guards against duplicate submissions, not crashes.

charge_id = await ctx.once(f"charge:{order_id}", lambda: charge_card(order_id))

It has the same crash window as a step, and a crash mid-retry inside a ctx.step re-runs the whole step.

Retrying a failed run

A run that ended in run:error (or was terminated) never gets a second terminal event. A retry is a new linked run that serves every step recorded before the failure and runs the failed step and everything after live. To try a different input, result or model with writes stopped, use a what-if.

handle = await client.retry("fulfill_order", failed_run_id)   # or handle.retry()

pyrula runs retry <run_id>                           # --dry-run prints the plan only
curl -X POST $BASE/fulfill_order/runs/<run_id>/retry # 202, 409 with a code when refused

client.plan_retry(name, run_id) returns the RetryPlan without starting anything.

The CLI targets --store (or PYRULA_STORE_URL): a SQLite file or Valkey (--namespace, --cluster). It doesn’t import your workflow code, so the retry carries no Concurrency or Rate key; use client.retry or the route when those limits must apply.

Where it starts. The cut is the step:error that escaped the body, or the terminal run:error if the body raised outside any step. The retry’s journal is the failed run’s journal before the cut, minus run:init, and all of it is served: finished steps, interrupt answers, timers, completed children, and step failures the body caught. The failed step and everything after run again.

The new run. Its id is the failed id plus .retry, so asking twice returns the same run. Provenance: trigger: "retry", parent_run_id, attempt + 1, source with retry_of, cut_seq, seeded_events. A failed retry can be retried again.

Writes are real. A step that started with no recorded outcome (worker died mid-step) is in doubt and runs again. RetryPlan.in_doubt, the CLI and the route response list them. Give non-idempotent writes an idempotency key.

Deploy first, then retry. Each claiming worker replays the copied prefix against its code and ends the retry with incompatible_upgrade on divergence (renamed or removed step, changed inputs). A worker still on old code can claim the retry and fail it again, so roll the fix to every worker first.

Refusals raise RetryRefused with a stable code (HTTP 409 on the route), before anything is written:

• retry_not_failed: the run completed, continued as new, or has not finished.
• retry_cancelled: the journal has a caller’s run:cancel. A cancel latched by an on_interrupt handler failure (interrupt_handler_failed) is engine-written, so that run is retryable.
• retry_counterfactual: the run was started by replay, case or eval. Use a what-if.
• retry_child_failure: the failure escaped from a child run. Retry the failed child, which is a run of its own.
• retry_recorded_wait_timeout: the run ended on wait_timeout. A timed-out wait the body caught does not refuse.
• retry_agent_run: agent turns are not retried; submit the turn again.
• retry_source_changed: the failed run no longer ends as it did, or a retry never received its prefix (retry the original run).

retry_source_missing (the failed run expired or was deleted before the claim) and retry_source_changed also appear as the retry run’s own run:error when its worker seeds it.

Cloud delivery guarantees

Your store is the source of truth; Cloud is a replica.

• Critical events (lifecycle, step:done / step:error, tool and usage results, park/resume, gap markers) are spooled to disk and dropped only on Cloud’s ack. Token deltas, llm:start, tool:start, step:pending and heartbeats are best-effort.
• A sweep at worker start and on each orphan scan re-sends any terminal run Cloud hasn’t confirmed.
• If a maxlen-trimmed ValkeyStore run is trimmed before Cloud has it, the runtime logs cloud_sweep_truncated with the count.

Delivery has no latency bound. See Connect to Pyrula Cloud.

History limits

ValkeyStore keeps maxlen events per run (default 100,000), trimming the oldest. A trimmed run can’t replay safely: TruncatedHistoryError with the missing count. ArchivingStore falls back to the full archived copy.

A run normally never gets there: it ends first at its event-count cap (max_event_count_per_run, default 50,000, and never more than the store keeps minus a margin of 1,000 or a tenth of maxlen, whichever is smaller, with a margin never under 2; maxlen must be at least 3) with stream:overflow and run:error (stream_too_large), and its history stays whole and readable. The ValkeyStore replay pages are also capped at replay_page_bytes (default 8 MiB) as well as replay_page_size entries, and each read from the server is sized from the largest event seen so far, so reading a run of large events does not build a reply of hundreds of them on the server (a read is at most 128 events, so the first one that meets events much bigger than any before it can still be large: 128 MB for megabyte events, the default payload limit, and more in proportion when that limit is lifted); a short page loses no event, and the cursor continues from the last one it returned.

Long-lived runs: ctx.continue_as_new() ends the run and starts a fresh one carrying state forward. Raising maxlen only moves the ceiling.

Address a chain by the id you started with. Status, results, queries and control reach the live run, and a signal, update, cancel or pause sent mid-hop carries to the next run. The start id works while the chain is open and for completed_ttl after it ends.

Step results

A step’s result is stored as JSON in step:done; ctx.step raises if it can’t serialize. Pyrula types use their plain JSON form. An annotated @step restores its declared result type on replay (sum types, typed errors, sealed cases).

Replay primitives

from pyrula.workflows import replay_all_events, reconstruct_state, ReplayState

replay_all_events reads a run’s full stream; reconstruct_state folds it into a ReplayState. See Determinism for what replay asks of your code.