# Replay and Recovery > step:pending before a step, step:done after. A crash resumes from the first step without a step:done. Source: https://docs.pyrula.io/workflows/concepts/replay-and-recovery/ `step:pending` goes in before a durable step, `step:done` after it succeeds. Recovery uses the pair to find the first unfinished step. ## After a crash When a worker dies or loses its lock, another worker claims the run, replays the stream and resumes at the first step with no `step:done`. Finished steps return their recorded results. ```python @workflow(name="fulfill_order") async def fulfill_order(ctx, order_id: str) -> str: charge_id = await ctx.step("charge", lambda: charge_card(order_id)) await ctx.step("email_receipt", lambda: send_receipt(order_id, charge_id)) return charge_id ``` ```text run:init step:pending charge step:done charge result="ch_1a2b" # worker crashes here --- another worker claims the run and replays the stream --- step:done charge # from the stream; the card is NOT re-charged step:pending email_receipt step:done email_receipt run:complete result="ch_1a2b" ``` ## The window where a step can run twice The crash that matters falls between `charge_card` returning and `step:done` reaching the store: ```text run:init step:pending charge # charge_card() has charged the card... # ...worker dies here, before step:done --- another worker claims the run and replays the stream --- step:pending charge # no step:done, so charge runs AGAIN step:done charge ``` Side effects are at-least-once. The window is the round trip that writes `step:done`; with `step_boundary_durability=False` it widens to every step in the unflushed batch ([Journal durability](https://docs.pyrula.io/workflows/guides/worker.md#journal-durability)). Close it with an idempotency key the downstream dedupes on. `run_id` and the step name are stable across replays: ```python charge_id = await ctx.step( "charge", charge_card, order_id, kwargs={"idempotency_key": f"{ctx.run_id}:charge"}, ) ``` ### `ctx.once` `ctx.once(key, fn)` stores `fn`'s result under your key and returns it on any later attempt, across runs. It guards against duplicate submissions, not crashes. ```python charge_id = await ctx.once(f"charge:{order_id}", lambda: charge_card(order_id)) ``` It has the same crash window as a step, and a crash mid-retry inside a `ctx.step` re-runs the whole step. ## Retrying a failed run A run that ended in `run:error` (or was [terminated](https://docs.pyrula.io/workflows/guides/cancel-and-terminate.md)) never gets a second terminal event. A retry is a new linked run that serves every step recorded before the failure and runs the failed step and everything after live. To try a different input, result or model with writes stopped, use a [what-if](https://docs.pyrula.io/workflows/guides/what-if.md). ```python handle = await client.retry("fulfill_order", failed_run_id) # or handle.retry() ``` ```bash pyrula runs retry # --dry-run prints the plan only curl -X POST $BASE/fulfill_order/runs//retry # 202, 409 with a code when refused ``` `client.plan_retry(name, run_id)` returns the `RetryPlan` without starting anything. The CLI targets `--store` (or `PYRULA_STORE_URL`): a SQLite file or Valkey (`--namespace`, `--cluster`). It doesn't import your workflow code, so the retry carries no `Concurrency` or `Rate` key; use `client.retry` or the route when those limits must apply. **Where it starts.** The cut is the `step:error` that escaped the body, or the terminal `run:error` if the body raised outside any step. The retry's journal is the failed run's journal before the cut, minus `run:init`, and all of it is served: finished steps, interrupt answers, timers, completed children, and step failures the body caught. The failed step and everything after run again. **The new run.** Its id is the failed id plus `.retry`, so asking twice returns the same run. Provenance: `trigger: "retry"`, `parent_run_id`, `attempt` + 1, `source` with `retry_of`, `cut_seq`, `seeded_events`. A failed retry can be retried again. **Writes are real.** A step that started with no recorded outcome (worker died mid-step) is in doubt and runs again. `RetryPlan.in_doubt`, the CLI and the route response list them. Give non-idempotent writes an idempotency key. **Deploy first, then retry.** Each claiming worker replays the copied prefix against its code and ends the retry with `incompatible_upgrade` on divergence (renamed or removed step, changed inputs). A worker still on old code can claim the retry and fail it again, so roll the fix to every worker first. Refusals raise `RetryRefused` with a stable `code` (HTTP 409 on the route), before anything is written: - `retry_not_failed`: the run completed, continued as new, or has not finished. - `retry_cancelled`: the journal has a caller's `run:cancel`. A cancel latched by an `on_interrupt` handler failure (`interrupt_handler_failed`) is engine-written, so that run is retryable. - `retry_counterfactual`: the run was started by `replay`, `case` or `eval`. Use a [what-if](https://docs.pyrula.io/workflows/guides/what-if.md). - `retry_child_failure`: the failure escaped from a child run. Retry the failed child, which is a run of its own. - `retry_recorded_wait_timeout`: the run ended on `wait_timeout`. A timed-out wait the body caught does not refuse. - `retry_agent_run`: agent turns are not retried; submit the turn again. - `retry_source_changed`: the failed run no longer ends as it did, or a retry never received its prefix (retry the original run). `retry_source_missing` (the failed run expired or was deleted before the claim) and `retry_source_changed` also appear as the retry run's own `run:error` when its worker seeds it. ## Cloud delivery guarantees Your store is the source of truth; Cloud is a replica. - Critical events (lifecycle, `step:done` / `step:error`, tool and usage results, park/resume, gap markers) are spooled to disk and dropped only on Cloud's ack. Token deltas, `llm:start`, `tool:start`, `step:pending` and heartbeats are best-effort. - A sweep at worker start and on each orphan scan re-sends any terminal run Cloud hasn't confirmed. - If a `maxlen`-trimmed `ValkeyStore` run is trimmed before Cloud has it, the runtime logs `cloud_sweep_truncated` with the count. Delivery has no latency bound. See [Connect to Pyrula Cloud](https://docs.pyrula.io/getting-started/cloud.md#not-losing-events). ## History limits `ValkeyStore` keeps `maxlen` events per run (default 100,000), trimming the oldest. A trimmed run can't replay safely: `TruncatedHistoryError` with the missing count. `ArchivingStore` falls back to the full archived copy. A run normally never gets there: it ends first at its event-count cap (`max_event_count_per_run`, default 50,000, and never more than the store keeps minus a margin of 1,000 or a tenth of `maxlen`, whichever is smaller, with a margin never under 2; `maxlen` must be at least 3) with `stream:overflow` and `run:error` (`stream_too_large`), and its history stays whole and readable. The `ValkeyStore` replay pages are also capped at `replay_page_bytes` (default 8 MiB) as well as `replay_page_size` entries, and each read from the server is sized from the largest event seen so far, so reading a run of large events does not build a reply of hundreds of them on the server (a read is at most 128 events, so the first one that meets events much bigger than any before it can still be large: 128 MB for megabyte events, the default payload limit, and more in proportion when that limit is lifted); a short page loses no event, and the cursor continues from the last one it returned. Long-lived runs: `ctx.continue_as_new()` ends the run and starts a fresh one carrying state forward. Raising `maxlen` only moves the ceiling. Address a chain by the id you started with. Status, results, queries and control reach the live run, and a signal, update, cancel or pause sent mid-hop carries to the next run. The start id works while the chain is open and for `completed_ttl` after it ends. ## Step results A step's result is stored as JSON in `step:done`; `ctx.step` raises if it can't serialize. Pyrula types use their [plain JSON form](https://docs.pyrula.io/primitives/json-form.md). An annotated `@step` restores its declared result type on replay (sum types, typed errors, sealed cases). ## Replay primitives ```python from pyrula.workflows import replay_all_events, reconstruct_state, ReplayState ``` `replay_all_events` reads a run's full stream; `reconstruct_state` folds it into a `ReplayState`. See [Determinism](https://docs.pyrula.io/workflows/concepts/determinism.md) for what replay asks of your code.