# Errors and retries

> Retry a step with RetryPolicy, and choose Transient, Poison or Fatal to say what a failure means.

Source: https://docs.pyrula.io/workflows/guides/errors-and-retries/

A step that raises does not retry by default. The exception comes back out of
`ctx.step`, where the body can catch it. If nothing catches it, the run ends with
`run:error`.

```python
from pyrula.workflows import RetryPolicy, workflow

@workflow(name="sync")
async def sync(ctx, account: str) -> str:
    try:
        return await ctx.step(
            "pull",
            pull_account, account,
            retry=RetryPolicy(max_attempts=5, initial_backoff=0.5),
            timeout=30,
        )
    except ConnectionError:
        return "skipped: upstream down"
```

A failure the body catches is a recorded outcome. `step:error` stores the class (`module:qualname`), message, and args and attributes when JSON. On replay the same call raises the same exception, so the body takes the same branch and the step doesn't re-run.

Pyrula rebuilds the exception from the record, including `OSError` fields, `TimeoutError` (also the one `timeout=` raises), custom `__init__`, `__slots__` and `add_note` notes. The class must be importable from a loaded module.

Three kinds can't be rebuilt: an exception with a non-JSON attribute (HTTP response, client object), a pydantic `ValidationError`, and a class defined inside a function. Replay raises `pyrula.RecordedError` with the recorded `type_path` and `message`, and the retry policy isn't consulted. When the original class is loaded and isn't an `OSError`, `RecordedError` also subclasses it, so `except` on it still matches. Its attributes (`response`, `code`) are absent, so keep what the body reads JSON-able.

A failure the body doesn't catch ends the attempt and isn't replayed. On requeue (Kafka retry, or recovery after the owner was lost) the step runs again.

## Return values

The return value is journaled in `run:complete`, so it must be JSON, a pydantic model or a dataclass. Anything else ends the run with `run:error`, code `result_not_serializable`, naming the type, with no retry. `NaN`, `inf` and an `Enum` that isn't also a `str` fail this way.

## Suspension

`ctx.sleep`, `ctx.wait_for_signal`, `ctx.interrupt`, pause, cancel and `ctx.continue_as_new` stop the body with a `BaseException` subclass, so `except Exception` doesn't catch them. Don't wrap durable calls in `except BaseException`; clean up in `finally`.

Inside LangGraph, a pause or cancel reaching a node is re-raised as `GraphBubbleUp`, an `Exception`, so an `except Exception` in a node can swallow it. The run re-reads its controls at the next durable boundary and stops there.

## RetryPolicy

```python
RetryPolicy(
    max_attempts=3,          # counts the first try; 1 = no retry
    initial_backoff=0.1,     # seconds
    max_backoff=30.0,
    backoff_multiplier=2.0,
    retryable=None,          # tuple of exception types; None = every Exception
    jitter=0.5,              # fraction of each delay randomized; 0 = exact, 1 = full jitter
)
```

`retry=` and `timeout=` (seconds per attempt) work on `ctx.step` and `ctx.once`. A retried step records only its final outcome. Each wait is between `1 - jitter` and 1 times the computed delay, capped by `max_backoff`. `max_attempts` counts within one execution; a worker crash or lost lease starts a new one at attempt 1.

A sync function can't have both `timeout=` and `retry=`, because a thread can't be cancelled. Pyrula raises `ValueError`; make the function `async def` or drop `timeout=`.

## Saying what a failure means

Raise one of these from a step to override `retryable`:

| Raise | Retried | Meaning |
|---|---|---|
| `Transient` | yes, up to `max_attempts` | A passing condition: network blip, lock contention. |
| `Poison` | never | Bad input. Retrying can't help. |
| `Fatal` | never | Broken invariant or missing config. |
| anything else | per `retryable` | |

```python
from pyrula.workflows import Poison, Transient

def charge(order):
    resp = gateway.charge(order)
    if resp.status == 429:
        raise Transient("rate limited")
    if resp.status == 422:
        raise Poison(f"card rejected: {resp.reason}")
    return resp.id
```

In a workflow `Poison` and `Fatal` both fail the step at once. A run ending on one writes `run:error` with code `poison` or `fatal` (any other body exception writes `internal` for a workflow, `agent_error` for an agent).

Both take an optional `code=` of 1 to 64 characters from letters, digits and `_ . : -` (else `ValueError`). It lands in `run:error.user_code` at every capture level, so use an identifier such as `"card.declined"`, never data. A [Kafka binding](https://docs.pyrula.io/workflows/guides/kafka.md) then applies its own `on_error` disposition.

## When the attempt itself breaks

Some failures hit before the run can record `run:error`: the workflow isn't registered on this worker, or a corrupt event breaks replay. The worker retries up to `WorkerOptions.poison_max_retries` (3) within `poison_retry_ttl_s` (1 hour), then writes a terminal `run:error` so no worker claims the run again.

An attempt whose own code blocked the event loop past `max(3 × lock_ttl, timeout)` counts toward the same budget ([Worker](https://docs.pyrula.io/workflows/guides/worker.md)). Its quarantine is `run:error` code `max_retries`, `error_type` `EventLoopBlocked`, `detail.cause` `loop_blocked`.

## When the store refuses a write

A store refuses a batch that puts an event behind a run's terminal. If the run is still live, the same events would be refused again, so the attempt ends the run with `run:error`, code `event_refused`, and a `detail` holding the store's reason. The queue worker does not retry it; a [Kafka binding](https://docs.pyrula.io/workflows/guides/kafka.md) retries the record under its `RetryPolicy` like any other code. If the store already shows the run as ended (a late write of a sibling step), the attempt only stands down and writes nothing.

## When the store fails under a run

A store call that fails because the store was unreachable or timed out (connection error, timeout, out-of-memory refusal) doesn't fail the run. The worker abandons the attempt, appends `run:requeued` with `reason: store_unavailable`, and releases the run. The next claim replays the journal as after a crash: a committed `step:done` isn't rerun, an uncommitted one is. After a journal write with unknown outcome the attempt stops writing and never re-sends, so a step isn't recorded twice.

Each outage requeue counts against `max_store_requeues` (default 20), separate from the `max_retries` (3) a lost owner spends. Exhausting it ends the run with `run:error`, code `store_unavailable`, `detail.attempts`. Both budgets count requeues since the run last made progress (a new `step:done` or handled update).

If an outage outlasts the store's `lock_ttl`, the lock lapses and the orphan sweep requeues as `owner_gone`, spending `max_retries`.

Only transport failures count (connection and timeout errors, a loading, read-only or out-of-memory server, a cluster down). Any other store error is raised once and ends the attempt as a workflow error.

`InlineWorkflowRunner` requeues as `store_unavailable` and raises `StoreUnavailable`; a later `run(name, run_id)` resumes the run.

A [Kafka-driven run](https://docs.pyrula.io/workflows/guides/kafka.md#a-store-outage-holds-the-record) is held and claimed again without spending `max_retries` or reaching `on_error`. 

Runs that no worker can serve because of a [version](https://docs.pyrula.io/workflows/guides/versioning.md) mismatch wait by default. To fail them with `no_compatible_worker` after a timeout:

```python
from pyrula.workflows import WorkerOptions
from pyrula.workflows.versioning import StrandedPolicy

WorkerOptions(stranded_policy=StrandedPolicy.FAIL, stranded_timeout=300)
```

## Retry a failed run

`RetryPolicy` covers attempts inside one run. Once a run has ended in `run:error`, `client.retry(name, run_id)` (or `pyrula runs retry`) starts a new linked run that serves every recorded step and runs the failed step onward live ([Retrying a failed run](https://docs.pyrula.io/workflows/concepts/replay-and-recovery.md#retrying-a-failed-run)). A cancelled run is refused with `retry_cancelled`.

## Caller errors

| Exception | Raised by |
|---|---|
| `ChildWorkflowError` | `ctx.run_child` when the child fails |
| `QueryError` | a query for an unknown name |
| `RunNotFoundError` / `RunFinishedError` / `RunNotWaitingError` | a client call on an unknown run / a finished run / an interrupt that isn't open |
| `UpdateRejected` / `UpdateError` / `UpdateTimeout` | `update` when the validator rejects it / the handler raises / no worker applies it in time |
| `DurableInvocationError` / `DurableInvocationTimeout` / `DurableInvocationCancelled` | `run_and_await` and the integrations built on it |
| `ContinuationChainError` | a continuation chain can't be followed (the `reason` says why) |

All are importable from `pyrula.workflows`.