Errors and retries

A step that raises does not retry by default. The exception comes back out of ctx.step, where the body can catch it. If nothing catches it, the run ends with run:error.

from pyrula.workflows import RetryPolicy, workflow

@workflow(name="sync")
async def sync(ctx, account: str) -> str:
    try:
        return await ctx.step(
            "pull",
            pull_account, account,
            retry=RetryPolicy(max_attempts=5, initial_backoff=0.5),
            timeout=30,
        )
    except ConnectionError:
        return "skipped: upstream down"

A failure the body catches is a recorded outcome. step:error stores the class (module:qualname), message, and args and attributes when JSON. On replay the same call raises the same exception, so the body takes the same branch and the step doesn’t re-run.

Pyrula rebuilds the exception from the record, including OSError fields, TimeoutError (also the one timeout= raises), custom __init__, __slots__ and add_note notes. The class must be importable from a loaded module.

Three kinds can’t be rebuilt: an exception with a non-JSON attribute (HTTP response, client object), a pydantic ValidationError, and a class defined inside a function. Replay raises pyrula.RecordedError with the recorded type_path and message, and the retry policy isn’t consulted. When the original class is loaded and isn’t an OSError, RecordedError also subclasses it, so except on it still matches. Its attributes (response, code) are absent, so keep what the body reads JSON-able.

A failure the body doesn’t catch ends the attempt and isn’t replayed. On requeue (Kafka retry, or recovery after the owner was lost) the step runs again.

Return values

The return value is journaled in run:complete, so it must be JSON, a pydantic model or a dataclass. Anything else ends the run with run:error, code result_not_serializable, naming the type, with no retry. NaN, inf and an Enum that isn’t also a str fail this way.

Suspension

ctx.sleep, ctx.wait_for_signal, ctx.interrupt, pause, cancel and ctx.continue_as_new stop the body with a BaseException subclass, so except Exception doesn’t catch them. Don’t wrap durable calls in except BaseException; clean up in finally.

Inside LangGraph, a pause or cancel reaching a node is re-raised as GraphBubbleUp, an Exception, so an except Exception in a node can swallow it. The run re-reads its controls at the next durable boundary and stops there.

RetryPolicy

RetryPolicy(
    max_attempts=3,          # counts the first try; 1 = no retry
    initial_backoff=0.1,     # seconds
    max_backoff=30.0,
    backoff_multiplier=2.0,
    retryable=None,          # tuple of exception types; None = every Exception
    jitter=0.5,              # fraction of each delay randomized; 0 = exact, 1 = full jitter
)

retry= and timeout= (seconds per attempt) work on ctx.step and ctx.once. A retried step records only its final outcome. Each wait is between 1 - jitter and 1 times the computed delay, capped by max_backoff. max_attempts counts within one execution; a worker crash or lost lease starts a new one at attempt 1.

A sync function can’t have both timeout= and retry=, because a thread can’t be cancelled. Pyrula raises ValueError; make the function async def or drop timeout=.

Saying what a failure means

Raise one of these from a step to override retryable:

Raise	Retried	Meaning
Transient	yes, up to max_attempts	A passing condition: network blip, lock contention.
Poison	never	Bad input. Retrying can’t help.
Fatal	never	Broken invariant or missing config.
anything else	per retryable

from pyrula.workflows import Poison, Transient

def charge(order):
    resp = gateway.charge(order)
    if resp.status == 429:
        raise Transient("rate limited")
    if resp.status == 422:
        raise Poison(f"card rejected: {resp.reason}")
    return resp.id

In a workflow Poison and Fatal both fail the step at once. A run ending on one writes run:error with code poison or fatal (any other body exception writes internal for a workflow, agent_error for an agent).

Both take an optional code= of 1 to 64 characters from letters, digits and _ . : - (else ValueError). It lands in run:error.user_code at every capture level, so use an identifier such as "card.declined", never data. A Kafka binding then applies its own on_error disposition.

When the attempt itself breaks

Some failures hit before the run can record run:error: the workflow isn’t registered on this worker, or a corrupt event breaks replay. The worker retries up to WorkerOptions.poison_max_retries (3) within poison_retry_ttl_s (1 hour), then writes a terminal run:error so no worker claims the run again.

An attempt whose own code blocked the event loop past max(3 × lock_ttl, timeout) counts toward the same budget (Worker). Its quarantine is run:error code max_retries, error_type EventLoopBlocked, detail.cause loop_blocked.

When the store refuses a write

A store refuses a batch that puts an event behind a run’s terminal. If the run is still live, the same events would be refused again, so the attempt ends the run with run:error, code event_refused, and a detail holding the store’s reason. The queue worker does not retry it; a Kafka binding retries the record under its RetryPolicy like any other code. If the store already shows the run as ended (a late write of a sibling step), the attempt only stands down and writes nothing.

When the store fails under a run

A store call that fails because the store was unreachable or timed out (connection error, timeout, out-of-memory refusal) doesn’t fail the run. The worker abandons the attempt, appends run:requeued with reason: store_unavailable, and releases the run. The next claim replays the journal as after a crash: a committed step:done isn’t rerun, an uncommitted one is. After a journal write with unknown outcome the attempt stops writing and never re-sends, so a step isn’t recorded twice.

Each outage requeue counts against max_store_requeues (default 20), separate from the max_retries (3) a lost owner spends. Exhausting it ends the run with run:error, code store_unavailable, detail.attempts. Both budgets count requeues since the run last made progress (a new step:done or handled update).

If an outage outlasts the store’s lock_ttl, the lock lapses and the orphan sweep requeues as owner_gone, spending max_retries.

Only transport failures count (connection and timeout errors, a loading, read-only or out-of-memory server, a cluster down). Any other store error is raised once and ends the attempt as a workflow error.

InlineWorkflowRunner requeues as store_unavailable and raises StoreUnavailable; a later run(name, run_id) resumes the run.

A Kafka-driven run is held and claimed again without spending max_retries or reaching on_error.

Runs that no worker can serve because of a version mismatch wait by default. To fail them with no_compatible_worker after a timeout:

from pyrula.workflows import WorkerOptions
from pyrula.workflows.versioning import StrandedPolicy

WorkerOptions(stranded_policy=StrandedPolicy.FAIL, stranded_timeout=300)

Retry a failed run

RetryPolicy covers attempts inside one run. Once a run has ended in run:error, client.retry(name, run_id) (or pyrula runs retry) starts a new linked run that serves every recorded step and runs the failed step onward live (Retrying a failed run). A cancelled run is refused with retry_cancelled.

Caller errors

Exception	Raised by
ChildWorkflowError	ctx.run_child when the child fails
QueryError	a query for an unknown name
RunNotFoundError / RunFinishedError / RunNotWaitingError	a client call on an unknown run / a finished run / an interrupt that isn’t open
UpdateRejected / UpdateError / UpdateTimeout	update when the validator rejects it / the handler raises / no worker applies it in time
DurableInvocationError / DurableInvocationTimeout / DurableInvocationCancelled	run_and_await and the integrations built on it
ContinuationChainError	a continuation chain can’t be followed (the reason says why)

All are importable from pyrula.workflows.