Surviving Worker Loss

The Quickstart shows a run suspend and resume inside one process. This is the crash the quickstart leaves out: a worker dies with a run in hand, and a second worker, watching the same store, finishes it. Done for real, with two OS processes, not simulated.

This uses the agent scaffold from pyrula init --agent (see the Agents Quickstart), because pyrula worker is the CLI process that claims and runs turns against a shared store. The same claim, lock, and requeue mechanism runs underneath every workflow, agent-shaped or not.

Start Valkey

docker run -d --name pyrula-valkey -p 6379:6379 valkey/valkey:8

Start two workers

In two separate terminals, from the directory pyrula init --agent scaffolded into, both pointed at the same store and the same agent:

pyrula worker --agent app:support --valkey-url redis://localhost:6379 \
  --llm-ref app:llm --lock-ttl 5 --orphan-scan-interval 5

--agent app:support resolves against the current directory, so this works from wherever app.py lives. Pass --app-dir to import from somewhere else. --llm-ref app:llm points the worker at the fake model the scaffold already builds in app.py; a worker needs an explicit --llm-ref whenever the agent it’s running declares one, even the scripted offline one.

--lock-ttl is how long a worker’s claim on a run is good for before it needs renewing. --orphan-scan-interval is how often a worker checks for claims nobody renewed. The defaults (30s and 60s) are fine in production; turned down here so the handoff below happens in seconds instead of minutes.

Submit a run

From a third terminal, in the same directory:

python -c "
from pyrula.agents import PyrulaWorker, ValkeyStore
from app import support

submitter = PyrulaWorker(agents=[support], store=ValkeyStore(url='redis://localhost:6379'), cloud=False)
run_id = submitter.submit('support', message='Where is order A-1042?')
print(run_id)
"

One of the two pyrula worker processes claims it, and its terminal logs the lookup_order tool call. Against the fake model this finishes in well under a second, so to catch it mid-run you need to kill fast, or just submit again if it finishes before you get there.

Kill the worker holding the run

Find the PID of the terminal that logged the claim and kill it the way a crash would, not the way a graceful shutdown would (Ctrl-C lets a worker finish its in-flight turns and hand back its slot cleanly, which defeats the point here):

kill -9 <pid>

Nothing hands the run off immediately. It sits claimed by a worker that no longer exists until the claim’s TTL runs out and the surviving worker’s orphan scan notices. With --lock-ttl 5 --orphan-scan-interval 5 that’s a matter of seconds.

What the surviving worker does

The handoff is marked by a RUN_REQUEUED event on the run’s stream: the surviving worker’s orphan scan finds a claim past its TTL, writes RUN_REQUEUED, and picks the run up itself. Watch its terminal and the run completes there. If lookup_order had already finished before the kill, it is read back from the journal, not run again.

Until that RUN_REQUEUED handoff happens, a run’s step:done events are fenced to the worker that claimed it. A worker that lost its claim (or never had it) cannot write a step:done for that run. That fencing is what makes the takeover safe: two workers never both believe they own the same run at once, so a step can’t be recorded twice by two different workers racing each other.

Why this matters

This is the same guarantee the Quickstart shows with a suspend and resume in one process, extended across a process that no longer exists. A step that already wrote its result is never re-run, whether the resume is a scheduled sleep waking up or a second worker taking over after the first one is gone.