Surviving Worker Loss

Kill a worker mid-run and watch a second one finish it from the shared store. The steps use the agent scaffold from pyrula init --agent (Agents Quickstart) and pyrula worker; plain workflows use the same claim, lock and requeue path.

Start Valkey

docker run -d --name pyrula-valkey -p 6379:6379 valkey/valkey:9.1.3

Start two workers

In two terminals, from the scaffolded directory, against the same store and agent:

pyrula worker --agent app:support --store redis://localhost:6379 \
  --llm-ref app:llm --lock-ttl 5 --orphan-scan-interval 5

--agent app:support resolves against the current directory (--app-dir overrides). --llm-ref app:llm is the scaffold’s fake model; a worker needs an explicit --llm-ref whenever its agent declares one. --lock-ttl and --orphan-scan-interval default to 30 s and 60 s; lowered here so the handoff takes seconds.

Submit a run

From a third terminal:

python -c "
from pyrula.agents import PyrulaWorker, ValkeyStore
from app import support

submitter = PyrulaWorker(agents=[support], store=ValkeyStore(url='redis://localhost:6379'), cloud=False)
run_id = submitter.submit('support', message='Where is order A-1042?')
print(run_id)
"

One worker claims it and logs the lookup_order tool call. The fake model finishes in under a second, so kill fast or submit again.

Kill the worker holding the run

Use kill -9; Ctrl-C lets the worker drain in-flight turns, which skips the crash:

kill -9 <pid>

The run stays claimed by the dead worker until its TTL lapses and the survivor’s orphan scan notices, seconds here.

Recovery

The survivor’s orphan scan finds the expired claim, writes run:requeued to the run’s stream and picks the run up. If lookup_order had finished, it’s read back from the journal, not rerun.

Every write is fenced in the store to the worker’s claim: it commits only while that lease still holds the run’s lock and the run hasn’t ended. A worker whose claim expired (even one that was just paused) gets its writes refused and stands down, so two workers never both record the same step.

Backends that keep lock and journal together (MemoryStore, ValkeyStore) fence atomically. CompositeStore splits them across two backends and checks the lock before a second call to append, so fencing is best-effort: a lock that moves in between can let one stale write through. Use a single-backend store if that gap matters.