# Surviving Worker Loss

> Kill a worker mid-run and watch a second one finish it without repeating completed steps.

Source: https://docs.pyrula.io/workflows/guides/surviving-worker-loss/

Kill a worker mid-run and watch a second one finish it from the shared store. The steps use the
agent scaffold from `pyrula init --agent` ([Agents Quickstart](https://docs.pyrula.io/agents/quickstart.md)) and
`pyrula worker`; plain workflows use the same claim, lock and requeue path.

## Start Valkey

```bash
docker run -d --name pyrula-valkey -p 6379:6379 valkey/valkey:9.1.3
```

## Start two workers

In two terminals, from the scaffolded directory, against the same store and agent:

```bash
pyrula worker --agent app:support --store redis://localhost:6379 \
  --llm-ref app:llm --lock-ttl 5 --orphan-scan-interval 5
```

`--agent app:support` resolves against the current directory (`--app-dir` overrides).
`--llm-ref app:llm` is the scaffold's fake model; a worker needs an explicit `--llm-ref` whenever
its agent declares one. `--lock-ttl` and `--orphan-scan-interval` default to 30 s and 60 s;
lowered here so the handoff takes seconds.

## Submit a run

From a third terminal:

```bash
python -c "
from pyrula.agents import PyrulaWorker, ValkeyStore
from app import support

submitter = PyrulaWorker(agents=[support], store=ValkeyStore(url='redis://localhost:6379'), cloud=False)
run_id = submitter.submit('support', message='Where is order A-1042?')
print(run_id)
"
```

One worker claims it and logs the `lookup_order` tool call. The fake model finishes in under a
second, so kill fast or submit again.

## Kill the worker holding the run

Use `kill -9`; `Ctrl-C` lets the worker drain in-flight turns, which skips the crash:

```bash
kill -9 <pid>
```

The run stays claimed by the dead worker until its TTL lapses and the survivor's orphan scan
notices, seconds here.

## Recovery

The survivor's orphan scan finds the expired claim, writes `run:requeued` to the run's stream and
picks the run up. If `lookup_order` had finished, it's read back from the journal, not rerun.

Every write is fenced in the store to the worker's claim: it commits only while that lease still
holds the run's lock and the run hasn't ended. A worker whose claim expired (even one that was
just paused) gets its writes refused and stands down, so two workers never both record the same
step.

Backends that keep lock and journal together (`MemoryStore`, `ValkeyStore`) fence atomically.
`CompositeStore` splits them across two backends and checks the lock before a second call to
append, so fencing is best-effort: a lock that moves in between can let one stale write through.
Use a single-backend store if that gap matters.