# Deploy a worker

> Run a workflow or agent worker in a container with probes, a clean SIGTERM drain, and Compose, Kubernetes and Render recipes.

Source: https://docs.pyrula.io/getting-started/deploy-a-worker/

A worker claims runs from the store and executes them. Sleeps, signal waits, children and
schedules need one ([Production checklist](https://docs.pyrula.io/getting-started/production-checklist.md)). The files
below live in [`examples/deploy/`](https://github.com/pyrula-io/pyrula/tree/main/examples/deploy)
and use the `/livez` and `/readyz` [health probes](https://docs.pyrula.io/workflows/guides/worker.md#health-probes).

## Entry point

| You serve | Command | Package |
|---|---|---|
| Plain `@workflow` functions | `pyrula worker --workflow app:process_order` | `pyrula-workflows` |
| `@agent` functions | `pyrula worker --agent app:support --llm-ref app:llm` | `pyrula-agents` |

One process serves workflows or agents, not both ([CLI reference](https://docs.pyrula.io/reference/cli.md#pyrula-worker)).
`--workflow` takes `module:attr` and repeats; a bare module name serves every workflow the
module registers, and mixing the two forms is refused.

## Build the image

Both Dockerfiles are multi-stage, run as uid 10001 and read connection settings from the
environment.

```dockerfile
# syntax=docker/dockerfile:1
# A production workflow worker. Build from this directory:
#   docker build -t my-workflow-worker .
# Base image pinned by tag and digest; bump both together.
ARG PYTHON_IMAGE=python:3.13-slim-bookworm@sha256:93498f36dc7e24ef233bfdf6cc17d4b8bd32d94ee737143e906460d5f5a914c5

FROM ${PYTHON_IMAGE} AS build
# Only for installing before the first release: point pip at a directory of wheels
# (--build-arg PIP_FIND_LINKS=...). Unset, pip uses the package index.
ARG PIP_FIND_LINKS
ARG PIP_TRUSTED_HOST
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_NO_CACHE_DIR=1
RUN python -m venv /opt/venv
COPY requirements.txt /tmp/requirements.txt
RUN /opt/venv/bin/pip install -r /tmp/requirements.txt

FROM ${PYTHON_IMAGE}
# Fixed uid so a Kubernetes runAsNonRoot check can verify it. The home directory is
# the only writable path: it holds the Cloud connector's event spool.
RUN useradd --system --uid 10001 --home-dir /var/lib/pyrula --create-home \
        --shell /usr/sbin/nologin pyrula
COPY --from=build /opt/venv /opt/venv
WORKDIR /app
COPY app.py .
ENV PATH=/opt/venv/bin:$PATH \
    PYTHONUNBUFFERED=1 \
    XDG_STATE_HOME=/var/lib/pyrula \
    PYRULA_WORKER_HEALTH_PORT=8081 \
    PYRULA_WORKER_HEALTH_HOST=0.0.0.0
# Connection settings (PYRULA_STORE_URL, PYRULA_POSTGRES_DSN, PYRULA_CLOUD_API_KEY)
# come from the environment at run time; nothing secret is baked into the image.
USER 10001
EXPOSE 8081
HEALTHCHECK --interval=10s --timeout=3s --start-period=20s --retries=3 \
    CMD ["python", "-c", "import urllib.request as u; u.urlopen('http://127.0.0.1:8081/readyz', timeout=2)"]
# SIGTERM starts the drain: /readyz turns 503, in-flight runs finish within
# --shutdown-grace, then the process exits 0.
STOPSIGNAL SIGTERM
ENTRYPOINT ["pyrula", "worker"]
CMD ["--workflow", "app:process_order"]
```

The agent worker differs in its entry point and its model key:

```dockerfile
# syntax=docker/dockerfile:1
# A production agent worker, served by `pyrula worker`. Build from this directory:
#   docker build -t my-agent-worker .
# Base image pinned by tag and digest; bump both together.
ARG PYTHON_IMAGE=python:3.13-slim-bookworm@sha256:93498f36dc7e24ef233bfdf6cc17d4b8bd32d94ee737143e906460d5f5a914c5

FROM ${PYTHON_IMAGE} AS build
# Only for installing before the first release: point pip at a directory of wheels
# (--build-arg PIP_FIND_LINKS=...). Unset, pip uses the package index.
ARG PIP_FIND_LINKS
ARG PIP_TRUSTED_HOST
ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_NO_CACHE_DIR=1
RUN python -m venv /opt/venv
COPY requirements.txt /tmp/requirements.txt
RUN /opt/venv/bin/pip install -r /tmp/requirements.txt

FROM ${PYTHON_IMAGE}
# Fixed uid so a Kubernetes runAsNonRoot check can verify it. The home directory is
# the only writable path: it holds the Cloud connector's event spool.
RUN useradd --system --uid 10001 --home-dir /var/lib/pyrula --create-home \
        --shell /usr/sbin/nologin pyrula
COPY --from=build /opt/venv /opt/venv
WORKDIR /app
COPY app.py .
ENV PATH=/opt/venv/bin:$PATH \
    PYTHONUNBUFFERED=1 \
    XDG_STATE_HOME=/var/lib/pyrula \
    PYRULA_WORKER_HEALTH_PORT=8081 \
    PYRULA_WORKER_HEALTH_HOST=0.0.0.0
# Connection settings (PYRULA_STORE_URL, ANTHROPIC_API_KEY, PYRULA_CLOUD_API_KEY) come
# from the environment at run time; nothing secret is baked into the image.
USER 10001
EXPOSE 8081
HEALTHCHECK --interval=10s --timeout=3s --start-period=20s --retries=3 \
    CMD ["python", "-c", "import urllib.request as u; u.urlopen('http://127.0.0.1:8081/readyz', timeout=2)"]
# SIGTERM starts the drain: /readyz turns 503, in-flight turns finish within
# --shutdown-grace, then the process exits 0.
STOPSIGNAL SIGTERM
ENTRYPOINT ["pyrula", "worker"]
CMD ["--agent", "app:support", "--llm-ref", "app:llm"]
```

```bash
cd examples/deploy/workflow-worker
docker build -t my-workflow-worker .
```

Before the first release is published, build the wheels yourself and pass
`--build-arg PIP_FIND_LINKS=<url or path>` (plus `PIP_TRUSTED_HOST=<host>` for plain HTTP).

## Configuration

| Variable | Meaning |
|---|---|
| `PYRULA_STORE_URL` | `valkey://host:port/db`, or `sqlite:////path.db` for workers on one host. Workers, clients and APIs in an environment share the URL and namespace. |
| `PYRULA_VALKEY_NAMESPACE` | Key namespace, so environments can share one instance. |
| `PYRULA_POSTGRES_DSN` | Optional archive of finished runs; workflow entry point only. Run `pyrula db migrate` first. |
| `PYRULA_WORKER_HEALTH_PORT`, `PYRULA_WORKER_HEALTH_HOST` | Probe listener. The images set `8081` and `0.0.0.0`; unset, probes are off. |
| `PYRULA_WORKER_ID` | Stable worker id (workflow entry point), e.g. the pod name. |
| `PYRULA_CLOUD_API_KEY` | Optional. Reports runs to [Pyrula Cloud](https://docs.pyrula.io/getting-started/cloud.md). |
| `XDG_STATE_HOME` | Cloud connector spool location. The images use `/var/lib/pyrula`. |
| `ANTHROPIC_API_KEY` | The agent example's model key. |

Tuning flags (`pyrula worker --help`) mirror
[`WorkerOptions`](https://docs.pyrula.io/workflows/guides/worker.md#concurrency-and-background-loops).

## Docker Compose

```yaml
# A workflow worker and Valkey on one host. From this directory:
#   docker compose up --build
# The agent worker is behind a profile and needs a model key:
#   ANTHROPIC_API_KEY=... docker compose --profile agents up --build
name: pyrula-workers

services:
  valkey:
    image: valkey/valkey:9.1.3
    # noeviction: an eviction policy deletes live run journals under memory pressure.
    # appendonly: without persistence a restart loses every in-flight run.
    command: ["valkey-server", "--appendonly", "yes", "--maxmemory-policy", "noeviction"]
    volumes:
      - valkey-data:/data
    healthcheck:
      test: ["CMD", "valkey-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

  workflow-worker:
    build: ./workflow-worker
    depends_on:
      valkey:
        condition: service_healthy
    environment:
      PYRULA_STORE_URL: valkey://valkey:6379/0
      PYRULA_VALKEY_NAMESPACE: production
      # Optional: PYRULA_POSTGRES_DSN archives finished runs; run `pyrula db migrate` first.
      # Optional: PYRULA_CLOUD_API_KEY reports runs to Pyrula Cloud.
      PYRULA_CLOUD_API_KEY: ${PYRULA_CLOUD_API_KEY:-}
    volumes:
      - workflow-spool:/var/lib/pyrula
    # The drain takes up to shutdown_grace (30 s) + 3 x cancel_timeout (5 s) + the Cloud
    # connector's drain deadline (5 s), 50 s in all; the stop grace period outlasts it.
    stop_grace_period: 60s
    restart: unless-stopped

  agent-worker:
    profiles: ["agents"]
    build: ./agent-worker
    depends_on:
      valkey:
        condition: service_healthy
    environment:
      PYRULA_STORE_URL: valkey://valkey:6379/0
      PYRULA_VALKEY_NAMESPACE: production
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:?set ANTHROPIC_API_KEY to run the agent worker}
      PYRULA_CLOUD_API_KEY: ${PYRULA_CLOUD_API_KEY:-}
    volumes:
      - agent-spool:/var/lib/pyrula
    stop_grace_period: 60s
    restart: unless-stopped

volumes:
  valkey-data:
  workflow-spool:
  agent-spool:
```

## Single-host Compose

On one machine the store can be a SQLite file on a named volume, with the same image:

```yaml
# A workflow worker on one host with a SQLite store: no Valkey service. From this directory:
#   docker compose -f docker-compose.single-host.yml up --build
# More worker containers on this host share the same file:
#   docker compose -f docker-compose.single-host.yml up --build --scale workflow-worker=3
name: pyrula-single-host

services:
  workflow-worker:
    build: ./workflow-worker
    environment:
      # The file lives on the named volume, on this host's own disk. A bind mount from a
      # network share, or the same volume used from a second host, is refused at open.
      PYRULA_STORE_URL: sqlite:////var/lib/pyrula/store/pyrula.db
      # Optional: PYRULA_CLOUD_API_KEY reports runs to Pyrula Cloud.
      PYRULA_CLOUD_API_KEY: ${PYRULA_CLOUD_API_KEY:-}
    volumes:
      # The store and the Cloud connector's event spool, shared by every container here.
      - pyrula-data:/var/lib/pyrula
    # The drain takes up to shutdown_grace (30 s) + 3 x cancel_timeout (5 s) + the Cloud
    # connector's drain deadline (5 s), 50 s in all; the stop grace period outlasts it.
    stop_grace_period: 60s
    restart: unless-stopped

volumes:
  pyrula-data:
```

`--scale workflow-worker=3` runs three workers on the one file, sharing its single writer
([Choosing a store](https://docs.pyrula.io/getting-started/choosing-a-store.md#one-writer-at-a-time)). Back up with
`pyrula store backup` from inside a container. Kubernetes and Render pods can land on different
nodes, and a second node opening the file is refused, so those recipes use Valkey.

## Kubernetes

```yaml
# A workflow worker Deployment, with the probes and drain time matched to the worker.
# Swap the image and the container args for the agent worker (`pyrula worker --agent ...`).
# The Secret holds PYRULA_STORE_URL and, optionally, PYRULA_CLOUD_API_KEY and
# PYRULA_POSTGRES_DSN:
#   kubectl create secret generic pyrula-worker \
#     --from-literal=PYRULA_STORE_URL=valkey://valkey.default.svc:6379/0
apiVersion: apps/v1
kind: Deployment
metadata:
  name: workflow-worker
spec:
  replicas: 3
  selector:
    matchLabels: { app: workflow-worker }
  template:
    metadata:
      labels: { app: workflow-worker }
    spec:
      # The worker drains for up to shutdown_grace (30 s) + 3 x cancel_timeout (5 s) + the
      # Cloud connector's drain deadline (5 s), 50 s in all; the pod gets longer before SIGKILL.
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
      containers:
        - name: worker
          image: registry.example.com/my-workflow-worker:1.0.0
          args: ["--workflow", "app:process_order", "--shutdown-grace", "30"]
          ports:
            - { name: health, containerPort: 8081 }
          env:
            - { name: PYRULA_WORKER_HEALTH_PORT, value: "8081" }
            - { name: PYRULA_WORKER_HEALTH_HOST, value: "0.0.0.0" }
            - { name: PYRULA_VALKEY_NAMESPACE, value: production }
            # A stable id per pod, so logs and shard leases name the same worker across restarts.
            - name: PYRULA_WORKER_ID
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
          envFrom:
            - secretRef: { name: pyrula-worker }
          # Liveness says the event loop turns and, on an agent worker, that no background
          # loop has died. A store outage must not restart pods, so it is readiness, not
          # liveness, that watches the store.
          livenessProbe:
            httpGet: { path: /livez, port: health }
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet: { path: /readyz, port: health }
            periodSeconds: 5
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { memory: 512Mi }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            # The Cloud connector's event spool. emptyDir survives a container restart but not
            # a pod reschedule; use a PersistentVolumeClaim to keep unsent events across both.
            - { name: spool, mountPath: /var/lib/pyrula }
      volumes:
        - name: spool
          emptyDir: {}
---
# A voluntary disruption (node drain, upgrade) evicts a pod with SIGTERM and the drain above.
# Evicting every pod at once would stop all claiming until replacements are ready.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: workflow-worker
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: workflow-worker }
```

- `/livez` answers while the event loop turns, so a Valkey outage makes pods not ready
  instead of restarting them. It returns 503 `loop_failed` once a background loop dies; a
  workflow worker then exits non-zero and an agent worker waits for the probe to restart it.
  `/readyz` also covers the store.
- `terminationGracePeriodSeconds` must exceed the drain ([Shutdown](#shutdown)).
- Memory scales with `--max-concurrent-runs` and payload size; the resource numbers are a
  starting point.
- A worker needs no Service. The probe is unauthenticated, so keep it off any externally
  reachable one.

## Render

```yaml
# Render Blueprint: a workflow worker and the Valkey it needs. Render's Key Value is
# Valkey-compatible; `noeviction` keeps live run journals from being evicted.
services:
  - type: worker
    name: workflow-worker
    runtime: docker
    dockerContext: ./workflow-worker
    dockerfilePath: ./workflow-worker/Dockerfile
    numInstances: 2
    # Render sends SIGTERM and waits this long before SIGKILL; it must outlast the drain
    # (shutdown_grace 30 s + 3 x cancel_timeout 5 s + the Cloud connector's 5 s = 50 s).
    maxShutdownDelaySeconds: 60
    envVars:
      - key: PYRULA_STORE_URL
        fromService:
          type: keyvalue
          name: pyrula-valkey
          property: connectionString
      - key: PYRULA_VALKEY_NAMESPACE
        value: production
      # Optional. Set in the dashboard; never commit a key.
      - key: PYRULA_CLOUD_API_KEY
        sync: false
      - key: PYRULA_RUNTIME_NAME
        value: render-workflow-worker

  - type: keyvalue
    name: pyrula-valkey
    plan: starter
    maxmemoryPolicy: noeviction
    ipAllowList: []
```

A Render background worker has no HTTP health-check path, so `/readyz` is not consulted. Render
restarts the instance when the process exits, which a workflow worker does when a loop dies.

## Shutdown

On SIGTERM or SIGINT the worker:

1. turns `/readyz` to 503 (`status: draining`) and keeps answering `/livez`;
2. stops claiming runs;
3. waits up to `--shutdown-grace` (30 s) for in-flight runs, then cancels the rest within one
   `--cancel-timeout` (5 s), leaving them for another worker to recover; it then waits up to
   `--cancel-timeout` twice more for store calls, sync steps and sync tools still in a thread
   (after stopping its loops and at the end), abandoning a hung one and queueing its run for
   another worker;
4. releases its shard leases so other workers take them immediately;
5. gives the Cloud connector `PYRULA_CLOUD_DRAIN_DEADLINE_S` (5 s) to flush, then exits 0.

A second SIGTERM or SIGINT during the drain quits at once with 143 or 130; its runs recover
like a crash's, once their locks lapse on `lock_ttl`.

The orchestrator's kill timer must exceed the sum, 50 s with the defaults (the recipes use 60).
Raise it along with `--shutdown-grace`.

## Scaling

- Replicas are independent: start more with the same image, URL and namespace.
- Each workflow's queue is split into `--shard-count` shards (64). Workers lease shards and
  rebalance every `--shard-lease-interval` (10 s).
- Raise `--max-concurrent-runs` before adding replicas when runs wait on I/O. Add replicas for
  CPU or to survive a node loss.
- Drain a workflow before changing its `--shard-count`
  ([Scaling across shards](https://docs.pyrula.io/workflows/guides/worker.md#scaling-across-shards)).
- After a crash, runs recover on the orphan scan (lock TTL plus up to `--orphan-scan-interval`,
  60 s) and shards after `--shard-lease-ttl` (30 s). A running step is run again, so
  side-effecting steps should be idempotent.

## Limits

- Valkey is required for workers on more than one host; SQLite serves one host
  ([Choosing a store](https://docs.pyrula.io/getting-started/choosing-a-store.md)), and Postgres only archives finished
  runs. The server must be Valkey 7.2+ or Redis 6.0+, persistent, with `noeviction`
  ([checklist](https://docs.pyrula.io/getting-started/production-checklist.md)).
- Without a persistent volume at `/var/lib/pyrula`, unsent Cloud events are lost with the
  container ([details](https://docs.pyrula.io/getting-started/cloud.md#not-losing-events)).