# Replay checks in CI > Replay a suite of saved cases against a pull request's code with no model keys, and fail the build when the change stops reproducing them. Source: https://docs.pyrula.io/workflows/guides/replay-ci/ `pyrula ci run` checks that a PR's code still does what it did in the runs you saved as cases. It runs in your CI job against your checkout, using only each case's recording. Cloud pins the suite and keeps the result. ## What a pass means The code gets every recorded model and tool result it asks for; a call the recording can't answer stops the case. A case passes when the code made the same calls with the same inputs, reached the same end, and every pinned assertion held. It says nothing about how a live model would answer today. | Outcome | Meaning | Fails the build | | --- | --- | --- | | `compatible` | Same calls, inputs and end; assertions held. | no | | `diverged` | A call does not match the recording: `input_changed`, `unexpected_call` or `missing_call`. | yes | | `raised` | Your code raised where the recording did not. | yes | | `assertion_failed` | The run matched, but a pinned assertion did not hold. | yes | | `error` | The case could not be checked (`bundle_missing`, `store_mismatch`, `target_not_registered`, `kind_mismatch`, `case_not_ready`, `recorded_error` or `internal`). | yes (exit 3) | | `timeout` | The case or the suite ran out of time. | yes (exit 5) | A tool that failed in the recorded run replays as that failure and is not itself a failure. Anything that is not your change, such as a flaky store, is an `error`, never a fail. ## Set up 1. Save the runs to guard as cases, from the dashboard or `pyrula case save` ([Saved cases](https://docs.pyrula.io/workflows/concepts/saved-cases.md)). 2. Make a suite: a named list of cases for one project and environment. Cases join explicitly, so a new case never gates builds by accident. 3. Mint a CI token in Settings, API tokens (it can read the project and start and report suite runs). Store it as the secret `PYRULA_TOKEN`. 4. Store the fixture store URL as `PYRULA_FIXTURE_STORE_URL`. It is never sent to Cloud. A store that is not the one a case was saved to is `store_mismatch`. 5. Map each case's target (a registered workflow or agent) to an import path in `pyproject.toml`: ```toml [tool.pyrula.ci] app_dir = "src" # added to sys.path before targets are imported deps = "app.deps:deps" # optional: the deps object your agents expect llm = "app.llm:client" # optional: a replay-only run never calls it [tool.pyrula.ci.targets] support_agent = "app.agents:support_agent" checkout = "app.workflows:checkout" ``` `--target NAME=MODULE:ATTR` overrides a configured target. An unlisted target ends as `error` with `target_not_registered`. 6. Add the Action. Install your project first; the check imports your code. ```yaml - uses: actions/setup-python@v5 with: {python-version: "3.13"} - run: pip install -e . - uses: pyrula-io/pyrula/actions/replay-check@v0.1.0 with: suite: checkout-regressions project: ${{ vars.PYRULA_PROJECT }} token: ${{ secrets.PYRULA_TOKEN }} fixture-store-url: ${{ secrets.PYRULA_FIXTURE_STORE_URL }} ``` The Action installs `pyrula-cloud-client` if absent (version from `client-version`), adds the Markdown report to the job summary, annotates each failing case, and sets outputs `verdict`, `suite-run-url`, `json-path`, `junit-path`. Other inputs: `environment` (default `production`), `cloud-url`, `timeout`, `idempotency-suffix`. Outside GitHub (`--environment` defaults to `production`): ```bash export PYRULA_TOKEN=... # never a flag export PYRULA_FIXTURE_STORE_URL=... pyrula ci run --suite checkout-regressions --project proj_123 \ --json report.json --junit report.xml --summary report.md ``` ## Timeouts and idempotency Cases run one at a time, replay-only with a model-less runtime. `--case-timeout` defaults to 120 seconds; `--timeout` for the suite defaults to the deadline less a minute, and cases starting after it are `timeout`. Keep targets async: Python cannot interrupt synchronous code that never yields. `--deadline-seconds` (60 to 21600, default 1800) is how long Cloud holds the run open; a late report is refused. `--verbose` adds engine logs and tracebacks. The run key is `GITHUB_RUN_ID-GITHUB_RUN_ATTEMPT-GITHUB_JOB`. Legs of one matrix job share it, so give each a distinct `idempotency-suffix` input (`--idempotency-suffix`) or the second leg's report is refused. Outside GitHub pass `--idempotency-key`, or each invocation is a new run. A key already reported exits 3 with the run's link. A suite run counts against the plan's evaluation runs per month. ## Exit codes | Code | Meaning | | --- | --- | | 0 | Every case is `compatible`. | | 1 | Any case `diverged`, `raised` or `assertion_failed`. | | 2 | Usage or configuration error, found before any network call. | | 3 | A case could not be checked, with no failures. Also a run key already reported. | | 4 | Infrastructure: Cloud unreachable, token or plan refusal, unreachable fixture store, or rejected report, with no failures. | | 5 | A timeout, with no failures. Also a run key that expired before reporting. | When several apply, precedence is 2, 1, 4, 5, 3. ## Fork pull requests The job runs the PR's code with your token and fixture store credential, so run it only on code you'd let read them. GitHub withholds secrets from a fork's `pull_request` workflow, so the command sees no `PYRULA_TOKEN` and exits 4. Skip forks (`if: github.event.pull_request.head.repo.full_name == github.repository`) or gate behind a protected environment after reading the diff. Never use `pull_request_target` for this. Use a read-only fixture store credential, so a malicious change can't alter the recordings that gate later builds. ## Outputs No output holds a recorded or produced value, only case names, targets, call keys, reason codes and counts, so public job logs are safe. ### JSON report `--json` writes `schema: "pyrula.replay-ci/1"`: ```json { "schema": "pyrula.replay-ci/1", "check": "replay_compatibility", "statement": "Deterministic replay-compatibility check: ...", "verdict": "fail", "exitCode": 1, "suite": {"id": "suite_1", "name": "checkout-regressions"}, "suiteRunId": "run_1", "url": "https://cloud.example/...", "project": "proj_123", "environment": "production", "startedAt": "2026-10-26T10:00:00Z", "finishedAt": "2026-10-26T10:00:05Z", "code": {"sha": "abc123", "ref": "refs/pull/7/merge", "pullRequest": 7}, "pyrulaVersion": "0.1.0", "counts": {"total": 3, "passed": 2, "failed": 1, "errored": 0, "timedOut": 0}, "cases": [ { "caseId": "case_1", "name": "refund flow", "kind": "agent", "target": "support_agent", "fixtureHash": "9f2c...", "outcome": "diverged", "reasonCode": "diverged", "reasonMessage": "tool:toolu_01: called with different input", "divergence": {"key": "tool:toolu_01", "reason": "input_changed"}, "calls": {"reused": 3, "simulated": 0, "unexpected": 1, "unconsumed": 0}, "assertions": [{"index": 0, "op": "status_completed", "passed": true, "reason": null}], "durationMs": 12 } ] } ``` `verdict` is `pass`, `fail`, `error`, `timeout` or `infrastructure`. A run that never started has no cases and an `error` object with `code` and `message`. `calls.unconsumed` counts recorded calls the code never made. ### JUnit One `` named for the suite and one `` per case. A fail is a `` (or `raised:CODE`, `assertion_failed`); an error or timeout is an ``. ## Ordering assertions `actions_match_original` and `action_not_executed` compare the run's actions with the original's, which Cloud supplies at start. Tool calls count as `tool:NAME`, workflow writes as `step:ID`, and a call served from the recording is `reused`, so a recorded write never counts as executed. If Cloud no longer has the original history, the case is an `error`. ## See also [Saved cases](https://docs.pyrula.io/workflows/concepts/saved-cases.md#replay-only) covers `pyrula case run --replay-only` for one case locally. [Versioning and deploys](https://docs.pyrula.io/workflows/guides/versioning.md) covers `replay_check`, which asks whether in-flight runs can resume on new code.