# LLM providers

> Which Pyrula client to use for OpenAI, Azure OpenAI, Gemini, Bedrock, Ollama, vLLM, Groq, OpenRouter and LiteLLM, whether you keep your own SDK code or let an agent call the model, and what each path journals.

Source: https://docs.pyrula.io/getting-started/recipes/llm-providers/

There are two ways to reach a model. Pick by what you already have:

- **An `@agent`** calls the model through `ctx.llm`, with a client from `pyrula.agents.llm`
  ([Calling an LLM](https://docs.pyrula.io/agents/authoring/llm.md)). That is the `native` row of the
  [guarantee matrix](https://docs.pyrula.io/agents/integrations/guarantees.md).
- **Your own SDK code** keeps its calls and swaps one import for a drop-in from
  `pyrula.agents.compat`. Inside a Pyrula run the call is journaled and replays without
  the provider; outside a run it is a plain passthrough. Which capabilities hold is in the
  matrix rows `compat.openai`, `compat.anthropic`, `compat.genai` and `compat.bedrock`;
  this page does not restate them.

The drop-ins are not async-only. Each async client has a sync twin (`OpenAI`, `Anthropic`,
`genai.Client`, a boto3 client through `journaled()`), and a sync client journals when it is
called on a thread of the run: `await ctx.to_thread(fn)`
([Sync code and threads](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#sync-code-and-threads)).

## Which path for which provider

| Provider | Agent (`ctx.llm`) | Your own SDK code | Notes |
|---|---|---|---|
| OpenAI | `OpenAILLM` | `compat.AsyncOpenAI`, `compat.OpenAI` | [The OpenAI drop-in](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#the-openai-drop-in) lists the journaled, passed-through and refused surfaces |
| Azure OpenAI | `OpenAILLM(base_url=...)` | `compat.AsyncAzureOpenAI`, `compat.AzureOpenAI` | A deployment name is not a model id; see [Prices](#prices) |
| Gemini | `GeminiLLM` | `compat.genai.Client` | Native Gemini, with Vertex AI; no OpenAI-compatible endpoint needed ([Gemini](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#gemini)) |
| Amazon Bedrock | `BedrockLLM` | `journaled(boto3.client("bedrock-runtime"))` | Converse API; `AnthropicBedrock` and `BedrockOpenAI` also have drop-ins |
| Ollama, vLLM, Groq, OpenRouter, Together | `OpenAILLM(base_url=...)` | `compat.AsyncOpenAI(base_url=...)` | Any server that speaks Chat Completions; see [OpenAI-compatible servers](#openai-compatible-servers) |
| Anything else LiteLLM supports | `LiteLLM(model="provider/model")` | none | See [LiteLLM](#litellm) |

## What each client is built from

These blocks construct the clients and make no request, so they run without keys or a network.
They need `pip install "pyrula[agents,gemini,bedrock]"`: the extras bring `google-genai` and
`boto3`, and the `openai` SDK installs with `pyrula[agents]`. The LiteLLM block also needs
`pyrula[litellm]`.

```python
import boto3

from pyrula.agents.compat import AsyncAzureOpenAI, AsyncOpenAI, genai, journaled
from pyrula.agents.llm import GeminiLLM, OpenAILLM

# A local server needs no key: a base_url with no key is accepted.
ollama = OpenAILLM(model="llama3.1", base_url="http://localhost:11434/v1")
assert ollama.base_url == "http://localhost:11434/v1"

# The same class reaches any hosted OpenAI-compatible API; the URL and key come from the provider.
router = OpenAILLM(model="openai/gpt-4o", base_url="https://openrouter.ai/api/v1", api_key="sk-or-example")

# Your own SDK code: the drop-in is a subclass of the SDK's client, with its constructor.
mine = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="unused")
azure = AsyncAzureOpenAI(
    azure_endpoint="https://example.openai.azure.com", api_version="2024-10-21", api_key="example"
)

# Gemini, natively and as a drop-in.
gemini = GeminiLLM(model="gemini-2.5-flash", api_key="example")
gemini_client = genai.Client(api_key="example")

# A boto3 client you built is converted in place.
bedrock = journaled(boto3.client("bedrock-runtime", region_name="us-east-1"))
```

## OpenAI-compatible servers

`OpenAILLM` and the OpenAI drop-ins take a `base_url`, so Ollama, vLLM, Groq, OpenRouter and
Together AI work through the same code as OpenAI. What this page can say about them:

- **Journaling and replay** are the OpenAI paths' own: the same surfaces are journaled and the
  same ones refused. Nothing in them is specific to a host.
- **No Pyrula test runs against these servers, or against Azure through `OpenAILLM`.** Take
  the Azure v1 base URL from Azure's documentation; the drop-in Azure clients are the documented path ([Azure OpenAI and Bedrock](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#azure-openai-and-bedrock)).
  The wire tests use the OpenAI shape, with one
  case for vLLM's `continuous_usage_stats`. A server that implements Chat Completions less
  faithfully than OpenAI (tool calls, streamed usage) behaves as that server does.
- **A reply with no `usage`** is metered at zero by the drop-ins, with a warning, so a spend cap
  does not count that call.
- **The rate-limit queue is per host.** A client with another `base_url` adds that host to its
  default `rate_limit` key, so a local server and OpenAI do not share a queue
  ([Drop-in clients](https://docs.pyrula.io/agents/authoring/llm.md#drop-in-clients-and-the-langchain-chat-model)).

## Prices

`max_cost_usd` prices a call by model id. A model that is not in the default price table stops
the run before the call with `unpriced_model`, which is what a local or routed model id is.
Give the id a price:

```python
from pyrula.agents import AgentLimits
from pyrula.pricing import ModelPrice

AgentLimits(
    max_cost_usd=2.50,
    prices={"llama3.1": ModelPrice(input_per_mtok=0.0, output_per_mtok=0.0)},
)
```

An Azure deployment name or a Bedrock inference profile ARN names no model until a reply comes
back, so a call to one is not checked for a price the first time, and the reply's model prices
the next ([Spend caps](https://docs.pyrula.io/agents/authoring/agents.md#spend-caps)). Giving that id a price in
`prices` always works.

## Sync code that is not a drop-in

If your code calls the provider's own sync SDK (`openai.OpenAI()`), there are three options, from
most to least journaled:

1. **Swap the import** to the sync drop-in and call it on a thread of the run:
   `await ctx.to_thread(fn)`. The call is journaled as a model call and replays without the
   provider.
2. **Move to the async client** (`AsyncOpenAI`) and `await` it in the body.
3. **Call the SDK inside a step.** The step's return value is what is journaled, so a resumed run
   returns it and does not call the provider again. A step is a plain function whose result is
   data you return, not an SDK object.

```python
import asyncio
import json
import threading

import httpx
import openai

from pyrula.workflows import MemoryStore, WorkerOptions, WorkflowWorker, step, workflow
from pyrula.workflows.sync import WorkflowClient

provider_calls = []


def provider(request: httpx.Request) -> httpx.Response:
    # A stand-in for the provider, so this block runs offline.
    provider_calls.append(json.loads(request.content))
    return httpx.Response(200, json={
        "id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": "llama3.1",
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": "Pong."}}],
        "usage": {"prompt_tokens": 9, "completion_tokens": 2, "total_tokens": 11},
    })


@step
def ask(question: str) -> str:
    client = openai.OpenAI(
        api_key="unused",
        base_url="http://llm.test/v1",
        max_retries=0,
        http_client=httpx.Client(transport=httpx.MockTransport(provider)),
    )
    reply = client.chat.completions.create(
        model="llama3.1", messages=[{"role": "user", "content": question}]
    )
    return reply.choices[0].message.content


@workflow(name="docs_recipe_sync_provider")
async def ping(ctx, question: str) -> str:
    return await ask(question)


store = MemoryStore()
worker = WorkflowWorker(
    workflows=[ping], store=store, options=WorkerOptions(claim_poll_interval=0.05)
)
threading.Thread(target=lambda: asyncio.run(worker.run()), daemon=True).start()

with WorkflowClient(store) as client:
    run = client.submit("docs_recipe_sync_provider", {"question": "Ping?"})
    assert run.result(timeout=30) == "Pong."
    # Only step events: no `llm:done`, which is what the spend caps count.
    assert "llm:done" not in {event.kind.value for event in run.events()}
assert len(provider_calls) == 1
```

The fidelity of option 3 is the step's: the result is journaled whole and replayed, with none of
the per-call detail the drop-ins record (streamed chunks, tool-call structure, usage). It also
does not take part in the spend caps, which count `llm:done` events. A crash between the
provider's reply and the step's record calls the provider again, as for any step
([The window where a step can run twice](https://docs.pyrula.io/workflows/concepts/replay-and-recovery.md#the-window-where-a-step-can-run-twice)).

## LiteLLM

`LiteLLM(model="provider/model")` routes through the `litellm` package (`pip install "pyrula[litellm]"`)
to any provider it supports, as an agent client. There is no LiteLLM drop-in for your own calls.
The `litellm` extra does not install on Python 3.15 yet. It forwards `api_base` and `api_key` on
every call:

```python
from pyrula.agents.llm import LiteLLM

LiteLLM(model="gemini/gemini-2.5-flash", api_key="example")
LiteLLM(model="openai/llama3.1", api_base="http://localhost:11434/v1", api_key="unused")
```

LiteLLM's own retries are switched off (`max_retries=0` on every call), because `AgentLLM` owns
retries and the rate limiter must see each attempt.