LLM providers

There are two ways to reach a model. Pick by what you already have:

• An @agent calls the model through ctx.llm, with a client from pyrula.agents.llm (Calling an LLM). That is the native row of the guarantee matrix.
• Your own SDK code keeps its calls and swaps one import for a drop-in from pyrula.agents.compat. Inside a Pyrula run the call is journaled and replays without the provider; outside a run it is a plain passthrough. Which capabilities hold is in the matrix rows compat.openai, compat.anthropic, compat.genai and compat.bedrock; this page does not restate them.

The drop-ins are not async-only. Each async client has a sync twin (OpenAI, Anthropic, genai.Client, a boto3 client through journaled()), and a sync client journals when it is called on a thread of the run: await ctx.to_thread(fn) (Sync code and threads).

Which path for which provider

Provider	Agent (ctx.llm)	Your own SDK code	Notes
OpenAI	OpenAILLM	compat.AsyncOpenAI, compat.OpenAI	The OpenAI drop-in lists the journaled, passed-through and refused surfaces
Azure OpenAI	OpenAILLM(base_url=...)	compat.AsyncAzureOpenAI, compat.AzureOpenAI	A deployment name is not a model id; see Prices
Gemini	GeminiLLM	compat.genai.Client	Native Gemini, with Vertex AI; no OpenAI-compatible endpoint needed (Gemini)
Amazon Bedrock	BedrockLLM	journaled(boto3.client("bedrock-runtime"))	Converse API; AnthropicBedrock and BedrockOpenAI also have drop-ins
Ollama, vLLM, Groq, OpenRouter, Together	OpenAILLM(base_url=...)	compat.AsyncOpenAI(base_url=...)	Any server that speaks Chat Completions; see OpenAI-compatible servers
Anything else LiteLLM supports	LiteLLM(model="provider/model")	none	See LiteLLM

What each client is built from

These blocks construct the clients and make no request, so they run without keys or a network. They need pip install "pyrula[agents,gemini,bedrock]": the extras bring google-genai and boto3, and the openai SDK installs with pyrula[agents]. The LiteLLM block also needs pyrula[litellm].

import boto3

from pyrula.agents.compat import AsyncAzureOpenAI, AsyncOpenAI, genai, journaled
from pyrula.agents.llm import GeminiLLM, OpenAILLM

# A local server needs no key: a base_url with no key is accepted.
ollama = OpenAILLM(model="llama3.1", base_url="http://localhost:11434/v1")
assert ollama.base_url == "http://localhost:11434/v1"

# The same class reaches any hosted OpenAI-compatible API; the URL and key come from the provider.
router = OpenAILLM(model="openai/gpt-4o", base_url="https://openrouter.ai/api/v1", api_key="sk-or-example")

# Your own SDK code: the drop-in is a subclass of the SDK's client, with its constructor.
mine = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="unused")
azure = AsyncAzureOpenAI(
    azure_endpoint="https://example.openai.azure.com", api_version="2024-10-21", api_key="example"
)

# Gemini, natively and as a drop-in.
gemini = GeminiLLM(model="gemini-2.5-flash", api_key="example")
gemini_client = genai.Client(api_key="example")

# A boto3 client you built is converted in place.
bedrock = journaled(boto3.client("bedrock-runtime", region_name="us-east-1"))

OpenAI-compatible servers

OpenAILLM and the OpenAI drop-ins take a base_url, so Ollama, vLLM, Groq, OpenRouter and Together AI work through the same code as OpenAI. What this page can say about them:

• Journaling and replay are the OpenAI paths’ own: the same surfaces are journaled and the same ones refused. Nothing in them is specific to a host.
• No Pyrula test runs against these servers, or against Azure through OpenAILLM. Take the Azure v1 base URL from Azure’s documentation; the drop-in Azure clients are the documented path (Azure OpenAI and Bedrock). The wire tests use the OpenAI shape, with one case for vLLM’s continuous_usage_stats. A server that implements Chat Completions less faithfully than OpenAI (tool calls, streamed usage) behaves as that server does.
• A reply with no usage is metered at zero by the drop-ins, with a warning, so a spend cap does not count that call.
• The rate-limit queue is per host. A client with another base_url adds that host to its default rate_limit key, so a local server and OpenAI do not share a queue (Drop-in clients).

Prices

max_cost_usd prices a call by model id. A model that is not in the default price table stops the run before the call with unpriced_model, which is what a local or routed model id is. Give the id a price:

from pyrula.agents import AgentLimits
from pyrula.pricing import ModelPrice

AgentLimits(
    max_cost_usd=2.50,
    prices={"llama3.1": ModelPrice(input_per_mtok=0.0, output_per_mtok=0.0)},
)

An Azure deployment name or a Bedrock inference profile ARN names no model until a reply comes back, so a call to one is not checked for a price the first time, and the reply’s model prices the next (Spend caps). Giving that id a price in prices always works.

Sync code that is not a drop-in

If your code calls the provider’s own sync SDK (openai.OpenAI()), there are three options, from most to least journaled:

1. Swap the import to the sync drop-in and call it on a thread of the run: await ctx.to_thread(fn). The call is journaled as a model call and replays without the provider.
2. Move to the async client (AsyncOpenAI) and await it in the body.
3. Call the SDK inside a step. The step’s return value is what is journaled, so a resumed run returns it and does not call the provider again. A step is a plain function whose result is data you return, not an SDK object.

import asyncio
import json
import threading

import httpx
import openai

from pyrula.workflows import MemoryStore, WorkerOptions, WorkflowWorker, step, workflow
from pyrula.workflows.sync import WorkflowClient

provider_calls = []


def provider(request: httpx.Request) -> httpx.Response:
    # A stand-in for the provider, so this block runs offline.
    provider_calls.append(json.loads(request.content))
    return httpx.Response(200, json={
        "id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": "llama3.1",
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": "Pong."}}],
        "usage": {"prompt_tokens": 9, "completion_tokens": 2, "total_tokens": 11},
    })


@step
def ask(question: str) -> str:
    client = openai.OpenAI(
        api_key="unused",
        base_url="http://llm.test/v1",
        max_retries=0,
        http_client=httpx.Client(transport=httpx.MockTransport(provider)),
    )
    reply = client.chat.completions.create(
        model="llama3.1", messages=[{"role": "user", "content": question}]
    )
    return reply.choices[0].message.content


@workflow(name="docs_recipe_sync_provider")
async def ping(ctx, question: str) -> str:
    return await ask(question)


store = MemoryStore()
worker = WorkflowWorker(
    workflows=[ping], store=store, options=WorkerOptions(claim_poll_interval=0.05)
)
threading.Thread(target=lambda: asyncio.run(worker.run()), daemon=True).start()

with WorkflowClient(store) as client:
    run = client.submit("docs_recipe_sync_provider", {"question": "Ping?"})
    assert run.result(timeout=30) == "Pong."
    # Only step events: no `llm:done`, which is what the spend caps count.
    assert "llm:done" not in {event.kind.value for event in run.events()}
assert len(provider_calls) == 1

The fidelity of option 3 is the step’s: the result is journaled whole and replayed, with none of the per-call detail the drop-ins record (streamed chunks, tool-call structure, usage). It also does not take part in the spend caps, which count llm:done events. A crash between the provider’s reply and the step’s record calls the provider again, as for any step (The window where a step can run twice).

LiteLLM

LiteLLM(model="provider/model") routes through the litellm package (pip install "pyrula[litellm]") to any provider it supports, as an agent client. There is no LiteLLM drop-in for your own calls. The litellm extra does not install on Python 3.15 yet. It forwards api_base and api_key on every call:

from pyrula.agents.llm import LiteLLM

LiteLLM(model="gemini/gemini-2.5-flash", api_key="example")
LiteLLM(model="openai/llama3.1", api_base="http://localhost:11434/v1", api_key="unused")

LiteLLM’s own retries are switched off (max_retries=0 on every call), because AgentLLM owns retries and the rate limiter must see each attempt.