Pre-release: the packages are not on PyPI yet, so pip install does not resolve. Build from source to try it today.

Calling an LLM

Inside an agent, the model goes through ctx.llm. Every call, token and tool result is recorded and replay reads it back. A call in flight when the worker died has no record, so replay makes it again and the provider may bill both.

@agent
async def support(ctx: AgentContext, message: str) -> str:
    return await ctx.llm.run(
        [{"role": "user", "content": message}],
        tools=[lookup_order],
        system="You are a helpful support assistant.",
    )

Call	Tools	Returns
`ctx.llm.run(messages, tools=, system=)`	runs the tool loop	the final message’s text
`ctx.llm.stream(messages, tools=, system=)`	runs the tool loop	async iterator of events (tokens, tool calls, tool results)
`ctx.llm.complete(messages, system=)`	none	the reply text
`ctx.llm.structured(messages, Model, system=, max_retries=2)`	none	a validated `Model` instance (forced tool call, re-asked on validation failure)

tools are @tool functions; the loop runs until the model stops asking for them. run returns only the last message; narration before a tool call is in the event stream.

async for event in ctx.llm.stream(messages, tools=[search]):
    ...

Providers

from pyrula.agents.llm import OpenAILLM, AnthropicLLM, GeminiLLM, BedrockLLM, LiteLLM

OpenAILLM(model="gpt-6-luna")
AnthropicLLM(model="claude-sonnet-4-6")
GeminiLLM(model="gemini-2.5-flash")   # needs pyrula[gemini]
BedrockLLM(model="us.anthropic.claude-sonnet-4-6-v1:0", region="us-east-1")   # needs pyrula[bedrock]
LiteLLM(model="...")   # anything LiteLLM supports

Pass the client on @agent(llm=...), to create_app / PyrulaWorker, or per run. Imports are lazy, so install the provider’s extra. The litellm extra does not install on Python 3.15 yet.

Shared constructor options on AnthropicLLM / OpenAILLM:

from pyrula.agents.llm import AnthropicLLM, LLMRate

AnthropicLLM(
    model="claude-sonnet-4-6",
    max_tokens=8192,                          # default
    llm_call_timeout=120.0,                   # per attempt; default None (the turn `timeout` still bounds it)
    max_retries=2,                            # retries after the first attempt; default 2
    rate_limit=LLMRate(rpm=50, tpm=40_000),   # pace live calls; see Rate limits
)

AnthropicLLM also takes prompt_caching=True and cache_ttl="5m"; OpenAILLM takes base_url, temperature, top_p, seed and parallel_tool_calls.

Gemini

GeminiLLM runs on the google-genai SDK (pip install "pyrula[gemini]", google-genai>=1.46,<3). It authenticates with api_key= (or GEMINI_API_KEY / GOOGLE_API_KEY), or against Vertex AI with vertexai=True, project=..., location=... and application default credentials (or credentials=); a prebuilt google.genai.Client goes in client=. Build that client with HttpOptions(retry_options=HttpRetryOptions(attempts=1)): AgentLLM owns retries, and the adapter sets one attempt on the client it builds itself. provider is gcp.gemini or gcp.vertex_ai in telemetry.

from pyrula.agents.llm import GeminiLLM

GeminiLLM(
    model="gemini-2.5-pro",
    vertexai=True, project="my-project", location="europe-west4",
    max_tokens=8192,               # max_output_tokens; default 8192
    temperature=0.2, top_p=None, seed=None,
    thinking_config={"include_thoughts": True},   # thought summaries arrive as thinking blocks
    config={"media_resolution": "MEDIA_RESOLUTION_LOW"},   # any other GenerateContentConfig field
)

Messages keep the Anthropic shape the other clients take, so a PDF or image goes in as a document or image block: {"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": ...}} is sent inline, and {"type": "url", "url": "gs://bucket/file.pdf", "media_type": "application/pdf"} as a file URI (media_type is read from the extension when omitted, and required when the URL has none). A tool_result carries text only; another part type is refused. Structured output (ctx.llm.structured) uses the same forced final_result tool as the other clients, with function-calling mode ANY. Gemini 3 models require the thought signature they attach to a function call to come back with it; the adapter keeps it on the assistant message’s block (signature) and sends it back. A journal written without one replays unchanged.

Usage maps prompt_token_count plus tool_use_prompt_token_count, less cached_content_token_count, to input, the cached tokens to cache reads, and candidates_token_count plus thoughts_token_count to output. Cache storage is billed hourly by Google and is not metered. A prompt or answer Gemini refuses ends the call with GeminiBlockedError and is not retried; a malformed function call, or a stream that ends without a finish reason, is retried. A 429 waits for the retryDelay Google puts in the error details. with_model() returns a copy on another Gemini model, so a what-if can swap Flash for Pro. Automatic function calling is off: tools are declared only, and @tool functions run in pyrula’s loop.

To keep your own google.genai calls and make them durable, use the drop-in pyrula.agents.compat.genai instead: see Gemini.

Amazon Bedrock

BedrockLLM runs on boto3’s bedrock-runtime Converse API (pip install "pyrula[bedrock]", boto3>=1.38), so one client covers Claude, Nova, Llama, Mistral and the other Converse models. There is no key: credentials come from boto3’s own chain (environment, shared profile, SSO, an instance or task role), and the adapter accepts, stores and journals none. model is a model id, an inference-profile id (us.anthropic.claude-sonnet-4-6-v1:0) or an ARN; region= and profile= pick the region and profile when the environment does not. No credentials or no region raises ConfigurationError at construction, and a 403 from AWS ends the turn with a message naming the credential chain and model access, without retries. provider is aws.bedrock in telemetry.

boto3 is blocking, so each call, and each event of a stream, runs on a thread of the loop’s default executor; a Bedrock call holds one thread while it waits. The client the adapter builds makes one attempt (AgentLLM owns retries) and waits up to read_timeout=300 seconds for bytes, since botocore’s own 60 s cuts a long non-streamed answer. A prebuilt boto3.client("bedrock-runtime") goes in client= and its retry configuration is left as it is: give it Config(retries={"total_max_attempts": 1}), or botocore retries under pyrula’s retries. A throttling or 5xx error (ThrottlingException, ServiceUnavailableException, InternalServerException, ModelNotReadyException), including one sent as an event inside a stream, and a dropped connection are retried before the first block (always in the default safe mode); validation and access errors are not.

Messages keep the Anthropic shape the other clients take: text, tool_use, tool_result (is_error becomes status: "error"), image and document blocks. Image and document bytes go inline from a base64 source, or by reference from an s3:// url source; any other url is refused, since Converse does not fetch it. A document’s title becomes its Converse name, with characters Converse refuses replaced by spaces. Consecutive messages of one role are merged, as Converse requires alternating roles, and an empty tool result is sent as (no output). tool_choice maps to auto, any or a named tool; none is refused. Anything else in additional_model_request_fields= (Claude’s {"thinking": {"type": "enabled", "budget_tokens": 4000}}), guardrail_config= or config= (any other Converse parameter, applied last) goes through unchanged. Reasoning streams as thinking blocks and goes back on the next request only with its signature, which the assistant message keeps; redacted reasoning is dropped and not sent back.

Prompt caching is on by default for the Claude models that support it (3.5 Haiku, 3.7 Sonnet, the 4 series and later: system, tools and the end of the transcript) and for Nova (system and transcript, not tools), and off for every other model and for an opaque id (an application inference profile, provisioned, custom or imported model), because Converse rejects a cache point on a model without caching. prompt_caching=True or False overrides that.

Usage maps inputTokens (already excluding cached tokens) to input, cacheReadInputTokens and cacheWriteInputTokens to the cache counts, and outputTokens to output, from the metadata event after messageStop. A guardrail_intervened or content_filtered stop raises BedrockBlockedError and model_context_window_exceeded raises BedrockContextWindowError (code context_window_exceeded), both after the call’s usage is metered and neither retried; tool arguments that are not JSON, which a max_tokens cut produces, raise BedrockResponseError; a stream that ends without messageStop is retried. Converse replies name no model, so a call is priced by the id you asked for, then its canonical id (claude-sonnet-4-6 for the profile above): Claude on Bedrock is priced at the Anthropic row. The packaged table prices these Bedrock-native models at the US East on-demand standard rates AWS publishes (its Price List API, version 20261102165015): Amazon Nova Pro, Lite, Micro and 2 Lite (amazon.nova-pro-v1:0, canonical nova-pro), Meta Llama 4 Maverick and Scout and Llama 3.3 70B (meta.llama4-maverick-17b-instruct-v1:0), and Mistral Large 3 and Mistral Large (mistral.mistral-large-3-675b-instruct, mistral.mistral-large-2402-v1:0). Nova lists cache read and write rates; the others list no cache price, so their cache tokens are priced at the input rate. Rates in other regions differ. Nova Premier, Nova 2 Pro, Cohere and the other Llama and Mistral models have no row; add one with prices= or --prices, keyed by the canonical id (nova-pro), or the model is refused with unpriced_model under max_cost_usd. A profile id finds its model’s context window through the canonical id; Bedrock’s window for a model can be smaller than the model maker’s, so set context_window= when it is. A Claude profile id defaults to 200,000 tokens, since Bedrock serves the 1M window only with a beta flag in additional_model_request_fields; set context_window= with the flag. A forced tool_choice (structured output uses one) is sent only to Claude, Nova and Mistral Large; on any other model Converse accepts auto only, so the tool is offered and the model decides, and a reply without the call goes through the usual validation retry. A history that holds tool calls sent without tools gets a toolConfig naming them, which Converse requires. with_model() returns a copy on another Bedrock model. aioboto3 is not supported.

To keep code that uses Anthropic’s own AnthropicBedrock or AnthropicVertex client and make it durable, swap the import to pyrula.agents.compat: see Anthropic on Bedrock and Vertex. A call is priced at the Anthropic row by the canonical form of its model id, as BedrockLLM does.

To keep your own boto3 converse calls and make them durable, convert your client with journaled(client): see Amazon Bedrock (boto3).

Rate limits

rate_limit=LLMRate(rpm=..., tpm=...) paces live calls. Every call, including complete() and structured(), reserves a slot in the store before it reaches the provider, and a caller whose slot is ahead waits for it. Clients that share a key (default: the provider and model) share one queue across processes and workers, so the limit holds however many callers wait. The store must support it (memory, SQLite and Valkey do); a store that does not raises ConfigurationError instead of running unmetered.

from pyrula.agents.llm import AnthropicLLM, LLMRate

AnthropicLLM(
    model="claude-sonnet-4-6",
    rate_limit=LLMRate(
        rpm=50,
        tpm=40_000,
        max_wait=60.0,       # longest a call waits for its slot; None waits as long as it takes
        overflow="fail",     # "fail" ends the turn with llm_rate_limited; "proceed" calls anyway, unmetered
    ),
)

A call whose slot is more than max_wait seconds away is refused without taking a slot. With overflow="fail" the turn ends with the error code llm_rate_limited (LLMRateLimitedError); with overflow="proceed" the call runs, a warning is logged and the span carries pyrula.llm.rate_limit="unmetered".
A waiting call checks for cancel and pause about every 5 seconds, and a call cancelled while waiting gives its slot back.
tpm is charged an estimate when the slot is taken and corrected to the provider’s reported usage afterwards. A call larger than tpm is charged the limit, so it is not refused forever.
A provider 429 with Retry-After holds every caller on the key for that long (at most 60 seconds), refunds the failed attempt’s tokens, and the retry takes a new slot. When the hold ends the bucket starts empty: the first call goes out at that moment and the rest are spaced at the rate, with no burst.
The queue is shared through the store. MemoryStore limits one process and SQLiteStore one host; only ValkeyStore (and PostgresStore, which keeps its live state in Valkey) holds the limit across hosts. Two processes on a memory store each get the full rate.
A wait for a retry’s slot does not count against llm_call_timeout, which bounds each provider attempt only.

When a turn ends with `llm_rate_limited`

The slot was further than max_wait away. Any of these clears it:

Raise max_wait (or set it to None) so callers queue longer.
Lower the worker concurrency, so fewer calls compete for the key.
Raise rpm or tpm, if the provider account allows it.
Set overflow="proceed" to call the provider unmetered when the queue is full.

Nothing re-runs a refused turn: submit it again, or set max_wait=None to wait instead.

Custom clients

AgentLLM owns the retries: a custom LLMClient makes one attempt per stream or complete and lets the provider’s error out. Do not loop inside the client. AgentLLM retries what it judges transient, takes a rate slot for each attempt, and sleeps between attempts. To change how many times it retries, set max_retries on the client (an int of at least 0; a client that has none gets 2). Build the provider’s SDK client with max_retries=0, or its retries run underneath AgentLLM’s, unmetered. For a transient error of your own, raise a subclass of pyrula.contracts.errors.Transient: AgentLLM retries it; an exception it does not recognise is not retried.

Drop-in clients and the LangChain chat model

The pyrula.agents.compat clients and durable_chat_model take the same rate_limit:

from pyrula.agents.compat import AsyncOpenAI
from pyrula.agents.llm import LLMRate

client = AsyncOpenAI(rate_limit=LLMRate(rpm=500, tpm=200_000, key="openai-main"))

Inside a run, every live request reserves a slot before it is sent, through the store the run already has, so these clients share a queue with each other and with OpenAILLM or AnthropicLLM on the same key. The default key for AsyncOpenAI is OpenAILLM:<model>, for AsyncAnthropic AnthropicLLM:<model>, and for durable_chat_model <model class>:<model id>; a client with another base_url (Azure, a proxy, a local server) adds that host to its default key, natives and drop-ins alike (OpenAILLM@localhost:<model>), since it is another account. The same applies inside a ctx.step or @tool body, where the call is not journaled but still takes a slot. A replayed call takes no slot. A slot further than max_wait away ends the run with llm_rate_limited and the provider is not called; overflow="proceed" calls it anyway.

client.rate_limit reads the limit, and with_options(rate_limit=...) or copy(rate_limit=...) replaces it (None removes it).
With a rate_limit, the client sends requests through the SDK with its retries switched off and makes the max_retries the client was configured with itself, so each attempt takes its own slot and a 429 holds the account for its Retry-After. Without a rate_limit the SDK retries as usual. durable_chat_model cannot do this for an arbitrary inner model: its inner model’s own retries are unmetered, and a warning says so; set max_retries=0 on the inner model to count every attempt, which also removes those retries.
A streamed request holds its slot while the stream opens and is charged the estimate, then settled to the usage the stream reports when it ends. A stream stopped early keeps the estimate. TPM counts input plus output tokens, as for the native clients. A stream stopped early is not charged for the output it did generate. A chat stream in a plain @workflow (which does not ask for usage) keeps the estimate.
Outside a run (no run state reachable) the limit has no effect, and a warning says so once.
The refusal is journaled as the call’s outcome, so a body that catches LLMRateLimitedError takes the same branch after a resume, which does not ask the limiter again. A run the refusal ended asks again when it is retried. A refusal in a plain @workflow surfaces as LLMRateLimitedError from the call.
The LangGraph node (as_langgraph_node) submits a run and awaits it, so it makes no provider call of its own; the agent a worker runs for it is limited by its own client’s rate_limit. The MCP server and client call tools, not models, so they have nothing to limit.

Streaming mode

from pyrula.agents.stream.kinds import StreamingMode

OpenAILLM(model="gpt-6-luna", streaming_mode=StreamingMode.LIVE)

Mode	Blocks reach the run stream	A provider error mid-call
`SAFE` (default)	once the call finishes	retried, up to `max_retries` more attempts
`LIVE`	as they stream	retried until the first block has been written; after that the call fails

SAFE buffers up to max_buffer_bytes (5 MiB) per call; more ends the turn with BUFFER_OVERFLOW. Use LIVE when time-to-first-token matters more than retries.

Long conversations

Before each call the loop estimates prompt tokens. Over budget, it compacts the history; still over the model’s window, it ends the turn with context_window_exceeded.

Window: known for common Claude and GPT models. For others set context_window, or management is off (one warning per model).
Budget: (context_window - reserved_output_tokens) × compaction_fraction; defaults 4096 and 0.75.
Default compactor: drop_oldest keeps the first message and the 6 most recent, never splits a tool call from its result, and makes no LLM call, so it replays identically. Each compaction writes context:compacted.

from pyrula.agents import AgentLimits, agent
from pyrula.agents.llm.context import make_drop_oldest

@agent(limits=AgentLimits(
    context_window=128_000,
    compaction_fraction=0.6,
    history_compactor=make_drop_oldest(keep_recent=10),   # None disables compaction
))
async def support(ctx, message: str) -> str:
    ...

A custom compactor is async (CompactionContext) -> list[dict] with history, token_count, budget, window and model. It can call await ctx.summarize(messages, instructions=...), a journaled LLM call. Keep tool calls with their results. See Context compaction for what is recorded and a worked example.

Retries

One layer: AgentLLM retries a failed call, and the provider clients make a single attempt each. max_retries (default 2, so three attempts) is a constructor option on OpenAILLM, AnthropicLLM and LiteLLM; the SDK’s own retries are turned off.

Retried: status 408, 409, 429 and 5xx, connection and timeout errors, and an error the provider sends inside an otherwise successful stream. A response header x-should-retry decides either way before the status does. Other 4xx errors are not retried, and 401 and 403 end the turn as a configuration error. llm_call_timeout bounds each attempt, so a timed-out attempt is retried.

The wait is the provider’s retry-after-ms or Retry-After (capped at 60 seconds), else exponential backoff from 0.5 s, capped at 8 s, with jitter. The run’s cancel and pause are checked about every 5 seconds of the wait. complete and structured retry at any point in either mode. A streamed run retries at any point in SAFE mode, and in LIVE mode only while no block has been written. The pyrula.agents.compat drop-ins retry on the same rule.

durable_chat_model is the exception to one layer: LangChain’s own retries inside its inner model still run.

Replay safety

Raw SDK calls in an agent body are not recorded and fire again on replay. Go through ctx.llm (Determinism).

View this page as Markdown