# Calling an LLM > ctx.llm runs the model and tool loop and records everything for replay. Source: https://docs.pyrula.io/agents/authoring/llm/ Inside an agent, the model goes through `ctx.llm`. Every call, token and tool result is recorded and replay reads it back. A call in flight when the worker died has no record, so replay makes it again and the provider may bill both. ```python @agent async def support(ctx: AgentContext, message: str) -> str: return await ctx.llm.run( [{"role": "user", "content": message}], tools=[lookup_order], system="You are a helpful support assistant.", ) ``` | Call | Tools | Returns | |---|---|---| | `ctx.llm.run(messages, tools=, system=)` | runs the tool loop | the final message's text | | `ctx.llm.stream(messages, tools=, system=)` | runs the tool loop | async iterator of events (tokens, tool calls, tool results) | | `ctx.llm.complete(messages, system=)` | none | the reply text | | `ctx.llm.structured(messages, Model, system=, max_retries=2)` | none | a validated `Model` instance (forced tool call, re-asked on validation failure) | `tools` are [`@tool`](https://docs.pyrula.io/agents/authoring/tools.md) functions; the loop runs until the model stops asking for them. `run` returns only the last message; narration before a tool call is in the event stream. ```python async for event in ctx.llm.stream(messages, tools=[search]): ... ``` ## Providers ```python from pyrula.agents.llm import OpenAILLM, AnthropicLLM, GeminiLLM, BedrockLLM, LiteLLM OpenAILLM(model="gpt-6-luna") AnthropicLLM(model="claude-sonnet-4-6") GeminiLLM(model="gemini-2.5-flash") # needs pyrula[gemini] BedrockLLM(model="us.anthropic.claude-sonnet-4-6-v1:0", region="us-east-1") # needs pyrula[bedrock] LiteLLM(model="...") # anything LiteLLM supports ``` Pass the client on `@agent(llm=...)`, to `create_app` / `PyrulaWorker`, or per run. Imports are lazy, so install the provider's extra. The `litellm` extra does not install on Python 3.15 yet. Shared constructor options on `AnthropicLLM` / `OpenAILLM`: ```python from pyrula.agents.llm import AnthropicLLM, LLMRate AnthropicLLM( model="claude-sonnet-4-6", max_tokens=8192, # default llm_call_timeout=120.0, # per attempt; default None (the turn `timeout` still bounds it) max_retries=2, # retries after the first attempt; default 2 rate_limit=LLMRate(rpm=50, tpm=40_000), # pace live calls; see Rate limits ) ``` `AnthropicLLM` also takes `prompt_caching=True` and `cache_ttl="5m"`; `OpenAILLM` takes `base_url`, `temperature`, `top_p`, `seed` and `parallel_tool_calls`. ### Gemini `GeminiLLM` runs on the `google-genai` SDK (`pip install "pyrula[gemini]"`, `google-genai>=1.46,<3`). It authenticates with `api_key=` (or `GEMINI_API_KEY` / `GOOGLE_API_KEY`), or against Vertex AI with `vertexai=True, project=..., location=...` and application default credentials (or `credentials=`); a prebuilt `google.genai.Client` goes in `client=`. Build that client with `HttpOptions(retry_options=HttpRetryOptions(attempts=1))`: `AgentLLM` owns retries, and the adapter sets one attempt on the client it builds itself. `provider` is `gcp.gemini` or `gcp.vertex_ai` in telemetry. ```python from pyrula.agents.llm import GeminiLLM GeminiLLM( model="gemini-2.5-pro", vertexai=True, project="my-project", location="europe-west4", max_tokens=8192, # max_output_tokens; default 8192 temperature=0.2, top_p=None, seed=None, thinking_config={"include_thoughts": True}, # thought summaries arrive as thinking blocks config={"media_resolution": "MEDIA_RESOLUTION_LOW"}, # any other GenerateContentConfig field ) ``` Messages keep the Anthropic shape the other clients take, so a PDF or image goes in as a `document` or `image` block: `{"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": ...}}` is sent inline, and `{"type": "url", "url": "gs://bucket/file.pdf", "media_type": "application/pdf"}` as a file URI (`media_type` is read from the extension when omitted, and required when the URL has none). A `tool_result` carries text only; another part type is refused. Structured output (`ctx.llm.structured`) uses the same forced `final_result` tool as the other clients, with function-calling mode `ANY`. Gemini 3 models require the thought signature they attach to a function call to come back with it; the adapter keeps it on the assistant message's block (`signature`) and sends it back. A journal written without one replays unchanged. Usage maps `prompt_token_count` plus `tool_use_prompt_token_count`, less `cached_content_token_count`, to input, the cached tokens to cache reads, and `candidates_token_count` plus `thoughts_token_count` to output. Cache storage is billed hourly by Google and is not metered. A prompt or answer Gemini refuses ends the call with `GeminiBlockedError` and is not retried; a malformed function call, or a stream that ends without a finish reason, is retried. A 429 waits for the `retryDelay` Google puts in the error details. `with_model()` returns a copy on another Gemini model, so a what-if can swap Flash for Pro. Automatic function calling is off: tools are declared only, and `@tool` functions run in pyrula's loop. To keep your own `google.genai` calls and make them durable, use the drop-in `pyrula.agents.compat.genai` instead: see [Gemini](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#gemini). ### Amazon Bedrock `BedrockLLM` runs on `boto3`'s `bedrock-runtime` Converse API (`pip install "pyrula[bedrock]"`, `boto3>=1.38`), so one client covers Claude, Nova, Llama, Mistral and the other Converse models. There is no key: credentials come from boto3's own chain (environment, shared profile, SSO, an instance or task role), and the adapter accepts, stores and journals none. `model` is a model id, an inference-profile id (`us.anthropic.claude-sonnet-4-6-v1:0`) or an ARN; `region=` and `profile=` pick the region and profile when the environment does not. No credentials or no region raises `ConfigurationError` at construction, and a 403 from AWS ends the turn with a message naming the credential chain and model access, without retries. `provider` is `aws.bedrock` in telemetry. boto3 is blocking, so each call, and each event of a stream, runs on a thread of the loop's default executor; a Bedrock call holds one thread while it waits. The client the adapter builds makes one attempt (`AgentLLM` owns retries) and waits up to `read_timeout=300` seconds for bytes, since botocore's own 60 s cuts a long non-streamed answer. A prebuilt `boto3.client("bedrock-runtime")` goes in `client=` and its retry configuration is left as it is: give it `Config(retries={"total_max_attempts": 1})`, or botocore retries under pyrula's retries. A throttling or 5xx error (`ThrottlingException`, `ServiceUnavailableException`, `InternalServerException`, `ModelNotReadyException`), including one sent as an event inside a stream, and a dropped connection are retried before the first block (always in the default safe mode); validation and access errors are not. Messages keep the Anthropic shape the other clients take: text, `tool_use`, `tool_result` (`is_error` becomes `status: "error"`), `image` and `document` blocks. Image and document bytes go inline from a base64 source, or by reference from an `s3://` url source; any other url is refused, since Converse does not fetch it. A document's `title` becomes its Converse name, with characters Converse refuses replaced by spaces. Consecutive messages of one role are merged, as Converse requires alternating roles, and an empty tool result is sent as `(no output)`. `tool_choice` maps to `auto`, `any` or a named tool; `none` is refused. Anything else in `additional_model_request_fields=` (Claude's `{"thinking": {"type": "enabled", "budget_tokens": 4000}}`), `guardrail_config=` or `config=` (any other Converse parameter, applied last) goes through unchanged. Reasoning streams as thinking blocks and goes back on the next request only with its signature, which the assistant message keeps; redacted reasoning is dropped and not sent back. Prompt caching is on by default for the Claude models that support it (3.5 Haiku, 3.7 Sonnet, the 4 series and later: system, tools and the end of the transcript) and for Nova (system and transcript, not tools), and off for every other model and for an opaque id (an application inference profile, provisioned, custom or imported model), because Converse rejects a cache point on a model without caching. `prompt_caching=True` or `False` overrides that. Usage maps `inputTokens` (already excluding cached tokens) to input, `cacheReadInputTokens` and `cacheWriteInputTokens` to the cache counts, and `outputTokens` to output, from the `metadata` event after `messageStop`. A `guardrail_intervened` or `content_filtered` stop raises `BedrockBlockedError` and `model_context_window_exceeded` raises `BedrockContextWindowError` (code `context_window_exceeded`), both after the call's usage is metered and neither retried; tool arguments that are not JSON, which a `max_tokens` cut produces, raise `BedrockResponseError`; a stream that ends without `messageStop` is retried. Converse replies name no model, so a call is priced by the id you asked for, then its canonical id (`claude-sonnet-4-6` for the profile above): Claude on Bedrock is priced at the Anthropic row. The packaged table prices these Bedrock-native models at the US East on-demand standard rates AWS publishes (its Price List API, version 20261102165015): Amazon Nova Pro, Lite, Micro and 2 Lite (`amazon.nova-pro-v1:0`, canonical `nova-pro`), Meta Llama 4 Maverick and Scout and Llama 3.3 70B (`meta.llama4-maverick-17b-instruct-v1:0`), and Mistral Large 3 and Mistral Large (`mistral.mistral-large-3-675b-instruct`, `mistral.mistral-large-2402-v1:0`). Nova lists cache read and write rates; the others list no cache price, so their cache tokens are priced at the input rate. Rates in other regions differ. Nova Premier, Nova 2 Pro, Cohere and the other Llama and Mistral models have no row; add one with `prices=` or `--prices`, keyed by the canonical id (`nova-pro`), or the model is refused with `unpriced_model` under `max_cost_usd`. A profile id finds its model's context window through the canonical id; Bedrock's window for a model can be smaller than the model maker's, so set `context_window=` when it is. A Claude profile id defaults to 200,000 tokens, since Bedrock serves the 1M window only with a beta flag in `additional_model_request_fields`; set `context_window=` with the flag. A forced `tool_choice` (structured output uses one) is sent only to Claude, Nova and Mistral Large; on any other model Converse accepts `auto` only, so the tool is offered and the model decides, and a reply without the call goes through the usual validation retry. A history that holds tool calls sent without `tools` gets a `toolConfig` naming them, which Converse requires. `with_model()` returns a copy on another Bedrock model. aioboto3 is not supported. To keep code that uses Anthropic's own `AnthropicBedrock` or `AnthropicVertex` client and make it durable, swap the import to `pyrula.agents.compat`: see [Anthropic on Bedrock and Vertex](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#anthropic-on-bedrock-and-vertex). A call is priced at the Anthropic row by the canonical form of its model id, as `BedrockLLM` does. To keep your own boto3 `converse` calls and make them durable, convert your client with `journaled(client)`: see [Amazon Bedrock (boto3)](https://docs.pyrula.io/agents/integrations/durable-vs-not.md#amazon-bedrock-boto3). ## Rate limits `rate_limit=LLMRate(rpm=..., tpm=...)` paces live calls. Every call, including `complete()` and `structured()`, reserves a slot in the store before it reaches the provider, and a caller whose slot is ahead waits for it. Clients that share a `key` (default: the provider and model) share one queue across processes and workers, so the limit holds however many callers wait. The store must support it (memory, SQLite and Valkey do); a store that does not raises `ConfigurationError` instead of running unmetered. ```python from pyrula.agents.llm import AnthropicLLM, LLMRate AnthropicLLM( model="claude-sonnet-4-6", rate_limit=LLMRate( rpm=50, tpm=40_000, max_wait=60.0, # longest a call waits for its slot; None waits as long as it takes overflow="fail", # "fail" ends the turn with llm_rate_limited; "proceed" calls anyway, unmetered ), ) ``` - A call whose slot is more than `max_wait` seconds away is refused without taking a slot. With `overflow="fail"` the turn ends with the error code `llm_rate_limited` (`LLMRateLimitedError`); with `overflow="proceed"` the call runs, a warning is logged and the span carries `pyrula.llm.rate_limit="unmetered"`. - A waiting call checks for cancel and pause about every 5 seconds, and a call cancelled while waiting gives its slot back. - `tpm` is charged an estimate when the slot is taken and corrected to the provider's reported usage afterwards. A call larger than `tpm` is charged the limit, so it is not refused forever. - A provider 429 with `Retry-After` holds every caller on the key for that long (at most 60 seconds), refunds the failed attempt's tokens, and the retry takes a new slot. When the hold ends the bucket starts empty: the first call goes out at that moment and the rest are spaced at the rate, with no burst. - The queue is shared through the store. `MemoryStore` limits one process and `SQLiteStore` one host; only `ValkeyStore` (and `PostgresStore`, which keeps its live state in Valkey) holds the limit across hosts. Two processes on a memory store each get the full rate. - A wait for a retry's slot does not count against `llm_call_timeout`, which bounds each provider attempt only. ### When a turn ends with `llm_rate_limited` The slot was further than `max_wait` away. Any of these clears it: 1. Raise `max_wait` (or set it to `None`) so callers queue longer. 2. Lower the worker concurrency, so fewer calls compete for the key. 3. Raise `rpm` or `tpm`, if the provider account allows it. 4. Set `overflow="proceed"` to call the provider unmetered when the queue is full. Nothing re-runs a refused turn: submit it again, or set `max_wait=None` to wait instead. ### Custom clients `AgentLLM` owns the retries: a custom `LLMClient` makes one attempt per `stream` or `complete` and lets the provider's error out. Do not loop inside the client. `AgentLLM` retries what it judges transient, takes a rate slot for each attempt, and sleeps between attempts. To change how many times it retries, set `max_retries` on the client (an int of at least 0; a client that has none gets 2). Build the provider's SDK client with `max_retries=0`, or its retries run underneath `AgentLLM`'s, unmetered. For a transient error of your own, raise a subclass of `pyrula.contracts.errors.Transient`: `AgentLLM` retries it; an exception it does not recognise is not retried. ### Drop-in clients and the LangChain chat model The `pyrula.agents.compat` clients and `durable_chat_model` take the same `rate_limit`: ```python from pyrula.agents.compat import AsyncOpenAI from pyrula.agents.llm import LLMRate client = AsyncOpenAI(rate_limit=LLMRate(rpm=500, tpm=200_000, key="openai-main")) ``` Inside a run, every live request reserves a slot before it is sent, through the store the run already has, so these clients share a queue with each other and with `OpenAILLM` or `AnthropicLLM` on the same key. The default key for `AsyncOpenAI` is `OpenAILLM:`, for `AsyncAnthropic` `AnthropicLLM:`, and for `durable_chat_model` `:`; a client with another `base_url` (Azure, a proxy, a local server) adds that host to its default key, natives and drop-ins alike (`OpenAILLM@localhost:`), since it is another account. The same applies inside a `ctx.step` or `@tool` body, where the call is not journaled but still takes a slot. A replayed call takes no slot. A slot further than `max_wait` away ends the run with `llm_rate_limited` and the provider is not called; `overflow="proceed"` calls it anyway. - `client.rate_limit` reads the limit, and `with_options(rate_limit=...)` or `copy(rate_limit=...)` replaces it (`None` removes it). - With a `rate_limit`, the client sends requests through the SDK with its retries switched off and makes the `max_retries` the client was configured with itself, so each attempt takes its own slot and a 429 holds the account for its `Retry-After`. Without a `rate_limit` the SDK retries as usual. `durable_chat_model` cannot do this for an arbitrary inner model: its inner model's own retries are unmetered, and a warning says so; set `max_retries=0` on the inner model to count every attempt, which also removes those retries. - A streamed request holds its slot while the stream opens and is charged the estimate, then settled to the usage the stream reports when it ends. A stream stopped early keeps the estimate. TPM counts input plus output tokens, as for the native clients. A stream stopped early is not charged for the output it did generate. A chat stream in a plain `@workflow` (which does not ask for usage) keeps the estimate. - Outside a run (no run state reachable) the limit has no effect, and a warning says so once. - The refusal is journaled as the call's outcome, so a body that catches `LLMRateLimitedError` takes the same branch after a resume, which does not ask the limiter again. A run the refusal ended asks again when it is retried. A refusal in a plain `@workflow` surfaces as `LLMRateLimitedError` from the call. - The LangGraph node (`as_langgraph_node`) submits a run and awaits it, so it makes no provider call of its own; the agent a worker runs for it is limited by its own client's `rate_limit`. The MCP server and client call tools, not models, so they have nothing to limit. ## Streaming mode ```python from pyrula.agents.stream.kinds import StreamingMode OpenAILLM(model="gpt-6-luna", streaming_mode=StreamingMode.LIVE) ``` | Mode | Blocks reach the run stream | A provider error mid-call | |---|---|---| | `SAFE` (default) | once the call finishes | retried, up to `max_retries` more attempts | | `LIVE` | as they stream | retried until the first block has been written; after that the call fails | `SAFE` buffers up to `max_buffer_bytes` (5 MiB) per call; more ends the turn with `BUFFER_OVERFLOW`. Use `LIVE` when time-to-first-token matters more than retries. ## Long conversations Before each call the loop estimates prompt tokens. Over budget, it compacts the history; still over the model's window, it ends the turn with `context_window_exceeded`. - **Window:** known for common Claude and GPT models. For others set `context_window`, or management is off (one warning per model). - **Budget:** `(context_window - reserved_output_tokens) × compaction_fraction`; defaults 4096 and 0.75. - **Default compactor:** `drop_oldest` keeps the first message and the 6 most recent, never splits a tool call from its result, and makes no LLM call, so it replays identically. Each compaction writes `context:compacted`. ```python from pyrula.agents import AgentLimits, agent from pyrula.agents.llm.context import make_drop_oldest @agent(limits=AgentLimits( context_window=128_000, compaction_fraction=0.6, history_compactor=make_drop_oldest(keep_recent=10), # None disables compaction )) async def support(ctx, message: str) -> str: ... ``` A custom compactor is `async (CompactionContext) -> list[dict]` with `history`, `token_count`, `budget`, `window` and `model`. It can call `await ctx.summarize(messages, instructions=...)`, a journaled LLM call. Keep tool calls with their results. See [Context compaction](https://docs.pyrula.io/agents/authoring/compaction.md) for what is recorded and a worked example. ## Retries One layer: `AgentLLM` retries a failed call, and the provider clients make a single attempt each. `max_retries` (default 2, so three attempts) is a constructor option on `OpenAILLM`, `AnthropicLLM` and `LiteLLM`; the SDK's own retries are turned off. Retried: status 408, 409, 429 and 5xx, connection and timeout errors, and an error the provider sends inside an otherwise successful stream. A response header `x-should-retry` decides either way before the status does. Other 4xx errors are not retried, and 401 and 403 end the turn as a configuration error. `llm_call_timeout` bounds each attempt, so a timed-out attempt is retried. The wait is the provider's `retry-after-ms` or `Retry-After` (capped at 60 seconds), else exponential backoff from 0.5 s, capped at 8 s, with jitter. The run's cancel and pause are checked about every 5 seconds of the wait. `complete` and `structured` retry at any point in either mode. A streamed `run` retries at any point in `SAFE` mode, and in `LIVE` mode only while no block has been written. The `pyrula.agents.compat` drop-ins retry on the same rule. `durable_chat_model` is the exception to one layer: LangChain's own retries inside its inner model still run. ## Replay safety Raw SDK calls in an agent body are not recorded and fire again on replay. Go through `ctx.llm` ([Determinism](https://docs.pyrula.io/workflows/concepts/determinism.md)).