LLM runtime, caching and observability¶

All direct and LangGraph LLM calls use the shared AI-provider connector. Provider modules still create their own clients; common OpenAI-compatible behavior is centralized for request formatting, structured output, usage normalization and prompt-cache parameters.

OpenAI transport, Responses API helpers, compatibility utilities and error diagnostics live together under src/common/connectors/ai_providers/providers/openai/. Azure OpenAI and OVH reuse the compatibility utilities from that package.

For the functional mapping between providers, semantic profiles, concrete fallback models and business tasks, see LLM providers, models and tasks.

Model selection¶

Every text or multimodal generation workflow selects a semantic LLM profile. Runtime configuration retains the resolved catalogues of all providers, with tenant overrides applied only to the tenant's selected provider. When a workflow requires another provider API, the connector selects that provider's catalogue before resolving its model and reasoning settings. CV parsing uses the selected provider's mini profile; native Anthropic Lexi generation and token counting use reasoning. Business calls do not accept a concrete model override. Transport adapters accept already-resolved models and derive missing defaults from the central profile map. Audio, Realtime and embedding models keep their dedicated configuration selectors.

configuration_resolver.py applies the same configuration precedence to MongoDB and PostgreSQL: a valid client/request provider, a valid database default, then the code provider default. Models must belong to that provider's catalogue; missing or invalid selections use the database profile/default and finally a centralized code model. model_profiles.py completes LLM profiles individually, while model_defaults.py owns embedding metadata and speech/Realtime defaults. Provider changes do not retain the previous provider's model overrides. Catalogue reads remain asynchronous in async callers, and database errors propagate.

Prompt structure and caching¶

Cache-aware prompts put stable instructions before dynamic request context:

stable instructions shared by repeated calls

**Dynamic Request Context**
tenant, user, role, skills or request-specific values

Caching is not a local response cache: the request still reaches the provider. The connector requests provider-side caching by default for every direct and LangGraph LLM call. A call can opt out of AIHub directives with prompt_cache=False; for generation this does not disable a provider's implicit caching. Native Decisions explicitly disables caching when opted out or ineligible, as described below. The provider receives cache directives only when all of these are true:

• the provider is OpenAI or Anthropic;
• the call is not in batch mode, or uses Anthropic Message Batches;
• a stable system/developer prefix exists;
• that prefix meets the selected model's estimated minimum length (512 tokens for Claude Haiku/Sonnet 5.5 and Opus 5/5.5; 1,024 for OpenAI). Older configured Claude models retain their documented higher thresholds.

Short or fully dynamic generation prompts are sent without cache directives. Anthropic calls and Message Batches use a 5-minute cache boundary after the static instructions by default, without automatically caching the variable request suffix. Batch cache hits are best-effort because requests run asynchronously and may outlive the cache. Batch and cache discounts stack. Length remains a character-based estimate, not a provider token count; cache usage in the response is authoritative. Tools and output schemas can also contribute to the provider-rendered prefix.

OpenAI calls receive a stable cache key and, on supported models, an explicit breakpoint after the reusable instructions, including text-block system messages. Block metadata and the changing suffix are preserved. Eligible GPT-5.6/GPT-6 calls use explicit-only mode so the changing request suffix cannot trigger a billable cache write; older OpenAI models retain their automatic caching behavior. Anthropic receives an ephemeral cache boundary. If a provider or model rejects its cache parameters on a generation call, the connector logs the rejection and retries once without caching. Other errors are not retried by this fallback. Azure and OVH usage is normalized but AIHub does not send automatic cache directives for them. Azure may still apply its native implicit cache.

For Anthropic conversations, internal callers can additionally request prompt_cache_conversation=True. The LangGraph adapter does so when it replays assistant messages or tool results; text simulation turns explicitly opt in. This adds a request-wide automatic boundary for reusable conversation history, in addition to the stable system boundary. One-shot classifiers and generators remain static-only, and Message Batches never add this automatic boundary. The conversation option takes effect only when the shared cache policy enables caching. Monitor cache_read and cache_creation in normalized usage (and cache_read_tokens / cache_creation_tokens in provider call logs) to verify reuse and compare writes against later reads; configuration alone proves no hit.

The memory-update classifier, simulation feedback analysis, action plans and knowledge-answer synthesis keep their instructions in a stable system message. Preferences, conversation history, transcripts, evaluation criteria, profiles, objectives and response language are supplied afterward as request context. Objective suggestions place existing objectives after the explicit dynamic boundary; tenant-specific feedback categories remain part of their reusable instructions. This preserves output schemas and business rules while allowing requests with different input data to reuse the same instruction prefix. Short instructions are not padded merely to qualify for caching.

Native Decisions caching¶

Every native Decisions workflow uses the same prompt_cache.py eligibility and cache-key helpers as generation. Keys include the resolved model, task and stable prefix digest, fit the 64-character API limit, and exclude variable request data. Single-choice calls place their rules and existing choice definitions in shared developer input; their typed choices refer to those definitions. Multi-question calls reuse their existing shared_instructions. Safety instructions remain in the trusted prefix, and dynamic developer context follows the cache breakpoint. User evidence remains a separate user message. Question-specific instructions, allowed values and identifiers stay in questions, outside the reusable prefix. Multi-question choice descriptions also stay with their owning questions.

Eligible calls use prompt_cache_options={"mode": "explicit", "ttl": "30m"} and an input_text breakpoint after the stable developer prefix. All other Decisions calls use explicit mode without a breakpoint, preventing implicit cache writes of variable evidence. prompt_cache=False is available on both single- and multi-question helpers. SDK errors retain the normal bounded retry policy; there is no cache-rejection retry or switch to generation.

This covers guardrails, supervisor routing, memory, training search and scope, opportunity profile intent, objective relevance, language detection, Lexi assignments/group types/duplicates, skill relevance/levels and marketplace filter selection. Short rules, including many categorical batch workflows, may remain ineligible. A request with multiple questions is not a provider Batch API request: its stable prefix can be reused across sequential question chunks.

The minimum is 1,024 reusable input tokens; question tokens do not count. AIHub's character estimate only determines whether to request caching. Actual reads and writes reported by OpenAI determine the benefit. Cache misses and first writes can cost more than uncached input, so compare total costs across cold and warm calls, not just cache-hit rates. See the OpenAI Decisions caching guide.

Each llm_decision event includes prompt_cache, prompt_cache_requested, prompt_cache_reason, prompt_cache_stable_tokens, cache_read_tokens and cache_write_tokens, alongside task, model, duration and cost. LangSmith metadata includes eligibility without prompt text, and normalized usage retains cache_read/cache_creation. Input totals already include both categories; usage and costs are summed once per request chunk, never per question.

Output and reasoning policies¶

Calls without an explicit output limit default to 4,096 output tokens. Known long-output tasks default to 8,192 tokens. Lexi tasks keep their intentionally managed output budgets. Token limits do not select a reasoning level.

Model and reasoning are resolved from the semantic profile. An explicit tenant effort overrides AIProviders; an explicit provider effort overrides the central profile default. Missing effort uses tiny=none, mini/default=low and large/reasoning=medium. Model-only and effort-only tenant settings preserve the other configured field. Model inheritance does not copy another profile's effort. Unsupported model controls are omitted; configured efforts are never silently lowered. External GenAI request overrides remain supported.

Anthropic receives the selected effort through output_config.effort, including Opus normal, streaming and Message Batch requests. Haiku's none maps to disabled thinking. Provider adapters translate the resolved setting rather than choosing a task-specific effort. PDF/image extraction and CV parsing follow their large and mini profiles respectively; summaries inherit mini.

OpenAI GPT-5.4, GPT-5.5, GPT-5.6 and GPT-6 calls with tools, tool history, or retained Responses output items use the Responses adapter rather than disabling reasoning. It translates function schemas, named tool selection, JSON output formats and multimodal input, and normalizes completed Responses into the shared content/tool/usage contract. LangGraph stores opaque continuation items in assistant-message metadata and replays them only to their originating model, with store=False. Streams close on completion, provider failure and cancellation. Error events and failed responses propagate through centralized OpenAI logging; application deadlines identify responses.create for this path.

LangGraph profiles currently use:

The actual provider and model are resolved from each tenant's ai_configuration; the profile names are not fixed model names. The summary runtime keeps its dedicated 1,024-token limit while resolving the mini profile. Invalid tenant overrides return to the provider mapping. If that mapping itself cannot be resolved, provider-specific code fallbacks prevent a model from being sent to the wrong API. OpenAI is used when no LLM provider configuration is available.

Supervisor routing and guardrail classification use typed Decisions calls with the mini profile. Their explicit question names are destination and guardrail_decision. Only guardrail doubt/block outcomes invoke separate mini response generation, with a 512-token budget. Safety decision and response share one 15-second deadline and separate task labels for cost/latency attribution. Native Decisions requires an already-resolved compatible OpenAI profile and does not apply generation limits or reasoning settings. Capability selection retains the existing structured calls for other providers and unsupported model mappings, without changing the tenant configuration or switching to generation after a failed native call. See supervisor routing and guardrail decisions for validation and failure semantics.

Batched categorical decisions¶

ai_providers.decisions supports named independent choice questions, sequential chunks of at most 400 questions and 2–255 choices per question. All questions are validated before sending the first chunk. Response validation rejects missing, duplicate or unexpected question names, unknown choices, invalid probability distributions and refusals. Answers are reconstructed in requested order using stable application IDs; labels are never regenerated. A failed chunk does not return partial results. Cancellation propagates and each native request has a 15-second deadline covering all attempts and retry waits. Like ordinary OpenAI LLM calls, native Decisions uses the SDK's asynchronous exponential backoff with jitter and provider retry hints. max_retries=2 allows at most three attempts in total for connection errors, request timeouts, HTTP 408/409/429 and server errors, subject to the SDK's retry-control headers. Refusals, invalid decision content and configuration errors do not trigger another attempt. There is no additional application retry loop.

Native requests use the resolved mini profile. Only OpenAI's compatible model currently exposes this capability. A catalogue exceeding the choice limit keeps the complete catalogue on the existing generative path; it is never truncated. Empty assignments and single-type classifications can be deterministic. The connector records request-level usage once per chunk, not once per question, with zero output cost and input/cache costs from shared pricing. Probability and confidence remain diagnostic signals; business unknowns use explicit choices.

Marketplace prefill remains generative on large, after loading options for selected filters only. Language detection still uses py3langid first and invokes the optional model fallback only for ambiguous texts. Lexi's fixed-provider selection remains owned by its workflows; the global fit check and default skill-level generation resolve the central OpenAI mini profile when no configuration is supplied, matching their existing OpenAI defaults. An explicit Anthropic level-generation override remains on its original provider. Native support requires a resolved configuration; callers with unresolved configurations retain ordinary provider resolution.

Existing workflow error contracts remain in place: safety/router failures stop the turn; optional profile enrichment, memory and language detection retain their best-effort behavior. Invalid global-fit responses keep skills; invalid objective responses keep objectives. Marketplace invalid decisions become the same sanitized model-response error. These are domain error contracts, not provider retries. Training classification/clarification failures retain the existing best-effort search behavior. A native skill-level failure propagates to the owning workflow without partial levels, invented proficiency values or a generative retry. Training clarifications use mini, a 512-token limit and a 15-second call timeout; decisions and conditional generation have separate task names for cost/latency analysis.

OpenAI error diagnostics¶

The optional internal Chat Completions stream collector concatenates refusal fragments into response_metadata.refusal, matching non-streaming completions. It preserves the provider's finish reason and keeps refusal text separate from generated content and tool arguments. Structured-output callers can therefore diagnose an empty answer as a refusal. GenAI relays native SSE chunks directly.

The OpenAI provider emits an ERROR event named openai_error when an SDK call fails. The shared formatter covers client initialization, asynchronous chat, streaming consumption, embeddings, Decisions, Responses file requests, transcription and speech synthesis. LangGraph calls use the same provider through the shared connector.

Each event includes the operation, model, exception class, elapsed milliseconds, HTTP status and API error code/type when available. provider_request_id is the OpenAI x-request-id; it is separate from AIHub's request_id and client_id, which remain attached through the active request context. Transport failures include the underlying cause's class, such as ConnectTimeout, without its message. Identifiers must pass a bounded format check; prompts, files, audio, credentials, raw exception messages and response bodies are excluded.

Application-level OpenAI deadlines use the same event, with llm_task and timeout_seconds to identify the timed-out operation. Search plain logs for OpenAI request failed, or filter JSON logs on event=openai_error.

An event is emitted when the SDK raises, before any application-level retry. Intermediate retries performed internally by the SDK are not logged separately; an in-flight call has no failure event until it raises or its application deadline expires. Timeout values, retries, exception propagation and cancellation remain unchanged. Embedding batches retain their existing continuation behavior without duplicating the SDK failure event.

LangSmith traces¶

Graph tracing is active only when a LangSmith/LangChain API key exists and one of the supported tracing flags is true. Calls made through the shared connector emit an LLM run named call_llm_async only while an active LangGraph run is present. Direct calls from background processing, administration routes or tests still execute normally, but do not create standalone LangSmith runs.

Connector child runs include:

• llm_task, provider, actual model and token policy;
• input, output, total and reasoning-token usage when supplied by the provider;
• cache-read and cache-creation token details when supplied;
• input_cost, output_cost and total_cost in USD;
• cache policy metadata (prompt_cache_requested, reason and stable-token estimate).

Cost is computed from the actual model when known, otherwise from the configured model. A model may define a pricing block in its configuration to override the built-in list-price table. Unsupported models keep their usage data but do not receive an estimated cost.

In LangSmith, inspect the complete named graph run and its call_llm_async child run outputs or usage/cost fields. Tool name and tool-call identifiers remain on LLM children created inside graph tools. The connector suppresses the duplicate LangChain model span while retaining callbacks needed for streaming events.

Training Finder database searches add one retriever child run per hybrid-search branch. MongoDB uses training_mongo_pipeline; its input contains the aggregation pipeline with the query vector redacted. PostgreSQL uses training_postgres_query; its input contains the compiled SQL and bound parameters with the query embedding redacted. Both spans identify the vector or full_text search mode and expose only result count and elapsed milliseconds in their output, never the returned documents or rows.

Asynchronous generation¶

Text LLM requests use call_llm_async() and embedding requests use generate_embeddings_async(). Provider SDK implementations are asynchronous; there is no synchronous connector. File, streaming and provider batch workflows retain their dedicated asynchronous APIs and semantic model profiles.

MongoDB skill-inference maintenance awaits native async reads, generation and writes, with at most eight concurrent entities. The bounded batch queue applies backpressure, logs individual failures and continues with the remaining entities. Cancellation stops and awaits its workers and closes the cursor. The HTTP request continues to await completion. JOM spreadsheet insights also await the async LLM connector. PostgreSQL user reconciliation keeps its existing durable job flow.