Language detection and translation¶

AIHub centralizes runtime language detection and translation in src/common/utils. Entity standardization, Skills Library, Chatbot / Oria, MCP and Training Matching therefore use the same provider order and degraded behavior while keeping backend-specific persistence rules.

DeepL requests reuse one lazily created HTTP client per application event loop, including requests from concurrent imports. The unified lifespan closes it after background operations and consumers stop. Standalone async callers use connector_clients_scope() from src.common.connectors.shared_clients; the evaluation runner owns the same scope. Missing DeepL credentials do not create a transport. Requests retain their 120-second timeout, per-request authorization, batch limits and existing immediate tiny LLM fallback.

Language detection¶

Detection follows one shared decision chain:

1. Repeated inputs are deduplicated while preserving their first-seen order. py3langid evaluates the batch outside the asynchronous event loop using two detectors initialized once on the first detection. They share one loaded full model and its tokenizer tables, while the restricted detector owns its language-specific arrays. A lock coordinates concurrent first calls, and both models are fully configured before publication. Later detections only read the model state. Async callers load the models in their worker thread, keeping the event loop responsive. Model-loading failures propagate without invoking the LLM fallback; empty inputs and language-code normalization do not load the models. The broad model accepts a result with normalized probability at least 0.20 and a margin of at least 0.15 over the runner-up. Scores are model-specific and are not interchangeable with Lingua's relative-confidence distance.
2. A weaker broad winner can be accepted when the model restricted to the eight frontend language roots agrees: broad probability at least 0.05, restricted probability at least 0.35, and restricted margin at least 0.18. Otherwise, overlapping 2/3/4-word segments retain the existing dominant-language recovery: margin at least 0.08, length-weighted score at least 30, and winner ratio at least 1.4. The model's zxx label remains undetected. Every decision is filtered through the caller's allowed language set.
3. Outside Chatbot and MCP, py3langid is the only language detector. An inconclusive result remains null: translation workflows treat that text as requiring translation, while DeepL and the translation LLM fallback auto-detect its source language. Training Matching uses its explicit request-locale fallback.
4. Chatbot and MCP enable the structured language-detection LLM fallback for inconclusive user messages sent to graph agents. With compatible OpenAI mini, it sends one native choice per ambiguous text, selecting an allowed ISO code or explicit undetermined value. Other providers and unsupported model mappings retain the tenant's tiny structured call, with no prompt cache and the profile's reasoning effort (none by default). Both paths have a 15-second request timeout and keep the request locale if detection remains inconclusive. Native Decisions uses its API's default input caching and never retries through generation.

Entity standardization uses the local-only grouped entry point once for all unique candidate fields before resolving the MongoDB translation cache. The Skills Library applies the same rule to a query: an inconclusive result enters the translation flow instead of being assumed to be English.

The Chatbot restricts detection to its frontend locales and normalizes language roots such as pt and zh to the supported presentation locales pt-BR and zh-CN. Streaming resolves this language before opening the SSE response, so all progress labels and the final payload use one locale. MCP agent wrappers and the Training Matching entry point use the same detector before building their agent context.

The public ISO language-code set is unchanged. The model's Norwegian no label is normalized to nb. Maori (mi), Tswana (tn) and Tsonga (ts) have no direct class in py3langid 0.4.0, so local recognition of those languages is not equivalent to Lingua. Short labels can also remain undetermined more often; no additional LLM fallback is enabled by the migration. See the calibration corpus and results.

Translation provider chain¶

Dynamic translation uses the following best-effort sequence:

1. Empty text, or text whose explicit source and target locales are equal, is returned immediately. Repeated texts are deduplicated while preserving their first-seen order.
2. All texts in a provider-safe group are sent in one DeepL request through its text array. DeepL is attempted once per group. Its first network error, non-success HTTP status or malformed or incomplete response immediately ends that group's DeepL path; there is no retry or backoff.
3. If DEEPL_API_KEY is absent or a grouped DeepL attempt fails, the complete failed group is sent in one structured translation call through the tenant's tiny LLM profile. The call has a 30-second timeout, no prompt cache and a bounded output size. Its reasoning follows the profile (none by default), preserving configured overrides. Stable numeric indexes associate each returned translation with its source text.
4. If the LLM fails, omits an item or returns unusable content, the source text is preserved for the affected item so optional localization does not abort the business workflow. Valid items from a partially usable LLM response are still retained.

A standard entity therefore normally performs at most one DeepL call, or one DeepL call followed by one grouped LLM fallback, for all of its uncached text. The implementation creates the minimum additional groups only when DeepL's 50-text or 128 KiB request limits require it.

The common boundary logs provider type or safe HTTP status, never provider payloads, credentials or source text. Task cancellation is not converted into a fallback and continues to propagate.

Persistence and backend boundaries¶

The MongoDB cache is deliberately limited to translations toward English. Its lookup key is the source text, and writes use idempotent upserts protected by the unique translations_title_unique index. Entity-standardization cache writes and request-time Skills Library cache access are best-effort: a persistence failure does not discard an available translation or abort the search response. Standardization collects all unique non-English texts, reads the cache once with a MongoDB $in query, sends only cache misses to the grouped provider chain, then writes all successful new translations in one unordered bulk operation. Bulk skill import uses the same grouped lookup, translation and write pattern while retaining its import-level failure contract.

Every translation-cache read or write failure is logged by the owning workflow with the failed operation, workflow context and safe exception type. A failed read continues with an empty cache and translates directly; a failed write preserves the translated business result. Logs never include source text, credentials or raw exception details.

Deliberate exceptions¶

Static streaming-label dictionaries are normal localization assets and do not call translation providers. Existing skill-translation catalogues remain owned by their MongoDB or PostgreSQL repositories. Workflows whose LLM prompt already requires output in the user's language, such as localized rationale generation, generate that content directly instead of translating it a second time.

Concrete provider models behind tiny are documented in LLM providers, models and tasks.