Configuration¶

AI Hub combines environment-level infrastructure settings with an application configuration document. MongoDB deployments store one Config document in each tenant database. PostgreSQL deployments use one global configuration in the database selected by DATABASE_URL. Never commit real keys, tokens or connection strings.

Client configuration normalization¶

The administrator endpoint PUT /admin/api/v1/configuration/clients/normalize reconciles stored configuration with src/templates/config_file.json. It adds missing defaults, recursively removes keys outside the template and preserves existing scalar values and lists. A structural section that is not an object is rejected before writing. Normalization is idempotent: an already normalized configuration is not written again. Existing credential sanitization and SQL import identifier validation still apply.

The following dynamic mappings are preserved in full when present, including empty objects:

• hard_filters, chatbot.hard_filters and rules;
• ai_configuration.llm_profiles, including configured models and reasoning settings;
• skill_scores.weight_type;
• standardization.{user,opportunity,job,training}.skills_inferring.reference_ids.

If a dynamic section is absent, it receives its template default. The canonical template also includes the supported user/opportunity import field selectors, user/opportunity/feedback transformation client selectors, pool.additional_pool and jpm_scores controls, so normalization retains these active settings. The PostgreSQL initialization seed includes the same defaults for newly created configuration rows; existing rows change only when normalization is requested.

MongoDB configuration reads and replacements, client enumeration and PostgreSQL configuration access use asynchronous drivers. The request returns HTTP 202; completion counts and failures are emailed to the selected configuration's recipients (the template for all-client MongoDB normalization). See Administration for client selection, global processing, response counts and errors.

Docker image¶

The Dockerfile uses python:3.14-slim and installs runtime and documentation dependencies from requirements.txt, without retaining the pip download cache or dependency bytecode. It installs only the entries above the exact # Test dependencies marker; keep that marker unchanged and place every runtime or documentation dependency above it. Local development and CI quality jobs still install the complete file, including the trailing test and development tools. All versions remain defined in that single file. defusedxml is an explicit runtime dependency so XML parsing protections do not depend on the installation of development tools.

The runtime image excludes evaluation cases and results, and removes the bundled tests directories from NumPy, SciPy, pandas and scikit-learn. Runtime datasets and testing helper modules remain available. Native extension debug information is stripped while executable code and dynamic symbols are retained; binutils is installed and removed within the same build layer. Native source-level debugging therefore requires an image built without that stripping step. These packaging changes do not remove scientific libraries or alter their configured algorithms. Development and CI environments retain the full packages.

Development caches, tests and Python bytecode are also excluded from the build context. The image keeps the application sources, Alembic migrations, configuration and documentation tooling needed to generate the English site at /docs/ in a background subprocess after API initialization. See the documentation lifecycle. Python can generate bytecode normally when the container runs.

docker build -t aihub:local .
docker run --rm --env-file .env -p 127.0.0.1:8000:80 aihub:local

The database addresses in .env must be reachable from the container; localhost points to the container itself. Use the database service name on a shared Docker network, or host.docker.internal on Docker Desktop for a database running on the host. Check http://localhost:8000/health-check after startup. The image runs Gunicorn with uvicorn_worker.UvicornWorker, three workers and port 80. uvicorn-worker supplies the Gunicorn integration; uvicorn remains the explicitly pinned ASGI server dependency. The release Dockerfile also remains compatible with the Kaniko build used by CI. The staging and production build job uses gzip compression level 6 for newly built image and cache layers, reducing registry upload size at the cost of more compression CPU time. This changes transfer size, not the installed image contents, and does not replace diagnosis of Harbor or reverse-proxy errors. The runtime documentation builder reads API versions and dependencies directly from source files. Docker builds and Rundeck/Ansible source copies require no version build argument or release metadata file. Global release tags are created or verified by the CI version job on develop and release/*. Requirements-only, documentation-only and tests/ changes increment that tag without changing API versions.

Core environment variables¶

Deployment environment¶

FastAPI resolves one deployment environment during application startup. Set APP_IDENTIFIER_SHORT to the environment name. If it is absent, empty or equal to mutu ignoring case, AI Hub falls back to ENV. Startup fails if the resulting value is empty.

The resolved value is shared with every mounted application and all background work. It supplies the Swagger default for environment query parameters and scopes MongoDB database and generated collection names. An explicit API input must match it; a different or empty value returns 400. API keys do not carry or override this setting in either MongoDB or PostgreSQL mode.

MongoDB¶

MONGO_CONNECTION_STR='mongodb://user:password@host:27017'
MONGO_MAX_POOL_SIZE='20'
MONGO_PROVISIONING_TIMEOUT_SECONDS='120'

This connection provides every MongoDB database used by AI Hub, including common_data, reference_data and tenant databases. When DATABASE_URL is absent, it also stores LangGraph checkpoints and long-term memory. It must support the Search and Vector Search features used by embeddings and hybrid search. PwC deployments can additionally provide MONGO_PWC_STAGING_DSN and MONGO_PWC_PROD_DSN.

Application and administrative operations use native async MongoDB clients. Only the pinned LangGraph MongoDB adapters retain a dedicated synchronous client for the default DSN; no synchronous PwC clients are initialized. Shared document types live separately from client construction. See LangGraph persistence for lifecycle ownership.

MongoDB clients validate the connection with a ping during initialization. If the URI is rejected or authentication fails with MongoDB error code 18 and the password requires encoding, AI Hub retries once after URL-encoding only that component. Existing percent-encoded passwords are preserved. Successful sync and async connections produce an application-level info log without exposing the URI or credentials. Other connection, authorization and network errors are returned without this fallback.

AI Hub supplies the Certifi CA bundle to MongoDB clients by default. For a deployment that does not expose TLS, add tls=false to its URI. The connection factory detects this option and does not pass tlsCAFile, including for the sync, async and PwC clients:

MONGO_CONNECTION_STR='mongodb://user:password@mongo1:27017,mongo2:27017/?replicaSet=rs0&authSource=admin&tls=false'

Set the option on every non-TLS URI. Connections without an explicit tls=false retain the CA bundle and existing TLS behavior.

AI providers¶

Without DATABASE_URL, provider and model definitions are read from the MongoDB AIProviders collection. PostgreSQL deployments read the same logical objects from aihub_ai_providers. Alembic seeds the ovh, openai, anthropic and azure providers, with OpenAI as the default LLM and embedding provider. Voyage is not supported and its obsolete PostgreSQL configuration is removed by migration. Model dictionaries remain JSONB so provider-specific attributes are preserved, while provider identity and default flags are relational columns.

MongoDB deployments are not modified by Alembic. Remove any legacy voyage document from their AIProviders collection before selecting or exposing the available providers; the runtime no longer contains a Voyage client.

Provide credentials only for providers referenced by tenant configuration:

# OpenAI
OPENAI_API_KEY='...'

# Azure OpenAI
AZURE_OPENAI_API_KEY='...'
AZURE_OPENAI_URL='https://<resource>.openai.azure.com'

# Anthropic
ANTHROPIC_API_KEY='...'

# OVH AI Endpoints
OVH_API_KEY='...'

Live voice sessions use the direct OpenAI provider and therefore require OPENAI_API_KEY. OPENAI_CA_BUNDLE can select a custom trusted CA bundle for the Live WebSocket.

Direct outbound HTTP¶

AIHub-owned direct HTTP calls use HTTPX2 and verify certificates against the operating-system trust store. Minimal containers and deployments behind a TLS-inspecting corporate proxy must install their CA system-wide or configure SSL_CERT_FILE or SSL_CERT_DIR. HTTP clients owned by OpenAI, LangChain and other frameworks retain their own dependency and certificate behavior.

Environment and authentication¶

ENV='integration'
APP_IDENTIFIER_SHORT='integration'
AI_HUB_USER_API_KEY='<user-api-key>'
AI_HUB_ADMIN_API_KEY='<admin-api-key>'
JWT_PUBLIC_KEY_PATH='/run/secrets/jwt-public-key.pem'

JWT_PUBLIC_KEY_PATH is required only when clients send bearer JWTs. See authentication for the RS256 contract and Client-ID routing. Standard endpoints accept AI_HUB_USER_API_KEY or AI_HUB_ADMIN_API_KEY; administrator endpoints accept only AI_HUB_ADMIN_API_KEY. Each non-empty applicable variable is authoritative. If none is configured, MongoDB fallback is available only without DATABASE_URL; PostgreSQL mode rejects the request. The generated Zensical site, Swagger, ReDoc and OpenAPI schemas are public.

Request resource limits¶

Untrusted spreadsheet, CSV, JSON and other non-audio uploads are read in bounded chunks. The default limit is 10 MiB. Chat speech and speech-to-text uploads use a separate 25 MiB default. Oversized content returns HTTP 413, including when no Content-Length header is available.

Upload and LLM-heavy POST routes share a fixed-window IP quota checked before authentication, including requests with missing or invalid keys. After API-key validation, separate upload and LLM quotas also apply to the valid key across client addresses. Counters store SHA-256 digests of addresses and keys. SQLite shares counters between the three Gunicorn workers in one container; it performs its short transactions on worker threads. The file is not shared between pods, so multi-pod deployments must retain a global rate limit at the ingress or API gateway. Exceeding either quota returns 429 with Retry-After; unavailable counter storage returns 503 before business processing.

The IP quota uses the ASGI client address. AIHub does not parse Forwarded or X-Forwarded-For itself. Behind an ingress, configure Gunicorn's FORWARDED_ALLOW_IPS with the trusted proxy addresses or networks and ensure that the ingress sanitizes forwarding headers. The container's Uvicorn worker passes this trust configuration to its proxy middleware. Without it, clients behind a proxy may share the proxy's quota. Do not trust arbitrary forwarding headers or set FORWARDED_ALLOW_IPS=* on a directly accessible server. Tune the IP quota for legitimate clients sharing an address.

LangSmith¶

LANGSMITH_API_KEY='...'
LANGSMITH_PROJECT='aihub-integration'
LANGSMITH_TRACING='true'

The legacy LANGCHAIN_API_KEY, LANGCHAIN_TRACING and LANGCHAIN_TRACING_V2 names are also recognized. A key and at least one truthy tracing flag are required. These settings trace named LangGraph executions and their named children, including connector LLM calls and Training Finder database searches. LLM calls made outside an active graph are not sent to LangSmith. The PostgreSQL migration smoke job explicitly disables all tracing flags so its synthetic persistence graph cannot appear in an environment project. See LLM runtime for usage, cost, prompt-cache and database-search fields.

Blocking work pool¶

The shared run_blocking helper runs document extraction, catalogue indexing, chatbot job/opportunity vector deduplication, spreadsheet processing and local SFTP key/stream reads in a dedicated executor. SFTP network transfers use AsyncSSH directly. The SQLite request limiter keeps using asyncio's separate default executor.

Limits apply per application worker; three Gunicorn workers therefore allow up to twelve concurrent calls in these pools. Libraries can have additional internal threads. Requests beyond the admission limit wait asynchronously before submission; this bounds the executor queue, not the total number or memory of incoming HTTP requests. Invalid settings fail startup.

Cancelled queued calls are skipped when a thread reaches them. Their slots remain reserved until then, preventing cancelled entries from accumulating beyond the queue limit. An already running thread continues and retains its capacity until completion; Python cannot forcibly stop it. Its result or exception is retrieved even if its caller has disconnected. The lifespan drains the pool before closing database resources. Shutdown can therefore wait for a running synchronous operation; no hard timeout is implied. Request correlation context is copied into each submitted call.

Standalone asynchronous scripts that call these workflows must own the same scope (the Lexi evaluation command already does):

from src.common.blocking_work import blocking_work_scope, run_blocking

async def main():
    async with blocking_work_scope():
        result = await run_blocking(str, 42)

Batched MongoDB vector searches¶

MONGO_VECTOR_SEARCH_CONCURRENCY defaults to 8 and must be a positive integer. It bounds batched skill searches and the skill/opportunity/job similarity lookups in atlas_vector_search.py. Each call creates at most this many workers, and all these calls share the same active-query budget per application worker. The budget is not a limit on every MongoDB query or on PostgreSQL searches. With three Gunicorn workers, these MongoDB lookups can run up to 24 queries concurrently across the process group.

Input ordering, result dictionaries, cosine conversion, thresholds, candidate filters, projections and index selection are preserved. Ordinary query failures allow the other items to finish before the first observed exception is propagated; partial results are not returned. Request cancellation stops the workers and awaits cursor cleanup before returning, including if cancellation is repeated. Cursors are also closed when processing fails.

FastAPI owns the budget through its lifespan. Standalone scripts using these MongoDB workflows must enter mongo_search_scope() from src.common.connectors.mongodb.search_concurrency, alongside blocking_work_scope() when they also invoke synchronous processing.

Datalake and batch processing¶

Configure Azure Blob storage with:

BLOB_ENDPOINT='DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...'

When BLOB_ENDPOINT is absent or empty, Datalake writes are silently skipped.

Batch controls:

BATCH_MAX_CONCURRENCY='32'
BATCH_DATALAKE_UPLOAD_CONCURRENCY='8'

See batch processing for queueing, backpressure and file compatibility.

Outgoing email¶

All outgoing email, including administrative reports, Lexi results and error notifications, uses one shared asynchronous connector. A nonblank BREVO_API_KEY selects the Brevo HTTPS API:

MAIL_SENDER='aihub@example.com'
BREVO_API_KEY='...'

Use a Brevo API key, not an SMTP key. The sender address must be authorized in the Brevo account. With a key configured, MAIL_SENDER is required; a missing sender is a configuration error. Brevo failures never trigger SMTP delivery.

If BREVO_API_KEY is missing, empty or whitespace-only, the connector selects the legacy SMTP transport instead. This supports environments where Brevo is unavailable, including US staging and production. Configure all five variables:

MAIL_SENDER='aihub@example.com'
MAIL_SMTP_LOGIN='...'
MAIL_SMTP_KEY='...'
MAIL_SMTP_URL='smtp.example.com'
MAIL_SMTP_PORT='587'

MAIL_SMTP_URL is a hostname, without a URL scheme. SMTP uses authenticated STARTTLS with certificate verification, not implicit TLS. The port must be an integer between 1 and 65535; an invalid configured port is a configuration error. The password is preserved exactly as configured.

If any required SMTP variable is missing, empty or whitespace-only, the connector logs one warning listing the missing variable names and skips delivery without opening a network connection. The calling process continues, but no email is sent. Environment diagnostics expose only presence flags for the Brevo key and SMTP login, key, host and port; their values are never included. The SMTP path does not import the Brevo SDK.

Both transports support plain text, HTML and file attachments. On Brevo, JSON attachment names gain a .txt suffix, such as operation-123.json.txt, because Brevo does not accept .json attachments; their contents are unchanged. SMTP retains original attachment names, including .json, and safely encodes special characters in MIME attachment headers. Both retain attachment bytes and use results.csv when no filename is provided.

Each send owns and closes its connection, including on failure or cancellation. An overall 20-second deadline bounds delivery. The Brevo SDK alone retries transient connection/server/rate-limit failures, with at most three attempts; authentication errors are not retried. SMTP makes one attempt without automatic retries. Configured delivery errors propagate to the existing caller boundary; administrative reports isolate them from the operation result. Successful acceptance by the selected transport is logged without recipient details and does not guarantee inbox delivery.

Lexi batching¶

Lexi has separate controls because its intentionally large generation outputs must not use the generic LLM limits:

Anthropic generation additionally supports J2S_GENERATION_CLASSIC_MAX_TOKENS, J2S_GENERATION_MAX_TOKENS, J2S_GENERATION_BATCH_POLL_INTERVAL_SECONDS and J2S_GENERATION_BATCH_TIMEOUT_SECONDS. These are output/call-mode controls, not global chatbot limits.

Smart Mobility SQL imports¶

Numeric tenant and entity IDs are validated before opening the SQL connection and bound as integers. Decimal integer strings such as "0042" are accepted as ID 42; alphanumeric values, booleans, fractional numbers and values outside 0..9223372036854775807 are rejected. Searches by opportunity code, employee number or email preserve the original text, including leading zeros.

Opportunity imports retain custom_job.code as joboffer_code in imported records and code in standardized documents. Null values remain null; non-null values are stored as strings, preserving existing text, whitespace and leading zeros. The field is descriptive metadata for identifying offers and is not added to matching, inference, translation or embedding inputs. Existing MongoDB documents receive the source value on their next import and standardization.

Optional standardization.user.import_fields.contract_type, standardization.user.import_fields.relationship and standardization.opportunity.import_fields.contract_type accept integer IDs or decimal integer strings, including the disabled sentinel -1. Omit unused settings; explicit null or empty values are invalid. Updating configuration via PUT /admin/api/v1/configuration/clients rejects invalid values with HTTP 422 before persistence. PostgreSQL configuration writes apply the same validation; existing configurations written outside AIHub must also satisfy these rules.

standardization.user.import_fields.technical_classification and standardization.opportunity.import_fields.technical_classification follow the same identifier validation and disabled -1 convention. They select Smart Editor fields independently for users and offers. The SQL import and PostgreSQL projections read the selected option's company_entity_item.name when company_item_id is set; otherwise they read the association's name. An unresolved option is missing data, never its numeric option ID. Both joins are restricted to the entity's tenant and selected field. Single-valued fields are expected; if duplicate associations exist, the lowest association ID wins.

The resulting technicalClassification is a string, with "-1" for a null or empty value. MongoDB imports extract a numeric prefix followed by a colon from the resolved value: "19: Other subsidiaries - Executive" becomes "19". This applies to direct association values and selected options for both users and offers. Plain codes and values without that numeric prefix remain unchanged. Existing MongoDB documents require reimport to receive this normalization. PostgreSQL projections currently preserve source labels. Neither path uses attached-job grading; jobClassification remains independent.

For La Poste (NB-9232), the import settings are 6530 for users and 6529 for opportunities. These are tenant settings, not global defaults. Configure hard_filters.jom.user.technical_classification and hard_filters.jom.opportunity.technical_classification with each direction's existing additional_parameters.classification_level_matches. Replace the JOM job_classification rule when activating this change; configuring both rules would apply both restrictions. Preserve the separate JPM job_classification rules and their mappings. Existing MongoDB documents require reimport and stored matching results require recalculation; deploying code alone does not activate this tenant configuration or populate historical documents.

Offer activities preserve their source code without converting it to an integer. User completion, entity and hierarchy data remain available to reporting even when they are not stored in MongoDB user documents.

Active training imports always combine tenant and requested-ID filters with end IS NULL OR end >= NOW().

Job imports set status=true when job.is_active=1 and either status_id is null or the associated JobStatus.is_visible=1. A missing status therefore keeps an active job active. A non-null status reference with no matching JobStatus row, or a status whose visibility is null or false, does not activate the job. is_hidden is imported separately and does not affect status.

Smart Mobility reads use SQLAlchemy's async engine with the asyncmy driver. The connector supports a direct database connection or an SSH tunnel managed by asyncssh.

For a uniform direct connection, set:

SM_HOST='mariadb.internal'
SM_PORT='3306'
SM_USERNAME='...'
SM_PASSWORD='...'
SM_DATABASE='optional_database_override'

When SM_USERNAME is unset, the connector selects environment-specific values from the source database name:

SSH ports default to 22; SQL ports default to 3306 in these fallback branches. The current tunnel preserves the former permissive behavior by not verifying the SSH server host key (known_hosts=None). Deployments should therefore restrict network access and credentials appropriately.

The query result is converted to a pandas DataFrame and then records. Batch standardization is bounded, but the SQL result itself is materialized in memory.

PostgreSQL runtime and migrations¶

The PostgreSQL notification listener, independent job workers, SQLAlchemy runtime and Alembic migrations share the same database connection settings:

DATABASE_URL='postgresql://user:password@host:5432/sm_mutu'
PG_SCHEMA='mysql'
PG_LANGGRAPH_POOL_MAX_SIZE='10'
PG_EMBEDDING_POOL_MAX_SIZE='10'
PG_JOB_POLL_INTERVAL_SECONDS='10'
PG_JOB_WORKER_RETRY_DELAY_SECONDS='5'
PG_JOB_CANCELLATION_CLEANUP_TIMEOUT_SECONDS='5'
PG_JOB_SHUTDOWN_TIMEOUT_SECONDS='15'
PG_JOB_FORCE_CANCEL_TIMEOUT_SECONDS='1'
PG_EMBEDDING_LISTENER_RETRY_DELAY='5'
PG_EMBEDDING_NOTIFY_CHANNEL='aihub_embedding_jobs'
PG_EMBEDDING_BATCH_SIZE='25'
PG_EMBEDDING_MAX_RETRIES='3'
PG_EMBEDDING_LEASE_SECONDS='900'
PG_EMBEDDING_RETRY_BASE_SECONDS='10'
PG_EMBEDDING_RETRY_MAX_SECONDS='300'
PG_USER_INFERENCE_NOTIFY_CHANNEL='aihub_user_skill_inference_jobs'
PG_USER_INFERENCE_BATCH_SIZE='10'
PG_USER_INFERENCE_MAX_RETRIES='3'
PG_USER_INFERENCE_LEASE_SECONDS='900'
PG_USER_INFERENCE_RETRY_BASE_SECONDS='10'
PG_USER_INFERENCE_RETRY_MAX_SECONDS='300'
PG_USER_RECONCILIATION_PAGE_SIZE='500'
PG_USER_RECONCILIATION_MAX_RETRIES='3'
PG_USER_RECONCILIATION_LEASE_SECONDS='300'
PG_USER_SKILL_GAP_RECONCILIATION_PAGE_SIZE='500'
PG_USER_SKILL_GAP_RECONCILIATION_MAX_RETRIES='3'
PG_ENTITY_RECONCILIATION_NOTIFY_CHANNEL='aihub_entity_reconciliation_jobs'
PG_ENTITY_RECONCILIATION_PAGE_SIZE='1000'
PG_ENTITY_RECONCILIATION_MAX_RETRIES='5'
PG_ENTITY_RECONCILIATION_LEASE_SECONDS='300'
PG_SKILL_SEARCH_EMBEDDING_DIMENSION='1536'
PG_JOB_SEARCH_EMBEDDING_DIMENSION='1536'
PG_JOB_EMBEDDING_NOTIFY_CHANNEL='aihub_job_embedding_jobs'
PG_JOB_EMBEDDING_BATCH_SIZE='25'
PG_JOB_EMBEDDING_MAX_RETRIES='3'
PG_JOB_EMBEDDING_LEASE_SECONDS='900'
PG_JOB_EMBEDDING_RETRY_BASE_SECONDS='10'
PG_JOB_EMBEDDING_RETRY_MAX_SECONDS='300'
PG_OPPORTUNITY_SEARCH_EMBEDDING_DIMENSION='1536'
PG_OPPORTUNITY_EMBEDDING_NOTIFY_CHANNEL='aihub_opportunity_embedding_jobs'
PG_OPPORTUNITY_EMBEDDING_BATCH_SIZE='10'
PG_OPPORTUNITY_EMBEDDING_MAX_RETRIES='3'
PG_OPPORTUNITY_EMBEDDING_LEASE_SECONDS='900'
PG_OPPORTUNITY_EMBEDDING_RETRY_BASE_SECONDS='10'
PG_OPPORTUNITY_EMBEDDING_RETRY_MAX_SECONDS='300'
PG_TRAINING_SEARCH_EMBEDDING_DIMENSION='1536'
PG_TRAINING_ENRICHMENT_NOTIFY_CHANNEL='aihub_training_enrichment_jobs'
PG_TRAINING_ENRICHMENT_BATCH_SIZE='25'
PG_TRAINING_ENRICHMENT_MAX_RETRIES='3'
PG_TRAINING_ENRICHMENT_LEASE_SECONDS='900'
PG_TRAINING_ENRICHMENT_RETRY_BASE_SECONDS='10'
PG_TRAINING_ENRICHMENT_RETRY_MAX_SECONDS='300'
PG_MATCHING_NOTIFY_CHANNEL='aihub_matching_jobs'
PG_MATCHING_BATCH_SIZE='10'
PG_MATCHING_MAX_RETRIES='3'
PG_MATCHING_LEASE_SECONDS='1800'
PG_MATCHING_RETRY_BASE_SECONDS='30'
PG_MATCHING_RETRY_MAX_SECONDS='900'
PG_MATCHING_EXPORT_MAX_ROWS='250000'
PG_RAG_EMBEDDING_DIMENSION='1536'

DATABASE_URL is the canonical PostgreSQL connection string. SQLAlchemy and Alembic normalize it to the postgresql+psycopg:// dialect when required; direct psycopg configuration readers remove that SQLAlchemy-only driver marker.

PG_SCHEMA defaults to mysql and qualifies the PostgreSQL skill_translation, aihub_embedding_jobs, aihub_entity_enrichment_jobs, aihub_ai_providers, aihub_config and LangGraph persistence tables. The current SQLAlchemy models map the AIHub-owned tables and the read-only mysql.skill_search_v1, mysql.aihub_user_v1 and mysql.aihub_job_v1, mysql.aihub_opportunity_v1 and mysql.aihub_training_v1 views, plus the Chatbot-owned aihub_conversations, aihub_chat_message_log, aihub_events, aihub_documents, aihub_rag_source_files, aihub_rag_chunks and aihub_rag_metrics tables. The Alembic migration bootstrap assumes the existing relational business tables already exist; it does not create the source user, skill, job, opportunity, training or organization model. The user projection expects training-attendance associations in company_entity_item_formation, with the source user ID stored as text in name for a same-tenant company_entity whose flag_id is 4. PG_EMBEDDING_JOBS_TABLE can override the queue table, but defaults to aihub_embedding_jobs. PG_EMBEDDING_NOTIFY_CHANNEL independently overrides the notification channel and uses the same aihub_embedding_jobs default. Deployments that explicitly configured either variable with the former embedding_jobs value must update it when applying the renaming migration. One psycopg connection listens to all configured notification channels. Each queue has its own asyncio worker, so a slow matching or AI call does not block skill, user, opportunity, job or training work. Notifications wake only the workers bound to that channel; PG_JOB_POLL_INTERVAL_SECONDS also makes every worker poll periodically so committed jobs remain discoverable after a lost or coalesced notification. Every pass claims a bounded batch and immediately yields before claiming another one.

Long-running jobs receive a unique lease token and refresh heartbeat_at while external AI or matching work is running. A worker may finalize a row only while it still owns that token. Expired leases are reclaimed, retries use the per-domain bounded exponential backoff, and a clean cancellation normally returns the interrupted batch to pending immediately. Skill embeddings, entity enrichments and matching allow a newer pending row to coexist with an older processing row for the same entity. If reclaim, retry or cancellation finds that newer row, the older claim becomes terminal instead of attempting to create a duplicate pending key; the newer work remains queued. Parent reconciliation jobs use keyset pages instead of scanning a complete tenant in a trigger or one worker transaction. PG_JOB_WORKER_RETRY_DELAY_SECONDS applies only when a worker pass itself fails; the per-domain retry variables apply to failed durable jobs. Shutdown waits up to PG_JOB_SHUTDOWN_TIMEOUT_SECONDS for workers after cancellation. Lease and reconciliation-claim cleanup is bounded by PG_JOB_CANCELLATION_CLEANUP_TIMEOUT_SECONDS; work that cannot be returned within that deadline remains safe for normal lease-expiry recovery. A worker still running after the graceful deadline receives a second cancellation and gets PG_JOB_FORCE_CANCEL_TIMEOUT_SECONDS to stop.

The pgvector extension may live in a different schema: the embedding listener reads its actual schema from the PostgreSQL catalog and uses a fully qualified type cast when saving an embedding. Shared SQLAlchemy connections add the configured business schema and the installed vector, pg_trgm and unaccent extension schemas to their PostgreSQL search path.

PG_SKILL_SEARCH_EMBEDDING_DIMENSION defaults to 1536, constrains skill_translation.embeddings to a fixed pgvector dimension and must match the embedding model used by both the listener and skill-search requests. The skill search migration requires pgvector 0.8 or newer and installs the pg_trgm and unaccent extensions. It creates:

• the automatically maintained skill_search_v1 relational view;
• B-tree indexes for translation language and repository joins;
• GIN full-text and trigram indexes on normalized translation titles;
• an HNSW cosine index on translation embeddings.

skill_search_v1 and the skill-title embedding triggers include only clients whose setting.type is LIVE or DEMO and skills whose referentiel_id resolves to an existing repository. Repository attachment enqueues missing vectors, while client deactivation and repository removal clear pending work. Client activation does not scan existing skills; use the PostgreSQL initialization endpoints to enqueue that backlog. The migration reuses the existing tenant, repository, translation and partial queue indexes for these transitions.

When DATABASE_URL is set, the two single-skill endpoints use this PostgreSQL search path. An unset value keeps their existing MongoDB path. PostgreSQL search failures are surfaced to the caller and do not silently fall back to MongoDB. The PostgreSQL text branch mirrors Atlas sequential autocomplete with accent folding, prefixes and one-edit fuzzy matching. Its vector branch maps cosine distance to the Atlas score range before the shared weighted RRF merge.

The PostgreSQL user, opportunity, job and training views expose only rows owned by a setting.type of LIVE or DEMO. Their inference and enrichment queues apply the same tenant rule in addition to each entity's enabled, visibility, validity or active-date conditions. User reconciliation is processed with keyset pages in aihub_user_reconciliation_jobs. Tenant deactivation automatically schedules cleanup; activation and Alembic never scan every user. Existing users are queued through the explicit PostgreSQL initialization API. PG_USER_RECONCILIATION_PAGE_SIZE bounds each database page. The lease and retry settings reclaim interrupted parent and user-inference jobs after a worker or pod restart. Moving an individual entity into an eligible state enqueues its missing asynchronous work; moving it out removes pending work. JOM/JPM are intentionally different: tenant deactivation removes pending user matching jobs, but activation does not create an initial matching backfill. PostgreSQL JOM/JPM spreadsheet exports resolve identifiers inside the requested tenant, apply their per-entity limit in SQL and reject results exceeding PG_MATCHING_EXPORT_MAX_ROWS instead of materializing an unbounded score table.

Talent Finder searches the relational location, skills, attached role, department and textual experience fields with Atlas-like token and fuzzy semantics. Education, certification and total-experience-duration criteria are not supported. Work arrangement remains optional; when supplied, PostgreSQL returns no match if its relational projection cannot enforce that value.

Existing-tenant initialization is exposed under /admin/api/v1/postgres-initialization. Its optional client_id is setting.id; omitting it targets every LIVE or DEMO setting. The all route sequences skills, users, opportunities, jobs and trainings. A run is complete once reconciliation has inserted all required child job rows; the API does not wait for the embedding and inference workers to consume them.

PG_JOB_SEARCH_EMBEDDING_DIMENSION defaults to 1536, constrains job.embeddings and must match the embedding provider used by the PostgreSQL job worker. The migration creates the live aihub_job_v1 view and adds:

• B-tree indexes on (setting_id, is_active, status_id, id) and (setting_id, code, id) for tenant lists, counts and exact code lookups;
• a reverse (status_id, is_active, setting_id, id) index for status-visibility transition cleanup;
• the existing Job-to-family and jobskill indexes plus reverse family-item indexes required by family-change propagation;
• a partial HNSW cosine index for non-null job embeddings.

No full-text, trigram or JSON GIN index is added because current job searches are exact-ID, exact-code or vector operations. PostgreSQL triggers enqueue a deduplicated job_embedding task only for fields used by title + " - " + jobFamily + ": " + description: title, mission, formation, experience, results, note and the effective family associations/names. Skill data, classification, hidden/active/status fields and stored vectors do not trigger regeneration. PG_JOB_EMBEDDING_NOTIFY_CHANNEL controls the listener channel; the batch-size and retry variables control job claiming. Job standardization is MongoDB-only and rejects PostgreSQL mode. With PostgreSQL, the main Job API, chatbot job mapping and JPM text similarity use the relational repository without a MongoDB vector-search fallback. The view requires a LIVE or DEMO tenant; the embedding queue additionally requires an active job with a visible status.

PG_OPPORTUNITY_SEARCH_EMBEDDING_DIMENSION also defaults to 1536 and must match the embedding provider used by the PostgreSQL opportunity worker. The opportunity migration constrains all three custom_job vector columns to that dimension and rejects existing vectors with incompatible dimensions. It creates the live aihub_opportunity_v1 view, deterministic translation and public-skill projection, B-tree join/filter indexes, normalized-title GIN indexes and HNSW cosine indexes for:

• the source opportunity title;
• the inferred title;
• the reconstructed posting used by matching similarity.

With DATABASE_URL, the main opportunity list/count endpoints and the opportunity_embedding worker use this relational path. PostgreSQL triggers enqueue a single deduplicated job only after changes to the effective title or description, the attached job, its public-skill associations, public skill titles/locales or repository visibility. Changes to skill levels, location, validity, deletion state, organization fields and generated AIHub columns do not enqueue embedding work. PG_OPPORTUNITY_EMBEDDING_NOTIFY_CHANNEL controls the listener channel; the batch-size and retry variables control job claiming. Opportunity standardization is MongoDB-only and rejects PostgreSQL mode. PwC-specific fields remain MongoDB-only. Chatbot opportunity title search uses the relational inferred-title full-text and HNSW indexes when PostgreSQL is active: it retrieves 50 text and vector candidates concurrently, normalizes the vector score to the Atlas range and applies the same 50/50 weighted RRF. Structured title criteria generate embeddings before entering that hybrid path. The view requires a LIVE or DEMO tenant; pending embedding work additionally requires a non-deleted opportunity whose validity has not expired.

PG_TRAINING_SEARCH_EMBEDDING_DIMENSION defaults to 1536 and constrains the formation.aihub_enriched_title_embeddings vector used by Learning Finder. The training migration creates the live aihub_training_v1 view, reconstructing the training document from formation, organisme, training skills and training access-group entities. It adds:

• tenant/locale/date and join indexes for active accessible training searches;
• normalized-title and enriched-title GIN trigram and full-text indexes;
• a partial HNSW cosine index for non-null enriched-title vectors.

With DATABASE_URL, Learning Finder, Training Matching, rationale generation and development-objective training grounding use this relational repository without a MongoDB training-search fallback. PostgreSQL triggers enqueue one deduplicated training_enrichment job only when formation.name or formation.description changes, because those are the only fields used to generate the enriched title. Skill associations, organization, dates, duration, locale and AIHub-generated columns do not enqueue work. Training standardization is MongoDB-only and rejects PostgreSQL mode. PG_TRAINING_ENRICHMENT_NOTIFY_CHANNEL controls the listener channel; the batch-size and retry variables control job claiming. The view requires a LIVE or DEMO tenant; enrichment jobs additionally require a non-empty title and an active date window. Learning Finder's PostgreSQL text branch uses Atlas-compatible any-term full-text semantics plus per-token fuzzy matching; the training organization is used as the provider when the relational provider projection is empty. Its language-filtered hybrid pass still retries once without language constraints only when the first fused result is empty.

DATABASE_URL also selects LangGraph's official asynchronous PostgreSQL checkpointer and store. At application startup, their official setup() methods create or migrate checkpoint_migrations, checkpoints, checkpoint_blobs, checkpoint_writes, store_migrations and store, plus their official indexes, inside PG_SCHEMA. If PostgreSQL returns InsufficientPrivilege, AI Hub permits startup only after confirming that both stored migration versions match the installed packages and all required tables already exist. This allows a personal runtime account without CREATE to use a schema previously initialized by the service account. Alembic deliberately leaves these vendor-owned tables to LangGraph. Their names remain unchanged so AI Hub can use the package without overriding its SQL. The store uses exact namespace/key operations only, so no store_vectors table or vector index is created.

LangGraph owns a dedicated psycopg pool because its official implementations require autocommit, disabled prepared statements and dictionary rows. PG_LANGGRAPH_POOL_MAX_SIZE controls its maximum size and defaults to 10. Startup fails explicitly for every setup error except the validated InsufficientPrivilege case above. A personal account still fails when the schema is missing, outdated or inconsistent. It does not fall back to MongoDB, and it does not copy existing MongoDB checkpoints or memories.

Alembic also creates the AIHub-prefixed Chatbot persistence tables used when PostgreSQL resources exist. Chat history and event APIs write aihub_conversations, aihub_chat_message_log and aihub_events; document tools write aihub_documents; tenant RAG ingestion and retrieval use aihub_rag_source_files, aihub_rag_chunks and aihub_rag_metrics. PG_RAG_EMBEDDING_DIMENSION defaults to 1536, constrains aihub_rag_chunks.embedding and must match the tenant embedding model used for RAG sync and retrieval. The migration adds B-tree, partial feedback, trigram, full-text and HNSW indexes for the current Chatbot history, event, document and RAG query shapes. Conversation and message searches use all-token autocomplete with prefix and two-edit fuzzy matching across the same fields as their Atlas indexes. PostgreSQL RAG text retrieval uses Atlas-compatible any-term matching; both retain deterministic score and recency ordering.

Alembic creates aihub_config as a singleton and seeds it with a frozen copy of src/templates/config_file.json. When DATABASE_URL is set, get_config_file_async and get_ai_configuration_async read these two PostgreSQL tables and ignore MongoDB database separation. PostgreSQL errors are surfaced rather than triggering a MongoDB fallback. The Chatbot frontend config endpoints also use this backend: GET /chatbot/v1/configs reads the public chatbot section from aihub_config, and PUT /chatbot/v1/configs validates the requesting administrator through PostgreSQL user data before updating the same row. Without DATABASE_URL, both endpoints keep using the tenant MongoDB Config document.

The canonical chatbot.agents section lists the currently supported agents. Peer-feedback agents are no longer registered. Existing stored configuration keys do not enable removed agents; configuration normalization uses the current template. Historical Alembic revisions and their frozen seeds remain unchanged.

Alembic revision 20260827_23 removes the obsolete aihub_api_keys table. Protected HTTP, WebSocket and MCP operations accept X-API-Key against AI_HUB_USER_API_KEY or AI_HUB_ADMIN_API_KEY; administrator routes accept only AI_HUB_ADMIN_API_KEY. Generated documentation, Swagger, ReDoc and OpenAPI schemas remain public. A non-empty applicable environment key never falls back to MongoDB after a mismatch. If no applicable key is configured, authentication reads the MongoDB APIKeys collection only when DATABASE_URL is absent; PostgreSQL mode returns 403. MongoDB tenant routing prefers its clientId and falls back to the Client-ID header. Environment-backed keys also use that header when a client is needed. Stored data-source, environment and email fields remain ignored; routing uses the fixed smartmobility data source and the environment resolved at FastAPI startup.

The embedding listener resolves its provider and model from the PostgreSQL provider table in this mode without requiring a MongoDB connection. PG_EMBEDDING_AI_PROVIDER and PG_EMBEDDING_MODEL override that selection when set. If the shared configuration cannot be loaded or does not identify an embedding provider, the listener logs a warning and uses OpenAI as its final fallback.

Every PostgreSQL schema change must be delivered through an Alembic migration in alembic/versions/. Production and staging migrations are executed by the infrastructure workflow outside GitLab CI; the repository CI only validates the migration chain against an ephemeral pgvector-enabled PostgreSQL service.

Administrators can also trigger alembic upgrade head from the running service through POST /admin/api/v1/diagnostics/postgres/migrations/run. The endpoint uses the current process DATABASE_URL and PG_SCHEMA and returns HTTP 202. Template notification recipients are resolved before admission. A process-local lock rejects overlapping listener/migration operations, while a PostgreSQL advisory lock scoped by database and schema prevents overlap across workers, pods and CLI processes. Execution conflicts and failures are reported in the final email; successful reports include the target and current revision. The listener restarts only after successful migration. Cancellation waits for the migration thread to finish before releasing its operation lock. Alembic records internal migration details at DEBUG and executed SQL at INFO; SQL parameter values are hidden so configuration and persisted data are not exposed in application logs. The equivalent verbose CLI invocation is alembic -x debug=true upgrade head.

GET /admin/api/v1/diagnostics/postgres/jobs/summary returns an exact snapshot of the seven physical queues consumed by the PostgreSQL job runtime. It includes database-wide and per-table status counts, plus discriminator breakdowns such as entity type, matching type, reconciliation reason and initialization scope. finished is the sum of processed, failed, skipped and canceled rows; outstanding covers every remaining status, including pending, processing and waiting. The endpoint is administrator-protected, PostgreSQL-only and does not change or retry any job.

Alembic also installs database-side retention for these seven queues. The cleanup_aihub_job_history function deletes at most 10,000 rows from each queue per invocation when their terminal timestamp is older than three calendar months. It uses processed_at for embedding, enrichment and matching jobs, and completed_at for reconciliation and initialization jobs. Only processed, failed, skipped or canceled statuses supported by each queue are eligible; pending, processing and waiting rows are never removed. Deleting an expired initialization job also removes its steps through their existing cascade. Partial timestamp indexes keep each bounded deletion from scanning active work. Alembic builds these small partial indexes one queue at a time and commits after each one. A regular index build allows long-lived read snapshots to continue; it briefly pauses writes to the queue instead of waiting indefinitely for those snapshots as CREATE INDEX CONCURRENTLY would. Index lock acquisition is capped at 10 seconds and construction at five minutes, so contention produces an explicit migration error. A subsequent run repairs an invalid index left by an interrupted concurrent attempt with REINDEX INDEX.

PostgreSQL does not provide a native TTL scheduler. When the pg_cron extension is already installed, Alembic registers aihub_job_history_cleanup:<schema> at 03:17 every day in the timezone configured for pg_cron. If the extension is available but not installed, or unavailable, Alembic logs a warning without blocking the structural migration. Infrastructure must configure shared_preload_libraries and cron.database_name as required by its PostgreSQL installation, then run:

CREATE EXTENSION IF NOT EXISTS pg_cron;
SELECT mysql.schedule_aihub_job_history_cleanup();

Replace mysql with PG_SCHEMA when it differs. The scheduling helper is idempotent for the migration role's named job. A bounded cleanup can also be invoked manually with SELECT mysql.cleanup_aihub_job_history();; its optional arguments are the retention interval and per-queue batch size.

RabbitMQ¶

RabbitMQ producers and consumers share the same connection configuration. RABBITMQ_URL is the only connection setting. Supply the complete AMQP URL, including credentials, host, port and virtual host, for example:

RABBITMQ_URL="amqp://service:example-password@rabbitmq.example.com:5672/service"

Single-host URLs, including a load balancer or cluster endpoint, retain the native aio-pika connection behavior. To configure several nodes of the same cluster, separate their addresses with commas:

RABBITMQ_URL="amqp://service:example-password@rabbitmq-1.example.com:5672,rabbitmq-2.example.com:5672,rabbitmq-3.example.com:5672/service"

AIHub expands the host list before calling the AMQP client. Credentials, virtual host, TLS scheme and URL options apply to every endpoint. Bracket IPv6 addresses, for example [2001:db8::1]:5672. Commas in credentials, the virtual host or query parameters are not treated as host separators.

For multi-host URLs, initial connection attempts follow the configured order, with a 10-second connection timeout per endpoint. A failed endpoint is closed before trying the next. Initial attempts fail fast per endpoint even if the URL contains fail_fast=0. If every endpoint fails, producers report the connection failure and consumers retry the complete list after RABBITMQ_RECONNECT_INTERVAL. After a connection interruption, the robust connection selects the next endpoint, wrapping to the first after the last, and retains native channel and consumer restoration. Reconnection attempts use the same timeout and configured interval. This does not add message replay after a failed publication. Single-host URLs keep their existing timeout and retry behavior.

Percent-encode reserved characters in the URL credentials. When RABBITMQ_URL is absent (read as None), empty or whitespace-only, RabbitMQ is unavailable: the shared connector logs a warning and opens no connection. Consumers exit without registering queues or retrying. Score producers skip publication; queued client imports fail explicitly because RabbitMQ is required. Score publication reports the existing RabbitMQ failure result instead of logging a successful send. No hostname or credential environment variables are used as a fallback.

A nonblank URL still follows the configured single-host or multi-host connection behavior described above; configuration and connection failures are not treated as disabled RabbitMQ.

GET /admin/api/v1/diagnostics/environment exposes the boolean RABBITMQ_URL is present without returning the URL or its credentials. The flag is false for absent, empty or whitespace-only values. It reports configuration presence, not broker reachability.

Operational tuning includes RABBITMQ_PREFETCH_COUNT, RABBITMQ_HEARTBEAT, RABBITMQ_RECONNECT_INTERVAL and RABBITMQ_QUEUES_PREFIXES.

All RabbitMQ consumers acknowledge (ack) each completed processing attempt, including failures and malformed JSON. Processing exceptions reaching the RabbitMQ boundary are logged once with their traceback, then acknowledged without requesting requeue or dead-lettering. A failed message must be republished manually to retry it.

User, opportunity and job messages whose tenant lookup reports an unknown client database emit one concise ERROR with entity type, client ID, data source and environment, without an exception traceback. They return normally and are acknowledged as well. This includes a missing client registration or an unavailable registered database.

Cancellation during processing still propagates and leaves the message unacknowledged. An acknowledgement failure emits a warning. If the channel or connection closes before an acknowledgement reaches RabbitMQ, the broker can redeliver the message; this policy does not guarantee exactly-once processing.

Automatic startup and administrative controls allow RabbitMQ consumers in any zone except eastus, including proxmox, gra11, and an unset or empty ZONE. The local deployment environment remains excluded. Automatic startup logs when the local environment or the eastus zone prevents consumers from starting.

When DATABASE_URL is set, RabbitMQ consumers are disabled at application startup and the generic administration routes under /admin/api/v1/consumers/{start,stop,restart} reject PostgreSQL mode. Those generic routes remain MongoDB/RabbitMQ-only. The separately named /admin/api/v1/consumers/postgres-listener/{start,stop,restart} routes start, stop or restart the shared PostgreSQL notification listener and all of its independent job workers as one runtime. They never start RabbitMQ consumers.

RABBITMQ_QUEUES_PREFIXES retains its existing base-prefix contract. For example, sm_integration expands tenant prefixes sm_integration_nextgen and sm_integration_legacy. Queue names add the existing transport namespace: sm_sm_integration_legacy.user_profile. Queued initialization shares this exact construction with consumers and does not rename existing queues. The tenant's rabbitmqPrefix must match one of the expanded prefixes. The existing listener activation requires sm_ in RABBITMQ_QUEUES_PREFIXES.

Configured Smart Mobility prefixes create consumers for user profiles, job offers, trainings, feedback, skills and job resource actions in every environment. All consumers in one application instance share a single robust connection and use one channel per queue. aio-pika restores the channels, queues and subscriptions after a transport interruption. Training queues are declared passively: if a queue does not exist, only that consumer logs a warning and is not registered. An initial connection failure is retried according to RABBITMQ_RECONNECT_INTERVAL.

Logging¶

config/logging.yaml is the central logging configuration. Application logs written with Loguru and standard-library logs from Gunicorn, Uvicorn and third-party clients are sent through the same Loguru sink.

The default auto profile selects:

• console with colors when the process writes to an interactive terminal;
• plain without ANSI colors when output is captured by an IDE, process manager, container runtime or log collector.

Both profiles use the same concise single-line format. The json profile is available as an explicit opt-in and emits one compact object containing only the useful log fields; it does not expose Loguru's verbose internal text/record envelope. All profiles write to the process stream. File rotation and Loki transport remain the responsibility of the container or hosting platform, which avoids competing files and transports across Gunicorn workers.

The following environment variables can override operational settings without modifying the YAML file:

Request logging emits one completion event with the HTTP method, route template, status and duration. X-Request-ID is reused when safe, generated otherwise, returned in the response and propagated to logs emitted while the request is processed. Client-ID is propagated when present. Query strings, request bodies, API keys, bearer tokens and arbitrary headers are never added to the request event. Health and metrics paths can be excluded in the YAML configuration.

MongoDB tenant resolution emits a WARNING with event=client_database_resolution_failed when routing cannot select a client database. Search plain logs for Client database resolution failed. The message includes the effective client_id, data_source, environment and a reason: configuration_not_found, invalid_client_database (missing or non-string clientDB), database_not_found, or missing_client_id (a datasource or environment was supplied without a client ID). These values reflect the lookup after authentication/default resolution, not the full HTTP inputs. The structured lookup_client_id field preserves the effective value and its string/integer type separately from the middleware's header-derived client_id correlation.

Both synchronous and asynchronous lookups retain the active request context. Routing text is limited to 128 input characters and sanitized; oversized values and nonscalar objects are omitted. Successful lookups and calls without any routing parameters, used to access only common/reference databases, are silent. The diagnostic does not change HTTP responses, database queries or consumer continuation behavior. Consumer-specific skip messages remain available.

OpenAI failures emit a centralized ERROR event with event=openai_error, including safe provider metadata and the active request/client correlation. Search plain logs for OpenAI request failed. See OpenAI error diagnostics for supported operations, timeout details and retry visibility.

All configured profiles apply the same confidentiality filter from src/core/log_redaction.py before output or queueing. Nested credential fields (API keys, passwords, tokens, authorization/cookie headers, connection strings, private keys and signatures) and SQL parameter fields are replaced with [REDACTED], regardless of common casing or separator variants. The caller's objects are not modified. Ordinary event, request/client correlation, status and duration fields remain available.

Messages also redact recognizable credential assignments, Basic/Bearer tokens, JWTs, private-key blocks and SQLAlchemy SQL/parameter detail blocks. URLs retain their destination/path but lose user credentials, query values and fragments. Control characters are escaped to prevent forged lines or terminal sequences. An individual string over 8,192 characters is omitted whole. Structured context has a 16,384-character budget, 200 visited values, depth 6 and 50 entries per container; excessive content and cycles become [OMITTED]. Custom objects and binary values are omitted without calling their representation methods.

Exceptions retain bounded type and frame-location summaries, including causes and exception-group members (up to eight exceptions and 30 frames in total). For each traceback, the remaining frame budget retains its innermost frames, including the failure location. When outer frames are dropped, [OMITTED] appears before the retained frames. Later causes or group members keep their types but omit frames if the shared budget is exhausted. Raw exception messages, SQL parameters, source lines and local variables are never rendered from exception objects, even with LOG_BACKTRACE or LOG_DIAGNOSE enabled. Application exception propagation and HTTP responses are unchanged.

This filter cannot recognize every unlabeled secret, personal datum or token embedded in an arbitrary URL path or message. Call sites must still minimize payloads and use safe operation/reason codes. Direct print, independent sinks and external tracing systems are outside this configured logging boundary.

Backend response cookie¶

Every HTTP response emitted by AIHub carries a session cookie named SRV, whose value is read once from socket.gethostname() when the Main API middleware stack is initialized. No environment variable is required. Local runs report the computer hostname; container deployments report the container hostname, normally the pod hostname in Kubernetes. Workers sharing one host report the same value.

For an HTTPS request, a response includes a header such as:

Set-Cookie: SRV=aihub-7c8f9d-x2abc; HttpOnly; Path=/; SameSite=lax; Secure

The cookie is host-only, has no expiry or maximum age, and applies to all paths. Plain HTTP responses omit Secure, including local development. Behind a TLS terminating proxy, the scheme must be resolved by the server's trusted proxy configuration (FORWARDED_ALLOW_IPS); AIHub does not trust raw forwarded headers itself. Existing response cookies are preserved, and an incoming SRV value is never used to identify the serving backend.

The same behavior covers the Main API and all mounted applications, including MCP, documentation, redirects, CORS preflight, validation errors and unhandled HTTP 500 responses. Streaming and SSE responses send the cookie with their initial headers without buffering the body. WebSocket and application lifecycle messages pass through unchanged.

To diagnose a failing request, inspect its Set-Cookie response header in the browser Network panel or with an HTTP client. HttpOnly prevents JavaScript from reading the cookie, and the existing CORS policy is unchanged; cross-origin clients must not assume the browser persists it. Inspect the individual response because a later response may report another backend. Errors generated entirely by an ingress or proxy cannot receive this application cookie.

Tenant AI configuration¶

LLM routing is based on semantic profiles rather than model names in application code. The common ai_configuration document for each provider owns both the model catalogue and the default mapping. For example, an OpenAI document can contain:

See LLM providers, models and tasks for the functional profile-to-task mapping, provider capabilities and last-resort model table.

{
  "provider": "openai",
  "is_default_llm_provider": true,
  "is_default_embedding_provider": true,
  "llm_models": {
    "luna": {"llm_name": "gpt-6-luna"},
    "sol_6_1": {"llm_name": "gpt-6.1-sol"},
    "terra_6_1": {"llm_name": "gpt-6.1-terra"}
  },
  "llm_profiles": {
    "default": {"model": "luna", "reasoning_effort": "low"},
    "tiny": {"model": "luna", "reasoning_effort": "none"},
    "mini": {"model": "luna", "reasoning_effort": "low"},
    "large": {"model": "terra_6_1", "reasoning_effort": "medium"},
    "reasoning": {"model": "sol_6_1", "reasoning_effort": "medium"}
  },
  "llm_realtime_models": {
    "gpt_live_1": {"llm_name": "gpt-live-1"}
  },
  "default_llm_realtime_model": "gpt_live_1",
  "embedding_models": {
    "embedding_small": {
      "embedding_name": "text-embedding-3-small",
      "mongo_vector_index": "embedding_small_vector_index",
      "mongo_vector_path": "embedding_small_vector",
      "embedding_dimension": 1536,
      "threshold_delta": 0
    }
  },
  "default_embedding_model": "embedding_small",
  "embedding_profiles": {
    "lexi": "embedding_small"
  }
}

Azure uses the same profile names but maps them to its own deployment names. The generation connector uses AZURE_OPENAI_URL with /openai/v1/ and does not read per-model API versions. The current standard catalogue and profiles are:

{
  "provider": "azure",
  "is_default_llm_provider": false,
  "is_default_embedding_provider": false,
  "llm_models": {
    "gpt_5_4_mini": {"llm_name": "gpt-5.4-mini"},
    "gpt_5_2": {"llm_name": "gpt-5.2"},
    "gpt_4_1_nano": {"llm_name": "gpt-4.1-nano"}
  },
  "llm_profiles": {
    "default": {"model": "gpt_5_4_mini", "reasoning_effort": "low"},
    "tiny": "gpt_4_1_nano",
    "mini": {"model": "gpt_5_4_mini", "reasoning_effort": "low"},
    "large": {"model": "gpt_5_4_mini", "reasoning_effort": "medium"},
    "reasoning": {"model": "gpt_5_2", "reasoning_effort": "medium"}
  }
}

Keep other existing model entries and the audio, Realtime and embedding catalogues. Existing provider records need the explicit Azure configuration migration; changing the initialization seed alone does not update them. Only is_default_llm_provider and is_default_embedding_provider are used to choose defaults; the old is_default_provider field can be removed. This keeps task routing provider-independent.

See the provider guide for complete Azure OpenAI deployment and OVH model-ID examples, connection variables and current connector capabilities.

OVH uses the shared https://oai.endpoints.kepler.ai.cloud.ovh.net/v1 API for both generation and embeddings. Its catalogue stores model IDs in llm_name and embedding_name. Existing URL-based provider records need the explicit OVH configuration migration alongside deployment; changing the initialization seed alone does not update them. The BGE embedding models, dimensions and vector storage fields remain unchanged.

The tenant document normally selects only providers; profile overrides are sparse:

{
  "ai_configuration": {
    "llm_provider": "openai",
    "llm_profiles": {
      "large": "sol"
    },
    "embedding_provider": "openai",
    "embedding_model": "embedding_small"
  }
}

Each profile value can be a model-catalogue alias, an object such as {"model": "sol", "reasoning_effort": "high"}, or a complete model configuration whose model is declared in that provider's catalogue. Unknown references fall back like missing references. Unspecified tenant profiles inherit the selected provider defaults. The default, tiny, mini, large and reasoning profiles are available to all connector calls and LangGraph models. Translation fallback uses tiny, while conversation summaries use mini directly.

llm_profiles.default should be present in every provider document. The other profiles are optional. A missing tiny mapping uses the provider-specific built-in minimal model; mini and large inherit default, and reasoning inherits large. Unresolved profiles use provider-specific code defaults without replacing valid sibling profiles. Provider selection is client/request selection when valid, then a valid database default, then the code default (OpenAI for LLMs, Azure for embeddings). This applies to both MongoDB and PostgreSQL, including empty provider catalogues. A provider switch discards overrides belonging to the previous provider. Embedding and audio/Realtime models follow the same client model, database default, code default ordering within their model family. Database failures still propagate; this policy handles missing configuration.

Reasoning in AIProviders¶

Store reasoning beside the model inside llm_profiles. MongoDB uses common_data.AIProviders; PostgreSQL uses the existing JSONB llm_profiles column in aihub_ai_providers. No separate reasoning column is needed. For example, this fragment intentionally overrides the code's mini=low default:

{
  "llm_profiles": {
    "mini": {"model": "luna", "reasoning_effort": "medium"}
  }
}

Merge this fragment into the provider record whose llm_models declares luna; it is not a replacement for the complete provider document. A tenant may set "mini": {"reasoning_effort": "high"} in its own ai_configuration.llm_profiles to retain the provider's model while overriding its effort.

For each field, resolution follows explicit tenant configuration, then that profile in AIProviders, then the centralized profile default. GenAI additionally accepts an explicit external request effort with highest priority. Missing or JSON null effort inherits; the string "none" explicitly requests disabled reasoning where supported. Defaults are tiny=none, mini/default=low and large/reasoning=medium. Inheriting a model from another profile does not copy its reasoning level. Existing configured levels remain authoritative.

The chosen model must accept the requested control. Azure GPT-4.1 nano and OVH's current Llama large model omit it. Native OpenAI Decisions has no effort parameter, even though its model is selected through a profile. Generation, streaming, batches and document extraction otherwise consume the profile setting. See provider limitations and GenAI request controls.

Updating code or the initialization seed does not rewrite existing provider or tenant records. Review and apply the desired profile edits to the database used by the deployment separately. Keep unrelated models, embeddings and audio settings.

Continuous voice uses llm_realtime_model, now selecting gpt-live-1 through the OpenAI Live API. Its delegated Responses backend uses the OpenAI mini semantic profile, including configured reasoning effort. The separate Realtime transcription catalogue/selector has been removed. Existing stored selections require the Live migration procedure. File transcription still uses llm_speech_to_text_model (gpt-transcribe), and simulation MP3 generation uses llm_text_to_speech_model (gpt-4o-mini-tts). See the speech model table.

Model configurations may include an optional pricing block containing USD prices per million input, output, cache-read and cache-write tokens. Provider- or gateway-reported costs take priority; configured prices and the built-in table are fallbacks when the API response contains no complete cost information.

See the embedding model table for provider defaults, dimensions and the independent Lexi selection. Changing an embedding model or dimension requires rebuilding the corresponding stored vectors and aligning the MongoDB/PostgreSQL vector indexes. It must not be treated as a transparent runtime-only switch.

embedding_profiles selects a specific embedding model for one domain, independently from default_embedding_model/embedding_model. Only the lexi profile exists today: Lexi's duplicate-detection and provided-skills matching compute embeddings on the fly and never persist them, so that profile can move to a newer embedding model without the reindexing constraint above. The initialization seed selects text-embedding-3-small for OpenAI/Azure and bge-multilingual-gemma2 for OVH. PostgreSQL loads these profiles when the profile column is created, preserving existing mappings on replay. Seed changes do not rewrite provider rows in databases whose migrations have already run. If a provider document has no embedding_profiles.lexi entry, Lexi falls back to text-embedding-3-small in code. Other embedding consumers (RAG, matching, skills library, training) are unaffected by this profile and keep using default_embedding_model/embedding_model.

Chatbot RAG configuration¶

RAG reads chatbot.rag from the active tenant configuration backend: the tenant MongoDB Config document when DATABASE_URL is absent, or the singleton PostgreSQL aihub_config row when PostgreSQL resources exist. A representative configuration is:

{
  "chatbot": {
    "rag": {
      "enabled": true,
      "chunk_size": 1200,
      "chunk_overlap": 200,
      "sync_interval_seconds": 300,
      "num_candidates": 100,
      "min_vector_score": 0.0,
      "rerank_pool_size": 40,
      "hybrid_vector_weight": 0.65,
      "hybrid_text_weight": 0.35,
      "query_cache_ttl_seconds": 90,
      "background_sync": {
        "enabled": false,
        "interval_seconds": 300
      },
      "mongo": {
        "chunks_collection": "RagChunks",
        "files_collection": "RagSourceFiles",
        "metrics_collection": "RagMetrics",
        "vector_index_name": "rag_chunks_vector_index",
        "text_index_name": "rag_chunks_text_index",
        "vector_path": "embeddings",
        "embedding_dimension": 1536
      },
      "azure_datalake": {
        "container": "ai-hub",
        "source_paths": ["knowledge/faq"],
        "allowed_extensions": [".txt", ".md", ".json", ".csv"],
        "max_files_per_path": 120,
        "max_file_size_bytes": 1500000,
        "cache_ttl_seconds": 300
      }
    }
  }
}

The RAG query cache retains at most 256 entries per application process, shared across tenants and backends. Each insertion removes expired entries, then evicts the oldest stored entries if necessary. query_cache_ttl_seconds still controls validity; synchronization also invalidates the affected tenant's entries.

The mongo block applies only to the MongoDB backend. New-client provisioning creates the RAG classic, text-search and vector-search indexes from these settings. Existing tenants and newly configured collection or index names require POST /admin/api/v1/clients/indexes reconciliation before use. Inspect its read-only plan first; tenant reconciliation creates missing indexes, updates compatible definitions, replaces incompatible indexes, and drops indexes absent from the catalog. RAG sync and reindex only process documents and never create indexes. Index-management rights are needed by the administrative provisioning/reconciliation workflows.

When PostgreSQL resources exist, RAG uses aihub_rag_source_files, aihub_rag_chunks and aihub_rag_metrics instead of RagSourceFiles, RagChunks and RagMetrics. The admin reindex endpoint calls sync_rag_documents_to_postgres, and retrieval calls the PostgreSQL hybrid vector/text query path. The background RAG sync worker remains MongoDB-only and is not started in PostgreSQL mode; PostgreSQL deployments should use the tenant-scoped admin reindex path for explicit synchronization.

Matching and standardization settings¶

The same tenant document controls:

• use_matching.jom and use_matching.jpm;
• entity standardization, translation and skill inference;
• JOM/JPM hard filters and score rules;
• pool size and matching weights;
• chatbot activation and available agents;
• report recipients and streaming flags.

The nightly_batch template field is retained for compatibility but AI Hub no longer schedules a nightly matching batch. Use the batch or processing APIs.

skill_scores.missing_target_skill_level_as_acquired controls, by matching type and processing side, whether a target skill without a level (null or 999) receives a linear level score of 1. The setting defaults to false so existing tenants keep the former behavior. To enable it only for recruiters processing users for an opportunity, use:

A required job or opportunity skill level of 0 is always treated like a missing level (null or 999) and therefore receives a linear level score of 1. A user level of 0 remains a real level and does not satisfy a positive requirement.

{
  "skill_scores": {
    "missing_target_skill_level_as_acquired": {
      "jom": {
        "user": false,
        "opportunity": true
      }
    }
  }
}

Administrative background execution¶

Long administrative operations use report_mail_addresses from the tenant configuration; client creation, index reconciliation with apply=true and tenant-independent skill operations use the configuration template. Configure Brevo or the SMTP fallback as described in outgoing email for completion delivery. Empty recipient lists prevent task acceptance. Without a Brevo API key and complete SMTP configuration, delivery is skipped with a warning without preventing task execution.

Client creation awaits resource reconciliation and sends its confirmation before HTTP 200. Initial imports remain in the background, with one existing batch email per completed entity publication (jobs, opportunities, trainings, users). All five notifications use the template recipients captured at task admission. Successful runs have no additional final summary. Empty imports are reported as no_data; partial failures retain the global partial_failure report. Exceptions and cancellation retain their terminal report. These notifications do not change the task status to success until publication finishes. They do not track consumer completion. Resource-only requests have no background import task.

These tasks are not durable across process restarts. See background operations for acceptance, reports and shutdown semantics. The admin-only GET /admin/api/v1/operations/{task_id} endpoint checks in-memory states across sibling workers in the same container. Each worker retains active tasks and its latest 1,000 completed outcomes; there is no database or disk history. The runtime needs a writable OS temporary directory for private Unix socket endpoints (aih-ops-<uid>-<parent-pid>). Workers must share that directory and OS user. Cross-container lookup is not supported. A worker restart loses its outcomes.

MongoDB indexes¶

POST /admin/api/v1/clients waits for collection/index commands and client registration, then returns HTTP 200. Only the optional initial import is scheduled in the background; the returned task_id tracks its publication. With initialisation_import=false, task_id is null. Atlas Search/Vector builds may still be running after their commands succeed. Initial imports are enabled by default (initialisation_import=true). Resource preparation uses the same reconciliation engine as /clients/indexes, creating missing resources, updating compatible definitions, replacing incompatible indexes, and dropping obsolete or duplicate indexes in catalog-managed collections on every call. It registers the tenant in ClientConfig only after all collection/index commands succeed. Existing configuration and registration values are preserved. Index operation failures are reported and prevent registration until reconciliation succeeds. Retrying requires no stored progress. Initial imports run again when enabled; use initialisation_import=false for resource-only repair. This includes classic indexes plus the Search and Vector Search indexes for matching, Talent Finder, chat history, events and RAG. When rabbitmq_prefix is omitted, provisioning stores sm_<APP_IDENTIFIER_SHORT>_legacy. This value deliberately uses APP_IDENTIFIER_SHORT directly, including mutu, and does not fall back to ENV. An explicit prefix is preserved. Existing tenants require explicit reconciliation when an index definition changes. /clients/indexes includes PwC registrations identified by ClientConfig.dataSource=pwc, reuses their provisioning catalog and preserves indexes outside that catalog. POST /admin/api/v1/clients/indexes accepts an optional db_name query parameter and defaults apply to false: the request waits for the read-only plan and returns the full JSON report with HTTP 200, without a task or email. With apply=true, it returns HTTP 202 with a task ID and emails the completed background report. Without db_name, it processes every client database named by clientDB in common_data.ClientConfig, excluding PwC tenants, plus reference_data and its managed skill collections. Each database is processed once; the results are aggregated without stopping after an individual database failure. With apply=true, it reconciles managed collections and indexes for the selected scope. Explicit db_name still targets only that database; common_data requires explicit selection. Search and Vector Search definitions are updated in place when the type is unchanged; TTL-only changes use collMod where supported. Other conflicts are replaced, and obsolete or duplicate indexes are removed. The _id_ index is kept. Type replacement waits up to 60 seconds for Atlas to release the old name before creating the new index; a shorter provisioning deadline still applies. See MongoDB indexes.

MONGO_PROVISIONING_TIMEOUT_SECONDS sets a positive, finite driver timeout in seconds (default 120) for each synchronous provisioning step, including configuration/registration access and managed index inspection or mutation. It does not limit the whole background task or the initial imports. Shorter connection timeouts still apply. Timeout errors are reported and completed resources remain available for the next call; the deadline does not guarantee cancellation of a server-side index build. Atlas Search command acceptance is asynchronous and does not wait for queryability.

Configuration ownership¶

• requirements.txt: exact dependency versions;
• src/templates/config_file.json: default tenant behavior;
• alembic/seeds/20260817_04_default_config.json: frozen PostgreSQL default configuration seed;
• alembic/seeds/20260817_04_ai_providers.json: frozen PostgreSQL provider/model seed;
• MongoDB AIProviders or PostgreSQL aihub_ai_providers: runtime provider/model definitions, selected through DATABASE_URL;
• MongoDB APIKeys: fallback authentication records used only when DATABASE_URL is absent and the relevant environment API key is absent or empty;
• environment variables: primary API keys, credentials and infrastructure endpoints;
• config/logging.yaml: log profiles, dependency levels and request context;
• zensical.toml: documentation structure and rendering.

Legacy SFTP exports¶

Statistics delivery uses AsyncSSH at the version pinned in requirements.txt (see current dependency constraints). The existing PWC_SFTP_HOSTNAME, PWC_SFTP_PORT, PWC_SFTP_USERNAME, PWC_SFTP_PASSWORD, PWC_SFTP_PRIVATE_KEY_PATH, PWC_SFTP_PASSPHRASE and corresponding BPCE host/port/username/password variables retain their roles. A PwC private key is loaded with its optional passphrase; BPCE uses the existing password path.

The existing SFTP connector explicitly adds ssh-dss to host-key and signature algorithms. Its legacy permissive host-key policy is unchanged and remains confined to that connector. MariaDB SSH configuration is unaffected. Connection establishment has a 30-second deadline; connection, retention and transfer share a 300-second deadline. Retention still precedes the upload, remote file size is confirmed, and delivery failures remain best effort. Cancellation closes the owned resources and propagates.

DSA is verified with local test servers. Actual customer server negotiation must still be checked in staging; enabling DSA alone does not enable every obsolete cipher or key-exchange algorithm.

Excel readers and exports¶

Excel imports use Pandas with the Calamine engine (python-calamine); generated XLSX exports use XlsxWriter. CSV inputs retain their existing reader. Existing upload size and decompression limits remain in place.

• Readers consume stored cell values, including the workbook's cached formula results. They do not calculate formulas; files with missing or stale caches must be recalculated and saved in a spreadsheet application before import.
• Identifier readers explicitly use dtype=str, preserving zeros in text cells. A numeric cell displayed as 000123 still has the value 123; its display mask does not become part of the identifier. Other imports retain Pandas' normal type and missing-value inference, including its default NA tokens.
• Date cells use the workbook's 1900 or 1904 epoch. Excel dates have no timezone; callers must not infer UTC from a spreadsheet date alone.
• Chat attachments extract all nonempty sheets. Other import paths still read the first sheet. Leading and interior blank rows are retained; trailing blank areas do not represent records.
• Exports create a new workbook containing tabular data. Untrusted strings remain literal, including formula-like strings and URLs. Automatic hyperlinks are disabled. These paths do not edit a template or preserve macros, styles, charts, merged cells or validation rules from an uploaded workbook.

Calamine and XlsxWriter are not drop-in APIs for arbitrary OpenPyXL use: future work that edits existing workbooks or depends on advanced Excel features needs its own compatibility review. The validated scope is the application's current value-reading and fresh tabular-export workflows.