Architecture overview¶

AI Hub is one deployable service containing a main FastAPI application and six mounted sub-applications. The separation gives each functional domain its own routes and API contract while keeping a shared lifecycle, tenant configuration and infrastructure connections.

AI Hub process
└── Main FastAPI application (/)
    ├── Main AI services
    ├── Administration (/admin)
    ├── Chatbot / Oria (/chatbot)
    ├── Data Exchange (/data_exchange)
    ├── MCP server (/mcp)
    ├── Lexi enrichment (/lexi)
    └── GenAI completions (/genai)

Documentation lifecycle¶

FastAPI mounts /docs/ immediately. Once the normal application resources have initialized, src/core/documentation_runtime.py launches an owned background subprocess and yields startup without waiting for documentation. Fingerprinting, source copying, the cache lock and the strict Zensical build all run outside the API event loop. Health checks, Swagger and business endpoints remain available during the build.

application_metadata.py owns the shared application options and mount paths; application_routes.py registers the same routers for runtime and offline OpenAPI generation. Developers refresh the HTTP Markdown and interface links through python -m scripts.update_http_reference and version them under docs/api/http/ and docs/includes/http/. The local generator imports real routers in a controlled environment without starting application resources. The commit hook checks that these sources match the current contracts. The deployed documentation builder renders them directly: it neither imports routers and their business dependencies nor copies src/ into the build project. Swagger, ReDoc and live OpenAPI remain generated by the running applications.

The site contains public functional guides and generated HTTP contracts, served from one static directory with its search index and sitemap. Access requires no API key or browser session. Business API authentication remains independent.

Only a completed site is published. While it is being prepared, GET /docs/ returns HTTP 503 with Retry-After: 5, Cache-Control: no-store and an HTML page that refreshes automatically. HEAD returns the same headers without a body. Build failures leave a generic HTTP 503 page, log the internal error and do not stop the API. There is no automatic retry until the next application start. The static mount is excluded from OpenAPI; Swagger itself does not depend on it.

The content-addressed cache is outside the repository. A filesystem lock allows workers sharing that cache to reuse one successful build for the same sources; separate pods with separate filesystems each build their own copy. Both Docker and source/playbook launches use the same versioned docs/, requirements.txt and zensical.toml sources, without a committed site/ directory or CI site artifact. API versions are read directly from application source files and rendered into docs/includes/versions/ only inside the isolated build project. The dependency list is generated from requirements.txt there as well. The site title stays unversioned; global release tags are managed only by CI. Docker and copied-source deployments require neither Git nor an extra release metadata file to build the site. Documentation, API versions, requirements and build-tool changes invalidate the cache. Business implementation changes with identical HTTP references and API versions reuse it; publishing a Git tag does not change the documentation inputs.

The runtime build has a 120-second timeout, including time waiting for the cache lock. Shutdown or timeout terminates the builder's process group, allows up to five seconds for exit, then kills remaining descendants and reaps the process. The CLI and commit hook still fail on generation errors. Background generation uses CPU and memory within the deployment's resource limits; it removes event loop blocking but does not eliminate resource pressure.

Runtime layers¶

Language detection initializes its shared broad and restricted py3langid models once, on first use. Concurrent cold calls share a lock and publish the pair only after both models are configured. Async detection performs initialization and inference in its worker thread, keeping the event loop responsive. Model-loading failures propagate to the caller; they do not trigger the optional LLM fallback. The asynchronous single-text and batch APIs share the same local detection implementation. Loaded buffers remain resident for subsequent requests, with the restricted detector sharing the broad model's immutable arrays.

SciPy and scikit-learn are loaded on first use by chatbot job vector deduplication or Lexi catalogue retrieval, rather than during route or graph registration. These operations use the shared bounded blocking pool, including the first import, so the event loop remains available. Python retains the modules after that first use; the startup memory saving is not an eviction mechanism. See blocking-work limits for pool ownership and cancellation behavior.

HTTP / SSE / WebSocket
        |
FastAPI routers and request validation
        |
Business workflows (entities, matching, Skills Library, Lexi, LangGraph)
        |
Shared AI-provider, MongoDB, PostgreSQL, RabbitMQ and Datalake connectors
        |
Tenant databases and external services

Standard authentication accepts AI_HUB_USER_API_KEY or AI_HUB_ADMIN_API_KEY, while administrator authentication accepts only the administrator key. A configured non-empty applicable key is authoritative. When no applicable key and no DATABASE_URL are configured, API keys fall back to MongoDB; the key, isAdmin and clientId are retained from that record. MongoDB routing prefers the stored identifier and falls back to Client-ID, with the fixed smartmobility data source. PostgreSQL deployments instead use the global aihub_config and aihub_ai_providers tables in their only database, while authentication requires an applicable environment key. FastAPI resolves the one active environment at startup from APP_IDENTIFIER_SHORT, with ENV as the fallback when the former is absent, empty or mutu, and shares it with every mounted application. API inputs cannot select another environment. AI calls use a shared connector API while each provider keeps ownership of its own client. OpenAI-compatible providers share request formatting, token accounting and response normalization.

Main processing paths¶

Conversational path¶

Chat requests enter the Chatbot API. The LangGraph pre-supervisor gate executes safety checks, then either returns a boundary response or routes to the supervisor and a specialist agent. Career Chat can consult internal knowledge by transferring to its Knowledge Base Deep Agent, which combines RAG and configured public pages and delivers a supported answer directly. DATABASE_URL selects the official PostgreSQL checkpointer and long-term store; without it, LangGraph keeps its MongoDB-backed persistence.

See agent orchestration, the agent catalogue, the tool catalogue and chat streaming.

Chatbot PostgreSQL persistence¶

When PostgreSQL resources are available, Chatbot-owned application data uses AIHub-prefixed PostgreSQL tables instead of MongoDB collections: aihub_conversations, aihub_chat_message_log, aihub_events, aihub_documents, aihub_rag_source_files, aihub_rag_chunks and aihub_rag_metrics. The FastAPI dependencies create database-scoped PostgreSQL repositories for chat, history, events, simulation, document tools and RAG retrieval. Legacy non-null routing columns receive one internal fixed scope; request headers and API-key routing fields do not determine it. If PostgreSQL resources are absent, the existing MongoDB collections remain the active backend.

The GET /chatbot/v1/configs and PUT /chatbot/v1/configs endpoints follow the same backend selection. PostgreSQL deployments read and update the singleton aihub_config row, including the public chatbot frontend settings; MongoDB deployments continue to use the tenant Config document. RAG reindex uses sync_rag_documents_to_postgres in PostgreSQL mode and sync_rag_documents_to_mongodb in MongoDB mode.

Import and matching path¶

Imported users, jobs, opportunities and trainings are standardized before they are persisted. Matching can then be launched by consumers, processing endpoints or bounded batch endpoints. Matching batches stream MongoDB cursors through async workers. Import batches process materialized SQL/source records through the same bounded worker engine while sharing MongoDB, AI and Datalake clients. For RabbitMQ user messages, the stored user timestamp represents the latest semantic profile change: timestamp-only consumer refreshes retain the previous value. Batch imports always write their newly generated timestamp. JOM and JPM check their last successful, nonempty complete user calculation independently. A changed user, a newer opportunity (JOM) or job (JPM), or missing scores causes a complete calculation of that product on the next user message. Individual pair-score dates do not establish freshness. Empty results and failed writes remain eligible for recalculation on every subsequent message. Forced/manual processing and only_standardize retain their existing behavior.

With DATABASE_URL, JOM and JPM use the live PostgreSQL user, opportunity, job and skill projections. Scores are stored in aihub_user_opportunity_matching, aihub_opportunity_user_matching and user_job_matching; JPM writes directly to PostgreSQL instead of publishing score updates through RabbitMQ. PostgreSQL triggers enqueue only user-side recalculation jobs in aihub_matching_jobs when matching-relevant user data or last_login changes and the corresponding product is enabled in aihub_config. Opportunities and jobs do not retrigger matching automatically, and PostgreSQL does not create initial missing-score jobs. These jobs are limited to enabled users owned by LIVE or DEMO tenants. Tenant deactivation clears their pending work; tenant activation does not backfill matching jobs, so the next relevant user change (including last_login) remains the recalculation entry point. At execution, each queued product checks its freshness against the current user and the entire tenant catalogue. Source-change triggers maintain dates without creating additional matching jobs; related skills, Smart Editor values and attached-job changes participate in this check. See matching freshness for storage, empty-result and rollout details.

See the matching endpoint catalogue, the detailed JOM and JPM algorithms, and batch processing.

Search path¶

MongoDB opportunity and training hybrid searches execute Atlas full-text and vector queries concurrently. Their ranked result lists are fused locally with weighted reciprocal-rank fusion (RRF); MongoDB $rankFusion is not used. Atlas stage construction is centralized in atlas_full_text_search.py and atlas_vector_search.py under src/common/connectors/mongodb, while atlas_hybrid_search.py coordinates the independent queries and search_fusion.py owns the database-independent RRF primitives. Opportunity search uses the same hard-filter configuration as matching and can recalculate a user's JOM scores once when they are genuinely absent.

Single-skill and Experience-to-skills searches follow the MongoDB path when DATABASE_URL is absent. If PostgreSQL is configured, they instead query a project-owned relational view that joins each skill_translation to its skill, repository, tenant and skill group. Text and HNSW cosine searches run concurrently and use the shared skill RRF implementation. The view and its underlying indexes are managed by Alembic; PostgreSQL reflects source-table updates without an application synchronization job. The reusable SQL and ranking repository is owned by src/entity/skill, while the Skills Library and Talent Intelligence layers select the backend and adapt results to their contracts. PostgreSQL Experience searches also resolve existing-skill labels and requested translations from the relational view and do not inspect the MongoDB collection input.

The relational skill view and embedding triggers share one eligibility rule: the owning tenant must be LIVE or DEMO, and the skill must reference an existing repository. Row-level title and repository changes maintain the queue, and tenant deactivation removes pending jobs that are no longer actionable. Tenant activation does not scan existing skills; administrators use the explicit PostgreSQL initialization workflow for that backfill.

Relational user projection¶

With DATABASE_URL, user reads use the live aihub_user_v1 PostgreSQL view. The view joins the existing user, job, organization, experience, activity and skill tables, so changes to source rows are visible without an import or a materialized-view refresh. Search indexes are placed on the physical source tables, and the repository that exposes the document-compatible projection is owned by src/entity/user.

User visibility fields have separate ownership. accessGroup is reserved for PwC opportunity eligibility and is empty in the Smart Mobility PostgreSQL projection. trainingAccessGroup contains the user's Smart Editor training groups (flag_id = 3) and is compared only with the same field on trainings. MongoDB Smart Mobility imports populate trainingAccessGroup; PwC imports populate accessGroup and leave trainingAccessGroup empty.

Training history follows the source Smart Editor attendance model. For the user's tenant, entities with flag_id = 4 link formations to users through the text user ID stored in company_entity_item_formation.name. The view deduplicates formation links and exposes them as trainingHistory; a missing business completion date is returned as null. MongoDB-mode Smart Mobility imports use the same relation and document shape.

AIHub-owned user fields are deliberately separated from the source model: aihub_user_profile stores administrator status, the application user ID and the simulation acknowledgement date. Experience-based inferred skills are stored as user/skill links in aihub_user_inferred_skills; source text and vector scores are not persisted. PostgreSQL experience triggers enqueue an inference job only for insert/delete or an actual change to the experience user, title or description. The view is limited to LIVE and DEMO tenants, while inference jobs additionally require an enabled user.

Explicit initialization and user maintenance create durable parent runs in aihub_user_reconciliation_jobs; tenant deactivation uses the same mechanism for cleanup. Tenant activation and Alembic do not enumerate existing users. The PostgreSQL listener claims one keyset-paginated user page per time slice, creates missing AIHub profiles, refreshes skill gaps with a set-based query and links individual inference jobs to the parent run. Progress, child successes/failures and a heartbeat remain queryable after a restart; expired processing leases are reclaimed. Notifications use an empty payload, avoiding PostgreSQL's payload-size limit for large tenants.

Administrators start a manual run with GET /admin/api/v1/entity-maintenance/users/infer-skills and receive a jobId. In PostgreSQL mode its progress is available from GET /admin/api/v1/entity-maintenance/users/infer-skills/jobs/{job_id}. Tenant deactivation uses the same paginated path to remove pending inference work and derived skill-gap rows.

aihub_user_skill_gap contains positive gaps between the skills required by the user's current job and declared user-skill levels. PostgreSQL refreshes those rows when the user changes job or activation state, a relevant userskill changes, or the associated job's jobskill rows change. These operations are database-maintained and limited to enabled users from LIVE or DEMO tenants. The listener processes both reconciliation pages and the bounded asynchronous inferred-skill batches, yielding between batches so other entity queues remain serviceable.

Relational job projection¶

With DATABASE_URL, job API reads, chatbot job mapping and JPM text-vector comparisons use the live aihub_job_v1 PostgreSQL view. The view reconstructs the existing job document from job, JobStatus, jobskill, skill_search_v1 and the organization entities that define the job family. The physical job.embeddings column remains the only writable AIHub field.

PostgreSQL enqueues one deduplicated job_embedding task after an insert or an actual change to the title, mission, formation, experience, results, note or effective job family. Family changes are propagated from association, family item name and family-entity classification updates. Skill associations and levels, classification, visibility, activation, hidden state and the embedding column do not enqueue work because they are not part of the generated text. The worker embeds exactly title + " - " + jobFamily + ": " + description and writes the vector atomically. Both the view and queue require a LIVE or DEMO tenant; embedding work additionally requires an active job with a visible status. Job activation and status transitions add missing work or remove pending work according to that eligibility. Tenant deactivation removes pending work, while tenant activation requires explicit initialization for pre-existing jobs.

Alembic manages tenant/status, status/tenant and tenant/code B-tree indexes, both directions needed to join and propagate family changes, and a partial HNSW cosine index on non-null job vectors. The reverse status index supports visibility-transition cleanup without scanning every job. The view is not materialized, so it has no indexes of its own. Job standardization remains MongoDB-only and rejects execution in PostgreSQL mode; an unset DATABASE_URL preserves the existing MongoDB API, chatbot and JPM vector paths.

Relational opportunity projection¶

With DATABASE_URL, the main opportunity query and active-count endpoints read the live aihub_opportunity_v1 PostgreSQL view. The view reconstructs the Smart Mobility opportunity document from custom_job, its deterministic English-first translations, attached jobs, organization entities and public job skills. Skill titles and types reuse skill_search_v1, following the same localization rules as the relational user projection.

AIHub enrichment remains on the physical custom_job row. PostgreSQL triggers enqueue one deduplicated opportunity_embedding job when an active opportunity's effective title, description, attached-job skills or public skill titles can change. The trigger column lists deliberately exclude skill levels, location, validity, deletion state, organization fields and AIHub-generated columns. The shared PostgreSQL listener claims the job, infers the title, reconstructs the posting and atomically stores all three vectors plus the enrichment timestamp. Alembic also manages the join, normalized text, trigram, full-text and HNSW cosine indexes on the physical source tables.

The view and embedding queue include only LIVE and DEMO tenants. Queue eligibility also requires a non-deleted opportunity whose validity has not expired. Opportunity eligibility transitions enqueue only missing vectors or remove pending work that can no longer be processed. Tenant deactivation removes pending work; tenant activation does not scan existing opportunities.

The view deliberately excludes fields that exist only in the PwC MongoDB contract. Opportunity standardization is a MongoDB-only workflow and rejects execution when DATABASE_URL selects PostgreSQL. PwC and Chatbot Atlas opportunity search continue to use MongoDB; an unset DATABASE_URL preserves all existing MongoDB query and standardization paths.

Relational training projection¶

With DATABASE_URL, Learning Finder, Training Matching, rationale generation and development-objective training grounding use the live aihub_training_v1 PostgreSQL view instead of reading the MongoDB Trainings collection. The view reconstructs the existing training document shape from formation, organisme, formations_skills, skill_search_v1 and training access-group entities.

AIHub enrichment fields live on the physical formation row: aihub_enriched_title, aihub_enriched_title_embeddings and aihub_timestamp. PostgreSQL enqueues one deduplicated training_enrichment job after insert or an actual update of formation.name or formation.description; skills, organization, dates, duration, locale and AIHub-generated columns do not trigger regeneration. The shared PostgreSQL listener generates the enriched title from only the title and description, then stores the enriched title vector atomically.

The view is restricted to LIVE and DEMO tenants. Enrichment work uses the same tenant rule and additionally requires a non-empty title plus an active date window. Training eligibility transitions keep pending jobs aligned without regenerating vectors that already exist. Tenant deactivation removes pending work, while pre-existing trainings are backfilled only through explicit initialization.

Explicit PostgreSQL initialization¶

Alembic creates tables, columns, views, indexes, configuration seeds, trigger functions and the durable job queues. It does not scan existing business rows or populate embedding, enrichment or inference queues. This keeps a fresh migration bounded while row-level triggers make later imports self-initializing.

Administrators initialize existing data through PostgreSQL-only routes under /admin/api/v1/postgres-initialization. Separate skills, users, opportunities, jobs and trainings routes target one entity family; all runs them in that order. The optional client_id is a PostgreSQL setting.id. When omitted, the run targets every LIVE or DEMO setting.

aihub_initialization_jobs and aihub_initialization_steps persist the global sequence. Each step creates or joins a paginated reconciliation parent, which enumerates eligible rows and inserts only missing child jobs. Initialization is complete as soon as all child job rows have been deposited. It never waits for the embedding or inference workers to execute those jobs, even when processing the resulting backlog takes several days. Status is available from GET /admin/api/v1/postgres-initialization/runs/{job_id}.

The PostgreSQL listener and all of its workers start only when the database's Alembic revision matches the repository head. The administrator migration route stops the local runtime before upgrading and starts it again only after a successful migration, preventing workers from querying objects that have not been created yet. Runtime shutdown bounds notification-connection closure, worker termination and lease cleanup. If best-effort cleanup exceeds its deadline, the durable job remains protected by its lease and is reclaimed normally after expiry instead of blocking application termination indefinitely.

Alembic manages the tenant/locale/date filter index, training-skill and access-group join indexes, normalized-title trigram and full-text indexes, and a partial HNSW cosine index on non-null enriched-title vectors. Training standardization remains MongoDB-only and rejects PostgreSQL mode, so relational deployments use only the database-triggered enrichment job.

See MongoDB indexes.

Technology versions¶

Development, CI and production run Python 3.14. Source syntax and Ruff are therefore aligned with Python 3.14. The following dependency constraints are generated directly from requirements.txt at documentation build time. They describe the declared runtime, documentation, test and development dependencies, not a snapshot of transitive packages installed in a particular environment. Exact pins and allowed ranges remain unchanged.

aio-pika==10.2.2
aiohttp==3.14.5
aiosmtplib==5.1.3
anthropic==2.6.0
apscheduler==3.11.3
asyncmy==0.2.17
asyncssh==2.24.1
# Support encrypted OpenSSH private keys used by SFTP.
bcrypt==5.0.0
brevo-python==5.1.0
certifi>=2025.8.3
# Preserve hardened XML parsing independently of development-only dependencies.
defusedxml==0.7.1
azure-storage-blob==12.31.1
pyyaml==6.0.3
fastapi==0.162.0
greenlet==3.5.6
# Brevo's SDK requires the original httpx distribution alongside httpx2.
httpx==0.28.1
httpx2==2.16.0
gunicorn==26.2.0
jinja2==3.1.6
py3langid==0.4.0
loguru==0.7.3
lxml==6.1.3
numpy==2.5.4
openai==3.46.0
python-calamine==0.9.1
xlsxwriter==3.3.1
pandas==3.1.1
pydantic==2.14.3
pydantic-settings==2.16.2
pyjwt[crypto]==2.15.1
pymongo==4.19.0
psycopg[binary,pool]==3.3.8
alembic==1.20.1
pgvector==0.5.1
python-docx==1.2.0
python-pptx==1.0.2
python-dotenv==1.2.4
python-multipart==0.0.33
pillow==13.0.0
pypdf==7.0.0
pypdfium2==5.15.0
rapidfuzz==3.14.6
scikit-learn==1.9.1
sqlalchemy==2.1.8
# Deep Agents' Google SDK dependency currently requires tenacity <9.2.
tenacity==9.1.4
tiktoken==0.14.0
anyascii==0.3.3
uvicorn==0.54.0
uvicorn-worker==0.4.0
sse-starlette==3.6.0
prometheus-fastapi-instrumentator==8.1.0
rich==15.0.0
markdown==3.11.3
beautifulsoup4==4.15.0
python-snappy==0.7.3

# langchain
deepagents==0.7.34
langchain==1.4.9
langchain-anthropic==1.7.10
langchain-core==1.7.6
langchain-openai==1.7.4
langsmith==0.14.17

# langgraph
langgraph==1.2.20
langgraph-checkpoint-mongodb==0.5.1
langgraph-checkpoint-postgres==3.2.0
langgraph-checkpoint-sqlite==3.1.3
langgraph-store-mongodb==0.4.0

#mcp
fastmcp==4.1.6

# Documentation dependencies (also required at runtime)
zensical==0.1.0
pymdown-extensions==12.3

# Docker installs only the requirements above the following marker.
# Keep the marker unchanged and all runtime dependencies above it.
# Test dependencies
pytest==9.2.0
pytest-asyncio==1.4.1.post1
pytest-cov==7.1.0
pytest-xdist==3.8.0
execnet==2.1.2
mongomock>=4.3.0

# Dev dependencies
pre_commit==4.7.0
# Pin pre-commit's environment builder past CVE-2026-113725.
virtualenv==21.15.6
mypy==2.4.1
pyright==1.1.414
ruff==0.17.4
pylint==4.1.3
bandit[toml]==1.9.4
pip-audit==2.10.1
pandas-stubs==3.1.0.261029
types-PyYAML==6.0.12.20261103
types-defusedxml==0.7.0.20260504

Deployment¶

The repository contains the Docker image definition and GitLab CI configuration. Deployment-specific values, credentials and tenant configuration must be provided by the target environment; they are not stored in this documentation.

Blocking operations¶

Lexi referential analysis, GenAI PDF/DOCX parsing, resume validation and data URL preparation, local language detection, JSON loading, upload archive validation and SMTP TLS context creation share the bounded blocking pool. Native async provider calls and the cancellable CV extraction subprocess remain unchanged. The pool admits at most BLOCKING_WORKERS + BLOCKING_MAX_PENDING submitted jobs (four workers and four queued jobs by default). Cancellation keeps a running job's reservation until its worker actually finishes; shutdown drains all jobs.

Direct default-executor calls are limited to operations that must be independent of that pool: HTTP quota checks, administrative status directory preparation and peer enumeration, one documentation publication per lifespan, and joining the blocking executor itself. The event loop owns and drains the default executor. Its thread count is bounded, but its submission queue is not; quota and status requests do not acquire the business pool's admission slots. This separation preserves control access under business-pool saturation, not a global request backlog bound. The standalone AI evaluation CLI retains its separately owned metadata, checkpoint and accounting executor work.

Chat uploads use parser-provided size metadata or a bounded native async read when metadata is absent. File-payload and frontend-image Base64 processing run in the bounded pool, and missing frontend configuration templates use the shared async JSON loader. Cached JWT verification remains synchronous: a local synthetic RSA-2048 benchmark found its per-token cost lower than a worker round trip.

Embedding tokenization, input reduction and request batching use the shared bounded blocking pool, including lazy tokenizer loading. Provider calls remain native async operations with unchanged token budgets, model selection and order. Operational batch and export scripts own a blocking-work scope for their runtime. Lexi duplicate grouping, MongoDB/PostgreSQL skill-result deduplication and shared matching skill scoring also execute their matrix computations in that pool. Database sessions remain on the event loop; workers only receive materialized values and preserve the existing similarity thresholds and result ordering.

Prompt Engine row validation, grouped prompt preparation and result assembly, and Lexi skill information, fit checks and level-table transformations use the bounded pool. Result frames are worker-owned; model calls retain their original ordering, profiles and concurrency.

Matching spreadsheet exports and Lexi spreadsheet processing run outside the event-loop thread. MongoDB skill-inference maintenance uses native async I/O with at most eight concurrent entities. SFTP transfers use AsyncSSH; local key and upload-stream reads use the bounded blocking pool. Opportunity status synchronization also uses native async MongoDB reads and writes. Only CSV/JSON parsing and identifier validation use the blocking pool, with worker-owned bytes and streams that remain valid after request cancellation. Their callers still await completion: response schemas, spreadsheet safety checks, transfer ordering and existing error handling are preserved. This keeps other requests responsive while a synchronous library is working; it does not make Python CPU work parallel when the interpreter uses the GIL.