Testing guide¶

Pandas compatibility checks¶

Dependency upgrades must preserve the existing dataframe and Excel contracts under warnings-as-errors. The regression suites exercise sheet-by-sheet Calamine reads, leading zeros, mixed numeric/empty export values, nullable SQL imports, objective timeframe columns, person disambiguation and experience aggregation. Lexi inputs with no supported generation columns must yield no source rows. Use returned frames for renaming and dropping columns, pd.read_excel with the open workbook for sheet reads, and explicit object columns when mixing numbers with empty strings. These changes preserve source values rather than stringifying numeric data or ignoring deprecation warnings.

The test suite uses pytest, pytest-asyncio and pytest-xdist. Tests should run without live MongoDB, MariaDB, RabbitMQ, Datalake or AI-provider access unless they are explicitly marked as integration tests.

Warnings are test failures by default. This makes runtime deprecations blocking, while Pyright checks src, tests and scripts in basic mode and its reportDeprecated rule catches deprecated APIs and typing forms statically before tests run.

Running tests¶

# Complete local suite: four workers, with 100% line and branch coverage
bash scripts/run_tests.sh

# Adjust concurrency or debug sequentially (0 and 1 both disable workers)
AIHUB_TEST_WORKERS=2 bash scripts/run_tests.sh
AIHUB_TEST_WORKERS=1 bash scripts/run_tests.sh

# Focused areas
pytest tests/common/test_batch_processing.py
pytest tests/admin/
pytest tests/matching/
pytest tests/chatbot/

# Local HTML coverage
pytest --cov=src --cov-report=html
open htmlcov/index.html

The runner clears Git's repository-local environment variables before pytest. This isolates tests that create disposable repositories from the parent commit hook, especially in linked worktrees where Git exports an absolute GIT_DIR. All ordinary hook checks still run; the isolation does not skip validation.

The runner defaults to four isolated processes with --dist worksteal, which redistributes pending tests when a worker finishes early. Coverage from all workers is combined before enforcing the unchanged 100% threshold. Plain pytest remains sequential unless -n is supplied explicitly. Worker counts must be nonnegative integers; automatic CPU-based fan-out is not the default because each worker imports the application and owns its fixtures in memory. Parametrization must use deterministic sequences, never unordered sets.

When ALEMBIC_TEST_DATABASE_URL is set, the runner uses sequential execution even if multiple workers were requested. Migration smoke tests reset shared database objects and must not run concurrently against the same database. Keep their dedicated CI invocation sequential as well.

Protocol tests for evaluation campaigns explicitly opt into stable_checkout_sources to reuse real source discovery and parsed imports within each worker. They still recompute validation, dependency closures and evidence for every invocation. This fixture is only for tests that keep checkout sources unchanged; source-mutation and invalidation tests use uncached reads. Version tests copy a worker-local initial Git repository, including its complete .git directory, before each case so refs, indexes and commits remain isolated.

The complete suite is enforced by the local pre-commit hook for development commits and ordinary commits on release branches. Actual promotion merges into release/* skip this repeated suite; other local hooks remain active and GitLab runs downstream quality checks. The commit policy requires a pending merge. For detached promotion worktrees, command-scoped AIHUB_RELEASE_TARGET names the release destination and must match the fetched destination tip; it cannot exempt a commit on develop. Manual bash scripts/run_tests.sh always runs the suite. Its 100% line and branch threshold must not be lowered or met by excluding application code. GitLab CI runs the disposable PostgreSQL migration smoke suite only on release/int_pg. Its mandatory quality job runs pip check, pip-audit and all static analyses against one fully resolved Python 3.14 environment. Ruff, Mypy, Pyright, Pylint and Bandit also cover the operational Python modules under scripts/; a known package vulnerability or an incomplete audit fails the pipeline. Python CI jobs share a pip download cache keyed by requirements.txt, while each ephemeral container still installs the exact pinned environment.

PostgreSQL migration smoke test¶

Alembic migrations are validated in CI against an ephemeral PostgreSQL service with pgvector installed. The smoke test bootstraps the current relational business-table baseline, runs alembic upgrade head, and then verifies the embedding queue, triggers, skill_search_v1, aihub_user_v1 and aihub_job_v1, aihub_opportunity_v1 and aihub_training_v1 views. It checks their projected data, targeted job, opportunity and training triggers and relational, full-text, trigram and HNSW indexes. It then initializes the official LangGraph PostgreSQL backends, verifies their framework-managed tables and performs a real checkpoint and long-term-memory round-trip. The Job checks verify that title/description-component and effective-family changes enqueue work, while skills, classification, hidden/active/status fields and stored vectors do not. The Training checks verify that only formation.name and formation.description enqueue training_enrichment; skills, organization, dates, duration, locale and AIHub-generated fields do not. The eligibility checks verify that only LIVE and DEMO clients appear in the skill, user, opportunity, job and training views. They verify that Alembic creates no initial entity jobs, exercise the explicit global and client-scoped initialization workflow, and confirm that initialization finishes after child jobs are queued rather than processed. The smoke test also clears the Alembic version and replays the complete chain over existing objects. Tenant activation does not backfill existing entities; deactivation, entity-specific eligibility and skill-repository transitions keep pending work aligned. Matching checks confirm that deactivation clears pending JOM/JPM jobs while activation does not backfill them; a later last_login change remains the trigger. Retention checks verify all seven partial indexes, bounded deletion of terminal rows older than three months, preservation of recent and active jobs, and initialization-step cascade behavior. The idempotent replay deliberately interrupts a concurrent retention-index validation and verifies that the regular bounded migration repairs the resulting ready-but-invalid index.

Run it locally against a disposable PostgreSQL database with:

ALEMBIC_TEST_DATABASE_URL='postgresql://postgres:postgres@localhost:5432/aihub_test'
PG_SCHEMA='mysql'
PG_SKILL_SEARCH_EMBEDDING_DIMENSION='3'
PG_JOB_SEARCH_EMBEDDING_DIMENSION='3'
PG_OPPORTUNITY_SEARCH_EMBEDDING_DIMENSION='3'
PG_TRAINING_SEARCH_EMBEDDING_DIMENSION='3'
pytest --confcutdir=tests/common/connectors/postgres \
  tests/common/connectors/postgres/test_alembic_migration_smoke.py

The isolated command does not load the global FastAPI test fixtures, so it does not require APP_IDENTIFIER_SHORT or ENV. The test skips automatically when ALEMBIC_TEST_DATABASE_URL is unset. Do not point it at a shared environment; it drops and recreates the target schema as part of the bootstrap.

Test structure¶

tests/
├── auth/                    # API-key, JWT and authorization behavior
├── admin/                   # admin services and operational controls
├── chatbot/                 # APIs, RAG and agentic orchestration
│   └── agentic/            # graphs, agents, prompts, tools and middlewares
├── common/                  # providers, MongoDB, batches and shared utilities
├── data_exchange/           # users, opportunities and feedback
├── entity/                  # user, job, opportunity, training and skill domains
├── lexi/                    # Lexi endpoint and workflow contracts
├── matching/                # JOM, JPM and matching boundary contracts
├── skills_library/          # skill search and embedding administration
├── talent_intelligence/     # focused generation and inference workflows
└── conftest.py              # shared fixtures and environment isolation

Entity-level suites live under tests/entity/{user,job,opportunity,training,skill} alongside the matching and administration suites.

Async example¶

import pytest

from src.common.batch_processing import run_bounded_batch


@pytest.mark.asyncio
async def test_materialized_rows_are_processed():
    processed = []

    async def process_entity(entity):
        processed.append(entity["id"])
        return entity

    result = await run_bounded_batch(
        [{"id": "row-1"}, {"id": "row-2"}],
        process_entity,
        lambda entity: entity["id"],
        requested_concurrency=2,
        entity_label="row",
        total_items=2,
    )

    assert processed == ["row-1", "row-2"]
    assert result.number_of_success == 2
    assert result.fail_ids == []

Mocking guidelines¶

• use AsyncMock for awaited MongoDB/provider/storage calls;
• patch the symbol where the module under test imports it, not an obsolete generic database module;
• use async context-manager fakes for shared batch or application contexts;
• assert both successful results and failure IDs/status codes;
• verify shared resources such as datalake_writer are propagated when the behavior depends on connection reuse;
• reset environment variables with monkeypatch rather than relying on a local .env file.

Quality gates¶

ruff check .
ruff format --check .
pytest tests/<affected-area>/

The mandatory commit hook runs Mypy, Pyright, Pylint, Bandit and the complete 100%-coverage suite once. For relevant documentation changes it also creates or reuses one strict, content-addressed Zensical build outside the worktree. Application startup reuses that same cache. Use pre-commit run --all-files only for explicit repository-wide verification or quality-tool configuration changes.

Use names following test_<behavior>_<condition>_<expected_result> and add a regression test for every corrected bug or changed public contract.

Dependency migration regression checks¶

tests/common/connectors/test_sftp.py runs real, temporary localhost SFTP servers with DSA host keys and both password and DSA private-key authentication. It also exercises cancellation, transfer deadlines, retention, resource cleanup and remote-size confirmation. It never connects to customer servers.

tests/common/security/test_excel_compatibility.py covers Calamine/XlsxWriter round trips, both Excel date epochs, cached formula values, booleans, numeric values, leading-zero text identifiers, interior and leading empty rows, multiple sheets and the existing CSV reader. Export-safety tests inspect the actual XLSX XML to verify that untrusted strings never become formulas or automatic links.

Language threshold calibration¶

tests/common/fixtures/language_detection.json contains 113 synthetic cases: the eight frontend language roots, selected other supported languages, short skill labels, mixed technical vocabulary and non-language inputs. Each case records a human target (language), the result of the previous Lingua code (lingua) and the frozen py3langid outcome (detected). Lingua is not required to run these tests. No LLM call is made.

The same corpus checks that detectors sharing one loaded model produce exactly the same complete rankings and probabilities as independently loaded models. Concurrent detection also retains every calibrated outcome, and buffer-identity checks prevent accidental duplication of the full model and tokenizer tables.

Calibration used 55 cases; the remaining 58 cases were reserved for validation. A grid of 24,300 threshold combinations minimized incorrect non-null predictions, then maximized correct results (including correctly undetermined non-language inputs). Ties preserved the initial threshold proposal. The grid was:

These figures describe this small synthetic corpus, not production accuracy. Projektmanagement, Datenanalyse, Softwareentwicklung and Análisis de datos lose a local decision compared with Lingua; their top-ranked model labels differ, so lowering acceptance thresholds cannot recover the expected language safely. They remain explicit regression cases returning None, preserving the existing caller fallback. The fixture is also an offline gate for future model upgrades; expand it with representative, sanitized examples before recalibrating again.

Live AI quality evaluations¶

Versioned model-behavior cases, bounded local campaigns and the weekly GitLab campaign are described in AI quality evaluations. They are separate from the offline unit suite and never run as part of ordinary pytest or Git commit hooks. Manual and weekly campaigns default to one observation per scenario; use --repetitions N only for explicitly requested repeated experiments, with unchanged spending ceilings. The judge is enabled by default. An explicit --no-judge on plan, run or collect filters out scenarios with requires_judge: true before budgeting or execution. Reports distinguish these exclusions from failures and successes. An agent may choose the mode for authorized, focused development checks and must report its scope; this does not authorize additional spending.

During implementation, review evaluation impact for added, changed and removed AI features, including indirect changes to prompts and embedding consumers. The feature plan describes scenario inputs, expected behavior, boundary cases and variants; human approval of that plan approves the scenarios for execution. Implement them with the code and revise the plan for material expectation changes. Observed results still require separate approval before becoming a reference. The evaluation skill maintains cases, inventory, suite selection and adapter contracts; Python validation and commit preparation check this handoff. Preserve expectations for model migrations, remove resources dedicated to deleted features, and record a reason for no impact. This offline maintenance does not authorize paid campaigns or reference approval.

Run python -m scripts.ai_evals coverage to check the workflow inventory and nominal/boundary examples without provider calls. tests/unit/ai_evals/ checks selection, grading, metrics and configuration. tests/integration/ai_evals/ verifies adapters, graphs, transports and campaign/report workflows with simulated model replies from tests/fixtures/ai_evals/; shared helpers live under tests/support/ai_evals/. Both suites forbid live sockets. Run both with pytest tests/unit/ai_evals tests/integration/ai_evals --no-cov, with tracing disabled as required above. These technical tests remain part of the complete local test gate and do not establish model quality. Real evaluation scenarios and business context remain under evals/, with no simulated model replies. Use plan --campaign weekly to inspect the complete corpus requirements. The supplied configurations enable rubric judging, and policy call ceilings include headroom above the current full-corpus branch estimate; the meter still enforces token, USD and time limits.

For human review, run python -m scripts.ai_evals export-review --output ai-eval-artifacts/corpus-review.xlsx. Add --report <report.json> to include actual observations alongside expected behavior. This offline export preserves all case fields, rejects stale corpus/report combinations and never approves a reference or imports Excel annotations. See the Excel review workflow and the step-by-step review and approval guide, which explains when to accept, request changes or leave evidence inconclusive.

The embedding suite adds labelled retrieval and duplicate-pair examples, native dimension validation, input-token accounting and advisory threshold calibration. Run plan --suite embeddings before a paid campaign. Offline vector fixtures verify the harness; live model quality and database-index behavior require their own evidence.