Generate skills¶

POST /lexi/api/v1/jobs/generate_skills

This endpoint generates a job-to-skill matrix. It starts from a list of jobs supplied by Lexi and returns, through a callback, a list of skills for each job with their groups, definitions and optional levels.

Purpose¶

The goal is to create a catalog that Lexi can use:

• each job receives a defined number of skills;
• skills are generated in the requested language;
• skills are classified into the groups supplied as input;
• groups follow the implicit type distribution, such as Hard / Soft, as closely as possible;
• overly similar skills are avoided or merged;
• irrelevant skills are removed before the final result.

Request from Lexi¶

Lexi mainly sends:

In searchQueries, each job generally contains:

• a job identifier;
• a title;
• a description;
• a job family.

Immediate response¶

Generation can take time, so the endpoint immediately confirms that processing has started:

{
  "message": "The file is processing, Lexi will be notified once it is finished."
}

The complete result is sent to Lexi later through a success callback. An error callback is sent on failure.

Main stages¶

1. Prepare the jobs¶

AI Hub converts the received list into a working table. It checks that the main fields are present, adds an empty family when one is missing and removes unnecessary whitespace from text.

This stage does not create any skills yet. It only prepares clean input for the LLM calls.

2. Prepare the generation rules¶

AI Hub prepares the rules the LLM must follow:

• output language;
• number of skills per job;
• allowed groups;
• expected distribution between implicit types;
• client context;
• optional skill-label syntax rule.

When groups are supplied, they take priority. Types remain implicit: the final result exposes the group, not the type.

3. Connect groups to implicit types¶

Each group is associated with exactly one configured implicit type. Compatible OpenAI mini uses independent categorical decisions; other provider/model configurations keep the structured mini call and existing heuristic error behavior. A single allowed type requires no model decision on the native path.

Example:

This stage only guides the distribution. For example, if the input asks for 70% Hard skills and TECHNIQUE maps to Hard, generation should favor skills in the TECHNIQUE group.

4. Generate a global skill catalog¶

The first major LLM call receives all jobs at once. It does not yet generate skills job by job; it first creates a global catalog of possible skills for the entire scope.

This approach prevents every job from inventing its own variants and creates a more consistent shared base.

In particular, the LLM must:

• create skills in the correct language;
• avoid duplicates and overly similar wording;
• avoid treating certifications or diplomas as skills;
• keep labels short and readable;
• use the allowed groups;
• follow the type-induced distribution as closely as possible.

5. Complete the catalog when needed¶

After the first catalog, AI Hub checks whether enough skills were generated for the number of jobs and the requested number of skills.

If the catalog is too small, a second LLM call receives:

• the jobs;
• the catalog already created;
• the expected number of additional skills.

This second call must not recreate the catalog. It only adds new, useful and non-redundant skills.

6. Assign skills to jobs¶

Once the global catalog is ready, another LLM call assigns its skills to jobs.

At this stage, the model may no longer invent new labels. It must select only from the existing catalog.

This separation matters:

• the catalog ensures global consistency;
• assignment ensures relevance to each job.

For very large volumes, this stage can use the batch API. Otherwise, it uses a standard streaming call.

7. Check the assigned skills¶

A global quality check receives all jobs and their already assigned skills. Compatible OpenAI mini makes one keep/flag decision per indexed job-skill pair with the same global context. Other providers/models keep the existing large structured check. Invalid responses retain skills under the existing error contract; no automatic provider retry is added.

Its role is not to rewrite everything. It only identifies skills that are clearly irrelevant to a given job.

For example:

• a moderately generic skill can remain;
• a skill with no direct or indirect connection to the job is removed;
• a certification or accreditation is removed if it appears as a skill.

This stage acts as a second check before enrichment.

8. Generate definitions and levels concurrently¶

After the quality check, two processes can run concurrently:

• generation of skill definitions;
• generation of expected levels.

Definitions are generated once per unique skill, then reused wherever the skill appears.

Levels are generated per job on the scale requested as input. For example, with skillLevelScale = 4, every skill receives a level between 1 and 4. The job description is included in the level prompt when available. Skill definitions are not available in this flow because both processes run concurrently.

Compatible OpenAI mini now selects each required level as a categorical choice, with stable job/skill indices and sequential chunks of up to 400 questions. Unsupported configurations retain large structured level generation. This changes only the level sub-step; catalogue generation/assignment and the provided-catalogue Anthropic workflow retain their existing providers. See Generate skill levels for context and failure behavior.

9. Finalize the result¶

AI Hub prepares the format expected by Lexi:

• remove duplicate job/skill pairs;
• generate a unique skill code in SK00001 format;
• add the job code and job family;
• add the group, definition and level;
• construct the final JSON.

Response to Lexi¶

The success callback contains a list of jobs, each with its skills.

Simplified example:

{
  "organizationId": "...",
  "userEmail": "...",
  "referentialHRX": "...",
  "generationMode": "...",
  "data": [
    {
      "job_code": "JOB001",
      "job_title": "Digital project manager",
      "job_family": "Project management",
      "skills": [
        {
          "skill_code": "SK00001",
          "skill_title": "Project management",
          "skill_group": "MANAGEMENT",
          "required_skill_level": 3,
          "skill_definition": "Ability to ..."
        }
      ]
    }
  ]
}

Points to note¶

• The result is not returned in the immediate HTTP response.
• Quality depends heavily on the supplied job descriptions.
• The groups supplied as input structure the final result.
• The Hard / Soft distribution guides the groups, but the type is not returned in the final payload.
• Definitions and levels can increase the total processing time.

Client-provided skills catalogue¶

HRX's third creation option imports jobs and an independent skills list, then uses this endpoint to associate them. Existing generated requests keep their behavior when skills_source is omitted. skills_source is independent of generation_mode; provided catalogues currently support creation (new) only, at most 500 jobs and 10,000 skills. Missing, invalid or expired catalogues fail before background work is accepted. There is no fallback to generated skills.

The JSON request adds skills_source: "provided" and skill_catalog containing run_id (UUID), fingerprint (SHA-256 of exact JSON bytes), and url (HRX private Storage signed URL, 15 minutes). The snapshot has version: 1, run_id, organization_id, referential_id, and skills. Each skill contains skill_id, skill_code, skill_name, skill_description, skill_group, skill_category, skill_type (hard, soft, or null), and optional source. Missing textual metadata is empty. Labels are mandatory; duplicate IDs and client codes are rejected. search_queries must contain distinct UUID job_id values. Job titles may repeat.

AIHub downloads at most 20 MiB from the exact configured HRX Storage origin/path, without redirects. It checks the digest, schema, and organization/referential/run scope before acknowledging. Jobs and skills retain their UUIDs through retrieval, selection, relevance review, proficiency levels, and the callback. Skill names, codes, descriptions and groups are never generated or rewritten in this branch. The callback adds skills_source, catalog_run_id, catalog_fingerprint, and skill_id for each selection. In provided mode, data contains only job_code and skills entries with skill_id and required_skill_level; descriptions are not repeated across 500 jobs. HRX validates these IDs in a transaction and retains all imported skills, including unassigned ones. Repeated callbacks are idempotent. An empty selection is a successful business result.

Retrieval and budgets¶

The catalogue is embedded once per run using the existing embedding connector, via the lexi embedding profile (text-embedding-3-small by default, configurable in AIProviders/aihub_ai_providers — see Tenant AI configuration). Catalogue and query embeddings are computed on the fly and never persisted, so this profile can move independently from the rest of the platform's embedding consumers. In-memory normalized vector similarity and character TF-IDF rankings are fused with AIHub reciprocal-rank scoring (semantic 0.6, text 0.4). The job title, family, complete description and up to six mission fragments produce up to 80 distinct candidates. Retrieval first reserves candidates for the different queries and fills the remaining places by their total fused score. It then orders this selected pool by that same global score, breaking ties by catalogue index. Sorting does not replace reserved mission candidates with a global top-k. Negative statements remain in the full description and its fragments; fusion does not interpret exclusions as negative evidence. The selection model must therefore assess each candidate against the job context, even when an excluded concept ranks first. One supplemental search can retrieve at most 40 previously unselected candidates for uncovered needs, when the remaining budget allows it. Selection, relevance review and proficiency calls each contain one job. The number of skills requested is a maximum; hard/soft ratios are indicative and never filled artificially.

The model sees local numeric skill IDs, resolved to catalogue UUIDs by the server. Selection, relevance review and proficiency responses use provider-enforced JSON outputs. The provider receives the SDK-transformed selection schema; the original Pydantic schema and candidate checks still validate every result, including index, level and uncovered-needs limits. Every response is checked against the candidates actually included in that call. Descriptions in prompts use excerpts of at most 1,200 characters; stored originals remain intact. If a request is too large, selection candidates are reduced; an incompressible job or downstream request fails explicitly. This tradeoff must be measured on the pilot, especially for unusually long client definitions.

The Anthropic Models API supplies model input/output ceilings; unavailable model limits block this flow. The exact provider token counter counts each complete request using the generation model, with a 10% input margin. AIHub adds no output ceiling: each call uses the model's published maximum capacity. Estimated usage, combining the input count with its margin and the reported output usage, is tracked for observability. Embedding tokens are accounted separately by the existing connector. There are at most four concurrent LLM calls per process; catalogues and their jobs are processed sequentially to bound index memory. The Anthropic reasoning profile selects the same model for token counting, model-limit retrieval and generation. Configure its provider catalogue or tenant profile instead of a model environment variable.

Provider errors fail the run explicitly. Deployments with multiple processes must allocate provider quotas accordingly. Stable failures include PROVIDED_PROMPT_TOO_LARGE, PROVIDED_OUTPUT_TRUNCATED, and OUT_OF_CATALOG_SELECTION.

For local integration tests, set both HRX origins explicitly to the local Supabase API origin, usually http://127.0.0.1:54321. Plain HTTP is accepted only when ENV identifies a local or test environment. Missing local origins stop the flow before any catalogue download or HRX callback, preventing an accidental fallback to the production project.

Pilot evaluation¶

Supply a private JSON file with catalog (snapshot above), exactly 50 annotations (job plus expert-reviewed relevant_skill_ids), and optional context, target, and level_scale. An empty relevant set represents a negative job. Keep this file outside the repository. The evaluator checks identities before any provider call.

# Embeddings and text/semantic/hybrid recall@80; writes aggregate metrics only.
.venv/bin/python -m scripts.lexi.evaluate_provided_skills /private/pilot.json /private/metrics.json
# Also run paid selection, relevance and levels for all three search modes.
.venv/bin/python -m scripts.lexi.evaluate_provided_skills /private/pilot.json /private/metrics.json --run-llm

Reports compare recall@80, final precision, false positives on negative jobs, reserved LLM tokens and duration. Read embedding usage separately in connector metrics. This script never calls HRX or persists associations. No annotated client set is included; synthetic regression tests are not a relevance benchmark.

Deploy AIHub and the HRX backend contracts before the HRX frontend, which exposes this third option to every organization. Validate catalogue download, callbacks and the annotated evaluation together before that frontend release.

The generated lexi HTTP reference is the authoritative source for parameters, schemas, responses and examples.