Skip to content

Detect duplicates

POST /lexi/api/v1/find_duplicates proposes groups of interchangeable professional concepts. Related concepts remain separate: budget planning and budget tracking are not automatically duplicates.

Request and response

{"objects": ["Project management", "Managing projects", "Data analysis"]}

A representative response is:

[["Project management", "Managing projects"], ["Data analysis"]]

A JSON list at the body root remains accepted. The response is still a list of label groups, including singletons. An empty input returns []; repeated exact strings are collapsed while retaining their first occurrence order.

Frontend migration: remove threshold from object requests. It no longer controls matching and is rejected as an unknown field (HTTP 422), rather than silently ignored. No response-shape change is required.

Candidate retrieval and equivalence decisions

  1. Embed each distinct label with the configured lexi embedding profile (text-embedding-3-small by default). Vectors are ephemeral, not persisted.
  2. Retrieve five nearest neighbors per label using cosine similarity, over the complete supplied catalogue. A pair is proposed if either endpoint retrieves the other. Matrix work runs off the event loop in blocks of 128 rows.
  3. Classify candidate pairs with native OpenAI Decisions through the configured mini LLM profile. Each request contains at most 64 independent pair questions and only their evidence. Calls are sequential and reuse the shared connector's timeout, retries, usage accounting and response validation.
  4. Accept only duplicate. distinct and insufficient_context keep labels separate. Cosine scores and model confidence do not authorize grouping.
  5. Assemble groups in input order, requiring every internal pair to have a duplicate verdict. Missing or conflicting pair decisions prevent a join; A–B and B–C alone never authorize an A–B–C group.

The configured mini profile must support native OpenAI Decisions; incompatible providers or profiles fail explicitly. There is no switch to another model or to cosine grouping after a provider failure. An incomplete provider result is an error, not an empty successful detection. The endpoint suggests groups; it does not merge stored objects or choose a replacement label.

Labels are the only evidence available on this endpoint. The referential quality workflow additionally supplies stored descriptions and object types, within each referential. Exact-label duplicates remain part of its deterministic checks.

Limits and evaluation

Five neighbors bound the proposed pair count by five times the number of distinct labels. This is candidate retrieval, not exhaustive equivalence checking: a true duplicate outside those neighbors will not reach Decisions. Groups larger than the candidate graph supports may be split rather than merged without evidence. Large catalogues incur more decision calls; a valid empty result is different from unavailable analysis. No absolute cosine threshold is portable across models.

Evaluations separately measure candidate recall, categorical decision accuracy, and final group precision/recall/F1 and false merges. The production neighbor count is not automatically changed by offline calibration.

See referential quality, AI evaluations, and the generated HTTP reference.