Replay an AI evaluation¶

Diagnostic experiments resume a saved LangGraph execution, branch it with explicit changes, or judge an existing answer again. They use the regular evaluation plan and run commands, process isolation, provider metering and shared budget. Only run calls providers. No frontend or production persistence change is needed.

Capture and inspect¶

Fresh conversational and specialist-agent observations automatically store native SQLite checkpoints under checkpoints/<case-hash>/<repetition>/ alongside their report. Each observation owns its connection and database, including in parallel campaigns. A manifest records the synthetic case, model/profile configuration, source revision, runtime fingerprint, graph schema/topology and package versions. Native checkpoint metadata retains the tool trace prefix, model prompt evidence and synthetic graph-store contents. Request credentials and database clients are not captured. Files belong to campaign artifacts, not Git. For reused observations, inspect the original report identified by provenance; collections do not duplicate its checkpoint database.

The SQLite adapter owns serialization and transactions. Completed checkpoints survive cancellation; work after the last committed checkpoint may be missing. The original observation remains incomplete after a timeout or worker failure. Report mirrors include consistent SQLite backups, including committed WAL pages.

The enclosing evaluation graph uses synchronous checkpoint durability. Knowledge Base research remains ephemeral (checkpointer=False) and explicitly uses async durability so it does not inherit a synchronous wait for a nonexistent checkpoint. Replay can restart its enclosing research node; it cannot resume an internal research step that was never checkpointed.

Inspect an observation without provider calls:

.venv/bin/python -m scripts.ai_evals checkpoints \
  --report ai-eval-artifacts/report.json \
  --case agents.career_chat_agent.closing_fr --observation 0

Output includes checkpoint IDs, namespaces, steps, pending graph nodes, tool trace prefixes and pending interaction payloads. Repetitions are zero-based. Choose a checkpoint before the decision to investigate. Preserve its exact namespace, including generated identifiers, when selecting a nested specialist checkpoint. There is no checkpoint between two operations inside one node or middleware. A terminal checkpoint cannot execute additional work.

Inspection uses a temporary database copy. Execution copies the source into a new observation database and follows the native ancestor checkpoint map for nested graphs. Original reports and checkpoint files are never edited.

Replay a decision¶

Create an experiment specification, for example closing-replay.json:

{
  "mode": "replay",
  "report": "ai-eval-artifacts/report.json",
  "case": "agents.career_chat_agent.closing_fr",
  "repetition": 0,
  "checkpoint_id": "<checkpoint ID from inspection>",
  "checkpoint_ns": "<namespace from inspection, or an empty string>"
}

report is resolved relative to the specification file. Then plan and, when live execution is authorized, run with identical configuration and repetition options:

.venv/bin/python -m scripts.ai_evals plan --experiment closing-replay.json
.venv/bin/python -m scripts.ai_evals run --experiment closing-replay.json \
  --output ai-eval-artifacts/closing-replay

The current checkout supplies the candidate code and prompts. --config selects the candidate semantic model profiles and reasoning settings. State, tool-schema, graph-topology and package compatibility are checked before provider execution. The source scenario must still match; an incompatible graph requires a fresh full scenario rather than a guessed checkpoint migration.

A captured branch can itself be replayed. Its effective context and continuation criteria are retained unless explicitly replaced; compatibility still uses the original catalogue scenario. Each new report links to its immediate source.

This runs a new generation from the selected state; it is not a cached replay of the old model response. Model and tool counters start a new invocation allowance, while the campaign's physical-request and spending limits still apply.

Branch state or answer a HITL question¶

Use mode: "fork" for changes. Supported updates fields are summary, selected_objective and search_refinement. Native reducers and successor selection apply. user_context replaces the synthetic profile as a whole; it does not merge fields implicitly. Control-flow and arbitrary message edits are rejected.

For a pending topic clarification:

{
  "mode": "fork",
  "report": "ai-eval-artifacts/report.json",
  "case": "agents.training_finder_agent.boundary",
  "checkpoint_id": "<checkpoint with the pending interruption>",
  "checkpoint_ns": "<its namespace>",
  "assessment": "training_topic_answer",
  "resume": {
    "action": "answer",
    "message_id": "experiment-sql-answer",
    "answers": [
      {"questionId": "q1", "action": "answer", "message": "SQL"}
    ]
  }
}

Use the exact question IDs and cover every pending question. A skipped question has action: "skip" and an empty message. Cancellation uses action: "cancel" and answers: []. Every branch is independent. A batch of frontend questions corresponds to one native interruption, not a checkpoint per displayed question.

Continuation assessments live in evals/experiment_assessments.json:

These are diagnostic criteria, not additional automatically launched campaign cases. They preserve the intended language, apply to English and French, and are reviewed with the feature plan. A HITL resume requires an explicit assessment: the original scenario may correctly expect an interruption, while the continuation expects a result. The judge receives the experiment inputs as untrusted evidence. Do not weaken an assessment merely to accept an observed failure.

External data still comes from the scenario's declared synthetic fixtures, never from production. When changing a context, review whether those fixtures cover the new request; changing the topic does not magically create a new catalogue.

Judge the recorded answer again¶

{
  "mode": "rejudge",
  "report": "ai-eval-artifacts/report.json",
  "case": "chatbot.training_title.sparse_fr",
  "repetition": 0
}

This uses the saved output and prompt evidence with the current rubric and judge instructions. Generation and business tools do not run. The judge must be enabled. Older reports can be used when they contain that evidence; they cannot support graph replay without captured checkpoints. Deterministic assertions also run again. Only the judge provider is required and the call plan counts one judge invocation per observation. When rejudging a HITL continuation, select its reviewed assessment explicitly, since the original catalogue rubric may expect a pause.

Compare and validate¶

Each experiment produces a new report plus experiment-comparison.json. HTML and the review workbook identify the diagnostic scope and retain original status, output, judge verdict and configuration. Tool trajectories contain the prefix before the checkpoint followed by new work, not the abandoned original suffix. Costs and timings count only new execution; inherited steps are not fresh coverage. Workbook scenario rows show the effective diagnostic context and criteria, while their catalogue hashes continue to identify the source scenario.

Diagnostic results are excluded from ordinary evidence reuse, full-run comparison, cost-reference extraction and baseline approval. An initially failed observation remains failed even when a later experiment succeeds. Run the complete scenario after a useful correction, then assess its quality normally.

Forecasts extrapolate from full-scenario costs; they are not measurements of a particular suffix or a guarantee that a changed branch costs less. Missing cost evidence still requires explicitly authorized calibration. No experiment raises the configured ceilings or repeats automatically until it passes. Use --repetitions N only for the explicitly requested number of observations.

Offline tests cover native nested replay, forks, HITL answers/skips/cancellation, durable interruption, source immutability, parallel isolation, compatibility, real specialist adapters and judge-only execution. These establish technical behavior, not live model quality.