feature
Aug 18, 2026Replay captured proxy sessions against enabled models, measure classification agreement, and inspect each result before changing providers.
You can now test different models against frozen sets of your organization’s real proxy sessions in Model Analysis. Instead of comparing models with synthetic prompts or a single hand-picked request, you can replay the same captured workload against several enabled models and inspect where their outputs agree, differ, or fail.
Model Analysis is available to enterprise organization administrators. It is designed for questions such as:
Start by creating a frozen dataset from captured proxy sessions. Select sessions with one operation title or tag, optionally narrow the date range, and choose the exact sessions you want to include.
The dataset freezes the source sessions and evaluation configuration. You can run the same dataset against different target models later without silently changing the cohort. The dataset purpose is optional, so you can use it to record the migration, experiment, or product decision behind the comparison when that context is useful.
Freezing does not call a provider. Provider calls begin only after you prepare and explicitly confirm a run.
Model Analysis provides two evaluation modes.
Category agreement is for structured classification output such as intent, sentiment, routing, risk, or document type.
Choose the JSON field that contains the category and review the allowed labels. ReclaimLLM extracts that field from both the captured source response and every target response. The run reports source-to-target agreement for each target model as an exact count and percentage.
Each result also shows the source label and target label directly:
billing_question → billing_question
This makes the denominator visible. Failed, incompatible, or unscorable sessions do not disappear from the run summary.
Blind judging is for responses where exact field equality is not enough.
Choose an independent enabled model and provide a rubric. The judge receives the original normalized request and two randomized responses labeled A and B. It does not receive the source or target model identities, latency, cost, or generated diff.
The result records the preferred response along with short strengths and weaknesses for both outputs. The judge model must be different from the source and target models.
A run creates one case for every frozen session and target-model combination. Before any provider inference, ReclaimLLM checks compatibility and builds a confirmation screen.
The preflight shows:
An administrator must confirm the full run after reviewing those limits. Cost estimates are intentionally disabled for now because ReclaimLLM does not yet maintain a complete, versioned token-price table across supported models. Provider-call ceilings remain enforced.
ReclaimLLM downloads the captured session artifact, builds a normalized Chat Completions request, replaces the source model with the selected target, and forces non-streaming output.
Only supported request fields are retained. Unsupported fields are dropped and recorded as a controlled change. Requests with unsupported message content, multiple choices, streaming source output, or unusable source data are marked incompatible instead of being sent.
Candidate and judge calls use the organization’s configured credentials through the enterprise proxy. Credentials are never exposed to the browser, and analysis traffic is not captured again as ordinary production sessions.
The run view separates succeeded, incompatible, failed, and unscorable cases. It also groups compatibility and execution reasons, including a failure that affected only one session.
Open a successful result to inspect:
Full requests, responses, diffs, and judge evidence are stored as encrypted, region-matched artifacts. Decrypted evidence is restricted to authorized administrators and access is audited.
If a run fails, Rerun prepares a new run with the same frozen dataset and target models while preserving the failed run for diagnosis. You review the new preflight before confirming it.
Open Enterprise Dashboard → Model Analysis, create a frozen dataset, and start with a small representative cohort. The full usage guide is available in the Model Analysis documentation.
We are especially interested in provider-specific incompatibilities, structured-output edge cases, and evaluation workflows that are difficult to express with category agreement or blind judging. Send those cases through the product feedback channel so we can improve the replay and evidence model.