A counterfactual replay of 61 eligible sessions from an anonymized two-user software organization, measured over a rolling 15-day workload.
Executive result
676,061
of 4,106,180 text tool-result tokens removed
16.46%
text tool-result reduction
19.71%
equivalent tool-result capacity increase
Replaying the current ReclaimLLM implementation over the organization's eligible captured sessions reduced the replayed stream to 3,430,119 tokens without re-running historical commands or calling a model.
Every 1 million text tool-result tokens in this workload would require about 835,400 tokens after reduction. The remaining budget could carry roughly 197,100 additional tokens of equivalent original tool-result workload.
Capacity calculation
676,061 / 3,430,119 = 19.71%Organization workload
One user generated 99.99% of captured token volume. This is operationally a single-heavy-user case study, not evidence about adoption across a broad workforce.
| Measure | Fifteen-day value |
|---|---|
| Users | 2 |
| Sessions | 183 |
| Tool calls | 19,765 |
| Captured input tokens | 680,199,930 |
| Captured output tokens | 15,149,757 |
| Captured total tokens | 695,349,687 |
The captured-token total describes the organization; it is not Replay's denominator. Prompt caching, provider reporting coverage, and different accounting paths make it unsuitable for converting Replay directly into a total-bill reduction.
Replay evidence
Replay used o200k_base and runtime implementation 0.1.27. It evaluated range caching, shell compaction, and hash deduplication.
The 5-turn and 5-tool-call thresholds match the earlier session analysis. They are below the documented 10/10 evidence floor and are disclosed rather than treated as equivalent evidence.
| Eligibility stage | Sessions |
|---|---|
| Considered | 97 |
| Eligible and replayed | 61 |
| Excluded | 36 |
Of 36 exclusions, 29 lacked enough replay-eligible calls, 3 exceeded the allowable unresolvable share, and 4 lacked enough text-result tokens. Eligible share: 62.89%. Replay classified sample confidence as full.
| Source | Original tokens | Tokens removed | Reduction | Capacity increase |
|---|---|---|---|---|
| Codex | 937,064 | 375,750 | 40.10% | 66.94% |
| Claude | 2,701,205 | 266,303 | 9.86% | 10.94% |
| Unattributed | 396,653 | 32,379 | 8.16% | 8.89% |
| Gemini | 71,258 | 1,629 | 2.29% | 2.34% |
| Total | 4,106,180 | 676,061 | 16.46% | 19.71% |
Provider spread reflects workload mix, not inherent provider compressibility. Codex sessions contained more repeated reads and long shell/search outputs; Claude carried most original volume with a lower reducible share.
rg180,616 removed tokens
sed136,785 removed tokens
cd-prefixed shell136,688 removed tokens
Together these command families represented 67.17% of the replayed reduction.
Capacity projection
If workload mix and volume remained unchanged, the replay result scales to the following direct text tool-result reduction.
| Period | Original tool-result tokens | Tokens removed | Reduced tool-result tokens |
|---|---|---|---|
| Observed 15 days | 4,106,180 | 676,061 | 3,430,119 |
| Illustrative 30 days | 8,212,360 | 1,352,122 | 6,860,238 |
| Illustrative 365 days | 99,917,047 | 16,450,818 | 83,466,229 |
The 30-day and annual figures do not account for team growth, workload changes, seasonality, or agent behavior under enforcement.
Shadow telemetry cross-check
Live session rows provide an independent shadow-era measurement. They are not added to or averaged with Replay.
| Runtime telemetry measure | Value |
|---|---|
| Sessions with mechanism telemetry | 63 of 183 |
| Sessions with a measured non-zero reduction | 49 of 183 |
| Measured projected reduction | 4,807,340 tokens |
One search-shaping event contributed 4,082,635 tokens, or 84.93% of the runtime total. Removing it leaves 724,705 tokens across the rest of the telemetry cohort. One observation should not set the expected organization-wide rate.
Runtime telemetry uses the labeled chars_div_4_v1estimator; Replay uses a real tokenizer. Replay reported that the estimator undercounted comparable measurements by 13.56%. Different corpus, coverage, attribution, and outlier distributions keep these totals separate.
The organization confirms the rollout remained in shadow mode. No runtime total is presented as realized production savings.
Interpretation
Supported
On 61 eligible sessions from a 15-day captured workload, Replay of the current implementation removed 676,061 of 4,106,180 text tool-result tokens (16.46%). Within that portion of context, the same token budget could carry 19.71% more equivalent workload.
Not supported
Replay never invokes a model. It cannot observe path changes, retries, or turn-count effects, which may make real-world impact lower—or negative. Prompt caching can also change billable tokens without changing the raw context reduction measured here.
Recommended next measurement
Enable the current implementation for a bounded cohort and compare it with the shadow baseline. Keep provider/model groups separate and measure input tokens, cache reads and writes, turns, retries, task completion, and mechanism adoption. Success requires the reduction to persist without materially increasing turns, retries, or failed tasks.
Until then, 16.46% reduction and 19.71% equivalent capacity increase are the defensible case-study figures.