Verify RCLM's Token Savings on Your Own Sessions with Replay

We publish a number about our own product: RCLM's compression hooks remove roughly a quarter of the tool-result tokens sent back to the model in a typical coding session. That's not a number you should have to take on faith. As of this release, you can check it against your own sessions, from your own terminal, without running anything or trusting a dashboard rollup.

Four new MCP tools — replay_eligibility, replay_session, replay_corpus, and replay_compare — let your coding assistant reproduce the exact compression logic RCLM's hooks ship with, over sessions RCLM already captured, and report how many tool-result tokens it would actually have removed.

How it works

If you have the RCLM MCP server enabled, just ask:

Would compression help on my last 50 sessions?

That routes to replay_eligibility, which checks only cheap session metadata — turn count, tool-call count, whether the session finished — before deciding whether a full replay is even worth running. It never fetches the full session content at this stage, so it's fast, and it's meant to be called first.

Once a session or a set of sessions looks worth checking, replay_session and replay_corpus do the real work:

How much would compression have saved on session <id>?
What's the aggregate reduction across my Codex sessions this month?

Both import the same compression engine RCLM's hooks run in production — not a simplified reimplementation — and tokenize the before/after output with a real tokenizer rather than the rough chars / 4 estimate our own runtime telemetry uses for speed. replay_compare runs the same session set through multiple mechanism combinations in one call, so if you want to know what read-caching adds on top of exec-output compaction alone, you get both numbers side by side instead of two separate runs to reconcile by hand.

Every result reports a funnel: how many sessions were considered, how many were eligible, and why the rest were excluded. If most of your sessions are too short to show a compression effect, that's the finding — the tools say so plainly instead of stretching a small sample into an unstable percentage. Sessions below the eligibility bar come back as "insufficient data" naming the exact constraint that failed, never a number with a footnote attached.

One thing Replay deliberately does not do yet: convert any of this into a dollar figure. We don't yet have reliable cost data across enough sessions to make that conversion honest, so results are reported strictly in tokens. Every "helps" result also carries a fixed caveat: Replay never calls a model, so it can't see whether a real compressed run would change the agent's behavior, trigger more retries, or take a different path entirely. A measured token reduction here is not a promise about your bill.

Why this exists

Publishing a compression number invites a completely reasonable response: "sure, on your data — not mine." We could have answered that with a bigger dashboard, but the moment someone doubts a token-savings claim is usually mid-task, in a terminal, not in a browser tab. So Replay lives where the question comes up, and answers it using the actual production code path rather than a parallel implementation that could quietly drift from what really ships.

We also didn't want a verification tool to become a marketing instrument. That risk shaped most of the design: results are reported in the same units as the original claim (tool-result tokens, never savings or dollars), a session too small to say anything meaningful gets refused rather than padded into a number, and a null or negative result is treated as a legitimate, useful outcome — "compression doesn't do much for how you work" is exactly the kind of thing we'd rather you learn from a tool than from a bill.

Current limitations

• Token reduction only — not total model input, not billing, not live end-to-end savings.
• No cost conversion yet. That unlocks once provider usage data covers enough sessions to make a dollar figure defensible rather than a guess.
• Eligibility thresholds are still fairly loose by default and were set against a small internal sample — expect them to get recalibrated as more real usage comes in.
• Whether these tools actually get called depends on your assistant recognizing the intent behind a prompt like "verify your compression claim." If it doesn't reach for the right tool, naming it directly (replay_session, replay_corpus, etc.) works.
• Replay reproduces what would have happened to already-captured tool output — it can't observe how a live, compressed run might behave differently, for better or worse.

If your numbers look off, or the eligibility bar feels wrong for how you actually use your assistant, or the reduction Replay reports doesn't line up with what you'd expect from your bill — that gap is useful information, not noise. Tell us.