RCLM Bench: Local Comparisons for Coding Models and Compression Tools

RCLM Bench is now available as an open-source, local-first tool for comparing coding models and context-compression techniques under controlled conditions.

The source, installation instructions, and example manifests are available at github.com/ReclaimLLM/rclm-bench.

Bench separates two different questions that are often mixed together:

1. How do different models perform on the same coding task?
2. How much can a compression mechanism reduce captured tool-result tokens?

These questions require different inputs and produce different levels of evidence, so RCLM Bench implements them as separate modes.

Compare models on real coding tasks

Model mode starts from a clean Git repository and an immutable baseline commit. You define a task such as a feature, bug fix, or refactor, along with the model variants and commands that verify the result.

Bench creates an isolated worktree for each variant. Docker is the default execution environment, keeping agent writes away from the source checkout and preventing variants from sharing working state.

uv run rclm-bench init bench.yaml \
  --mode model \
  --repository /absolute/path/to/repository

uv run rclm-bench run bench.yaml

A feature or bug-fix comparison must include at least one task-specific verifier that fails at the baseline and passes after a correct implementation. Existing test suites can run alongside it to detect regressions.

The terminal interface shows model activity and pauses when the active agent asks a question. Human response time is separated from active execution time. After each attempt, Bench records the verifier result, code diff, reported token usage and cost when available, turns, tool calls, and human interventions.

Model variants currently run serially. This keeps interactive questions attributable to one active model, but it means parallel interactive model execution is not yet available.

Claude Code is the current coding-agent client. Different model IDs and compatible provider endpoints can be configured through it. Native Codex CLI execution is not implemented yet.

Compare compression and context plumbing

Plumbing mode works with an existing Claude Code or Codex JSONL session. It reads captured tool results and sends the same ordered inputs through a passthrough control and one or more compression treatments.

uv run rclm-bench init \
  --mode plumbing \
  --session /absolute/path/to/session.jsonl

uv run rclm-bench run plumbing.yaml

A directory can also be provided. Bench searches it recursively for JSONL files, detects Claude and Codex formats, and records source hashes for provenance.

External compression tools run as separate executables using a vendor-neutral JSONL protocol. The protocol defines a versioned handshake, ordered tool-result units, transformation statuses, frame limits, and timeouts. This keeps vendor dependencies outside the Bench installation and allows compression tools to evolve independently.

The repository includes configuration guidance for RCLM and an optional Headroom adapter. Every comparison can include a built-in passthrough treatment as the control. Reports use a common tokenizer, check that protected result envelopes remain unchanged, distinguish applied transformations from ineligible inputs and errors, and preserve the normalized plumbing.yaml configuration used for the run.

RCLM Bench does not require the ReclaimLLM backend, an account, or a hosted worker. Sessions, reports, worktrees, and the local run database remain on the user’s machine.

What plumbing results mean

Plumbing mode is an offline counterfactual. It answers questions such as:

• How many captured tool-result tokens did each treatment remove?
• How frequently was a treatment applied?
• Did the adapter preserve the expected result structure?
• Was the session large enough to meet the configured evidence floor?

It does not run the downstream model. It therefore cannot establish whether compression changes model decisions, retries, task correctness, turn count, provider billing, or total session cost.

A Codex JSONL file can be evaluated in plumbing mode, but it cannot be imported as a completed Codex arm in model mode. A fair model comparison requires both models to start from the same Git commit, task prompt, environment, and verifier.

Tools such as RTK that rerun and rewrite commands also cannot be faithfully evaluated from historical output alone. They belong in an isolated live model comparison; per-variant command-wrapper integration is not bundled in the current release.

Reading the reports carefully

Model reports currently describe one attempt per variant. They are useful for inspecting behavior and verification results, but they do not establish statistical significance or model equivalence.

Session logs and generated reports can contain source code, commands, local paths, or secrets. Bench stores them locally and ignores result directories by default, but users should still review artifacts before sharing them.

RCLM Bench is licensed under Apache 2.0. We welcome criticism, adapter proposals, unusual session formats, and reports of comparisons that the current evidence model handles poorly. The code and issue tracker are available at github.com/ReclaimLLM/rclm-bench.