Key Takeaways
- Net-new (greenfield) projects: Codistry ran ~48% cheaper per project and used about half the billable tokens.
- Existing (brownfield) codebases: a one-time index pays for itself in ~1.2 tasks, then savings climb toward ~48%.
- It’s structural, not a fluke: Codistry’s management of context holds true on local open-weight models (~41% fewer tokens, no caching involved) as well as cloud models (tested with Opus 4.8).
- Dollars matter more, not tokens: raw token counts can overstate the gap (much of Claude Code’s volume is cheap cache reads); we focus on token costs to demonstrate real savings.
| Project type | Fewer billable tokens* | Lower cost* |
|---|---|---|
| Greenfield (net-new project) | ~48% | ~48% |
| Brownfield (existing repo) | ~32% | ~48% |
| Average across both | ~40% | ~48% |
* vs Claude Code, per task, on Claude Opus 4.8. Billable tokens = input + output; cost prices every token type (cache reads/writes included), so cost savings can exceed token savings. Brownfield cost is the steady state after Codistry’s one-time codebase index amortizes; the first task on a fresh repo is roughly break-even, then savings climb toward ~48%.
Overview
If you use AI coding agents, you are paying for tokens and it is surprisingly hard to know how many. The agent, the model, the tools it calls, and the way it manages context all factor into the calculations. We wanted a clear, apples-to-apples picture, so we measured two agents on the same tasks with the same tooling and counted every token that actually reached the model. This post covers how we measured it and what we learned.
The hard part: Measuring without changing what you measure
Our first instinct was to put a gateway in front of the model and read the usage logs. But a typical gateway often translates between API formats and when it does, it rewrites requests and responses so one client can talk to many model providers. That translation changes the payload, which changes the token count. For a fair comparison, that’s exactly what we don’t want.
So we wrote a tiny transparent token-counting proxy instead. It sat between the coding agent and the model and forwarded every request and response byte-for-byte with no translation and no rewriting. It only observed the usage numbers the model itself reported, logging input, output, and cached tokens for every request. Because it never touched the payload, it didn’t skew the comparison, and the counts equal exactly what the model processed and exactly what we were billed for.
Test Methodology
We compared two agents: Claude Code and Codistry. Both were driven through their VS Code extensions, with an identical set of 6 MCP servers with a total of 72 tools (chrome-devtools, fetch, fetch2, git, pdf-reader, playwright) enabled.
Note that we originally configured 2 additional MCP servers: Azure DevOps Remote MCP and the GitHub Remote MCP within Codistry, but later learned that these were not supported by Claude Code, so they were removed from the test harness in order to keep the comparison fair.
Also, because we used the VS Code extensions rather than the command-line tools, each run was kicked off manually and followed the following process: start the proxy, open the editor with the run’s settings, paste the prompt, let the agent finish, then stop the proxy and total the logs.
We ran the test cases against two very different model backends so the results wouldn’t hinge on one environment:
- Round 1 - Open-weight model: qwen3.6-35b-a3b-mlx served locally via LM Studio.
- Round 2 - Cloud model: Claude Opus 4.8 on the Anthropic API.
And we used two tasks that mirror real work. One was greenfield and one was brownfield:
Blank project: build a small Python expression-evaluator library from scratch, with tests that must pass. Run 10 times per agent. The exact prompt that was used is below:
▸blank-project prompt (calc library)
Build a small Python library called `calc` that evaluates arithmetic expressions from a string. Requirements: - Single module `calc.py` with a function `evaluate(expression: str) -> float`. - Support +, -, *, /, parentheses, unary minus, and decimals. - Correct operator precedence and left-to-right associativity. - Raise ValueError with a clear message on invalid input (empty string, unbalanced parentheses, unknown characters, trailing operators). - Raise ZeroDivisionError on divide-by-zero. - Standard library only. No eval(), no external packages except pytest. Testing: - Write tests in `test_calc.py` using pytest. - Cover: precedence (e.g. "2+3*4" == 14), parentheses, nested parentheses, unary minus, decimals, whitespace handling, and every error case above. - Use pytest.raises for the error cases. Workflow: 1. Write calc.py and test_calc.py. 2. Run `pytest -q` yourself. 3. If any test fails, fix the code (or the test if it's wrong) and re-run. 4. Repeat until all tests pass. Show me the final passing test output. Keep it minimal and readable. Do not add features beyond the spec.
Existing project: analyze the real, ~143k-line PocketBase Go repository and design a Google Cloud Storage backend (documentation and a plan only, no code). Run once per agent. The exact prompt that was used is below:
▸existing-project prompt (PocketBase GCS backend)
DO NOT GENERATE ANY NEW CODE, BUT YOU CAN GENERATE DOCUMENTATION AND PLANS. Requirements: 1. Analyze the Existing Codebase - Review PocketBase's overall codebase structure and module layout. - Understand how filesystem backends are implemented today (e.g., Local, S3). - Identify where the S3 filesystem integration is defined, how it is initialized, configured, and used throughout the application. - Infer design patterns, abstractions, naming conventions, error-handling style, and coding standards used across the project. 2. Design a New Filesystem Backend for Google Cloud Storage Create a detailed design plan that includes: - The new package and file structure to match the existing architecture (e.g., /fs/gcs/ or similar). - Required structs, interfaces, and method implementations. - How configuration options should be provided (e.g., credentials, bucket name, project ID, optional CDN URL). - How initialization should occur within PocketBase's app startup sequence. - How this new backend should be wired into existing abstractions so the rest of PocketBase can treat it identically to any other filesystem. - Error handling, retries, timeouts, and security considerations (following PocketBase's existing patterns). - Required updates to documentation and example configuration. 3. Provide Output in an Organized Format Your final output should include: - High-level design summary: architecture, design decisions, key interfaces - A step-by-step implementation plan: including the sequence of changes and files to create or modify - The diff-style or complete file contents: new files, modified files, configs, documentation - Testing plan: how to verify feature correctness locally Important Requirements - Follow existing project conventions precisely (naming, error handling, package layout, function signatures, comments). - Keep the implementation modular and consistent with PocketBase's filesystem abstraction layer. - Do not introduce breaking changes. - Assume the feature may later support additional cloud providers — design clean abstractions. - Ensure the feature compiles and can be plugged in as an alternative filesystem backend with minimal changes required from PocketBase users.
PocketBase repository size (CLOC snapshot at test time)
PocketBase is a single-binary Go backend with an embedded Svelte admin UI totalling roughly 92,600 lines of Go and ~143,000 lines of code across all languages, providing a realistic medium-sized existing codebase for testing.
▸full CLOC snapshot
| Language | Files | Blank | Comment | Code |
|---|---|---|---|---|
| Go | 437 | 15,708 | 10,448 | 92,606 |
| Svelte | 171 | 2,935 | 506 | 20,081 |
| SCSS | 27 | 285 | 113 | 8,579 |
| TypeScript | 1 | 166 | 15,743 | 7,940 |
| CSS | 17 | 0 | 64 | 5,128 |
| JavaScript | 77 | 782 | 1,099 | 3,574 |
| Markdown | 8 | 1,249 | 0 | 2,422 |
| JSON | 2 | 0 | 0 | 2,213 |
| Other (SVG/YAML/HTML/…) | 40 | 35 | 15 | 464 |
| SUM | 780 | 21,159 | 27,988 | 143,007 |
Round 1: Open-weight model (qwen3.6-35b-a3b-mlx)
On the from-scratch task, Codistry finished each run using about 41% fewer tokens on average, largely because it re-sent less context between steps. It got there in more, smaller turns and took a bit longer in wall-clock time.
| Blank project (avg per run) | Claude Code | Codistry |
|---|---|---|
| Total tokens / run | 751,438 | 446,717 |
| Requests / run | 10.3 | 16.5 |
| Wall-clock / run | 1:43 | 2:40 |
On the task alone, Claude Code used 57.7% more tokens (2,256,744 vs 1,431,116). Codistry runs a one-time temporal index of a codebase before working in it (afterwards only commit deltas are re-indexed). Treated as a prerequisite, Codistry’s effective cost is 1,614,152, narrowing the gap and resulting in Claude Code using +39.8% more tokens. The index (183,036 tokens) amortizes across later tasks on the same repository.
Round 2: Cloud model (Claude Opus 4.8)
When using Opus 4.8 in the cloud, the raw token counts look very lopsided (446,446 Claude vs 74,374 Codistry), but we’ll look at exactly why in more detail: On the blank project, looking at pure input tokens + output tokens, Claude Code used about 2x the tokens per run. When you factor in the use of cached tokens however, Claude Code processed roughly 6x the tokens.
So why is the token gap so high, but the dollar gap doesn’t follow the same 6x trend? Most of Claude Code’s volume was cache reads, which Anthropic bills at one-tenth the input rate, so those tokens are very cheap. The expensive line item is output. Claude Code’s 6x token footprint becomes roughly a 1.9x cost difference, so Codistry came in about 48% cheaper per task.
The existing-repo task has a bit of an indexing nuance, so we will walk through the details. On average, per task, Codistry costs ~48% less (about half). However, when you factor in the one-time indexing (treating it as a pre-requisite in the testing) this brings the first PocketBase task to costing a bit more with Codistry (8.7% more to be exact), then the indexing cost amortizes over all future tasks. Since we only assigned the coding agents one task in the PocketBase test case, that single task didn’t show the true value of the Codistry context engine.
| PocketBase task (cloud, cost) | Claude Code | Codistry |
|---|---|---|
| Single Task only | $2.12 | $1.10 |
| With one-time codebase index | $2.12 | $2.30 |
| After 1.2 Tasks | $2.50 | $2.50 |
By the time you’ve done roughly 1.2 tasks on the PocketBase codebase, both agents have cost about $2.50 total. That’s the crossover point where every task after that, Codistry is cheaper, trending toward ~48% savings.