Do You Know What You're Paying For When You Use an AI Coder?

TL;DR. When you use an AI coder, most of your token spend goes to overhead, not work. In one real coder we instrumented, tool schemas ate 68% of the cost per turn, the system prompt took 20%, and actual tool results only 12%. Simple fixes like lazy-loading schemas cut cumulative cost by 47% with zero quality loss. The reason these fixes are rare is misaligned incentives, since the companies selling coders also sell the tokens those coders burn.

There's a structural conflict of interest in the AI-assisted programming market that almost nobody talks about. The firms selling tokens through an API are the same firms shipping the coders that consume those tokens. The less you understand about what gets consumed, the better their margins look.

I'm not describing a conspiracy. I'm describing incentives that point the wrong way. A company that profits from token volume has no economic reason to trim the consumption of a tool built on its own API. The predictable result is black boxes that chew through tens of thousands of tokens per turn while you have no idea where the cost lands.

What Happens Inside the Black Box

We cracked one of these boxes open. I won't name the product, but what we found is worth studying.

A typical AI coder, when it processes each of your messages, ships the model a bundle. That bundle includes the system prompt (fixed instructions on how the model should behave), the schemas for every available tool (detailed descriptions of each tool the model can call), the conversation history, results from tools run in earlier turns, and assorted metadata like project settings, persistent memories, and environment details.

Then we instrumented the flow to measure what each piece actually costs. The breakdown surprised even us.

In a short session of a few turns, the distribution looked roughly like this:

• 68% of the total cost came from tool schemas.
• 20% came from the system prompt.
• 12% came from actual tool results, meaning the productive work the model does for you.

Read that again. Nearly 70% of what you pay per interaction is tool descriptions the model needs so it knows what exists. Most of those tools won't be touched in that turn. You're paying the model to read the manual for 44 tools on every message, even when it uses 2 or 3.

The Invisible Cost of Schemas

We measured the fixed floor for tool schemas at roughly 30,500 tokens per turn. That's the minimum before any real work starts. Across a 15-turn session, that's 457,500 tokens spent on schemas alone. At current frontier-model prices, it's a large slice of the session bill, and the user hasn't done anything except talk.

The fix exists and it isn't fancy: load schemas on demand. Rather than send all 44 full schemas every turn, send only the 9 essentials (file reading, editing, search, command execution) and expose the rest when the model asks. The model gets a list of available tool names and, if it needs a specific one, it requests that schema in the moment.

When we shipped this, the numbers were clean. Schema tokens per turn dropped from 30,500 to 10,600, a 65% cut. Total input tokens per turn fell from 39,800 to 22,200, a 44% cut. We validated across a 15-turn session and saw zero quality degradation, zero cache failures, and 47% savings in cumulative total cost.

So if the fix is simple and the payoff is big, why isn't it the default? Look at the incentives.

The Anatomy of Waste

Tool schemas are just the loudest example. The waste comes in layers.

Tool results that live in context forever. When you ask the coder to read a file on turn 3, that file's content keeps getting sent to the model on turns 4, 5, 6, 7, and onward, until the context window overflows and forces a compaction. We set tighter temporal limits: results older than 8 turns get cleared automatically, instead of the default 15. That stops stale file reads, dead terminal output, and outdated search results from burning tokens on turns where they no longer matter.

Duplicate reads of the same file. If the coder reads a file twice in one session and nothing changed, the full content ships to the model again. We added deduplication. If the file hasn't changed since the last read (checked by timestamp), we return a short stub saying the content is unchanged. For partial reads with overlapping ranges, we compute the intersection and send only the new bytes.

Project instructions parked outside the cache zone. Most AI coders let you set persistent project instructions like code conventions, commit rules, and style preferences. These get injected as a user message rather than as part of the system prompt. The distinction is technical, but the money is real. The system prompt is cached after the first turn (re-read cost: 10% of full price), while user messages pay full price every turn. In a project with 3,000 tokens of instructions across a 100-turn session, that's 270,000 tokens billed at full price versus 30,000 tokens at cache price. Moving the instructions into the system prompt, placed after the stable content so the cache prefix stays valid, was a few lines of code with huge cumulative impact.

Memory manifests injected in full. Persistent memory systems that retain context across sessions usually inject the complete index of every memory on every turn. With 60 memory files, that's close to 3,000 tokens per turn just for the manifest. We added type-based tiering. Feedback and user-profile memories (compact, broadly relevant) stay always visible. Reference and project memories (bulky, situational) get filtered by recency, with only the 15 most recent in the manifest, and the rest reachable through on-demand search. Measured result: 913 fewer tokens per turn in the system prompt.

The Problem Isn't Technical, It's About Incentives

Every one of these optimizations is technically simple. None demand new research or a rearchitecture. They're adjustments any competent engineering team could ship in days.

The reason commercial coders don't ship them isn't a skills gap. It's that every token saved is revenue that never arrives. When your coder spends 30,500 tokens per turn on schemas that could be 10,600, the 19,900-token gap is provider revenue. Over a 100-turn session, that's nearly 2 million tokens, paid by you, without you knowing you paid.

Here's the insidious part. You have no way to check. Commercial coders don't give you a cost breakdown by component. They won't tell you how much of your bill is system prompt, how much is tool schema, how much is tool results, how much is real conversation. You get a total token count and a dollar figure at month's end. The opacity is functional.

What Should Exist

Every AI coder should offer, at a minimum:

1. A per-component breakdown per turn, showing exactly where your tokens go. We built this as an environment flag that emits a structured log per session, with proportional decomposition rescaled to the actual tokens the API reports. Overhead is zero when off and negligible when on.
2. Control over compaction aggressiveness, with user-configurable parameters and kill-switches to revert to default behavior if something breaks.
3. Lazy loading of tool schemas as the default, not a feature buried behind an experimental flag.
4. File-read deduplication and temporal cleanup of obsolete results.
5. Cumulative estimated cost visible in the interface in real time, with visual warnings when it crosses a reasonable threshold.

None of this is hard. All of it works against the financial interest of whoever sells the tokens.

The Future of Efficiency Belongs to Those Who Pay

This rhymes with the early days of cloud computing, when providers sold oversized instances and customers had no way to see they were paying for idle capacity. It took independent monitoring and cost-optimization tools before the market corrected.

With AI coders, we're one step behind that. Most users don't even know the problem exists. They accept the cost as a given because they can't see its composition. They assume the provider is optimizing on their behalf, when the economic incentive runs the other way.

The fix won't come from those who sell tokens. It'll come from those who pay for them, once they start demanding transparency, instrumentation, and control. Or it'll come from tools that choose to compete on efficiency instead of volume.

The numbers here aren't theoretical. They're real measurements from a real coder, before and after changes any team could make. The question isn't whether it's possible. The question is why nobody's doing it.

FAQ

Why do tool schemas cost so much in an AI coder?

Tool schemas are detailed descriptions of every tool the model can call, and most coders send all of them on every turn so the model knows what's available. In one real coder we measured, schemas made up 68% of the cost per turn, roughly 30,500 tokens, even though the model typically uses only 2 or 3 tools per message.

How much can lazy-loading tool schemas actually save?

In our test, sending only 9 essential schemas and loading the rest on demand cut schema tokens per turn from 30,500 to 10,600, a 65% reduction. Total input tokens per turn fell from 39,800 to 22,200, a 44% reduction, and cumulative session cost dropped 47% with zero quality degradation and zero cache failures.

Why don't commercial AI coders optimize token usage by default?

Because the companies selling the coders usually also sell the tokens the coders consume. Every token saved is revenue that doesn't arrive, so there's no economic reason to trim consumption. The optimizations are technically simple, but they run against the provider's financial interest.

Why does it matter whether instructions go in the system prompt or a user message?

The system prompt is cached after the first turn and re-read at about 10% of full price, while user messages pay full price every turn. For 3,000 tokens of project instructions over a 100-turn session, that's the difference between 270,000 tokens at full price and 30,000 tokens at cache price.

How can I tell where my AI coder's tokens are going?

Today you usually can't, because commercial coders don't publish a per-component cost breakdown. The right fix is a per-turn breakdown showing system prompt, tool schemas, tool results, and conversation separately, plus a live cumulative cost estimate in the interface. Until vendors offer that, you only see a total token count and a monthly dollar figure.