The Hidden Cost of Agentic AI: Understanding LLM Token Billing

TL;DR. LLM billing boils down to two main costs: input tokens (everything you send) and output tokens (everything the model spills out). Because models are stateless, your application has to resend the entire conversation history on every single turn (meaning input costs compound rapidly as sessions grow). Prompt caching mitigates this, but it comes with caveats like write surcharges, strict cache minimums, and TTL expirations.

That is the whole story in one paragraph. The rest of this post is the uncomfortable detail that shows up on your invoice.

What Happens When You "Call GPT"

When people say "I called GPT," what really happened is that a piece of software (the harness) makes an HTTP request to a REST endpoint.

We call that the harness. Your agent framework, your IDE plugin, your own Python script: those are all harnesses. The model does nothing until the harness hands it over a fully formed request.

💡 For a simple explanation of what a harness is, check this Aipster's article

Here is the part where it gets interesting. These HTTP endpoints are modeled based on the REST paradigm. One of the core tenets of this paradigm is the statelessness: every single request must contain all the context required to process since there is no recollection from past calls.

Being stateless make it easier to the inference provider to scale the service out: any server in a cluster can handle any incoming request without needing to have a fresh copy of session data. But it also have another consequence: the model has no memory of your last message and on every single turn, the harness has to resend the full history. Over and over again.

What Actually Goes in the Payload ...

When you interact with a model through code, you are almost always hitting the Completions Endpoint ( /v1/chat/completions).

In classical web development, sending a request to an endpoint performs an action or returns a resource: GET /users/123 fetches a user, POST /orders creates an order. You pay a tiny, fixed infrastructure cost for the HTTP round trip.

In the LLM world, the completion endpoint acts less like a standard web resource and more like a token-processing engine. It receives a conversation and outputs what it thinks would be the next round of it.

{
  "messages": [
    { "role": "system", "content": "You are a helpful software architect..." },
    { "role": "user", "content": "How do I implement a Shared Kernel in DDD?" },
    { "role": "assistant", "content": "A Shared Kernel represents a shared domain model..." },
    { "role": "user", "content": "Can you show me a Java implementation with Spring?" }
  ]
}

The snippet above exemplifies a typical request to the completions endpoints. Notice that the payload includes every hop in the conversation. It includes the system prompt (the single message whose role is system), every question the user has asked, and the responses the model has given (user and assistant roles, respectively).

... and What Goes Back ...

The response has a similarly structured JSON envelope, but it contains something very different: the model's newly generated message. Continuing the example above, the model might reply with something like this:

{
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<thinking>\nI should provide a minimal Shared Kernel example...\n</thinking>\n\nSure, here's a minimal implementation..."
      },
      "finish_reason": "stop"
    }
  ]
}

🚨 Some models might produce internal reasoning that may be hidden, exposed through a separate field, or represented through provider-specific metadata. For educational purposes, we assume it to be part of message.content field.

There are a couple of important things to notice here.

First, the model isn't returning a new conversation. It is returning a completion (hence the endpoint name): the next piece of text generated from the input you just sent. The message contains the model's contribution, with the role assistant identifying who produced it.

💡 The choices array exists because the API could, in theory, return more than one possible completion.

The finish_reason tells the harness why the generation has stopped. In the case of stop, it is the model having reached a natural stopping point. Other reasons include the model hitting a token limit or producing a tool call.

And that's it. From the API's perspective, the request is over.

... and Forth.

But the application isn't done.

The harness takes that response, does whatever it needs to do with it, and eventually sends another request.

If this is a simple chat application, the harness just waits for the human to type their next message. But if you are building an agentic workflow, the model often dictates the next step by triggering a tool.

Let's imagine that instead of returning plain text, the model's response included a request to invoke a compile_code tool to test its Java snippet. The harness intercepts that command, executes the local compiler, grabs the resulting error output, and immediately fires a new request back to the API:

{
  "messages": [
    { "role": "system", "content": "You are a helpful software architect..." },
    { "role": "user", "content": "How do I implement a Shared Kernel in DDD?" },
    { "role": "assistant", "content": "A Shared Kernel represents a shared domain model..." },
    { "role": "user", "content": "Can you show me a Java implementation with Spring?" },
    { 
      "role": "assistant", 
      "tool_calls": [
        {
          "id": "call_abc123",
          "type": "function",
          "function": { "name": "compile_code", "arguments": "..." }
        }
      ] 
    },
    { 
      "role": "tool", 
      "tool_call_id": "call_abc123", 
      "content": "Execution failed. Error: cannot find symbol class Entity in package domain.shared." 
    }
  ]
}

Notice what just happened?. For the model to fix the bug, the harness had to resend everything.

The new payload includes the original system prompt, the user's initial questions, the previous context, the model's tool request, and the tool's execution results.

In an agentic application, this loop of generating code, testing it via tools, and passing the errors back to the model can happen dozens of times in seconds before a human ever sees the final output. With every single autonomous turn, the JSON payload gets heavier.

This compounding snowball of text is exactly why inference providers split your bill into two distinct buckets: Input Tokens and Output Tokens.

The Cost of Memory: Why Every Turn Costs More

To understand why LLM invoices explode, you first have to look at the raw unit of compute: the token. A token is a mathematical chunk of text roughly equivalent to four English characters ("cat" is one token; "indivisibility" is four).

💡 For more info on what a token is, check this article.

While tokens sound straightforward, providers do not treat them equally. Your bill is split into two distinct tiers based on how they are processed.

Output Tokens: What you (mostly) see

Simply put, output tokens are the words the model types back to you.

There is, however, a nuance to it: before committing to a final answer, modern reasoning models generate an internal monologue to map out the problem. A model might burn thousands of expensive output tokens before giving you a single line of actionable code and inference providers consider these "thought token" as output tokens.

Input Tokens: The Cumulative Weight

Input tokens represent everything you hand to the model.

🚨 Beware the Tool Payload: All the JSON schemas defining your tools and servers must be attached to the payload on every single request. These definitions are billed as input tokens, meaning a large toolkit will bloat your baseline costs before the model even generates a response.

Cost multiplier

Usually, input tokens carry a lower unit price. However, their volume compounds aggressively: because APIs are stateless, every turn forces your harness to re-send the entire conversation history.

This is where the math starts to break down for complex applications. If your system prompt and tool definitions total 5Ktokens, and an agent runs a 10-step autonomous loop to fix a bug, you are paying to process those exact same 5K tokens ten times.

To stop this financial (and computing) bleed, inference providers introduced a mechanism to give their stateless models a temporary memory: prompt caching.

Prompt Caching: A Temporary Fix for Stateless Amnesia

Imagine forcing an employee to read a 50-page company handbook from cover to cover every single time you asked them a quick question about the dress code. That is exactly what standard stateless APIs do to an LLM.

Prompt caching changes the game. Instead of making the AI process your massive system prompts and tool definitions from scratch on every single turn, the provider essentially keeps a "bookmarked" version of your text active in its short-term memory. It skips the heavy reading, which results in faster responses and drastically cheaper bills.

🚨 Usually, it is required the prompt to have a minimum number of tokens to be elegible for caching.

The Discount: Pennies on the Dollar

When the model successfully uses its "bookmark" (reads from the cache), the savings are massive. Providers typically offer steep discounts for these cached tokens—in Anthropic's case, a massive 90% off the standard input price.

Instead of paying full price every single time your agent loops or a user replies, you pay a fraction of a cent for the heavy context you’ve already sent. If you have a 5k token system prompt, getting a 90% discount on every subsequent turn transforms a prohibitively expensive agentic workflow into a highly affordable one.

The Ticking Clock (TTL)

But there is (always) a catch: this short-term memory is incredibly short-lived. You are always racing against a strict expiration timer known as the Time-to-Live (TTL).

Usually, the cache TTL is only 5 minutes. If your application goes quiet for six minutes: maybe your human user is just taking a moment to read the previous output, or they stepped away for coffee, or even model asks for a tool that takes time to complete. It doesn't matter, the AI throws out the bookmark. The next time they send a message, the model has forgotten everything, and you have to pay to process that massive payload all over again.

The Math in Action: A Tale of Two Loops

To see exactly why prompt caching is a lifesaver for agentic workflows, let’s run the numbers. We will use a fictional, but realistic, pricing tier for our model (priced per 1 million tokens):

Imagine an autonomous coding agent trying to fix a bug. It starts with a heavy payload: your system prompt, the tool schemas (like compile_code), and the user's initial codebase.

Let's set our Base Payload at 5,000 tokens.

1st Turn: The Initial Request

On the very first request, the cache is completely empty. The API has to process the entire payload from scratch. The model thinks for a moment and outputs a 500-token tool call to run the compiler.

At this point, the API writes those initial 5,000 tokens to the cache, and the TTL timer starts ticking.

2nd Turn: The Snowball vs. The Anchor

The compiler finishes and returns 500 tokens of error logs. The harness packages everything up and fires off the next request.

Because of the stateless nature of the API, the new payload is the original 5K tokens + the model's 500-token tool call + the 500 tokens of error logs. Our total input is now 6,000 tokens. The model processes this and outputs a final 1K-token code fix.

Here is how the bill diverges depending on whether you beat the 5-minute TTL timer:

Notice the shift, without caching, your input costs are already outpacing your output costs by Turn 2. With caching, your input costs actually dropped by over 80% compared to Turn 1, even though the payload got heavier. By the 10th turn of a complex agentic loop, the uncached payload might swell to 15K tokens, costing you $0.075 in input fees alone for a single turn. With caching, those tokens are safely bookmarked, keeping your input costs anchored to pennies.

The Hidden Toll: The Cache Write Surcharge

If a steep discount sounds too good to be true, it’s because it is. Sometimes, providers charge a premium above the standard input rate the very first time they process and bookmark your text.

Think of this write surcharge as a toll you pay on the first turn of a conversation. If your system prompt is 5K tokens, you might pay an extra 25% premium on the first turn to write it to the cache. You don't actually start saving money until the second turn. Because of this, if your agent solves the problem in a single step and immediately shuts down, caching could technically cost you more than a standard API call because you paid the write toll but never lived long enough to reap the read discount.

The Math in Action (redux)

To make the example reflect a provider that uses this surcharge model, let's update the pricing table to include a hypothetical 25% write premium:

1st Turn: The Initial Request (redux)

On the very first request, the cache is completely empty. The API has to process the entire payload from scratch and write it to memory. The model thinks for a moment and outputs a 500-token tool call to run the compiler.

The Bottom Line: Design for the Cache

Understanding how tokens are billed transforms prompt engineering from a creative exercise into a systems architecture problem. If you are building simple, single-turn chat apps, prompt caching is just a nice bonus. But if you are building autonomous agents that loop through tools and self-correct, caching is the only thing standing between you and a staggering API bill.

To actually reap these discounts and offset the write surcharges, you have to design your payloads to be cache-friendly. Because most caching mechanisms read from the top down, the order of your JSON array matters immensely:

Put static content first: Your system prompt, tool schemas, and heavy reference documents should always live at the very top of your payload. They rarely change, meaning they can stay safely bookmarked.

Put dynamic content last: The back-and-forth conversation history and the latest user queries should sit at the bottom. The moment a single token changes, the cache breaks for everything below it.

Mind the timer: Group your agent's background tasks closely together. If you know a tool will take 10 minutes to execute, be prepared to pay the cache write toll again when the agent wakes back up.

FAQ

Does prompt caching affect the quality of the model's output?

No, caching has zero impact on the quality of the generation. When a cache hit occurs, the inference provider simply reuses the previous computation. The text generated is exactly the same as if the prompt was processed from scratch.

Do all providers charge a "write surcharge" for caching?

No, caching mechanics vary heavily by provider. The article describes a system similar to Anthropic’s explicit caching, which offers massive read discounts (typically 90%) but charges a premium (e.g., +25%) to write to the cache in exchange for guaranteed hits. Others may choose not charge for cache writes.

What happens if I change a single character in my system prompt?

Caching systems read top-down. Altering even one character invalidates the cache for that token and every single token that follows it. If you inject a dynamic timestamp like Current time: 10:04 AM at the top of your system prompt, your cache will miss on every single request.

How long does the cached prompt actually live?

The default Time-to-Live (TTL) for most major providers is 5 minutes. Every time you get a successful cache hit, that 5-minute timer resets. Some providers offer extended TTLs, but opting into guaranteed extended storage usually requires paying a much higher write surcharge upfront.

Can I extend the cache lifetime beyond the default?

Some providers offer this as an option. Instead of the standard short-lived cache (typically a few minutes), you can pay a steeper write premium to keep the cached prefix alive for longer (e.g. up to an hour). Read costs stay the same either way. This is worth it when your workflow has gaps between requests that regularly exceed the default TTL.