A CFO I worked with last quarter opened our first meeting with a screenshot of an invoice. The AI vendor line had gone from $40,000 to $126,000 in a single billing cycle. Nobody on her team could explain why. Usage hadn't tripled. Headcount hadn't tripled. The product hadn't even shipped a major new feature. What had tripled, it turned out, was the length of a single internal system prompt, multiplied across nine million requests a month, multiplied again by a support bot that was quietly resending the entire chat history on every single turn.
Nobody had done anything wrong, exactly. They just had never been shown the meter. That's the pattern behind almost every one of these conversations I have with CXOs. The spending is rarely reckless; it's invisible, denominated in a unit most executives have heard mentioned in a pricing page and never actually looked in the eye.
That unit is the token. It is the single most important line item in your AI P&L that almost nobody outside engineering can define, price, or audit. This piece is my attempt to fix that: the exact walkthrough I give in the room, from "what even is a token" to the architecture decisions that separate a 15%-margin AI feature from a 70%-margin one.
You'll know precisely what a token is, and isn't, including why the same sentence in Hindi or in Python can cost three times as much as it does in plain English. You'll see exactly where a typical AI system's tokens disappear, from system prompts to invisible "reasoning" tokens you're billed for but never see. And you'll walk away with a three-tier reduction framework: one afternoon of easy wins, one sprint of architecture work, and one platform investment. Across the audits I've run, that sequence typically takes total token spend down 60–90% without degrading output quality, if you measure as you go.
That last caveat matters more than it sounds. Keep it in your pocket; we'll come back to it at the end.
What a token actually is (it is not a word)
Every slide deck that mentions tokens shows the same reassuring rule of thumb: roughly 4 characters, or ¾ of a word, is one token. That's true on average and useless in practice, because the model actually counts chunks from a fixed vocabulary it memorized during training, somewhere between 30,000 and 200,000 possible chunks depending on the model, rather than words or characters.
Think of it like a printing press from before word-processors existed. A print shop didn't cast a fresh block of lead for every single letter of every word, every time. It kept a case of precast common blocks: whole words like "the" and "and", common suffixes like "-ing" and "-tion", and reserved the fiddly letter-by-letter casting for rare words and names. A tokenizer does the same thing to text. Common English word-fragments get their own single block. Rare words, names, code, numbers, and non-English scripts get built up out of smaller, more expensive pieces.
This is the single most important mental model to walk away with: tokens are priced per chunk, and the tokenizer decides the chunk boundaries, not you, not your prompt engineer, and not the length of the English sentence you'd use to describe the idea. Two prompts that mean the same thing to a human can cost meaningfully different amounts to a model, purely based on vocabulary, punctuation density, and script.
Don't take my word for it. Type something below and watch it get chopped up.
Note: this is a simplified, illustrative approximation built to show the pattern. Real tokenizers (like OpenAI's tiktoken or Anthropic's) use vocabularies trained on trillions of characters via byte-pair encoding, and are more efficient than this demo. Even so, the underlying shape of the result holds true across every production tokenizer: short common words are cheap, and everything unusual costs more.
Not all tokens are created equal
The rule-of-thumb "100 tokens ≈ 75 words" was measured on English prose. The moment your organization operates in code, numbers, or a language other than English, that ratio quietly breaks, and almost never in your favor.
Code costs more than it looks like it should
Indentation, brackets, camelCase, snake_case, and semicolons are all things a tokenizer trained mostly on natural language treats as unfamiliar, expensive-to-encode text. A block of Python that a developer would call "short" routinely tokenizes 30–60% heavier than an English paragraph explaining the same logic in prose.
Global operations pay a language tax
Tokenizer vocabularies are trained on internet-scale text, which skews heavily English. Languages with different scripts, including Hindi, Arabic, Chinese, Japanese, and Korean, get far less vocabulary "real estate," so the same sentence, carrying the same meaning, gets chopped into noticeably more tokens. If your support bot, sales assistant, or agent workflow runs across multiple markets, your per-conversation cost varies meaningfully by language, and most finance teams have never modeled that into their unit economics.
Approximate token count for a sentence of equivalent meaning ("Our quarterly business review is tomorrow morning"), by form:
Illustrative figures based on typical BPE tokenizer behavior; exact counts vary by provider and vocabulary.
And it's not just text
Images, audio, and video get converted into tokens too. A single image can cost anywhere from a few hundred to well over a thousand tokens depending on resolution and the provider's "detail" setting, a fact that quietly turns "let the AI read the screenshot" into one of the more expensive things you can ask a model to do.
Token cost is a property of form, rather than meaning. The same request, in code instead of prose, in Hindi instead of English, with an image instead of a description, can cost 1.5–4x more for identical business value. If you're comparing AI feature costs across markets or across "should this be code or a description," you're implicitly comparing tokenizers, whether you meant to or not.
Closed models rent you intelligence. Open models sell you the recipe. Size decides how much either one costs.
Every model you can call sits on two independent axes, and confusing them is where a lot of AI budgeting goes wrong. The first axis is ownership: are the weights closed behind an API, or open for anyone to download and run? The second axis is size: is this a large, general-purpose flagship, or a small language model built to do one narrow thing cheaply and fast?
That second axis matters just as much as the first, and it's the one most budgets ignore. A small language model, typically a few hundred million to single-digit-billion parameters, trades broad reasoning for speed and price: often 10 to 50 times cheaper per token than its flagship sibling, fast enough to feel instant, and, for a well-scoped task like classifying a ticket or extracting a field, frequently just as accurate. Cross the two axes and you get four quadrants, not two:
Peak reasoning, available on day one, priced on someone else's card. No infrastructure to run, and effectively zero cost control beyond which model you call and how much you send it.
Near-frontier capability, if you have the GPUs and the operations maturity to run it well. The weights are free. The electricity, hardware, and headcount are not.
A fraction of the flagship price, no infrastructure to manage, and plenty capable for narrow, well-scoped work. Still billed per token, on the vendor's terms.
Small, fast, and often free to self-host. This is where most of the repeated, narrow steps buried inside a production agent should actually live, and it's the quadrant the reduction framework keeps steering you toward.
None of the four is "cheaper" in the abstract. A closed flagship model can be the cheapest option for a low-volume, high-stakes workflow, where engineering a self-hosting stack would cost more in salaries than the API bill ever would. A small model, open or closed, can be 10 to 50 times cheaper per token the moment you're running millions of narrow, repeatable calls. The table below is the one I open every engagement with. Filter it by the shape of your workload.
| Model | Type | Input $/1M tok | Output $/1M tok | Context window | Est. cost / task |
|---|
Pricing shown is approximate, standard (non-batch, non-cached) API pricing as publicly listed as of July 2026, and it changes frequently, so treat it as directional rather than a quote. Task-type token profiles are illustrative typical ranges observed across audits rather than a guarantee for your workload.
Where your tokens actually go
Here's the question that ends most "why is our AI bill so high" meetings before they start: almost none of your tokens are the question the user actually typed. A single request to a production agent is mostly invisible scaffolding, and every layer of it is billed at full price, every single call.
Picture a moderately complex customer-support agent handling one ticket. Here's roughly how a typical token budget breaks down before the model has said a single useful word back to the customer:
Now stretch that same shape across an agent, and every tool round-trip resends most of the stack above (schemas, history, and often the growing pile of previous tool outputs) again. Stretch it across a multi-agent system, and every hand-off between agents can mean re-transmitting the accumulated context a second, third, or fourth time, because most naive agent frameworks give each agent its own copy of "everything so far" rather than a shared, addressable memory.
That single sentence is why the reduction framework below is organized around plumbing rather than model selection. Picking a cheaper model helps. Fixing the plumbing helps ten times more.
The token cost & energy simulator
Cost isn't the only meter running. Every token also has a small, real energy footprint, one that's easy to hand-wave away at the level of a single chat message and much harder to ignore once you multiply it by a million calls a month. Plug in a shape of workload roughly like yours.
Energy estimates use blended Wh-per-token rates derived from published figures (OpenAI ~0.34 Wh/query, Google Gemini ~0.24 Wh/median text prompt, and peer-reviewed inference-energy studies showing a ~0.16–0.60 Wh/query interquartile range, with reasoning-heavy models running roughly an order of magnitude higher per output token). Actual energy draw varies significantly by hardware, batching, and data center efficiency, so treat this as a directional estimate for prioritization rather than a certified figure.
A three-tier stack for cutting token spend
This is the framework I actually run in engagements. It's deliberately ordered by effort rather than cleverness, because the biggest win is almost always the cheapest one to ship, and the most sophisticated technique is usually the last 15% of the curve rather than the first.
Illustrative cumulative reduction observed across typical token audits. Your mileage will vary by starting hygiene.
- Turn on prompt caching. Both OpenAI and Anthropic will serve repeated prefixes (your system prompt, your tool schemas, your few-shot examples) from cache at roughly a tenth of the fresh input price. If your system prompt is static across calls and you haven't explicitly enabled or structured for caching, this is usually the single largest lever in the entire audit.
- Trim the system prompt. Audit it line by line. In nearly every engagement I've run, 30–50% of the system prompt is dead weight: leftover debug instructions, a style guide nobody reads, three different phrasings of the same rule stacked on top of each other by three different people over a year.
- Set output limits on purpose. Don't ship with the provider's default max-token ceiling. Decide, deliberately, how long an answer should be, and ask for that. A habit of "explain in detail" in a prompt template quietly taxes every single call forever.
- Right-size the model per step. Classification, extraction, formatting, and routing rarely need your flagship model. Send those to a smaller or open-weight model and reserve the expensive one for judgment calls.
- Prune your retrieval top-k. Retrieving 20 chunks "to be safe" when the model only ever uses 2 or 3 of them is pure waste, paid on every single query. Measure actual chunk utilization before tuning k.
- Manage conversation memory instead of replaying it. Swap "resend the full transcript every turn" for a sliding window plus a rolling summary of everything older. History is usually the single fastest-growing line item in any chat product's token bill.
- Add semantic caching. Exact-match caching only catches identical repeat queries. Embedding-based semantic caching catches the near-identical ones too, so "what's your return policy" and "how do I return something" don't need two separate model calls.
- Route by complexity. Put a cheap, fast classifier in front of your flagship model. Let it triage: simple requests go to a small model, hard ones get escalated. Most production traffic is simpler than the model you're currently sending it to.
- Constrain output structure. JSON-schema or grammar-constrained decoding stops the model from padding a structured answer with unrequested prose, a surprisingly large source of wasted output tokens in API-driven, non-chat use cases.
- Discipline your agent's tool use. Batch tool calls where possible, cache tool outputs across a session, and stop shipping your entire tool library's schema on every turn, and send only the subset relevant to where the agent currently is in the task.
- Upgrade retrieval instead of widening it. Smaller, higher-precision chunks plus a reranker beat "retrieve more context and hope the model sorts it out," almost every time, on both cost and accuracy.
- Distill narrow agent subtasks into small, fine-tuned models. If a step in your agent is well-scoped and repeated thousands of times (classify this ticket, extract this field, format this response), it very often doesn't need a frontier model at all. Replace it entirely.
- Redesign multi-agent orchestration around shared memory. Instead of every agent holding, and resending, its own full copy of "everything so far," give the system a shared, addressable context: a blackboard each agent reads from and writes to, rather than a relay race of ever-growing messages.
- Self-host open-weight models for high-volume, latency-tolerant work. Once volume justifies the capex, owned infrastructure with quantization, batching, and speculative decoding can undercut per-token vendor pricing by an order of magnitude, at the cost of an ops team that now owns uptime.
- Govern tokens like cloud spend, because they are cloud spend. Per-team and per-feature token budgets, live cost dashboards, and anomaly alerts on sudden context growth. Most six-figure surprise invoices trace back to a monitoring failure more than a usage failure.
- Run continuous token-vs-accuracy evals. This is the tier where the anti-hallucination warning from the start of this piece becomes load-bearing: every technique above is also a lever that can strip away the grounding a model needs to answer correctly. A token-efficient system that hallucinates more is still a liability. It simply files its invoices later, as incidents and lost trust instead of API charges. Track accuracy against a fixed golden set every time you tighten the token budget, alongside cost.
Hallucinations and token consumption are among the biggest, hardest problems in AI today. Of the two, solving token consumption first is the more prudent move. Your product economics, your AI-native company's valuation, and your portfolio pricing are all heavily influenced by how effectively you manage token costs. Leaders who treat tokens as seriously as cloud spend or data governance are the ones who win.
Start your audit today: pick one high-traffic agent, instrument token usage, apply Tier 1, then layer up. The savings compound, the insights multiply, and suddenly your AI initiatives shift from mere innovation to genuine market mileage. In nearly every token optimization training and audit we've run, leaders left with a glint in their eyes: the spark of seeing a clear, actionable path to control costs and gain real competitive advantage.
The future of AI belongs to those who master tokens and hallucinations, far more than those who simply chase bigger models.
Drop a note to share your thoughts, challenges, or ideas: business@thegraas.com