In short
- A model is not a database. It generates text; it does not look things up unless you give it a tool that does.
- Tokens are the unit of text you are billed for, both in and out.
- The context window is working memory for one request. Everything counts toward it, including tool definitions and the answer itself.
- Hallucination is a predictable consequence of how models are trained and scored, not a glitch.
- Cost = calls × tokens per call × model tier, multiplied by every retry and loop.
Tokens: the unit you pay in
Text is split into tokens before the model sees it. A token is usually a common word, a word fragment or a punctuation mark.
Google's Gemini documentation gives the most quotable rule of thumb: one token is about four characters, and 100 tokens equal roughly 60 to 80 English words. Anthropic's pricing documentation gives a comparable estimate — approximately four characters or 0.75 words in English — and notes that the exact count varies by language and content type.
Three operator consequences follow:
- Your inputs are bigger than they look. A 4-page PDF, a CRM record and a 20-line system prompt add up quickly, and you pay for them on every single call.
- Language matters. Because counts vary by language, the same message in two languages can cost noticeably different amounts. If you run multilingual workflows, measure both.
- Estimate before you scale. Providers expose token-counting endpoints precisely so you can price a workload before running it across 50,000 records. Use them instead of generic third-party tokenizers, which may not match the model you are actually calling.
The context window: working memory, not memory
The context window is all the text the model can reference while producing one response — including that response. Anthropic's documentation describes it as working memory, explicitly distinguishing it from the data the model was trained on.
Two things surprise people:
Everything counts. Anthropic's docs spell it out: the system prompt, every message in the conversation, tool results, images and documents, plus your tool definitions all count toward the window, and so does the output the model generates. In agentic systems, tool definitions and accumulated tool results are frequently the largest hidden consumers — not the user's question.
More is not better. The same documentation notes that as the token count grows, accuracy and recall degrade, a phenomenon it names context rot. Curating what goes into context is as important as how much space you have.
And the blunt one: the model has no memory of your business. Between requests it retains nothing. Anything it appears to remember was re-sent in that request, retrieved from a store your system maintains, or summarized forward by a compaction step. If your workflow depends on the model "knowing" your pricing rules, those rules have to be in the request, every time — which is also why they cost you tokens, every time.
Practically:
- Put the instruction close to the data it applies to.
- Send the three relevant records, not the whole table.
- When conversations run long, summarize and drop, rather than letting history accumulate.
- Expect a hard error rather than graceful degradation if the input alone exceeds the window.
Why models make things up
A hallucination is a confident, fluent, wrong answer. It is not the model malfunctioning; it is the model doing exactly what it was optimized to do.
In Why Language Models Hallucinate, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang argue that models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty. Their analogy is a student on a difficult exam: guessing scores better than leaving the answer blank, so a system optimized to be a good test-taker learns to guess. They also argue that when incorrect statements cannot be distinguished from facts during pretraining, errors arise from ordinary statistical pressures. Their proposed fix is socio-technical: change the scoring of the benchmarks that dominate leaderboards, so uncertainty is not punished.
You cannot change how the industry scores models. You can change how your workflow scores answers:
- Ground the answer. Give the model the source data in the request and instruct it to use only that. This is what retrieval does; see RAG for sales teams.
- Make "unknown" a legal output. If your schema only permits a value, the model will produce one. Add
unknownornot_foundas an allowed enum value and you will be astonished how often it is the right answer. - Constrain the shape. A label from a fixed list can be validated by code. A paragraph cannot.
- Verify against a system of record. The CRM knows the phone number. The model is guessing at it.
- Measure. Twenty to fifty hand-labelled real examples tell you your actual error rate. A demo tells you nothing.
And the rule that matters most in operations: never let a model be the only thing standing between a guess and a customer. That is what the human in the loop is for.
What actually drives your bill
Model usage is priced per million tokens, with separate rates for input and output — and output is typically priced several times higher than input. Anthropic's pricing documentation shows that structure clearly, along with two levers worth knowing:
- Prompt caching. Reusing an identical prompt prefix across calls lets the provider read from cache at a fraction of the standard input price (Anthropic's standard cache-read multiplier is a tenth of base input, with a small premium charged on the write). Caching changes what you pay for those tokens — not whether they count toward the context window.
- Batch processing. Anthropic's Batch API processes large volumes asynchronously at a 50% discount on both input and output tokens. If a job does not need an answer in the next second, it probably belongs in a batch.
Pricing changes often and varies by provider, so treat the mechanics as stable and the numbers as something to look up. The five things that actually move your bill:
- Number of calls per record. This is the one people forget. A five-step chain costs five times a single call, before retries.
- Tokens per call. Mostly input: prompts, attached records, tool definitions, conversation history.
- Output length. Priced higher. "Write a detailed summary" is a budget decision.
- Model tier. The gap between the smallest and largest models in a family is usually a large multiple.
- Loops and retries. An uncapped agentic loop is an open invoice. Cap iterations, cap spend per day, alert on anomalies.
A budget worksheet (hypothetical numbers)
Imagine a workflow that processes 10,000 records a month. Each record sends about 2,000 input tokens and gets back about 300 output tokens, in a two-step chain. That is 20,000 calls, 40M input tokens and 6M output tokens a month.
Now imagine a price of $3 per million input tokens and $15 per million output tokens — invented figures, used only to show the shape of the calculation. That is $120 of input and $90 of output: $210 a month. Swap in a tier that costs three times more and you are at $630. Add an evaluator step that runs on every record and you are at roughly $840. Cache the static two-thirds of the prompt and the input line drops substantially.
The lesson is not the number. It is that the four decisions — how many calls, how much context, how long the output, which tier — are yours, and they compound.
MitHub's model tier test
Before choosing a model, answer five questions. If you cannot, you are choosing by reputation.
| # | Question | If the answer is... |
|---|---|---|
| 1 | Can I write down what a correct output looks like? | No → build the eval set first. Nothing else matters yet. |
| 2 | Is this classification/extraction, or open-ended reasoning? | Classification → start at the smallest tier. |
| 3 | What is the cost of one wrong answer? | High → larger tier plus a human gate, not one or the other. |
| 4 | How many times per day does it run? | Thousands → tier choice dominates your bill; test the cheap one properly. |
| 5 | Does it need current or private information? | Yes → the problem is retrieval and tools, not model size. |
Then follow three rules:
- Start small and move up only on evidence. Run the smallest tier against your eval set. Upgrade when it fails, not when you feel nervous.
- Make the model a configuration value, not a hardcoded string. Models are replaced constantly. Your workflow should survive a swap.
- Ignore leaderboard numbers for your decision. Public benchmarks measure generic tasks. Thirty examples of your work are worth more than any published score, and re-running them is how you evaluate a new model in an afternoon.
The operator's glossary
| Term | What it means in practice |
|---|---|
| Token | A chunk of text, roughly four characters in English. The billing unit. |
| Context window | Everything the model can see in one request, including its own answer. |
| System prompt | Standing instructions sent with every request. Counts as input tokens. |
| Inference | One run of the model. One call. |
| Structured output | A response constrained to a defined shape your code can validate. |
| Tool / function call | The model asking your system to do something, with arguments. |
| RAG | Retrieving relevant documents and putting them in the request as grounding. |
| Eval set | Your labelled examples. The only honest measure of whether a change helped. |
| Prompt caching | Paying less for a repeated prompt prefix. Does not free context space. |
| Batch | Asynchronous processing at a discount, for work that can wait. |
| Hallucination | A fluent, confident, wrong output. |
Where to go next
This is the floor, not the ceiling. Once the vocabulary is solid, the useful next questions are architectural: how to combine model calls into something reliable, in agentic workflows; when a model belongs in a step at all, in AI agents vs automation; and how to design the whole process around these costs and limits, in AI-first workflows. To apply it to revenue systems with real money attached, start with the Revenue Reverse Engineering faculty.