What a token is, and why it costs money

Every AI company bills in tokens. Nobody explains what one is, which makes the bill feel arbitrary. It is not arbitrary, and once you see the unit the pricing stops being mysterious.

A token is a chunk of text, not a word

Models do not read letters and they do not read words. They read tokens: fragments produced by chopping text up with a fixed vocabulary. Common words are usually one token. Longer or rarer words split into several. Spaces and punctuation are usually attached to the token beside them.

OpenAI's published rule of thumb for English is that one token is about four characters, and 100 tokens is about 75 words (OpenAI, What are tokens and how to count them).

That rule is for English prose. Code, other languages and unusual names all split into more tokens per character, which is why the same idea can cost more in one language than another. If you want to see it rather than take my word for it, paste text into OpenAI's tokenizer and watch it break apart.

Why it costs anything

A model is a very large pile of numbers. To produce the next token, the machine runs your text through that pile once. That is a real amount of arithmetic on a real GPU that somebody is renting by the hour, and it happens again for every single token.

This is the thing to hold on to, because it explains almost everything else about the industry:

Ordinary software costs almost nothing to serve one more user. AI costs real money every single time it is used.

Sending a tweet costs the platform a rounding error. Asking a model a question costs a measurable amount of electricity and rented hardware. That difference is why AI pricing looks like a utility bill rather than a subscription, and it is the root of why the economics are so strange.

Output costs more than input

Look at any provider's pricing page and you will see two numbers, and the output number is bigger. It is usually several times bigger.

The reason is mechanical. Your input can be processed in parallel, the whole prompt at once. Output cannot: the model produces one token, appends it, and runs again to produce the next. A 500-token answer means 500 sequential passes. You are paying for time on the machine, and generating is slower than reading.

A worked example, with made-up round numbers so the arithmetic is visible. Say a model charges $3 per million input tokens and $15 per million output tokens. Those are illustrative, not a quote. Check the current figures on the Anthropic or OpenAI pricing pages, because they change and I would rather you had today's number than mine.

• You send a 1,000-token prompt: 1,000 / 1,000,000 x $3 = $0.003
• You get a 500-token answer: 500 / 1,000,000 x $15 = $0.0075
• Total: about a cent.

A cent is nothing. That is exactly why this feels free, and exactly why it stops being free at scale. A million of those conversations is around $10,000.

The part that surprises people: conversations re-send themselves

Models have no memory between calls. None. The illusion of a chat that remembers what you said is produced by re-sending the entire conversation with every new message.

So the tenth message in a thread does not cost what the first one cost. It costs the first nine messages plus your new one, every time. Cost grows with the square of the conversation length, not in a straight line.

That is why a long thread gets sluggish and expensive, why "start a new chat" is real advice rather than superstition, and why every AI product spends engineering effort on summarising old messages instead of resending them.

What a context window actually limits

The context window is the maximum number of tokens the model can be given at once: the conversation, your documents, everything. When people say a model has a 200,000-token window they mean roughly 150,000 words of English in one go.

It is a ceiling, not a target. Filling it costs what filling it costs, and there is reasonable evidence that models attend less reliably to material buried in the middle of a very long context than to material at the start or end. If you want the research rather than the folklore, the original study is Lost in the Middle (Liu et al., 2023). How much that still applies to the newest long-context models is genuinely contested, and I am not going to pretend to a cleaner answer than the field has.

What to do with this

• Shorter prompts are cheaper, but output length is where the money actually goes.
• Start a new conversation when you change subject. You are not being tidy, you are cutting the bill.
• If a tool feels slow, it is usually generating a long answer, not thinking hard.
• When a company advertises a cheap per-token price, check the output price. That is the one you will actually spend.