What a token really costs · Part 5 of 6
A token isn't a fixed amount of text
The same paragraph is 59 tokens to GPT-5 and 81 to Claude Opus 5.5, so prices per million tokens don't compare directly.
A model doesn’t work in words. It works in tokens: pieces of text from a vocabulary that each model family builds for itself. Prices are per token, so the size of a token is part of the price.
I counted the same paragraph, the introduction on this site’s home page, with each model’s tokenizer. In English, most need 59 or 60 tokens and Claude Haiku 4.5 needs 61. Claude Sonnet 5.5 and Opus 5.5 need 81: 37% more than GPT-5 for the same words. GPT-5 is two generations old, but it is the newest OpenAI model whose tokenizer is public.
Language matters as much. In Swedish, the same paragraph takes 78 tokens with GPT-5 and 122 with Claude Opus 5.5.
Go deeper: three languages, ten models
The paragraph in English, Swedish and Simplified Chinese, counted by each tokenizer:
| Model | Tokenizer | English | Swedish | Chinese |
|---|---|---|---|---|
| GPT-5 | o200k_base (tiktoken 0.14) | 59 | 78 | 70 |
| GPT-4 | cl100k_base (tiktoken 0.14) | 59 | 99 | 109 |
| Claude Haiku 4.5 | Claude API, measured | 61 | 94 | 99 |
| Claude Sonnet 5.5 | Claude API, measured | 81 | 122 | 100 |
| Claude Opus 5.5 | Claude API, measured | 81 | 122 | 100 |
| DeepSeek V3.1 | deepseek-ai/DeepSeek-V3.1 | 60 | 92 | 60 |
| Qwen3 | Qwen/Qwen3-8B | 59 | 98 | 58 |
| GLM-4.5 | zai-org/GLM-4.5 | 59 | 88 | 57 |
| Mistral Small 3.1 | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | 59 | 87 | 81 |
| gpt-oss | openai/gpt-oss-20b | 59 | 78 | 70 |
Tokenizers built with a lot of Chinese text (Qwen, DeepSeek, GLM) need no more tokens for the Chinese paragraph than for the English one. GPT-4’s older tokenizer needs almost twice as many.
OpenAI and the open-weight models publish their tokenizers, so those counts come from the tokenizer files themselves (tiktoken 0.14 and Hugging Face tokenizers). Anthropic doesn’t publish Claude’s, so I sent each paragraph to each Claude model and compared the input tokens it reported with those of a one-character message.
Claude Haiku 4.5 uses Anthropic’s previous tokenizer. The one in Claude 4.7 and later models “produces approximately 30% more tokens for the same text”, according to Anthropic. For this English paragraph it’s 33% more.
The texts:
- English: finops.work is an independent notebook on AI tokenomics: what it takes to produce tokens, where they are consumed, and what value they create. I take one question at a time, go deep, and come back with something simple: real numbers, worked examples, one chart per idea.
- Swedish: finops.work är en oberoende anteckningsbok om AI:s tokenekonomi: vad som krävs för att producera tokens, var de förbrukas och vilket värde de skapar. Jag tar en fråga i taget, går på djupet och kommer tillbaka med något enkelt: riktiga siffror, genomräknade exempel, ett diagram per idé.
- Chinese: finops.work 是一本关于 AI 词元经济学的独立笔记：生产词元需要什么，词元在哪里被消耗，又创造了什么价值。我每次只研究一个问题，深入下去，再带回简单的东西：真实的数字、算好的例子、每个想法一张图。