Tokens are the currency of all this, and we love them for it: you pay in tokens on a cloud invoice and you pay in tokens in the context memory of the machine under your desk. The same text is not the same number of tokens in every language or in every model: measured across four document types, a text costs 1.17x more in Spanish and 1.29x more in German on gpt-oss-20b, and 1.33x and 1.62x on qwen3-8b.
Counting the tokens in a text tells you “how big is this”. The question that actually decides a budget is another one: how much more the same work costs just because of the language it is written in, and which model keeps that difference smallest.
How much does language actually add?
Measured with each model’s own tokenizer, over a parallel corpus in which the four texts say the same thing in the four languages. The ratio is the total tokens of those texts in that language divided by the total in English.
| Model | Vocabulary | Spanish | French | German |
|---|---|---|---|---|
gpt-oss-20b / gpt-oss-120b |
201,088 | 1.17x | 1.28x | 1.29x |
llama4-scout-17b |
202,048 | 1.17x | 1.28x | 1.30x |
qwen3.6-27b / qwen3.8-27b |
248,320 | 1.18x | 1.29x | 1.31x |
gemma4-12b (whole family) |
262,144 | 1.16x | 1.27x | 1.33x |
mistral-small-24b / nemotron-3.5-super-120b / nemotron-3.5-lightning-30b |
131,072 | 1.23x | 1.29x | 1.40x |
glm-5.2 |
154,880 | 1.28x | 1.40x | 1.49x |
flint-40b |
150,144 | 1.30x | 1.40x | 1.55x |
deepseek-v4-flash |
129,280 | 1.35x | 1.45x | 1.58x |
granite-4.2-8b |
100,352 | 1.31x | 1.47x | 1.61x |
qwen3-8b / qwen3-coder-30b |
151,936 | 1.33x | 1.46x | 1.62x |
Measured by Local AI Scope. Parallel corpus sha256[:16] 2f6c96b6ea0b161f, tokenized 2026-08-25 (granite-4.2-8b: 2026-09-24; nemotron-3.5-super-120b: 2026-10-17; nemotron-3.5-lightning-30b: 2026-10-20; flint-40b: 2026-10-23) with each model’s real tokenizer, add_special_tokens=False.
Why the penalty tracks the tokenizer, not the brand
The column sorts almost perfectly by vocabulary size, and it cuts straight across vendors. The large 2026 vocabularies (gpt-oss at 201k, Llama 4 at 202k, Qwen3.6 and 3.8 at 248k, gemma-4 at 262k) all land between 1.29x and 1.33x in German. The smaller ones (Granite 4.2 at 100k, Mistral Small and Nemotron 3.5 at 131k, Flint at 150k, Qwen3 at 152k, GLM-5.2 at 155k, DeepSeek at 129k) pay 1.40x to 1.62x.
So “which family” turns out to be the wrong question. qwen3-8b and qwen3.6-27b share a brand and sit at 1.62x and 1.31x in German, because they do not share a tokenizer generation. Two models from the same house can end up at opposite ends of this table — same surname, very different grocery bills.
One row runs against the grain and deserves a closer look: mistral-small-24b has a 131k vocabulary and still comes in at 1.29x in French, the best French ratio in its size class, while sitting at 1.40x in German. NVIDIA’s two Nemotron 3.5 models in our catalogue, nemotron-3.5-super-120b and nemotron-3.5-lightning-30b, land on exactly the same ratios, token for token, in all four languages. Not a coincidence, and not the badge on the model either: we opened the three tokenizer files, and Mistral’s own tekken.json, and found one tokenizer, Tekken, with the same 131,072-slot vocabulary in the same order and the same 269,443 merge rules. Only a few control tokens change names (22 at most), such as the markers for chat turns, reasoning, tool calls and images, never a piece of a word. Three models, two vendors, one tokenizer, one language surcharge: this section’s point, made by the table itself.
Is the penalty the same for every kind of text?
No, and treating it as a constant is the second way this goes wrong. Same model, same four languages, now broken out by document type on qwen3-coder-30b:
| Document | English | Spanish | French | German |
|---|---|---|---|---|
| Source code with comments | 210 tokens | 269 (1.28x) | 309 (1.47x) | 353 (1.68x) |
| Contract clauses | 186 tokens | 258 (1.39x) | 264 (1.42x) | 324 (1.74x) |
| Support reply | 155 tokens | 215 (1.39x) | 244 (1.57x) | 239 (1.54x) |
| Business email | 158 tokens | 198 (1.25x) | 215 (1.36x) | 230 (1.46x) |
Measured by Local AI Scope, same corpus.
What this changes about the cloud bill
Per-token pricing is quoted in a currency, not in a language. Take the same monthly workload, process it in German through a model with an older tokenizer, and it moves through 62% more tokens than the English version of that workload. The invoice follows the tokens, obediently.
That pushes toward running locally exactly in the markets where the arithmetic bites hardest — which, as far as we are concerned, is one more good excuse to run the model at home.
What this changes about local memory
Locally there is no invoice, so the cost shows up as context instead. Context memory is allocated in tokens, so 1.62x the tokens means reserving 1.62x the window for the same document, and that competes with the weights for the same GB.
Because runtimes allocate context in power-of-two steps, the cost does not creep up like a ramp; it behaves like a staircase. It is absorbed until it crosses a threshold and then doubles at once. An 11,000-word contract reserves 16k in English and 32k in Spanish on gemma4-12b, gpt-oss-20b and mistral-small-24b alike. On a machine that was comfortable, that is invisible; on one that was not, it removes models from the shortlist.
Opens on YouTube (no cookies) only when you press play. More videos on its sheet →
The fit calculator does that part of the arithmetic for you.
What is measured here and what is not
This page measures the cost of a language in tokens. It does not measure the quality of a model’s answers in that language: that requires running the models on a GPU and evaluating outputs, which we have not done, and calling a formula a benchmark is precisely the mistake this site exists to avoid.
So the claim we stand behind is narrow and useful: the language you work in changes how many tokens the same work takes, by a factor we have measured per model, and that changes both the invoice and what fits on your machine.
Run your own text
The Model Finder applies these ratios to your document size, your machine and your language, and reports the context each of the 26 models can actually hold. It runs in your browser; the text you describe never leaves it. Per-model figures live on each model sheet, with the language column on every one, and the runtime profiles cover the program that will load them.