Methodology and evidence labels
A number without an origin and without a date is not evidence; it is a claim. This page explains where every figure on Local AI Scope comes from, what each label means, and — just as important — what is not measured.
1. The evidence labels
Every published figure carries one of these five labels. They are distinguished by text and icon, never by color alone.
| Label | What it means exactly | What it may support |
|---|---|---|
Measured by Local AI Scope |
We ran it ourselves, under a written, reproducible protocol: the version, the machine, the configuration and the date are all published | Yes, including a recommendation, stating the interval and the number of repetitions |
Community verified |
Submitted by someone outside and passed the site’s consistency checks; we did not run it ourselves | Yes, with the lower confidence shown visibly |
Estimated |
The result of a formula, not of a run. The formula is published and can be redone | As guidance. Never presented as a benchmark |
Vendor declared |
Stated by the manufacturer in a primary source: specification sheet, configuration file or license | For declared facts (size, license, maximum context). Not for performance claims |
Stale |
The figure is past its expiry: a newer version exists, or the price is old | It supports no top recommendation. It stays visible, with its date, as history |
2. Where each figure comes from
Model size
The size of the file that is actually published for that quantization, not parameters multiplied by bits. Label: Vendor declared.
Context memory
Computed layer by layer, using the attention pattern each model declares in its official configuration file. This matters more than it sounds: 2026 models no longer use full attention on every layer, and the generic formula most calculators use treats all layers alike. In one checked case that generic formula asked for 12.00 GiB where the real calculation over the published configuration gives 2.31 GiB — an overestimate of 5.2×. In plain terms: other tools tell someone with 12 GB that a model will not fit, when it will. Label: Estimated, with the formula published.
Language cost
This is the site’s own measurement. Each model’s real tokenizer — the program that chops text into the units you are billed for — is run over an in-house, versioned parallel corpus: the same texts, of four different kinds, in all four languages. The comparison is how many units it costs to say exactly the same thing in each language. Label: Measured by Local AI Scope.
The result is that the language penalty depends not on the model family but on the generation of its tokenizer, and that it also varies with the kind of text: with the same model, a contract in German can cost considerably more than an email in German. That is why the working language is an input to the tool and not a footnote.
Speed
A declared estimate, derived from the memory bandwidth and compute throughput the manufacturer states, with assumed efficiencies. It never decides whether a model fits; it can tip the recommended route when a machine is borderline, and where it does, the page says so on that same line. Label: Estimated.
3. How a measurement is identified
A speed is not a figure unless you know what it was measured on. The identity of a measurement is the full combination: model, version, artifact, quantization, runtime and its version, operating system, driver, hardware, task, language, protocol and date. That way a new configuration is not mistaken for a new figure, and an improvement is not recorded as a regression.
4. The controls
A method tested only against the cases it gets right is not a method. Every version of the calculation engine passes two opposite controls before anything is published:
- A positive control: a combination that must fit, and has to come out as fitting.
- A negative control: a combination that cannot fit, and has to come out rejected, with the shortfall stated.
If either fails, nothing is published.
5. What is not measured
What is measured is the cost of a language, not the quality in that language. Knowing that a model fits, and what it costs to work in German, says nothing about how well it answers in German: that requires running the models and grading their output under a protocol of our own, and that has not been done yet. No page on this site claims “measured linguistic quality”. When that measurement exists, it will appear here with its protocol, its date and its label.
6. Expiry and updates
Every field has its own expiry: a price goes stale sooner than a license. When a figure passes it, it is marked Stale, stops supporting headline recommendations, and stays visible with its date instead of disappearing. Every published figure carries the date it was last checked.
7. Reproducibility and corrections
The aim is that anyone can redo the calculations: formulas are published, primary sources are linked, and the measurement corpus is versioned with its checksum. If you reproduce a figure and get something else, that is valuable information — write to the address in the legal notice. Corrections are published with their date, as set out in the editorial policy.