How Metered compares models

Metered does not rank models on list price per 1,000,000 native tokens. Metered ranks stacks on the cost to finish the same official work. The rank uses $ / MU. A stack with a low list price can have a high $ / MU. The $ / MU is high if the stack uses more native tokens to finish the work.

What an MU is

1 MU is 4 Unicode characters after NFC and LF. An MU is not a native token. Two models can read the same characters and use different token counts. The rank does not use the tokenizer. The rank uses one fixed quantity of official work.

Official suite work-2026.08-py4 is 3 Harbor coding jobs: a durable work queue, a rate limiter, and a JSON Patch subset. The agent does not see the hidden verifier. After the agent exits, we grade a git patch in a Docker container. That container has no network. This split is the same as DeepSWE.

The suite is 4293.5 MU. That value is 17174 characters of fixed prompts and reference answers, divided by 4. The lock does not grow when a model writes more text. Fertility, thinking, retries, and failed attempts change only the bill.

The ranking: $ / MU

The rank value is $ billed divided by Work MU. This value is available only when a stack finishes every official task on work-2026.08-py4. A failed job is not cheaper than a complete finish. We divide $ billed by 4293.5. We do not divide by 1,000,000 tokens. We do not divide by 1,000,000 MU.

1 MU        = 4 Unicode characters (NFC, LF)
Work MU     = 17174 / 4 = 4293.5   (work-2026.08-py4)
$ billed    = uncached_in × P_in
            + cache_write × P_write
            + cache_hit   × P_hit
            + billed_out  × P_out
$ / MU      = $ billed / 4293.5
$ / pass    = $ billed / tasks that passed
Tokens/pass = (uncached_in + billed_out) / passed

A complete finish that costs $1 gives a $ / MU of 1 / 4293.5. Do not change the suite size to 1,000,000 units. $ / MU is the cost after the work. $ / MU is not a list price per 1,000,000 tokens.

Token counts come from the suite adapter for that harness. Examples are the Claude JSON result, Codex turn.completed, Gemini session stats, and OpenCode step_finish. The runner writes usage.json. Uncached input, cache writes, cache reads, visible output, and thinking all enter the bill. Thinking that is already inside output is not added again. A complete finish with no counted tokens is not $0 / MU. That finish has no bill. That finish does not rank.

$ / pass and Tokens / pass stay on the row. These values show the cost and the token count for the job. Burn vs leanest compares Tokens / pass to the lowest Tokens / pass among complete stacks. We bill thinking tokens at the output rate.

Each Effort value is a separate row. Do not average Effort values. A row with high Effort is not the same as a row with default Effort.

Why a partial pass does not rank

The suite can have more than one official task. Do not sort on $ / pass after a short suite. A stack that finishes only a short task can then rank first. That rank is wrong. We show the pass count on every row. We give $ / MU and a sort key only to a complete finish.

The local runner retries each task until the task passes, or until the attempt count reaches MAX_ATTEMPTS. If MAX_ATTEMPTS is 0, the runner does not stop on attempt count. The runner continues until the task passes. Every attempt stays in the bill. That bill is the cost to correct the work.

Token efficiency

Tokens / pass is (input + output + thinking) / passed on the same suite. Failed attempts stay in the token count. 1.00× burn is the complete stack with the lowest Tokens / pass. 5.50× means the stack used 5.50 times as many tokens to finish the jobs.

This number includes the tokenizer. You do not need a second token value to see extra token use.

Encoding fertility

Fertility is a check on encoding. Fertility is not the sort key. Ahia et al. (EMNLP 2023) show that the same text gives a different API bill under different tokenizers. The TensorZero (2026) study finds 2.65 times more native tokens on tool schemas for Claude Opus 4.7 than for GPT-5.4. In that study, a 2 times list-price difference is a 5.3 times input bill.

If two models read the same characters, each model can use a different number of native tokens.

1 MU = 4 Unicode characters (NFC, LF)
Fertility = nativeTokens / MU
TruePrice = ListPrice × Fertility

Basket basket-2026.08-preview has 6 slices:

• English prose (30%): Default product text, documents, and essays.
• Code (25%): This slice is source code. Most production token use is source code.
• Structured (15%): JSON, YAML, and wire formats.
• Tool schemas (10%): Function-call payloads. Fertility is often higher on this slice.
• CJK (10%): Chinese, Japanese, and Korean text. Many English tokenizers use more tokens on this slice.
• Instructions (10%): System prompts and product instructions.

We count with official tables when we have them (o200k_base, cl100k_base). If we do not have a table, we use a lab count API or a pasted native count. Estimates have a label.

What a row is

A row is one model, one harness, and one endpoint. The model is the weights. The harness is the program that runs the model. The endpoint is the service that you pay. GPT with ChatGPT, GPT with OpenCode, and GPT with the API are 3 evals. The same harness on OpenRouter and on the lab endpoint can give 2 bills for the same token use.

What each column means

Home and Stacks share one slim table. Lab, harness, Effort, and endpoint sit under the model name. Tokens / pass, sticker out, and the rest of the breakdown live on the model page.

Stack

Stack is the model name. The line under it is lab, harness, Effort when it is not default, and the endpoint. The model is the weights. The harness is the program that runs the official job. The endpoint is the service that billed the tokens. List prices and lab identity come from models.dev.

Passed

Passed shows two numbers: tasks that passed, and official tasks. The value uses the form passed / tasks. 1/1 means the stack finished the one official job. 0/1 means the stack did not finish the official job.

Attempts

Attempts is the number of harness calls on that run, including retries after a failed hidden verifier. 1 means the job passed on the first call.

Time

Time is decode time at a reference speed of 80 tokens per second. It is not wall clock. Wall clock varies with the model speed and the hardware running it, so the board compares runs at one fixed speed. The basis is billed output: output plus thinking, with thinking counted once. Input tokens are excluded because they are prefilled, not decoded, so the 80 tokens per second rate does not apply to them. A run with no output tokens has no Time.

Complete and incomplete

Stacks can filter on complete and incomplete. Complete means every official task passed. Incomplete means the stack has a published run that did not pass every task. Only complete rows can have $ / MU. The pass count stays on every row.

$ / MU

$ / MU is $ billed divided by Work MU. Work MU for this suite is 4293.5. This value is available only for a complete finish that has a token bill. A complete finish with 0 counted tokens does not rank.

$ / pass

$ / pass is $ billed divided by the number of tasks that passed. Failed attempts stay in $ billed. An incomplete run can have $ / pass if at least 1 task passed. This value is not the sort key.

Tokens / pass

Tokens / pass is (input + output + thinking) / passed. Failed attempts stay in the token count. Cache tokens that count as input stay in the input count. This value shows the number of native tokens the stack used per passed task. Home and Stacks omit this column. The model page shows it.

Burn vs leanest

Burn vs leanest is Tokens / pass for this row, divided by the lowest Tokens / pass among complete rows. The leanest stack is the complete stack with the lowest Tokens / pass. 1.00× means this row used the same number of tokens as the leanest complete stack. 2.00× means this row used 2 times as many tokens to finish the same official work. A row that is not complete has no Burn vs leanest.

Encoding

Encoding is fertility on the text basket. Fertility is native tokens / MU for the same characters. A value of 1.00 is 4 characters per native token. A higher value means the tokenizer uses more native tokens for the same text. Encoding does not change the rank.

List

List is sticker in: the published rate for 1,000,000 input tokens on that endpoint. This is not $ / MU.

Sticker out

Sticker out is the list price for 1,000,000 output tokens on that endpoint. We bill thinking tokens at this rate. The model page shows this value on the endpoint table.

Charts

Charts use the same numbers as the tables. Ranked bars and the sticker scatter include only complete finishes with a $ / MU. The official job matrix shows pass or fail on each sealed task, plus attempts. A missing cell means the run was typed in without a package. Charts do not rank incomplete runs. They do not average Effort settings.

Public API

The index is also JSON. These routes are read-only. They do not accept eval packages. Upload still happens at /eval.

• /api/v1/stacks — ranked stacks, same order as the site
• /api/v1/models/<slug> — one model, its runs, and per-task rows
• /api/v1/suite — suite version, Work MU, and official task names

Home, Stacks, and the model page poll those routes so a newly published run appears without a refresh. The runner lockfile with prompts and hashes stays at /eval/suite.

How an eval is run

1. Freeze the suite in its own repo (work-2026.08-py4). Hash the instruction and the expected answers. Do not hash the CLI flags.
2. Clone that repo. Run python3 -m metered_suite init. That command writes harness.yaml from the CLIs on PATH. A run will not start without that file. Then run python3 -m metered_suite <harness> --model … --effort …. The harness name must be a key in harness.yaml. Set the harness, the model, and the Effort. Approve flags live in that file.
3. The runner seeds a Docker agent workspace with only the official repo. Hidden tests stay out of the workspace. After the harness exits, collect a git patch. Grade the patch in a new container with no network. This split is the same as DeepSWE. Retry each failed patch until the task passes or until MAX_ATTEMPTS.
4. After the CLI exits, the suite harvests token counts from stdout, stderr, usage.json, and session files. Keep input, output, thinking, and cache from every attempt in the bill. If there are no tokens, there is no $ / MU.
5. The package includes hashes of the jobs, the answers, and the totals. Upload the package at /eval. Edited totals fail verification. Swapped prompts fail verification. A changed pass flag fails verification.

Suite-verified does not mean the API bill is correct. Suite-verified means the package is a complete sealed run of our jobs. The totals match the jobs and the answers on the fixed prompts. A user with reputation 40 and no prior rejects publishes to Stacks on submit. Other accounts wait for an admin. One harness for every model removes one difference. Several harnesses for one model are valid if each harness has a name.

What we do not include

• A 7:2:1 cache blend that we did not measure on your usage.
• Intelligence scores. Pass or fail is a test that we can check. Pass or fail is not an intelligence score.
• Images, audio, and video. Those types use different meters.

Encoding fertility counts the same text with each tokenizer. That value is encoding. That value is not a job.