Eval

Upload a sealed metered-eval/1 package from the official suite repo. Anyone can read how. Only signed-in users can upload. Reputation 40+ with a clean record publishes on submit. New accounts wait for review.

How to eval

Clone the suite. Run init to write harness.yaml from CLIs on PATH. Then run the CLI with a harness from that file, plus model and effort. Docker is required: the agent works in an isolated checkout and the hidden verifier grades a git patch with no network. Do not edit tasks/.

1. clone and initwork-2026.08-py4 / 0.2.0

git clone https://github.com/danecwalker/metered-suite.git
cd metered-suite
python3 -m metered_suite init

2. runsealed package

python3 -m metered_suite <harness> --model <sku> --effort max
# Docker required. Harness names come from harness.yaml

3 official jobs, hash b3737f13ab78d7af… / /eval/suite
Official suite work-2026.08-py4. Pass a harness from harness.yaml, --model, and --effort. Approve flags live in harness.yaml. The suite harvests usage.json. Same SKU, different effort or harness, different row. Reputation 40+ can file a SKU that models.dev does not know yet, and publishes a clean package without an admin click.