HEMRAWhy HEMRA
A benchmark score is a model inside a harness.
Claude Code, Codex, Gemini CLI and your own agent loop can all run the same model and get different scores. The harness is the prompts, the tools, the permissions, the MCP servers and the loop around the model. Change any of it and the number moves. HEMRA exists so you can tell whether it moved because of your change.
Where agent numbers go wrong
Every one of these happens in practice, and each one can make a harness look better or worse than it is.
The task set drifted
A dataset was updated between two runs, or one run used a subset. The two scores describe different exams.
Someone re-scored it
A home-grown grader replaced the benchmark's own, or the verifier changed. The numbers look comparable and are not.
An outage counted as a failure
A container that never started, a rate limit or a login wall was counted as the agent failing the task.
The agent said it was done
"Task complete" is the agent's opinion. Finishing a run tells you nothing about whether the task passed.
Different scales were added up
Pass rates from two datasets, or from two reward definitions, were averaged into one headline number.
Something else changed too
The agent updated itself, the model alias moved, or a tool server changed its tools. The run did not record which.
What we believe
These are the rules HEMRA is built around. Each is enforced in code and covered by tests, not just written down.
- Completion is not success. How a run ended and whether the task passed are separate facts, shown separately.
- The benchmark's scorer is the authority. HEMRA imports official scores unchanged and keeps the original files byte for byte. A native run is scored by the task's own verifier.
- Compare only like with like. Results pair task by task only when the scorer matches. Anything that differs is flagged, never averaged away.
- Record everything, overwrite nothing. Every attempt, event and artifact is kept. A retry is a new attempt, and only infrastructure failures are retried.
- Pin what you ran. Datasets are pinned by digest, harness versions and models are pinned, and MCP tool sets are compared by digest.
- Your machine, your data. No account and no telemetry. The store is a folder you own, and secrets never enter it.
- Any harness, any language. A small, versioned protocol on stdio, with drivers for agents that have their own interface.
Who it is for
People building agent harnesses
Change a prompt, a tool or a permission rule, and see task by task what it did against the version you had yesterday.
Teams choosing an agent
Run Claude Code, Codex, Gemini CLI, Goose or OpenCode on your own tasks, with the same model, in the same sandbox, and compare.
Benchmark and eval maintainers
Keep your scorer authoritative. HEMRA runs agents against Harbor-format tasks and imports Harbor and SWE-bench results without re-grading them.
What HEMRA is not
Saying no keeps the numbers honest.
- Not a benchmark. It runs benchmarks; the tasks and the scorers belong to their authors.
- Not a replacement for Harbor or SWE-bench scoring. Their verdicts are imported, never recomputed.
- Not a hosted leaderboard. Results live on your machine and leave it only when you export them.
- Not an agent framework. Bring any agent; HEMRA measures it.
- Not a hostile multi-tenant sandbox. The Docker sandbox is hardened for your own runs, but a container is not a boundary for code that is out to escape it.
About
HEMRA stands for Harness Evaluation, Measurement & Runtime Architecture. It is built by Umai Tech in Stockholm.
Designed in the open
Every design starts as an RFC, and every decision that ships is recorded as an ADR, with what was left out and why. HEMRA is in early development (v0.1, Linux and macOS), and the interfaces will still change.