HEMRAHEMRA documentation
HEMRA runs AI agent harnesses on pinned benchmark tasks, records every attempt on your machine, and compares harnesses task by task with the benchmark's own scorer. It is a single hemra command, a web dashboard and a Mac app over one local store.
These pages describe what is on main. HEMRA is in early development (v0.1, Linux and macOS), so commands and file formats can still change between versions.
Where to start
- Install the
hemracommand, and the optional pieces: Docker, Node.js for the SDK drivers, and the Mac app. - Follow the quick start: pin a dataset, run a harness on it, and read the result.
- Read Concepts for the four nouns everything is built from: dataset, harness, sandbox and run.
Guides
| Guide | What it covers |
|---|---|
| Datasets | Pinning tasks from a folder, the Harbor registry, SWE-bench or Hugging Face |
| Harnesses | Harness files: your own program, Claude Code, Codex, Pi, ACP agents, SDK agents, computer use and Harbor agents, plus MCP servers |
| Sandboxes | Where a trial runs: this host, Docker, Modal, Fly.io Machines or Fly Sprites, and what each enforces |
| Runs | Run files, attempts, the queue, retries, detaching and cancelling |
| Results and compare | Reports, hemra compare, imports, exports and the dashboard |
| Judges | Rubric judges inside a benchmark, and HEMRA judges you add yourself |
| Remote workers | Running queued jobs on other machines |
| Mac app | The desktop app: download, updates and what it does |
Reference
- Command line: every command and option, generated from
hemra --help. - Trust and privacy: what HEMRA stores, how it handles credentials, how runs are isolated, and what leaves your machine.
- The harness protocol, for writing a harness in any language, and the TypeScript SDK.
The rules HEMRA holds to
- Completion is not success. A harness that finished has not passed. Only a scorer decides that: the benchmark's own, or a task's verifier.
- Official scores are imported, never recomputed. Rewards from Harbor or SWE-bench are copied as they are, and their files are kept byte for byte.
- Compare like with like. Tasks pair only when both sides used the same task definition, scorer and reward. Anything else is flagged, not averaged in.
- Everything stays on your machine. HEMRA has no account and no telemetry. What leaves the machine is what your harnesses and their models send.