HEMRAConcepts
HEMRA is built from four nouns. Every command works on one of them.
| Noun | What it is | Commands |
|---|---|---|
| Dataset | A set of tasks, copied into the store and pinned by digest | hemra dataset add, list, show, rm |
| Harness | The agent under test, described by one TOML file | hemra harness add, list, show, check |
| Sandbox | Where a harness runs: this host, Docker, or a managed sandbox | [sandbox] in a run file, --sandbox, hemra doctor |
| Run | Harnesses x datasets x attempts, recorded under one ID | hemra run, runs, show, compare, cancel, export |
Inside a run
- A trial is one repetition of one task by one harness.
-k 3makes three trials per task. - An attempt is one execution of a trial. A trial gets another attempt only after an infrastructure failure, and only if the run allows retries. A harness that failed or timed out is a result, and is never retried.
- A score is a graded result for a trial, with its source:
official(a benchmark tool such as Harbor),verifier(the task's owntests/test.sh),imported, orjudge(a HEMRA judge). - A run cell is one harness on one dataset in a run, named
run/dataset/harness.hemra comparetakes cells or imports as its two sides.
How a trial ends
HEMRA keeps two things apart that are easy to mix up:
- How the run ended. Did the harness finish (
completed), report a failure, time out, break the protocol, or hit an infrastructure problem? This comes from the runner. - Whether the task was solved. This comes only from a scorer. A harness that says "done" has not passed.
Infrastructure, scorer and cancellation problems are never counted as task failures. The dashboard and the reports show them apart.
The harness protocol
A harness speaks a small JSON-lines protocol on stdin and stdout: it reads initialize, answers ready with its identity and version, reads run.start with the task, emits events, and finishes with exactly one run.completed or run.failed. Every frame is checked before it is recorded. You only write this yourself for a via = "program" harness; HEMRA's drivers speak it for agent CLIs, SDKs, ACP agents and products run through their UI. The protocol is specified in RFC-0002, and the TypeScript SDK handles it for you.
The store
All of this lives in one store directory: SQLite for runs, trials, attempts and scores, JSONL event logs written as events arrive, and artifacts stored by SHA-256. A store survives restarts. Queued work resumes, and an attempt that was running when the process died is closed as an infrastructure failure.
Pinned versions
A comparison is only as good as what was held fixed. HEMRA records the digest of every dataset and harness file a run used, the agent version each attempt observed against the one the file pins, the model, the sandbox and its image. The run report flags unpinned or drifting agents, differing models and mismatched scorers.