HEMRAArchitecture
A small Rust runtime around one protocol
The CLI, the Mac app and the dashboard are thin. The work happens in library crates they share, so a run started from the app is the same run the CLI would record. Design lives in RFCs; what is built is recorded in ADRs.
hemra-run; the scheduler leases each queued trial and the engine runs it in a sandbox, while events stream into the store. The dashboard and the app read through the same read-only API. Solid boxes are HEMRA crates; dashed boxes run outside the runner's process.What happens in a run
There are two paths. HEMRA runs the agent itself (the native path), or a benchmark tool runs it and HEMRA imports the result.
HEMRA runs the agent
hemra runresolves pinned datasets and harness files, then records the run, its spec and every planned trial in one transaction.- One job per trial goes into the durable queue. A scheduler leases each job and starts an attempt.
- The engine starts
hemra-driverand the agent in the task's own image, with a cleared environment, and speaks the harness protocol with it on stdio. - Every accepted event is appended to the attempt's log as it arrives. MCP tool calls go through the proxy and are logged by the runner.
- The container is removed, artifacts are hashed by the runner and stored by digest.
- The task's unchanged verifier scores a snapshot of the agent's work in a separate container.
The benchmark tool runs it
- For a
via = "harbor"harness,hemra runsupervises oneharbor runper dataset, on exactly the run's selected tasks. - Harbor installs the agent into each task's container, runs it and runs the verifier.
- HEMRA keeps the job directory byte for byte and imports each trial's official reward unchanged, filed under the run.
hemra importdoes the same for a Harbor job or SWE-bench run made elsewhere, and detects which it was given.- A native run of the same task directory pairs with a Harbor run in
compare, because both carry the same task checksum.
Rules the code enforces
These hold across every crate and are covered by tests.
Completion is not success
A harness finishing means only that it finished. Pass or fail comes from a scorer, never from the runner or the agent's own claim.
One protocol, two implementations
The harness protocol is versioned JSON lines on stdio. Rust and TypeScript replay the same golden transcripts, so they cannot drift. Errors use stable codes.
Attempts are append-only
A retry is a new attempt with a new ID. Only infrastructure failures are retried; failures and timeouts are results.
Like with like
Tasks pair only with the same scorer fingerprint and reward names. Differing models, versions and tool sets are flagged, and nothing is added up across datasets.
Artifacts are untrusted
The runner computes every digest. The API serves artifacts as inert downloads under a sandbox CSP, and the views render agent output as text only.
No secrets in records
The harness environment is cleared except PATH and named entries. Stores, run files and queued jobs keep variable names, never values.
Where to read more
| Topic | Document |
|---|---|
| Every crate and package, and the execution paths in detail | docs/architecture.md |
| The harness protocol | RFC-0002 |
| The user model: datasets, harnesses, sandboxes, runs | RFC-0008 |
| Native drivers | RFC-0007 |
| What is stored, credentials, isolation, what leaves the machine | Trust and privacy |
| Decisions in force | ADR index |