HEMRA

Runs

A run is harnesses x datasets x attempts, recorded under one ID. Describe it with flags, or in a run file when it has more than one harness or dataset.

hemra run -H claude-code -H codex -d tb -k 3 --baseline claude-code
hemra run run.toml

Run files

name = "harnesses"            # the run's label (default `run`)
baseline = "claude-code"      # every other harness is compared with it (default: the first)
harnesses = ["claude-code", "pi"]   # harness files in harnesses/ next to this file
datasets = ["hemra-offline", "text-tools"]   # pinned: `name` or `name@<digest prefix>`
attempts = 3                  # per task; each is its own trial (default 1)
concurrency = 2               # trials at once (default 1)
model = "claude-sonnet-4-5"   # every harness must use it, or the run is refused
timeout_secs = 900            # per attempt (default: the task's agent timeout, else 900)
max_infra_retries = 1         # retries after infrastructure failures only (default 0)
tasks = ["hemra/*"]           # only tasks matching these patterns
exclude = ["slow-solve"]      # skip these
require = ["network_block"]   # see Sandboxes

[sandbox]                     # see Sandboxes
image = "alpine:3.20"

[verifier]                    # what task verifiers may reach (default: nothing)
network = "bridge"
env = ["OPENAI_API_KEY"]

[[judge]]                     # a HEMRA judge, see Judges
name = "quality"

Unknown keys are errors. Flags given next to a file replace its values (-H, -d, -k, -t, -x, --baseline, --concurrency, --sandbox), or add to them (--require, --verifier-env, --judge). The command line reference lists them all.

What happens

  1. Resolve. Every dataset is resolved to its digest and selected tasks, every harness file is loaded and its digest taken, and the sandbox is checked against require. Task images are built. A problem here stops the run before anything is recorded.
  2. Record. The run, its spec and every planned trial and job are recorded in one transaction.
  3. Run. Each program, cli, sdk, acp or computer-use trial is a job in the store's durable queue. Each harbor harness runs one supervised Harbor job per dataset, imported under the run as it finishes.
  4. Score. Driver trials on Harbor tasks are scored by the task's verifier, and by any HEMRA judges the run names. Harbor's own rewards are imported as they are.
  5. Report. One table per dataset, every harness against the baseline, with flags for anything that makes a comparison unfair.

hemra run exits with a non-zero status when a cell did not finish.

The queue

Queued work survives restarts. --detach records and queues the run, prints its ID and returns. Anything that works the store's queue then runs it: hemra serve, the Mac app, or remote workers.

hemra run run.toml --detach
hemra serve --concurrency 2       # works the queue while serving the viewer
hemra runs                        # see it progress
hemra cancel <run-id>             # stop its queued and running jobs

Ctrl-C on hemra run, or hemra cancel, cancels the run's unfinished attempts. They stay cancelled, and a cancelled trial is never counted as a failed one.

Retries

Only infrastructure failures are retried, up to max_infra_retries, each as a new attempt. A harness that failed or timed out is a result. Repetitions (-k) are separate trials, not retries.

Secrets

Harness files, run files and queued jobs name environment variables; they never hold values. Whatever runs the job reads the values from its own environment: hemra run, hemra serve, the Mac app or a worker. No value is written to the store. A harness can still print a secret into its own output, so keep keys out of prompts and tool output too.

Edit this page on GitHub