Quick start

This walks through a first run from a clone of the repository, using the sample harness and the one-task example dataset. It needs no API key and no Docker. See Install first.

1. Pin a dataset

A dataset is a set of tasks, copied into the store and pinned by digest so a run records exactly which task bytes it used.

hemra dataset add examples/tasks --name examples
hemra dataset show examples

2. Check a harness

A harness is one TOML file that says what runs. examples/harness-sh/echo-sh.toml runs a POSIX shell script that speaks HEMRA's harness protocol:

via = "program"
command = ["./echo.sh"]      # relative to this file
version = "0.1.0"            # what echo.sh reports in `ready`

hemra harness check runs it on one trivial task and compares the version it observed with the one the file pins:

hemra harness check examples/harness-sh/echo-sh.toml --task examples/harness-sh/task.json

The status it prints says whether the harness finished. It does not say whether a task passed: that is a scorer's job.

3. Run it

A run is harnesses x datasets x attempts. Here: one harness, one dataset, three attempts per task, on this host:

hemra run -H echo-sh --harness-dir examples/harness-sh -d examples -k 3 --sandbox local

hemra run records the run, queues one job per trial and works the queue until the run ends, then prints its report. Without --sandbox local, harnesses run in a Docker container (see Sandboxes).

4. Read the results

hemra runs                        # newest first
hemra show <run-id> --report      # the report again
hemra show <attempt-id> --events  # one attempt and its event log
hemra serve                       # the trace viewer at http://127.0.0.1:7878/

The sample harness is not scored, so its trials show as unscored. Runs of Harbor tasks with a driver harness in Docker are scored by each task's own verifier.

5. Compare two harnesses

hemra compare pairs two sides task by task. Without Harbor installed, try it on the recorded Harbor jobs in the repository: the oracle agent runs each task's reference solution, and the nop agent does nothing.

hemra import tests/harbor/fixtures/harbor-0.24.0/oracle --label oracle
hemra import tests/harbor/fixtures/harbor-0.24.0/nop --label nop
hemra compare oracle nop

Run a real agent: hemra harness add my-agent --via cli --agent claude-code writes a harness file for Claude Code. Set ANTHROPIC_API_KEY and run it like the sample. See Harnesses.
Pin a real benchmark, such as Terminal-Bench 2.1 from Hugging Face: hemra dataset add hf:harborframework/terminal-bench-2.1/tasks --name tb. See Datasets.
Compare several harnesses on several datasets with a run file. See Runs.

Edit this page on GitHub