Results and compare

Everything a run produces is in the store: the run and its report, every trial and attempt, each attempt's event log and artifacts, and the scores.

hemra runs                          # runs, imports and loose attempts, newest first
hemra show <run>                    # the run as JSON
hemra show <run> --report           # its report as text
hemra show <attempt> --events       # an attempt with its event log
hemra export <run> --out ./export   # a self-contained copy: manifests, events, blobs

show, export and cancel take a run's ID or label, an import's ID or label, a job ID or an attempt ID.

The report

A run's report has one table per dataset. Each harness is compared with the baseline on the tasks both ran, and anything that makes the comparison unfair is flagged rather than averaged in: differing models, an unpinned or drifting agent version, an agent other than the one pinned, an incomplete cell, trials nothing graded, or scorers that differ.

Compare

hemra compare a b pairs two sides task by task and reports b - a. A side is an import (by ID or label) or a run cell: run, run/harness or run/dataset/harness, the run by ID or label. hemra compare <run> alone prints the run's report.

hemra compare oracle nop
hemra compare strong/judged/claude-code weak/judged/claude-code --criteria
hemra compare <run>/tb/claude-code <run>/tb/codex --json

Pairing rules:

Tasks pair only when both sides used the same scorer fingerprint and reward names. Anything else is listed as not comparable.
Tasks with a single 0/1 reward pair on pass rates; tasks with other rewards pair on mean reward and are summed apart.
Pass rates are shown with their denominators. Infrastructure failures, scorer failures and cancelled trials are counted separately and never as failed tasks.
Trials judged by different judge models never pair.

--criteria lists every rubric criterion that both sides were judged on. --judge NAME compares by a HEMRA judge's scores instead of the task's own. See Judges.

Imports

Results produced elsewhere come in with hemra import. HEMRA detects the format, copies the official rewards as they are, and keeps the tool's files byte for byte:

hemra import jobs/2026-10-14__12-00-00 --label earlier                 # a Harbor job directory
hemra import ./swebench-run --swebench-version 5.0.2 --label agent-a   # a SWE-bench harness run

For SWE-bench, only the instances in the run report's resolved_ids are passed; every other submitted instance is failed, except what the harness itself attributes to the environment or could not grade. Imports pair with run cells of the same task directories, so a run of Claude Code through HEMRA's driver compares with a Harbor job of another agent.

The dashboard and the trace viewer

hemra serve serves a read-only API over the store and a trace viewer at http://127.0.0.1:7878/, and works the queue. With --ui apps/dashboard/dist (after npm run build in a clone) it serves the web dashboard instead: runs and their reports, attempts with live traces, imports, compare, datasets, harnesses, scorers and judges.

The API listens on loopback only. A non-loopback --listen needs --expose, because the API has no authentication. Artifacts are served as downloads, never rendered.

Edit this page on GitHub