HEMRACommand line reference
Every hemra command and its options, as hemra <command> --help prints them. This page is generated from the binary: run HEMRA_UPDATE_DOCS=1 cargo test -p hemra-cli --test cli_docs to refresh it, and do not edit it by hand. For what the commands are for, start with Concepts and the quick start.
hemra
Harness Evaluation, Measurement & Runtime Architecture
Usage: hemra <COMMAND>
Commands:
runs List runs, imports and attempts recorded outside a run, newest first
show Print a run (its report), import, job or attempt
export Write a self-contained copy of a run, job, attempt (manifest.json, events.jsonl, blobs/) or imported job (import.json, raw/ with the original files)
cancel Cancel a run's unfinished jobs, or one queued or running job
doctor Check what this machine can run, list execution providers and their capabilities, and with --prune remove containers a killed runner left behind
worker Run attempts leased from `hemra serve --workers-listen` on this machine, until interrupted
import Import results produced elsewhere: a Harbor job directory or a SWE-bench harness run, format detected, official scores kept as-is
compare Compare two imports or run cells on the tasks they share (`b` against `a`), or every harness of one run against its baseline
serve Serve the read-only HTTP API and the trace viewer (or, with `--ui`, the web dashboard) over a store, and work its queue (Ctrl-C or SIGTERM cancels running attempts and requeues their jobs)
run Run harnesses x pinned datasets as one run, through the durable queue, and compare every harness with a baseline (RFC-0008 §7)
harness Create, list, show and check harness files: the agents under test (RFC-0008 §5)
judge Create, list, show and check HEMRA judges: general judges a run applies to any dataset, apart from a dataset's own (RFC-0009 §4.3)
dataset Pin datasets in the store by digest, and list, show or remove them
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print versionhemra runs
List runs, imports and attempts recorded outside a run, newest first
Usage: hemra runs [OPTIONS]
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
--limit <LIMIT> Most rows to print [default: 50]
--json Print the runs as JSON
-h, --help Print helphemra show
Print a run (its report), import, job or attempt
Usage: hemra show [OPTIONS] <ID>
Arguments:
<ID> Run ID or label, imported job ID or label, job ID, or attempt ID
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
--events Include the event log (attempts only)
--report For a run: print its report (each harness against the baseline, per dataset) as text instead of the run as JSON
-h, --help Print helphemra export
Write a self-contained copy of a run, job, attempt (manifest.json, events.jsonl, blobs/) or imported job (import.json, raw/ with the original files)
Usage: hemra export [OPTIONS] --out <OUT> <ID>
Arguments:
<ID> Run ID or label, imported job ID or label, job ID, or attempt ID
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
--out <OUT> Directory to write; it must not exist yet
-h, --help Print helphemra cancel
Cancel a run's unfinished jobs, or one queued or running job
Usage: hemra cancel [OPTIONS] <ID>
Arguments:
<ID> Run ID or label, job ID, or the ID of the attempt a job is running
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
-h, --help Print helphemra doctor
Check what this machine can run, list execution providers and their capabilities, and with --prune remove containers a killed runner left behind
Usage: hemra doctor [OPTIONS]
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
--prune Remove containers HEMRA started but never cleaned up (for example after `hemra` itself was killed). Containers of runs still in progress on this host are left alone unless --all is given
--all With --prune, also remove containers whose hemra process is still running
--dry-run With --prune, only list what would be removed
--json Print everything, provider isolation statements included, as JSON
--docker-bin <DOCKER_BIN> The docker CLI to use [env: HEMRA_DOCKER=] [default: docker]
--harbor <HARBOR> Harbor executable [env: HEMRA_HARBOR=] [default: harbor]
-h, --help Print helphemra worker
Run attempts leased from `hemra serve --workers-listen` on this machine, until interrupted
Usage: hemra worker [OPTIONS] --controller <CONTROLLER> --id <ID>
Options:
--controller <CONTROLLER> Controller base URL, e.g. https://controller.internal:7878
--id <ID> Stable worker identity
--token-env <TOKEN_ENV> Name of the environment variable holding the enrollment token [default: HEMRA_WORKER_TOKEN]
--ca-cert <CA_CERT> PEM CA certificate to trust for the controller instead of the system roots
--concurrency <CONCURRENCY> Most attempts to run at once [default: 1]
--label <LABELS> Label reported at registration (KEY=VALUE, repeatable)
--work-dir <WORK_DIR> Parent directory for attempt workspaces and the event spool [default: .hemra-worker]
--until-idle Exit once the controller has no job and nothing is running
-h, --help Print helphemra import
Import results produced elsewhere: a Harbor job directory or a SWE-bench harness run, format detected, official scores kept as-is
Usage: hemra import [OPTIONS] <DIR>
Arguments:
<DIR> A Harbor job directory (jobs/<job-name>), or the directory the SWE-bench harness ran in (it holds logs/run_evaluation/)
Options:
--store <STORE>
Store directory [env: HEMRA_STORE=] [default: .hemra]
--format <FORMAT>
Skip detection and read the directory as this format [possible values: harbor, swebench]
--label <LABEL>
Label used by `hemra compare` (default: the job directory name or the run ID)
--allow-untested-version
Accept output from a tool version this adapter has not been tested against (for SWE-bench, also a run of unknown version)
--swebench-version <SWEBENCH_VERSION>
SWE-bench: the swebench release that produced the run; the harness does not record it
--run-id <RUN_ID>
SWE-bench: the run to import, when the directory holds several
--report <REPORT>
SWE-bench: the run report, `<model>.<run_id>.json` (default: found in the directory or its report/)
-p, --predictions <PREDICTIONS>
SWE-bench: the predictions file the harness was given, kept with the import
-h, --help
Print helphemra compare
Compare two imports or run cells on the tasks they share (`b` against `a`), or every harness of one run against its baseline
Usage: hemra compare [OPTIONS] <A> [B]
Arguments:
<A> Baseline: an import's ID or label, or a run cell (`run`, `run/harness` or `run/dataset/harness`, the run by ID or label). Alone, a run: its report, every harness against the baseline
[B] Candidate, named like `a`. Deltas are `b - a`
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
--json Print the full comparison as JSON
--criteria List every rubric criterion paired on both sides (RFC-0009)
--judge <NAME> Compare the scores of this HEMRA judge (ADR-0043) instead of the task's own. Both sides must be run cells it judged
-h, --help Print helphemra serve
Serve the read-only HTTP API and the trace viewer (or, with `--ui`, the web dashboard) over a store, and work its queue (Ctrl-C or SIGTERM cancels running attempts and requeues their jobs)
Usage: hemra serve [OPTIONS]
Options:
--store <STORE>
Store directory [env: HEMRA_STORE=] [default: .hemra]
--listen <LISTEN>
Address to listen on. The API has no authentication [default: 127.0.0.1:7878]
--expose
Allow a non-loopback `--listen` address, exposing the store unauthenticated
--ui <UI>
Serve this built web dashboard (`apps/dashboard/dist`) instead of the built-in trace viewer [env: HEMRA_UI=]
--concurrency <CONCURRENCY>
Most queued attempts to run at once on this machine. `0` only serves the API and leaves the queue to `hemra run` or remote workers [default: 1]
--work-dir <WORK_DIR>
Parent directory for attempt workspaces (default: <store>/work)
--workers-listen <ADDR>
Also lease queued jobs to remote `hemra worker` daemons on this address. Anything but loopback needs --tls-cert and --tls-key. This listener is separate from the API's and only speaks the worker protocol
--tls-cert <TLS_CERT>
PEM certificate chain for the worker listener's HTTPS
--tls-key <TLS_KEY>
PEM private key for the worker listener's HTTPS
--token-env <TOKEN_ENV>
Name of the environment variable holding the enrollment token workers present [default: HEMRA_WORKER_TOKEN]
--lease-ttl-secs <LEASE_TTL_SECS>
Seconds a worker's lease lives without a heartbeat [default: 30]
-h, --help
Print helphemra run
Run harnesses x pinned datasets as one run, through the durable queue, and compare every harness with a baseline (RFC-0008 §7)
Usage: hemra run [OPTIONS] [FILE]
Arguments:
[FILE] Run file (TOML). Flags given next to it override its values
Options:
--store <STORE>
Store directory [env: HEMRA_STORE=] [default: .hemra]
-H, --harness <HARNESSES>
Harness to run, by name (`harnesses/<name>.toml`). Repeatable; replaces the run file's `harnesses`
-d, --dataset <DATASETS>
Pinned dataset, `name` or `name@<digest prefix>`. Repeatable; replaces the run file's `datasets`
-k, --attempts <ATTEMPTS>
Attempts per task: each is its own trial
--baseline <BASELINE>
Harness every other harness is compared with (default: the first)
--concurrency <CONCURRENCY>
Most trials running at once
-t, --task <TASKS>
Only tasks matching this pattern (`*` matches anything). Repeatable; replaces the run file's `tasks`
-x, --exclude <EXCLUDE>
Skip tasks matching this pattern. Repeatable; replaces the run file's `exclude`
--label <LABEL>
Label for `hemra compare` and `hemra show` (default: the run file's `name`, else `run`)
--sandbox <docker|local|modal|fly|sprites>
Where queued harnesses run. `local` runs them on this host and is recorded as such, `modal` in a Modal Sandbox, `fly` in a Fly.io Machine (program harnesses, `[sandbox] image` from a registry) and `sprites` in a Fly Sprite (Fly's base image); the default is the run file's, else `docker`
--require <CAPABILITY>
Refuse the run unless the sandbox, as configured, offers these capabilities (comma-separated or repeated), e.g. network_block,resource_limit. `hemra doctor` shows what each sandbox offers
--verifier-network <NETWORK>
Network for task verifiers, such as `bridge`, for verifiers that call models (default: the run file's `[verifier] network`, else `none`)
--verifier-env <NAME>
A host variable the task's `[verifier.env]` templates may read, such as a judge model's API key. Repeatable; adds to the run file's `[verifier] env`. Names are recorded, values never are
-J, --judge <NAME>
A HEMRA judge (`judges/<name>/`) that scores every queued trial beside the task's own verifier, as a score of its own. Repeatable; adds to the run file's `[[judge]]`
--judges <DIR>
Directory of judges (default: `judges/` beside the harness directory)
--id <ID>
Run ID (default: `<label>-<8 hex digits>`). An ID is used once
--detach
Record and queue the run, then return without working the queue
--json
Print the report as JSON
--harbor <HARBOR>
Harbor executable, for `via = "harbor"` harnesses [env: HEMRA_HARBOR=] [default: harbor]
--allow-untested-version
Run a Harbor version this adapter has not been tested against
--docker-bin <DOCKER_BIN>
The docker CLI to use [env: HEMRA_DOCKER=] [default: docker]
--harnesses <HARNESSES_DIR>
Directory of harness files (default: `harnesses/` next to the run file, else `--harness-dir`)
--harness-dir <DIR>
Directory of harness files; `--harness NAME` reads `<dir>/NAME.toml` [env: HEMRA_HARNESS_DIR=] [default: harnesses]
--allow-host-auto-approve
Allow a harness with `permission = "auto"` to approve every agent action on this host (`--sandbox local` only)
--driver-bin <DRIVER_BIN>
The driver program: for `cli` harnesses the `hemra-driver` binary (default: next to `hemra`, then `PATH`), for `sdk:<agent>` the built driver, e.g. `packages/drivers/claude-agent-sdk/dist/main.js` (default: `hemra-driver-<agent>` on `PATH`) [env: HEMRA_DRIVER_BIN=]
--sandbox-bin <SANDBOX_BIN>
The bridge to a managed sandbox, e.g. `packages/sandboxes/modal/dist/main.js` (default: `$HEMRA_SANDBOX_MODAL_BIN`, `$HEMRA_SANDBOX_FLY_BIN` or `$HEMRA_SANDBOX_SPRITES_BIN`, else `hemra-sandbox-<kind>` on `PATH`)
-h, --help
Print helphemra harness
Create, list, show and check harness files: the agents under test (RFC-0008 §5)
Usage: hemra harness [OPTIONS] <COMMAND>
Commands:
add Write a new harness file from a template: `<dir>/<name>.toml`
list List harness files, with any that do not load and why
show Show one harness as loaded: digests, referenced files, variable names
check Run one trivial task with the harness and report the agent version it observed against the pinned one. `harbor` harnesses are checked statically and against the installed Harbor
help Print this message or the help of the given subcommand(s)
Options:
--harness-dir <DIR> Directory of harness files; `--harness NAME` reads `<dir>/NAME.toml` [env: HEMRA_HARNESS_DIR=] [default: harnesses]
-h, --help Print helphemra harness add
Write a new harness file from a template: `<dir>/<name>.toml`
Usage: hemra harness add [OPTIONS] --via <VIA> <NAME>
Arguments:
<NAME>
Options:
--harness-dir <DIR> Directory of harness files; `--harness NAME` reads `<dir>/NAME.toml` [env: HEMRA_HARNESS_DIR=] [default: harnesses]
--via <VIA> How the harness runs
--agent <AGENT> The agent, for `cli` (claude-code, codex or pi), `sdk` and `harbor` (default: claude-code, or claude-agent-sdk for `sdk`)
--force Replace an existing file
-h, --help Print helphemra harness list
List harness files, with any that do not load and why
Usage: hemra harness list [OPTIONS]
Options:
--harness-dir <DIR> Directory of harness files; `--harness NAME` reads `<dir>/NAME.toml` [env: HEMRA_HARNESS_DIR=] [default: harnesses]
--json
-h, --help Print helphemra harness show
Show one harness as loaded: digests, referenced files, variable names
Usage: hemra harness show [OPTIONS] <HARNESS>
Arguments:
<HARNESS> A name in the harness directory, or a path to a harness file
Options:
--harness-dir <DIR> Directory of harness files; `--harness NAME` reads `<dir>/NAME.toml` [env: HEMRA_HARNESS_DIR=] [default: harnesses]
--json
-h, --help Print helphemra harness check
Run one trivial task with the harness and report the agent version it observed against the pinned one. `harbor` harnesses are checked statically and against the installed Harbor
Usage: hemra harness check [OPTIONS] <HARNESS>
Arguments:
<HARNESS> A name in the harness directory, or a path to a harness file
Options:
--harness-dir <DIR>
Directory of harness files; `--harness NAME` reads `<dir>/NAME.toml` [env: HEMRA_HARNESS_DIR=] [default: harnesses]
--task <TASK>
Task JSON to run instead of the built-in one-line task
--store <STORE>
Record the attempt in this store [env: HEMRA_STORE=]
--timeout-secs <TIMEOUT_SECS>
[default: 300]
--allow-host-auto-approve
Allow a harness with `permission = "auto"` to approve every agent action on this host
--driver-bin <DRIVER_BIN>
The driver program (see `hemra run --help`) [env: HEMRA_DRIVER_BIN=]
--harbor <HARBOR>
Harbor executable, for `via = "harbor"` [env: HEMRA_HARBOR=] [default: harbor]
--docker-image <IMAGE>
Run the harness in this Docker image. The harness program and arguments are then paths inside the container; the workspace is mounted at /hemra/workspace and the artifact directory at /hemra/artifacts
--docker-pull <DOCKER_PULL>
When to pull the image [default: missing] [possible values: missing, always, never]
--docker-network <DOCKER_NETWORK>
Container network. The default `none` blocks all network access [default: none]
--docker-memory <DOCKER_MEMORY>
Memory limit, in bytes or with a k/m/g suffix. `0` removes the limit
--docker-cpus <DOCKER_CPUS>
CPU limit (fractions allowed). `0` removes the limit
--docker-pids <DOCKER_PIDS>
Process limit inside the container. `0` removes the limit
--docker-user <DOCKER_USER>
`--user` for the container (default: your own uid:gid)
--docker-read-only
Make the image's root filesystem read-only (/tmp stays writable)
--docker-mount <DOCKER_MOUNT>
Extra bind mount HOST:CONTAINER[:ro|rw], read-only unless `:rw` is given
--docker-bin <DOCKER_BIN>
The docker CLI to use [env: HEMRA_DOCKER=] [default: docker]
-h, --help
Print helphemra judge
Create, list, show and check HEMRA judges: general judges a run applies to any dataset, apart from a dataset's own (RFC-0009 §4.3)
Usage: hemra judge [OPTIONS] <COMMAND>
Commands:
add Write a new judge from a template: `<dir>/<name>/judge.toml` and a starting rubric for each thing it reads
list List judges, with any that do not load and why
show Show one judge as loaded: what it reads, its models, rubric and digest
check Run the judge once on a small built-in attempt, through its gateway and the judge runtime (needs Docker and the judge's key), and print what it scored. Calls the model a few times
help Print this message or the help of the given subcommand(s)
Options:
--judge-dir <DIR> Directory of judges; `--judge NAME` reads `<dir>/NAME/judge.toml` [env: HEMRA_JUDGE_DIR=] [default: judges]
-h, --help Print helphemra judge add
Write a new judge from a template: `<dir>/<name>/judge.toml` and a starting rubric for each thing it reads
Usage: hemra judge add [OPTIONS] --reads <READS> <NAME>
Arguments:
<NAME>
Options:
--judge-dir <DIR> Directory of judges; `--judge NAME` reads `<dir>/NAME/judge.toml` [env: HEMRA_JUDGE_DIR=] [default: judges]
--reads <READS> What it reads, comma-separated: output (the agent's final answer), artifacts, trajectory (steps and tool calls), workspace (the files it left)
--provider <PROVIDER> The model provider its gateway reaches [default: openai]
--model <MODEL> The pinned judge model, `<provider>/<model>` (default: openai/gpt-5.5 or anthropic/claude-sonnet-5-5)
--force Replace an existing judge's files
-h, --help Print helphemra judge list
List judges, with any that do not load and why
Usage: hemra judge list [OPTIONS]
Options:
--json
--judge-dir <DIR> Directory of judges; `--judge NAME` reads `<dir>/NAME/judge.toml` [env: HEMRA_JUDGE_DIR=] [default: judges]
-h, --help Print helphemra judge show
Show one judge as loaded: what it reads, its models, rubric and digest
Usage: hemra judge show [OPTIONS] <JUDGE>
Arguments:
<JUDGE> A name in the judge directory, or a path to a judge directory
Options:
--json
--judge-dir <DIR> Directory of judges; `--judge NAME` reads `<dir>/NAME/judge.toml` [env: HEMRA_JUDGE_DIR=] [default: judges]
-h, --help Print helphemra judge check
Run the judge once on a small built-in attempt, through its gateway and the judge runtime (needs Docker and the judge's key), and print what it scored. Calls the model a few times
Usage: hemra judge check [OPTIONS] <JUDGE>
Arguments:
<JUDGE>
Options:
--docker-bin <DOCKER_BIN> The docker CLI [env: HEMRA_DOCKER=] [default: docker]
--judge-dir <DIR> Directory of judges; `--judge NAME` reads `<dir>/NAME/judge.toml` [env: HEMRA_JUDGE_DIR=] [default: judges]
--json
-h, --help Print helphemra dataset
Pin datasets in the store by digest, and list, show or remove them
Usage: hemra dataset [OPTIONS] <COMMAND>
Commands:
add Copy a set of tasks into the store and pin it by digest
list List pinned datasets, newest version first within a name
show Show one pinned dataset and its tasks
rm Remove one pinned version, or every version of a name
help Print this message or the help of the given subcommand(s)
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
-h, --help Print helphemra dataset add
Copy a set of tasks into the store and pin it by digest
Usage: hemra dataset add [OPTIONS] <SOURCE>
Arguments:
<SOURCE> A directory of Harbor-format tasks or one task directory; `harbor:<name>[@<version>]`, downloaded from the Harbor registry; or `swebench:<dataset>[@<split>]` (verified, lite, full, multimodal, a Hugging Face name or a local .json/.jsonl/.parquet file), converted to tasks; or `hf:<org>/<repo>[@<revision>][/<subdir>]`, Harbor-format tasks in a Hugging Face dataset repository, pinned at the revision's commit (`HF_TOKEN` for gated repositories, `HF_ENDPOINT` for a mirror)
Options:
--name <NAME> Name to add it under (default: the directory's, registry dataset's or repository's name, or `swebench-<dataset>`)
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
--harbor <HARBOR> Harbor executable, for `harbor:` sources [env: HEMRA_HARBOR=] [default: harbor]
--allow-untested-version Download with a Harbor version this adapter has not been tested against
--python <PYTHON> Python with `swebench` installed, for `swebench:` sources other than a local .json or .jsonl file [env: HEMRA_SWEBENCH_PYTHON=] [default: python3]
-h, --help Print helphemra dataset list
List pinned datasets, newest version first within a name
Usage: hemra dataset list [OPTIONS]
Options:
--json
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
-h, --help Print helphemra dataset show
Show one pinned dataset and its tasks
Usage: hemra dataset show [OPTIONS] <DATASET>
Arguments:
<DATASET> `name` (newest version) or `name@<digest prefix>`
Options:
--json
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
-h, --help Print helphemra dataset rm
Remove one pinned version, or every version of a name
Usage: hemra dataset rm [OPTIONS] <DATASET>
Arguments:
<DATASET> `name` (every version) or `name@<digest prefix>`
Options:
--store <STORE> Store directory [env: HEMRA_STORE=] [default: .hemra]
-h, --help Print help