Harnesses

A harness is the agent under test, described by one TOML file. Its via says how it runs. hemra run -H <name> reads harnesses/<name>.toml, or the file of that name in --harness-dir.

hemra harness add my-agent --via cli --agent claude-code   # writes harnesses/my-agent.toml
hemra harness list
hemra harness show my-agent      # as loaded: digests, referenced files, variable names
hemra harness check my-agent     # one trivial task, observed version against the pinned one

Kinds of harness

`via`	Runs	Agents tested with it
`program`	Any program that speaks the harness protocol itself	Your own, in any language; `@hemra/sdk` for TypeScript
`cli`	An agent's command line, through `hemra-driver`	Claude Code, Codex, Pi
`sdk`	An agent SDK, through a TypeScript driver	Claude Agent SDK, Codex SDK
`acp`	Any Agent Client Protocol agent, through `hemra-driver`	Claude Code's and Codex's ACP adapters, Gemini CLI, Goose, OpenCode
`computer-use`	An agent product through its web UI, in Chromium	ChatGPT agent mode
`harbor`	A Harbor agent, as one supervised Harbor job per dataset	Any Harbor agent, such as `oracle`, `nop` or `terminus-2`

Every field of a harness file is recorded with the run, and its digest is part of the run's spec. Harness files name environment variables, never their values.

Your own program

via = "program"
command = ["./echo.sh"]      # relative to this file; in a container, a path inside it
version = "0.1.0"            # what the harness reports in `ready`
env = ["MY_MODEL_KEY"]       # passed through by name

The harness's environment is cleared except PATH and the variables the file names. To write one, read the harness protocol or start from the TypeScript SDK and its example.

Agents through a driver

A cli, sdk or acp file is an agent profile, handed to the driver unchanged. For Claude Code:

via = "cli"
agent = "claude-code"
version = "2.1.302"          # expected; the observed version is what gets recorded
model = "claude-sonnet-4-5"
max_turns = 20
permission = "allowlist"     # deny | allowlist | auto
allowed_tools = ["Bash", "Read", "Write", "Edit"]
env = ["ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL"]

The common fields:

Field	Meaning
`agent`	The agent: `claude-code`, `codex` or `pi` for `cli`; `claude-agent-sdk` or `codex-sdk` for `sdk`; the agent's own name for `acp`
`version`	The version you expect. The version the agent reports is recorded, and a difference is flagged
`model`	The model. Pin it: a run can require every harness to use the same one
`permission`	`deny` refuses every tool request, `allowlist` allows `allowed_tools` only, `auto` approves everything (on this host only with `--allow-host-auto-approve`)
`allowed_tools`	Tool names, or ACP tool kinds such as `execute`, `edit` and `read`
`max_turns`	A turn limit, where the agent has one (not Codex, not ACP agents)
`env`	Variables passed to the agent, by name
`auth`	`operator` uses your own login and configuration instead of an isolated `HOME`
`capture`	`metadata` keeps text and tool payloads out of the event log
`[files]`	Files copied into the agent's isolated `HOME`, relative to the harness file
`bin`, `extra_args`	For `acp`: the agent program and its arguments, such as `bin = "gemini"` and `extra_args = ["--acp"]`

Each agent runs with its own HOME, so your own configuration and logins are not used unless auth = "operator" says so. The driver reports what the agent did as events and keeps its raw output as the raw-transcript artifact. It never turns the agent's own claim of success into a score.

The repository has a working file for every agent in examples/: driver-claude-code, driver-codex, driver-pi, driver-claude-agent-sdk, driver-codex-sdk, driver-acp and driver-computer-use. ACP agents that approve their own tools by default (Goose and OpenCode) need the configuration file their example copies in, so that permission decides.

The sdk drivers are built from the repository with Node.js 22 (npm ci && npm run build), and found with --driver-bin, for example --driver-bin packages/drivers/claude-agent-sdk/dist/main.js.

Agent products through their UI

A computer-use harness operates a product's web UI with a scripted recipe, in a browser profile you signed in to. It types only the task's instruction and the fixed replies in [ui], and only runs tasks marked metadata.shareable = true. A login wall, a broken recipe or a product outage is an infrastructure error, never a failed task.

Harbor agents

via = "harbor"
agent = "terminus-2"
model = "openai/gpt-5.5"
version = "2.0.0"            # the agent version Harbor installs; an unpinned one is flagged
# configs = ["agent.yaml"]   # Harbor config files, relative to this file
# [kwargs]                   # extra agent arguments, recorded with the run: no secrets

A run gives each dataset to Harbor as one supervised job, keeps Harbor's job directory byte for byte, and imports its official rewards. Harbor is pinned: HEMRA is tested against Harbor 0.24.

MCP servers

A harness file can list MCP servers. The agent reaches them only through HEMRA's proxy, which starts each server beside the runner, probes its tools before the attempt and records every tools/call in the attempt's event log.

[[mcp]]
name = "docs"
command = ["./docs-server.py"]   # relative to this file
version = "1.2.0"                # expected serverInfo.version
env = ["DOCS_TOKEN"]

Allow its tools like any other: allowed_tools = ["mcp__docs"] allows every tool of the docs server. A server that fails its probe fails the attempt before the agent starts, and two runs with different tool sets are never paired silently. computer-use harnesses take no MCP servers.

Edit this page on GitHub