Trust and privacy

What HEMRA v0.1 trusts, what it writes to disk, how it handles credentials, how agent runs are isolated, and what leaves the machine. This is the "documented trust assumptions, privacy boundaries and operational limits" item of RFC-0001 MVP 5. It describes the code on main, not the RFCs' intended end state; where the two differ, the gap is listed under Known gaps.

Summary

• HEMRA is a single-operator, local tool. It trusts the operator's configuration and the benchmark's official scorer. It does not trust harness output, artifacts or agent-generated code, and it is not a sandbox for hostile code.
• HEMRA itself makes no network calls and sends no telemetry. Anything that leaves the machine is sent by a harness, by Docker, or by Harbor, under the operator's control.
• The store keeps task inputs, every protocol event, the stderr tail, artifact bytes and Harbor's job files, unredacted. Treat a store and any export from it as being as sensitive as the most sensitive prompt, tool output or file a run touched.
• Environment variable values are never persisted; only their names. But a harness can still leak a credential into an event, its stderr or an artifact, and HEMRA does not redact.

Trust model

Out of scope for v0.1 (RFC-0001 non-goals, RFC-0003 §3): multi-tenant or public runners, and hostile code that tries to escape Docker. RFC-0004 and RFC-0006 cover those later and require VM-grade isolation first.

What HEMRA stores

The store (--store, default .hemra, or $HEMRA_STORE)

Layout from crates/hemra-storage/src/lib.rs:

A store directory that HEMRA creates is owner-only (0700); an existing directory keeps the permissions the operator gave it. Blobs are written read-only. Nothing is encrypted at rest.

Attempts are append-only: there is no command that edits or deletes a stored attempt, event log or blob, and no retention or garbage collection yet. Deleting data means deleting files from the store by hand (see Operating guidance).

Outside the store

• hemra harness check runs its trial in a throwaway $TMPDIR/hemra-check-* workspace, which is removed when the check ends. With --store the attempt's events and artifacts are kept in the store; without it, nothing of the trial is kept.
• hemra run prints its report to stdout (--json for everything). Where that goes is up to the operator's shell.
• hemra export writes a self-contained copy of a run, an attempt (manifest, record, events.jsonl, artifact blobs) or of an imported Harbor job (import.json and Harbor's raw files) to a directory. Exports are not redacted; they contain the same sensitive data as the store.

Credentials

What the code does:

• Local harnesses start with an empty environment plus PATH and the variables the harness file names in env (local_command in hemra-engine/src/runner.rs). Host variables such as ANTHROPIC_API_KEY or AWS_* are not inherited unless passed explicitly.
• Only variable names are persisted. The manifest records env_keys; values never reach SQLite, the event log or exports (test: docker_manifest_pins_the_image_and_keeps_no_env_values).
• Docker harnesses receive values by name, not argv. The docker client runs with a cleared environment except the variables it needs (PATH, HOME, USER, TMPDIR, XDG_CONFIG_HOME, DOCKER_*) plus the harness entries, and the argv carries --env NAME only, so values do not appear in ps output for the docker process or in the recorded container settings. Names the client itself reads are refused as harness variables.
• MCP servers start with an empty environment too (ADR-0026): PATH plus the names in their [[mcp]] env, read from hemra's environment; an unset name stops the run before anything starts. Manifests keep the names only.
• **run.start carries no secrets** by contract (RFC-0002 §13): it holds the task, the run context and runtime paths.
• Harbor runs with the operator's full environment. hemra run does not clear the environment of the harbor process it starts for a via = "harbor" harness, because Harbor's agents read their model API keys from it (for example ANTHROPIC_API_KEY for anthropic/... models). HEMRA does not write any key into hemra-config.json.

What it does not do:

• No redaction. If a harness echoes a key into a log event, a tool result, stderr or an artifact, HEMRA stores it as-is. RFC-0002 §13 treats redaction as defense in depth that is not yet built.
• Agent kwargs are persisted. A harness file's kwargs go into hemra-config.json and into the run spec in the store, which are kept. Never put a key there; use the environment Harbor inherits.
• No secret store. There is no keychain, vault or secret-mount integration yet.

Isolation of agent runs

Local provider (default)

The harness is a child process in its own process group, running as the operator's user, in the operator's file system and network namespace. HEMRA gives it:

• a cleared environment (above) and a working directory set to the trial workspace;
• protocol limits: 1 MiB per line, 64 MiB of events, a 15-minute default deadline, a 30-second handshake timeout, and escalation from run.cancel to SIGTERM to SIGKILL on the whole group;
• artifact checks: only regular files under the artifact directory, no symlink escapes, quotas, runner-computed SHA-256.

It does not restrict file access, network access or system calls. A local harness can read ~/.ssh, ~/.aws, browser profiles or anything else the operator can read, and can talk to any host. Use the local provider only for harnesses and agents you trust as much as your own shell.

Docker provider (the hemra run default, or --docker-image)

hemra run runs program harnesses in the Docker sandbox unless given --sandbox local, with the image from the run file's [sandbox] table; hemra harness check --docker-image does the same for one trial. The same protocol session runs inside docker run -i with the RFC-0003 §12 defaults (hemra-engine/src/docker.rs, ADR-0006):

Each loosening (network, memory = "0", :rw mounts in [sandbox], or --docker-network, --docker-memory 0, --docker-user on harness check) is an explicit operator setting and is recorded in the manifest, so a stored attempt shows the isolation it actually ran with.

Docker shares the host kernel. It is a reasonable boundary against accidents and against agents doing something unexpected, not against code deliberately trying to escape (RFC-0001 invariants, RFC-0004 §12). A harness that needs a model API must be given network access (for example network = "bridge"), and from then on it can reach any host, not just the model provider.

Modal sandbox ([sandbox] kind = "modal", experimental)

hemra run can run program harnesses in a Modal Sandbox (ADR-0044) through the hemra-sandbox-modal bridge. The harness runs as root inside a gVisor sandbox on Modal's machines, with no network unless network = "open", and CPU and memory limits. What leaves the machine: the trial's workspace, every [sandbox] mounts directory (copied, regular files only), the task input in run.start, and the values of the harness's environment variables, which Modal holds for the exec. Artifacts come back as regular files only, checked by name and within the trial's limits, before the runner verifies them. Modal's credentials (MODAL_TOKEN_ID, MODAL_TOKEN_SECRET or ~/.modal.toml) are passed to the bridge only, never to the harness, and a harness variable with a Modal or bridge name is refused. A sandbox HEMRA could not stop is stopped by Modal when its lifetime (the trial's timeout plus 15 minutes) runs out.

Fly.io Machine ([sandbox] kind = "fly", experimental)

Program harnesses can also run in a Fly Machine (ADR-0045): a Firecracker microVM in the Fly app named by [sandbox] app, running as root inside it, always with outbound network (Fly cannot block it, so the run file must say network = "open" and --require network_block refuses). The same things leave the machine as for Modal, sent to Fly: the workspace, mounts directories, task input and the harness's variable values. The values travel in a script written into the Machine over SSH and deleted as it starts, never in a command line. FLY_API_TOKEN is passed to the bridge and flyctl only, never to the harness. Use a deploy token scoped to the sandbox app (fly tokens create deploy -a <app>), not an organization token. A Machine HEMRA could not destroy destroys itself when its lifetime runs out.

Fly Sprite ([sandbox] kind = "sprites", experimental)

Program harnesses can also run in a Fly Sprite (ADR-0047): a Firecracker microVM from Fly's own base image, running as the Sprite's unprivileged user. Its network policy is set before anything is copied in. By default it denies all outbound traffic. With [sandbox] egress = ["github.com", ...] it allows only those domains, and network = "open" sets no policy. The same things leave the machine as for Fly Machines: the workspace, mounts directories, task input and the harness's variable values. The values are written in a script through the Sprites filesystem API and deleted as the harness starts, never put in a URL or a command line. SPRITES_TOKEN is passed to the bridge only, never to the harness. A Sprite has no lifetime. One that HEMRA could not delete (HEMRA killed mid-trial) stays asleep with its disk, including the workspace copy, until you delete it. Every such Sprite is named hemra-... and labelled hemra-owned.

MCP servers ([[mcp]] in a harness file)

HEMRA runs each declared MCP server itself, on the runner's host, outside any sandbox, and the harness reaches it only through a Unix socket in a private temporary directory (mounted read-only into a Docker container) (ADR-0026). The server has the host access of the user running hemra, so an agent in a --network none container can still act on the network or the filesystem through the tools it was given. List only servers you would run yourself. A server's arguments and results are recorded like any tool call.

Task verifiers (driver harnesses in the Docker sandbox)

The task's verifier runs in its own container on a snapshot of the agent's, with the same hardening, --network none and only the literal values of the task's [verifier.env] (ADR-0012, ADR-0019). Verifiers that call models (LLM judges) need the operator to open them per run: [verifier] network and env in the run file, or --verifier-network and --verifier-env NAME (ADR-0036). Only the named host variables are read, their values reach the container as --env NAME through the docker client's environment, and the store records the names, never the values. Opening the network lets the task's test code reach any host with the keys you named, so do that only for datasets whose tests you would run yourself. A multi-step task runs each step in a container started from the snapshot of the step before it. When a verifier grades with a rubric (Harbor's rewardkit), its reward-details.json is kept with the trial's files, and the store keeps each criterion's score, the judges' model names and their reasoning, cut to 600 characters (ADR-0042). That reasoning is model output about the agent's work: it can quote the work, and it is shown as text only.

A HEMRA judge (judges/<name>/, applied with [[judge]] or --judge, ADR-0043) runs after the task's verifier, in its own container from HEMRA's judge runtime image, with --network none, a read-only root and the same hardening. It sees only what its judge.toml reads (the agent's last message, its artifacts, a trajectory built from its events, a copy of its working directory) and the instruction. It reaches its model only through HEMRA's model gateway, a Unix socket served by the host that forwards only the provider's model-call paths to the judge's base_url, adds the key there, and refuses calls beyond max_calls. The key is never in the container, the store or the transcripts. Each call's request and response bodies (the agent's work, the rubric, the judge's answer) are kept with the trial's files under trials/<attempt>/judges/<name>/calls/, and the score records each call's model, status, duration and tokens.

Requiring isolation (--require)

Each provider reports the isolation it enforces as configured (ADR-0013): Docker offers network_block only with --network none and resource_limit only with memory, CPU and PID limits all set, and the local provider offers neither. hemra run --require network_block,resource_limit refuses to start, recording nothing, when the run's sandbox does not offer every flag, and a stored manifest lists the flags that were required. hemra doctor --json prints each provider's capabilities and a one-line statement of its isolation.

Harbor (hemra run with via = "harbor" harnesses)

HEMRA supervises the harbor CLI as an ordinary local process: its own process group, a timeout, SIGINT/SIGTERM/SIGKILL escalation on cancel. Isolation of the agent and verifier is Harbor's, configured through Harbor's own environment settings; HEMRA does not add Docker flags to Harbor's containers. HEMRA retains Harbor's output without following symlinks, since the job directory can hold agent-written files.

What is sent off-machine

HEMRA's own crates send nothing on their own: no telemetry, crash reporting or update checks. The one exception is the signed desktop app's update check (below). Off-machine traffic comes only from the components below, each started by an operator command or an opt-in setting:

hemra import, hemra dataset (except dataset add harbor: and swebench:, which download through Harbor or the datasets library, and hf:, which downloads from HF_ENDPOINT or huggingface.co and its CDN, sending HF_TOKEN to the endpoint only and never recording it), hemra runs, hemra show, hemra export, hemra cancel, hemra compare and hemra doctor work entirely on local files, the local Docker daemon and --version probes of local tools. hemra serve sends nothing out itself, though the queued attempts it runs reach whatever their harness reaches; it listens on 127.0.0.1:7878 by default, has no authentication, and refuses a non-loopback address unless given --expose (ADR-0010). Anyone who can reach it can read the whole store, including transcripts and artifacts.

Operating guidance

• Keep stores private. New stores are created 0700; if you point --store at a directory that already exists, chmod 700 it yourself. Do not commit a store or export to a repository.
• Run harnesses you did not write in the Docker sandbox (the hemra run default, not --sandbox local), and keep network = "none" unless the harness needs a model API.
• Pass credentials only by name, in a harness file's env, never in task input, harness arguments, agent kwargs or Harbor config files: those are persisted.
• Use dedicated, low-limit API keys for evaluation runs; assume anything given to a harness can end up in its transcript.
• Before sharing an export, read events.jsonl and the artifacts. There is no redacted export yet.
• To delete one attempt by hand, remove its row from attempts, attempt_transitions and artifacts, its events/<attempt>.jsonl, and any blob no other row references. Deleting the whole store directory is the reliable way to delete everything.
• Runs without --store leave $TMPDIR/hemra-* behind on purpose; delete them once you have what you need, or pass --work-dir somewhere you manage.

Known gaps

Tracked against the RFCs; none of these are built yet.

Edit this page on GitHub