Datasets

A dataset is a set of tasks in Harbor's task format, copied into the store and pinned by digest. A run names datasets by name and records the full digest it used, so you always know which task bytes a result came from, even if the source changes later.

hemra dataset add <source> [--name NAME]
hemra dataset list
hemra dataset show NAME            # its digest and tasks
hemra dataset rm NAME[@DIGEST]     # one version, or every version of a name

Sources

Source	Example	What is pinned
A local directory	`hemra dataset add examples/tasks --name examples`	One task directory, or a directory whose subdirectories are tasks
The Harbor registry	`hemra dataset add harbor:terminal-bench@2.0`	The registry dataset, downloaded with Harbor
SWE-bench	`hemra dataset add swebench:verified`	SWE-bench instances converted to tasks (also `lite`, `full`, `multimodal`, a Hugging Face name, or a local `.json`, `.jsonl` or `.parquet` file)
Hugging Face	`hemra dataset add hf:harborframework/terminal-bench-2.1/tasks`	Harbor-format tasks in a Hugging Face dataset repository, at one commit

A hf: source is hf:<org>/<repo>[@<revision>][/<subdirectory>]. HEMRA resolves the revision to a commit, downloads only at that commit, and checks every file against the hub's own hashes before pinning. Set HF_TOKEN for gated repositories, and HF_ENDPOINT for a mirror.

A task

A task is a directory with task.toml and instruction.md, as Harbor lays it out. Most also have environment/Dockerfile, the image the agent works in, and tests/test.sh, the verifier that scores the attempt and writes a reward.

hello-file/
  task.toml          name, version, timeouts, resources, metadata
  instruction.md     what the agent is asked to do
  environment/       the task's image (Dockerfile)
  tests/test.sh      the verifier: writes the reward

A task can have several steps ([[steps]] in task.toml). Each step is its own session in the same sandbox, started from the previous step's snapshot and scored by its own verifier, and the rewards combine by the task's strategy.

A swebench: task has no tests/: it is graded by the official SWE-bench harness on the agent's patch.

Versions and digests

Adding the same tasks again under the same name reports the existing pin. Adding changed tasks pins a second version.
name means the most recently added version, and name@<digest prefix> exactly one.
A pinned version is never changed in place. Symlinks and special files are refused, so a pinned dataset cannot point outside the store.
Pick tasks within a run with -t/--task and -x/--exclude patterns, where * matches anything.

Edit this page on GitHub