Judges

Some tasks are graded by a model, not a script. HEMRA has two kinds of model judge, and keeps them apart.

A judge explains or adds a score. It never changes the task's own score, and a judge that fails is an evaluation failure, never a failed task.

Dataset judges

A task's verifier can grade with Harbor's rewardkit rubrics. HEMRA reads the rubric's details (reward-details.json) into the score: each judge model, and every criterion's value, weight, reasoning and panel votes. The reward itself still comes only from reward.json.

hemra compare strong/judged/claude-code weak/judged/claude-code --criteria

lists each criterion both sides were judged on, b - a. Trials judged by different models never pair. A verifier that calls a model needs a network and its key, which the run must allow: --verifier-network bridge --verifier-env OPENAI_API_KEY (see Sandboxes).

HEMRA judges

A HEMRA judge is a directory with a judge.toml and a rewardkit rubric. hemra judge add writes one:

hemra judge add quality --reads output,trajectory --provider openai --model openai/gpt-5.5
hemra judge list
hemra judge show quality
hemra judge check quality       # judges a built-in sample attempt (needs Docker and the key)

# judges/quality/judge.toml
description = "Is the final answer correct and well argued?"
version = "1"
reads = ["output", "trajectory"]   # output | artifacts | trajectory | workspace
max_calls = 200                     # model calls per trial
timeout_sec = 600

[model]
provider = "openai"                 # openai | anthropic
key = "OPENAI_API_KEY"              # a name, never a value
base_url = "https://api.openai.com/v1"

reads decides what the judge can see: the agent's last message (output), its verified artifacts, its trajectory (steps and tool calls, from the event log), or the files it left in its workspace. The rubric's criteria name pinned <provider>/<model> judges.

Apply judges to a run with --judge quality (repeatable) or in the run file:

[[judge]]
name = "quality"
datasets = ["legal"]   # default: every dataset in the run

After the task's verifier has scored a trial, each judge runs in its own container with no network, seeing only what reads declares. It reaches its model through a gateway on the host that holds the key, allows only the provider's model endpoints, limits the number of calls, and records every call. Compare cells by one judge's scores with:

hemra compare strong/judged/claude-code weak/judged/claude-code --judge quality

The run report shows HEMRA judges in a section of their own, and the dashboard labels them apart from dataset judges. Judge reasoning is shown as text only.

Edit this page on GitHub