@hemra/sdk · TypeScript

Your agent loop, measured like any other.

Wrap your agent in a few lines and HEMRA can run it, record it and compare it with Claude Code, Codex or anything else. The SDK speaks the harness protocol for you: framing, the handshake, event order, limits, cancellation, artifacts and the terminal message.

Get started SDK on GitHub

Node.js 22 or later. Not on npm yet: use it from the HEMRA repository's workspace.

my-harness.ts

import { defineHarness, main } from "@hemra/sdk";

const harness = defineHarness({
  id: "my-agent",
  version: "0.1.0",
  async run(ctx) {
    ctx.log("info", "agent started");
    ctx.emit({ kind: "tool.call", callId: "c1",
               data: { tool: "shell" } });
    const answer = await runAgent(ctx.task.input,
                                   { signal: ctx.signal });
    ctx.emit({ kind: "tool.result", callId: "c1",
               data: { exit_code: 0 } });
    return { answer };
  },
});

await main(harness);

What the SDK does for you

You write run(ctx). Everything the runner checks is handled, and checked again in your tests with the runner's own rules.

Protocol, handled

stdin and stdout framing, the initialize → ready → run.start handshake, sequence numbers and timestamps, and exactly one terminal message.

Limits enforced early

emit throws ProtocolViolation before a line would exceed max_line_bytes or the per-trial event quota, instead of the runner failing the trial later.

Cancellation as a signal

ctx.signal aborts on a timeout, a user cancel or a closed stdin. Pass it to anything slow and the trial ends as cancelled, cleanly.

Artifacts, safely

artifactPath gives a safe path inside the artifact directory; artifact checks the path, hashes the file and sends artifact.created. The runner still computes its own digest.

stdout stays clean

Under main, anything written to stdout, console.log included, goes to stderr, because stdout carries only protocol frames.

Same rules as Rust

The frame parser and SessionValidator are a port of the Rust hemra-protocol crate, tested against the same golden transcripts, so the two cannot drift.

From file to compared run

Three steps: the harness above, a harness file next to it, and the runner.

1. Describe it in a harness file

A harness is one TOML file. via = "program" means it speaks the protocol itself.

# my-harness.toml
via = "program"
command = ["./dist/my-harness.js"]   # relative to this file

2. Check it, then run it

check runs one trivial task and compares the version it reports with the pinned one.

npm ci && npm run build
hemra harness check my-harness.toml --task task.json
hemra run -H my-harness --harness-dir . \
  -d <pinned dataset> --sandbox local
hemra compare <run>

The context

Everything a harness gets from the runner, in ctx.

Member	What it is
`task`	`{ id, input, metadata? }` from `run.start`
`experimentId`, `attemptId`, `context`	Non-secret run context
`artifactDirectory`, `workspaceDirectory`	Absolute directories the runner controls
`limits`, `runnerCapabilities`	From `initialize`
`signal`	Aborted on cancel, timeout or closed stdin
`emit({ kind, data, callId?, spanId?, parentSpanId? })`	Writes one event
`log(level, message, data?)`	Shorthand for a `log` event
`artifactPath(rel)`, `artifact(rel, mediaType)`	Write and register an artifact; returns its ID

Ending a trial

Return a value: run.completed with it as the output.
Return result(output, { summary, finalArtifacts }) to fill the optional fields.
Throw HarnessFailure(code, message, { retryable }): run.failed with your code.
Throw anything else: run.failed as a harness error, or as cancelled if the signal was aborted.

Completed is not passed. run.completed means your harness finished. Whether the task passed is decided by the task's verifier or the benchmark's scorer, never by the harness.

Test it without a runner

runHarnessInMemory drives your harness through a full session and checks every frame against the runner's rules.

import { runHarnessInMemory } from "@hemra/sdk/testing";

const run = await runHarnessInMemory(harness, {
  task: { id: "t1", input: { q: "hi" } },
});
assert.equal(run.violation, undefined);
assert.equal(run.exit.terminal, "run.completed");

Options cover limits, protocol versions, runner capabilities and cancelAfterEvents for cancellation.

Not TypeScript? Speak the protocol

The harness protocol is versioned JSON lines on stdin and stdout, in any language. These three lines are a complete, valid harness side of a trial, taken from the golden transcripts both implementations are tested against.

Read the protocol (RFC-0002)

{"protocol":"1.0","trial_id":"trial-001","type":"ready","message_id":"h-ready",
 "selected_protocol":"1.0","harness":{"id":"fixture","version":"0.1.0"},"capabilities":["cancel"]}
{"protocol":"1.0","trial_id":"trial-001","type":"event","message_id":"h-e1","seq":1,
 "timestamp":"2026-10-14T10:00:01Z","event":{"kind":"log","data":{"level":"info","message":"step 1"}}}
{"protocol":"1.0","trial_id":"trial-001","type":"run.completed","message_id":"h-done",
 "output":{"summary":"ok"}}