HEMRARemote workers
hemra serve --workers-listen leases the store's queue to hemra worker daemons on other machines, over HTTPS on a listener of its own. Each worker runs leased attempts with the same runner and streams their events and artifacts back. The serving process is the only one that writes the store.
export HEMRA_WORKER_TOKEN=... # the shared enrollment secret, never an argument
hemra serve --concurrency 0 --workers-listen 0.0.0.0:7879 --tls-cert cert.pem --tls-key key.pem
# on each worker machine
export HEMRA_WORKER_TOKEN=...
hemra worker --controller https://controller.internal:7879 --id gpu-1 --ca-cert ca.pem --concurrency 4A listener on anything but loopback needs --tls-cert and --tls-key. --concurrency 0 leaves all the work to workers; any other value also works the queue on the serving machine.
What workers run
- Workers take jobs that run on the host:
--sandbox localruns. Jobs with a Docker sandbox or a task verifier stay queued for a local scheduler, such ashemra runorhemra serveon the store's own machine. - Environment variables a harness names are read from the worker's environment. Credentials never cross the wire.
- Labels (
--label KEY=VALUE) are recorded at registration. They are not matched against jobs yet, so any worker may take any job it can run.
Leases and fencing
A worker holds a lease on each job it runs and renews it with heartbeats. A worker that stops heartbeating loses its lease (--lease-ttl-secs, 30 by default): its attempt ends as an infrastructure failure, and the job is retried by its max_infra_retries as a new attempt. A result that arrives on an old lease is refused and never stored, so a slow worker cannot overwrite a newer attempt.
In the Mac app
The Mac app can serve the same worker listener for the store it has open, after you confirm it in a dialog. Its enrollment token is kept in a file only you can read. See Mac app.