ReLab

Terms

Workflow

A named set of steps and the dependencies between them. It is a definition; it does not run on its own.

Run

One execution of a workflow. Its history is the journal, and its last event is terminal: once a run has finished, its story cannot change.

Task

One step of a run. A task can be attempted several times; each attempt is recorded separately.

Worker

A process that claims tasks and runs their handlers. ReLab's crash tests kill real worker processes with SIGKILL.

Lease

A time-bounded hold on a task. One worker holds it, and renews it while it works. If the renewals stop, the hold expires — which is what makes recovery work when a machine loses power and nothing gets the chance to say so.

Reaper

The part of the engine that expires leases nobody renewed and returns that work to the queue. It is why a killed worker's task comes back without anyone reporting the death.

Heartbeat

A worker saying it is alive. Three missed beats make it SUSPECT; five make it LOST and release its leases. One missed beat is never treated as failure.

Journal

The append-only event history of a run. A state change and the event describing it are written in one transaction, and sequence numbers are gapless — so a gap means data was lost, not that something was skipped.

Replay

Rebuilding a run's state from its journal alone, with a pure reducer that cannot read the database. If replay and the database disagree, the journal is not a faithful account and that is a bug worth finding.

Idempotency key

The name a handler gives a side effect — a charge, an email — so the system can tell whether it has already been recorded. A retry that asks for an effect already recorded under the same key does not perform it again.

Dead letter

A task that exhausted its attempts. It is kept, not discarded: an abandoned task is evidence, and the run's failure has to be explainable afterwards.

Scenario

A run with a fault injected on purpose, from a fixed seed, so the same break can be reproduced. The fault degrades the real system rather than simulating a degradation.

At-least-once

What ReLab actually guarantees: a task may run more than once, and an effect already recorded under a key is not performed again. There is a window between performing an effect and recording it in which a crash can cost a duplicate. ReLab does not claim exactly-once, because it does not have it — decision 0005 says why.

A run

One execution of a workflow, from acceptance to a terminal event.

Run created RUN_CREATED

The workflow was accepted and its tasks were written down.

Run queued RUN_QUEUED

The run's first tasks became available for a worker to claim.

Run started RUN_STARTED

A worker picked up the first task in the run.

Workflow completed RUN_SUCCEEDED

Every task finished. This event is the run's last: a finished run's story cannot change.

Workflow failed RUN_FAILED

A task ran out of attempts and the run could not continue.

Workflow cancelled RUN_CANCELLED

The run was stopped on purpose before it finished.

A task

One step of the workflow. Most of a recovery story is told here.

Task ready to run TASK_SCHEDULED

The task's dependencies are satisfied, so it entered the queue.

Task claimed by a worker TASK_LEASED

One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.

Task running TASK_STARTED

The handler began executing. Each attempt writes one of these.

Task finished TASK_SUCCEEDED

The handler returned without an error and the result was recorded.

Task failed TASK_FAILED

The attempt returned an error. Whether it is retried depends on attempts remaining.

Retry scheduled TASK_RETRY_SCHEDULED

The task will be offered again after a backoff delay.

Worker stopped responding TASK_LEASE_EXPIRED

Nobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.

Task returned to the queue TASK_REQUEUED

The work a vanished worker was holding became claimable again.

Task gave up TASK_DEAD_LETTERED

The task exhausted its attempts and will not be tried again.

A worker

A process that claims tasks and heartbeats while it holds them. Only WORKER_LOST reaches a run's journal today: a worker's own comings and goings are state in the workers table, and the journal describes what happened to the run.

Worker joined WORKER_REGISTERED

A worker process announced itself and began heartbeating. Defined, but not written to a run journal today — this one is state in the workers table, so you will not find it on a run’s timeline.

Worker heartbeat WORKER_HEARTBEAT

The worker is alive. One missed beat never counts as failure. Defined, but not written to a run journal today — this one is state in the workers table, so you will not find it on a run’s timeline.

Worker doubted WORKER_SUSPECT

Three heartbeats missed. Its work is not reclaimed yet: a worker that stopped answering has not necessarily stopped working. Defined, but not written to a run journal today — this one is state in the workers table, so you will not find it on a run’s timeline.

Worker declared gone WORKER_LOST

The holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.

The break, and what stopped it costing twice

The two events that are the reason this project exists.

Failure injected on purpose FAULT_INJECTED

ReLab degraded the real system here. This is the break, and everything after it is the recovery.

Duplicate effect prevented SIDE_EFFECT_SKIPPED

The retry asked to perform an effect already recorded under the same key, so it was not performed a second time.

Statuses

What a run, task, or worker status means, in one clause. The same words appear on every table in the dashboard.

SUCCEEDED

finished, with every task done

FAILED

stopped because a task ran out of attempts

CANCELLED

stopped on purpose

RUNNING

in progress

QUEUED

waiting for a worker

CREATED

accepted, not yet queued

PENDING

waiting on a dependency

READY

claimable now

LEASED

held by a worker

RETRYING

failed, waiting for its next attempt

DEAD

out of attempts

HEALTHY

heartbeating

SUSPECT

missed beats, work not yet reclaimed

LOST

gone, leases released

STOPPED

shut down and said so

Where these are defined

• DATA.md the schema as implemented, including every event type
• Reliability guarantees stated precisely, with the limitations kept
• Back to the overview a real run, start to finish