ReLab

One real failure, start to finish

A worker ran the workflow's charge step — a recorded effect standing in for a payment — and was killed before it could acknowledge the task. Nobody told the system it had gone. Every step below is an event from that run’s journal, at its real sequence number and its real time.

1. Run started#6 23:31:09.439
   RUN_STARTED

   A worker picked up the first task in the run.

2. Failure injected on purpose charge#11 23:31:09.648
   FAULT_INJECTED

   ReLab degraded the real system here. This is the break, and everything after it is the recovery.

3. Worker declared gone#12 23:31:11.020
   WORKER_LOST

   The holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.

4. Worker stopped responding charge#13 23:31:11.220
   TASK_LEASE_EXPIRED

   Nobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.

5. Task returned to the queue charge#14 23:31:11.220
   TASK_REQUEUED

   The work a vanished worker was holding became claimable again.

6. Task running charge#16 23:31:11.255
   TASK_STARTED

   The handler began executing. Each attempt writes one of these.

7. Duplicate effect prevented charge#17 23:31:11.257
   SIDE_EFFECT_SKIPPED

   The retry asked to perform an effect already recorded under the same key, so it was not performed a second time.

8. Task finished charge#18 23:31:11.259
   TASK_SUCCEEDED

   The handler returned without an error and the result was recorded.

9. Workflow completed#23 23:31:11.299
   RUN_SUCCEEDED

   Every task finished. This event is the run's last: a finished run's story cannot change.

Open the whole run, all 23 events worker-crash-after-effect

Why this exists

Who this is for

How a killed worker’s task comes back

The mechanism, once, in order. Every stage names the event that records it, so the picture can be checked against a real journal rather than believed.

1. 1

   A worker holds the task

   The worker holds a lease and renews it while it works. It performs the workflow's charge step — a recorded effect standing in for a payment — and writes it to the ledger under a key.

   TASK_LEASED
2. 2

   The worker is killed

   SIGKILL, mid-task. No shutdown, no handover, nothing gets the chance to report it. The renewals simply stop.

   FAULT_INJECTED
3. 3

   The hold expires

   Nobody renewed the lease, so the reaper concludes the holder is gone. This is the same path a machine losing power takes.

   TASK_LEASE_EXPIRED
4. 4

   The task comes back

   The work is claimable again, and another worker takes it under a new attempt number. Nobody had to notice the death for this to happen.

   TASK_REQUEUED
5. 5

   The charge is not repeated

   The retry asks for an effect already recorded under that key, so it is not performed a second time.

   SIDE_EFFECT_SKIPPED
6. 6

   The run finishes

   Every task done, the whole thing recorded in one gapless sequence that replay can rebuild without the database.

   RUN_SUCCEEDED

Nothing in this sequence depends on the dying worker reporting anything, which is the point: a process that has been SIGKILLed, or a machine that has lost power, gets no chance to. What each of these words means.

How a ReLab run works

1. 1

   Run a workflow

   A multi-step workflow runs across worker processes, each task held by one worker at a time.

2. 2

   Break something

   A scenario injects a real degradation from a fixed seed: a worker crash, a duplicate delivery, latency, an HTTP error, a database disconnect.

3. 3

   Record what happened

   Every run and task state change, and the event describing it, are written in one transaction, in a gapless sequence.

4. 4

   Replay the history

   A pure reducer rebuilds the run’s state from the journal alone, with no access to the database it came from.

5. 5

   Verify the result

   Assertions check the recovery: the task came back, the side effect was not repeated, the run reached the state the journal says it reached.

Underneath: PostgreSQL as the only datastore and the coordination point, task leases renewed by the worker holding them, a reaper that releases the leases of workers that stopped answering, an idempotency ledger keyed per effect, and an append-only event log. The guarantees, stated precisely.

Now

A live worker is one that is HEALTHY or SUSPECT: still heartbeating, or doubted but not yet reclaimed. A dead letter is a task that exhausted its attempts. Nothing here is running: these are the counters as they stood when the recording was made.

Latest runs

Go deeper

• README what it is, and a demo you can run
• Glossary every event in a journal, in plain language
• Architecture engine, workers, replay, fault injection
• Reliability guarantees stated precisely, with the limitations kept
• Benchmarks measured, on stated hardware
• Decision records what was decided, and what was rejected