ReLab

Break it. Watch it recover.

← all runs

Run

Status
SUCCEEDED
Workflow
effectful v1
Run id
c68a432f-a112-40a3-b1af-2433dad7c7f9
Seed
31337
Scenario
duplicate-delivery-on-charge
Duration
39ms

This run was disrupted and finished anyway

1 side effect already performed was suppressed by the idempotency ledger, so the retry did not repeat work that had already happened. All of it is in the 18 events below.

What happened

The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.

  1. Run started#6 23:31:07.030
    RUN_STARTED

    A worker picked up the first task in the run.

  2. Failure injected on purpose charge#13 23:31:07.047
    FAULT_INJECTED

    ReLab degraded the real system here. This is the break, and everything after it is the recovery.

  3. Duplicate effect prevented charge#14 23:31:07.048
    SIDE_EFFECT_SKIPPED

    The retry asked to perform an effect already recorded under the same key, so it was not performed a second time.

  4. Task running report#16 23:31:07.052
    TASK_STARTED

    The handler began executing. Each attempt writes one of these.

  5. Task finished report#17 23:31:07.054
    TASK_SUCCEEDED

    The handler returned without an error and the result was recorded.

  6. Workflow completed#18 23:31:07.054
    RUN_SUCCEEDED

    Every task finished. This event is the run's last: a finished run's story cannot change.

What the journal proves

3
Attempts
1
Workers involved
1
Faults injected
0
Leases expired
0
Tasks requeued
0
Retries scheduled
0
Workers lost
1
Effects suppressed
0
Dead-lettered
0.01s
Recovery

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Each task in the run, with the attempt it reached and the worker that last held it.
StatusTaskAttemptWorkerError
SUCCEEDEDcharge1/360026864·
SUCCEEDEDimport1/360026864·
SUCCEEDEDreport1/360026864·

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.

1 of 18 recorded events, oldest first.
SeqTimeWhat happenedTaskWorkerTechnical detail
1323:31:07.047Failure injected on purposeFAULT_INJECTEDcharge·
fault_type=duplicate-delivery

ReLab degraded the real system here. This is the break, and everything after it is the recovery.

Event
FAULT_INJECTED
Sequence
13
Task
charge
Worker
·
Occurred at
2026-09-02T23:31:07.047482Z
{
  "v": 1,
  "draw": 0,
  "seed": 31337,
  "scenario": "duplicate-delivery-on-charge",
  "fault_type": "duplicate-delivery",
  "fault_point": "after-task-finish"
}
RECORDINGfive real runs exported from real PostgreSQL by scripts/record-demo.sh, crashes delivered by SIGKILLrelab 241c136-dirty, recorded