ReLab

← all runs

Run

Status

SUCCEEDED

Workflow

effectful v1

Run id

96ea5bfd-1536-42c2-ac1b-72b7101efeb7

Seed

1337

Scenario

worker-crash-after-effect

Duration

1.89s

This run was disrupted and finished anyway

1 worker holding work for this run went away. 1 lease expired and the work came back through the reaper. 1 side effect already performed was suppressed by the idempotency ledger, so the retry did not repeat work that had already happened. All of it is in the 23 events below.

What happened

The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.

Run started#6 23:31:09.439
RUN_STARTED
A worker picked up the first task in the run.
Failure injected on purpose charge#11 23:31:09.648
FAULT_INJECTED
ReLab degraded the real system here. This is the break, and everything after it is the recovery.
Worker declared gone#12 23:31:11.020
WORKER_LOST
The holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.
Worker stopped responding charge#13 23:31:11.220
TASK_LEASE_EXPIRED
Nobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.
Task returned to the queue charge#14 23:31:11.220
TASK_REQUEUED
The work a vanished worker was holding became claimable again.
Task running charge#16 23:31:11.255
TASK_STARTED
The handler began executing. Each attempt writes one of these.
Duplicate effect prevented charge#17 23:31:11.257
SIDE_EFFECT_SKIPPED
The retry asked to perform an effect already recorded under the same key, so it was not performed a second time.
Task finished charge#18 23:31:11.259
TASK_SUCCEEDED
The handler returned without an error and the result was recorded.
Workflow completed#23 23:31:11.299
RUN_SUCCEEDED
Every task finished. This event is the run's last: a finished run's story cannot change.

What the journal proves

Attempts

Workers involved

Faults injected

Leases expired

Tasks requeued

Retries scheduled

Workers lost

Effects suppressed

Dead-lettered

1.65s

Recovery

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Each task in the run, with the attempt it reached and the worker that last held it.
Status	Task	Attempt	Worker	Error
SUCCEEDED	charge	2/3	611a59e4	·
SUCCEEDED	import	1/3	611a59e4	·
SUCCEEDED	report	1/3	305c1862	·

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.

1 of 23 recorded events, oldest first.
Seq	Time	What happened	Task	Worker	Technical detail
17	23:31:11.257	Duplicate effect prevented`SIDE_EFFECT_SKIPPED`	charge	·	attempt=2 idempotency_key=96ea5bfd-1536-42c2-ac1b-72b7101efeb7:charge:external-charge The retry asked to perform an effect already recorded under the same key, so it was not performed a second time. Event SIDE_EFFECT_SKIPPED Sequence 17 Task charge Worker · Occurred at 2026-09-02T23:31:11.257794Z { "v": 1, "attempt": 2, "first_attempt": 1, "idempotency_key": "96ea5bfd-1536-42c2-ac1b-72b7101efeb7:charge:external-charge" }