ReLab

← all runs

Run

Status

FAILED

Workflow

data-pipeline v1

Run id

29780383-a221-4743-8fd1-72660de1e6d5

Seed

Scenario

upstream-down-throughout

Duration

2.97s

3 of 4 tasks did not succeed

This run was disrupted and did not finish

3 tasks exhausted its attempts and was dead-lettered. All of it is in the 28 events below.

What happened

The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.

Run started#6 23:31:04.036
RUN_STARTED
A worker picked up the first task in the run.
Failure injected on purpose validate#11 23:31:04.049
FAULT_INJECTED
ReLab degraded the real system here. This is the break, and everything after it is the recovery.
Retry scheduled validate#13 23:31:04.050
TASK_RETRY_SCHEDULED
The task will be offered again after a backoff delay.
Task running validate#16 23:31:04.893
TASK_STARTED
The handler began executing. Each attempt writes one of these.
Task gave up validate#25 23:31:06.990
TASK_DEAD_LETTERED
The task exhausted its attempts and will not be tried again.
Workflow failed#28 23:31:06.990
RUN_FAILED
A task ran out of attempts and the run could not continue.

What the journal proves

Attempts

Workers involved

Faults injected

Leases expired

Tasks requeued

Retries scheduled

Workers lost

Effects suppressed

Dead-lettered

2.94s

Recovery

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Each task in the run, with the attempt it reached and the worker that last held it.
Status	Task	Attempt	Worker	Error
DEAD	analyze	0/3	·	·
SUCCEEDED	import	1/3	db3c6da3	·
DEAD	report	0/3	·	·
DEAD	validate	3/3	·	injected http-error fault: upstream returned 503

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.

5 of 28 recorded events, oldest first.
Seq	Time	What happened	Task	Worker	Technical detail
13	23:31:04.050	Retry scheduled`TASK_RETRY_SCHEDULED`	validate	·	attempt=1 next_attempt=2 delay_ms=816 The task will be offered again after a backoff delay. Event TASK_RETRY_SCHEDULED Sequence 13 Task validate Worker · Occurred at 2026-09-02T23:31:04.050011Z { "v": 1, "run_at": "2026-09-02T23:31:04.866937481Z", "attempt": 1, "delay_ms": 816, "next_attempt": 2 }
19	23:31:04.896	Retry scheduled`TASK_RETRY_SCHEDULED`	validate	·	attempt=2 next_attempt=3 delay_ms=2070 The task will be offered again after a backoff delay. Event TASK_RETRY_SCHEDULED Sequence 19 Task validate Worker · Occurred at 2026-09-02T23:31:04.896821Z { "v": 1, "run_at": "2026-09-02T23:31:06.967797438Z", "attempt": 2, "delay_ms": 2070, "next_attempt": 3 }
25	23:31:06.990	Task gave up`TASK_DEAD_LETTERED`	validate	·	reason=attempts exhausted: injected http-error fault: upstream r… The task exhausted its attempts and will not be tried again. Event TASK_DEAD_LETTERED Sequence 25 Task validate Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "reason": "attempts exhausted: injected http-error fault: upstream returned 503", "attempts": 3 }
26	23:31:06.990	Task gave up`TASK_DEAD_LETTERED`	analyze	·	reason=upstream task validate did not succeed: The task exhausted its attempts and will not be tried again. Event TASK_DEAD_LETTERED Sequence 26 Task analyze Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "reason": "upstream task validate did not succeed: ", "attempts": 0 }
27	23:31:06.990	Task gave up`TASK_DEAD_LETTERED`	report	·	reason=upstream task validate did not succeed: The task exhausted its attempts and will not be tried again. Event TASK_DEAD_LETTERED Sequence 27 Task report Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "reason": "upstream task validate did not succeed: ", "attempts": 0 }