ReLab

← all runs

Run

Status

FAILED

Workflow

data-pipeline v1

Run id

29780383-a221-4743-8fd1-72660de1e6d5

Seed

Scenario

upstream-down-throughout

Duration

2.97s

3 of 4 tasks did not succeed

This run was disrupted and did not finish

3 tasks exhausted its attempts and was dead-lettered. All of it is in the 28 events below.

What happened

The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.

Run started#6 23:31:04.036
RUN_STARTED
A worker picked up the first task in the run.
Failure injected on purpose validate#11 23:31:04.049
FAULT_INJECTED
ReLab degraded the real system here. This is the break, and everything after it is the recovery.
Retry scheduled validate#13 23:31:04.050
TASK_RETRY_SCHEDULED
The task will be offered again after a backoff delay.
Task running validate#16 23:31:04.893
TASK_STARTED
The handler began executing. Each attempt writes one of these.
Task gave up validate#25 23:31:06.990
TASK_DEAD_LETTERED
The task exhausted its attempts and will not be tried again.
Workflow failed#28 23:31:06.990
RUN_FAILED
A task ran out of attempts and the run could not continue.

What the journal proves

Attempts

Workers involved

Faults injected

Leases expired

Tasks requeued

Retries scheduled

Workers lost

Effects suppressed

Dead-lettered

2.94s

Recovery

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Each task in the run, with the attempt it reached and the worker that last held it.
Status	Task	Attempt	Worker	Error
DEAD	analyze	0/3	·	·
SUCCEEDED	import	1/3	db3c6da3	·
DEAD	report	0/3	·	·
DEAD	validate	3/3	·	injected http-error fault: upstream returned 503

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.


{
  "v": 1,
  "seed": 7,
  "scenario_hash": "2c06a5fafde1e39af9e65c579ccbdbf08679be67e2d91b5d14dd2983bb0e231d",
  "scenario_name": "upstream-down-throughout",
  "workflow_name": "data-pipeline",
  "definition_hash": "d0c1b1539e630e362b4c1acb18f66e38edd787006e2efc1435bfa3bb17bb2e97",
  "workflow_version": 1
}
{
  "v": 1,
  "attempt": 0,
  "scheduled_at": "2026-09-02T23:31:04.020500779Z"
}
{
  "v": 1,
  "task_count": 4
}
{
  "v": 1,
  "attempt": 1,
  "lease_expires_at": "2026-09-02T23:31:34.028667542Z"
}
{
  "v": 1,
  "attempt": 1,
  "handler": "import_csv"
}
{
  "v": 1
}
{
  "v": 1,
  "output": {
    "rows": 1000
  },
  "attempt": 1,
  "artifacts": [
    {
      "name": "imported.csv",
      "size": 9,
      "sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64",
      "content_type": "text/csv"
    }
  ],
  "duration_ms": 0
}
{
  "v": 1,
  "attempt": 0,
  "depends_on": [
    "import"
  ],
  "scheduled_at": "2026-09-02T23:31:04.039478796Z"
}
{
  "v": 1,
  "attempt": 1,
  "lease_expires_at": "2026-09-02T23:31:34.044949701Z"
}
{
  "v": 1,
  "attempt": 1,
  "handler": "validate_rows"
}
{
  "v": 1,
  "draw": 0,
  "seed": 7,
  "params": {
    "status": 503
  },
  "scenario": "upstream-down-throughout",
  "fault_type": "http-error",
  "fault_point": "after-task-start"
}
{
  "v": 1,
  "error": "injected http-error fault: upstream returned 503",
  "attempt": 1,
  "retryable": true,
  "duration_ms": 0
}
{
  "v": 1,
  "run_at": "2026-09-02T23:31:04.866937481Z",
  "attempt": 1,
  "delay_ms": 816,
  "next_attempt": 2
}
{
  "v": 1,
  "attempt": 1,
  "scheduled_at": "2026-09-02T23:31:04.888795832Z"
}
{
  "v": 1,
  "attempt": 2,
  "lease_expires_at": "2026-09-02T23:31:34.890548215Z"
}
{
  "v": 1,
  "attempt": 2,
  "handler": "validate_rows"
}
{
  "v": 1,
  "draw": 0,
  "seed": 7,
  "params": {
    "status": 503
  },
  "scenario": "upstream-down-throughout",
  "fault_type": "http-error",
  "fault_point": "after-task-start"
}
{
  "v": 1,
  "error": "injected http-error fault: upstream returned 503",
  "attempt": 2,
  "retryable": true,
  "duration_ms": 0
}
{
  "v": 1,
  "run_at": "2026-09-02T23:31:06.967797438Z",
  "attempt": 2,
  "delay_ms": 2070,
  "next_attempt": 3
}
{
  "v": 1,
  "attempt": 2,
  "scheduled_at": "2026-09-02T23:31:06.981373147Z"
}
{
  "v": 1,
  "attempt": 3,
  "lease_expires_at": "2026-09-02T23:31:36.98300296Z"
}
{
  "v": 1,
  "attempt": 3,
  "handler": "validate_rows"
}
{
  "v": 1,
  "draw": 0,
  "seed": 7,
  "params": {
    "status": 503
  },
  "scenario": "upstream-down-throughout",
  "fault_type": "http-error",
  "fault_point": "after-task-start"
}
{
  "v": 1,
  "error": "injected http-error fault: upstream returned 503",
  "attempt": 3,
  "retryable": false,
  "duration_ms": 0
}
{
  "v": 1,
  "reason": "attempts exhausted: injected http-error fault: upstream returned 503",
  "attempts": 3
}
{
  "v": 1,
  "reason": "upstream task validate did not succeed: ",
  "attempts": 0
}
{
  "v": 1,
  "reason": "upstream task validate did not succeed: ",
  "attempts": 0
}
{
  "v": 1,
  "detail": "3 of 4 tasks did not succeed",
  "reason": "tasks dead-lettered"
}

28 of 28 recorded events, oldest first.
Seq	Time	What happened	Task	Worker	Technical detail
1	23:31:04.020	Run created`RUN_CREATED`	·	·	no payload fields The workflow was accepted and its tasks were written down. Event RUN_CREATED Sequence 1 Task · Worker · Occurred at 2026-09-02T23:31:04.0205Z { "v": 1, "seed": 7, "scenario_hash": "2c06a5fafde1e39af9e65c579ccbdbf08679be67e2d91b5d14dd2983bb0e231d", "scenario_name": "upstream-down-throughout", "workflow_name": "data-pipeline", "definition_hash": "d0c1b1539e630e362b4c1acb18f66e38edd787006e2efc1435bfa3bb17bb2e97", "workflow_version": 1 }
2	23:31:04.020	Task ready to run`TASK_SCHEDULED`	import	·	attempt=0 The task's dependencies are satisfied, so it entered the queue. Event TASK_SCHEDULED Sequence 2 Task import Worker · Occurred at 2026-09-02T23:31:04.0205Z { "v": 1, "attempt": 0, "scheduled_at": "2026-09-02T23:31:04.020500779Z" }
3	23:31:04.020	Run queued`RUN_QUEUED`	·	·	task_count=4 The run's first tasks became available for a worker to claim. Event RUN_QUEUED Sequence 3 Task · Worker · Occurred at 2026-09-02T23:31:04.0205Z { "v": 1, "task_count": 4 }
4	23:31:04.028	Task claimed by a worker`TASK_LEASED`	import	db3c6da3	attempt=1 One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number. Event TASK_LEASED Sequence 4 Task import Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.028667Z { "v": 1, "attempt": 1, "lease_expires_at": "2026-09-02T23:31:34.028667542Z" }
5	23:31:04.036	Task running`TASK_STARTED`	import	db3c6da3	attempt=1 The handler began executing. Each attempt writes one of these. Event TASK_STARTED Sequence 5 Task import Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.036525Z { "v": 1, "attempt": 1, "handler": "import_csv" }
6	23:31:04.036	Run started`RUN_STARTED`	·	·	no payload fields A worker picked up the first task in the run. Event RUN_STARTED Sequence 6 Task · Worker · Occurred at 2026-09-02T23:31:04.036525Z { "v": 1 }
7	23:31:04.039	Task finished`TASK_SUCCEEDED`	import	db3c6da3	attempt=1 duration_ms=0 The handler returned without an error and the result was recorded. Event TASK_SUCCEEDED Sequence 7 Task import Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.039478Z { "v": 1, "output": { "rows": 1000 }, "attempt": 1, "artifacts": [ { "name": "imported.csv", "size": 9, "sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64", "content_type": "text/csv" } ], "duration_ms": 0 }
8	23:31:04.039	Task ready to run`TASK_SCHEDULED`	validate	·	attempt=0 The task's dependencies are satisfied, so it entered the queue. Event TASK_SCHEDULED Sequence 8 Task validate Worker · Occurred at 2026-09-02T23:31:04.039478Z { "v": 1, "attempt": 0, "depends_on": [ "import" ], "scheduled_at": "2026-09-02T23:31:04.039478796Z" }
9	23:31:04.044	Task claimed by a worker`TASK_LEASED`	validate	db3c6da3	attempt=1 One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number. Event TASK_LEASED Sequence 9 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.044949Z { "v": 1, "attempt": 1, "lease_expires_at": "2026-09-02T23:31:34.044949701Z" }
10	23:31:04.047	Task running`TASK_STARTED`	validate	db3c6da3	attempt=1 The handler began executing. Each attempt writes one of these. Event TASK_STARTED Sequence 10 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.047864Z { "v": 1, "attempt": 1, "handler": "validate_rows" }
11	23:31:04.049	Failure injected on purpose`FAULT_INJECTED`	validate	·	fault_type=http-error ReLab degraded the real system here. This is the break, and everything after it is the recovery. Event FAULT_INJECTED Sequence 11 Task validate Worker · Occurred at 2026-09-02T23:31:04.049284Z { "v": 1, "draw": 0, "seed": 7, "params": { "status": 503 }, "scenario": "upstream-down-throughout", "fault_type": "http-error", "fault_point": "after-task-start" }
12	23:31:04.050	Task failed`TASK_FAILED`	validate	db3c6da3	attempt=1 error=injected http-error fault: upstream returned 503 duration_ms=0 The attempt returned an error. Whether it is retried depends on attempts remaining. Event TASK_FAILED Sequence 12 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.050011Z { "v": 1, "error": "injected http-error fault: upstream returned 503", "attempt": 1, "retryable": true, "duration_ms": 0 }
13	23:31:04.050	Retry scheduled`TASK_RETRY_SCHEDULED`	validate	·	attempt=1 next_attempt=2 delay_ms=816 The task will be offered again after a backoff delay. Event TASK_RETRY_SCHEDULED Sequence 13 Task validate Worker · Occurred at 2026-09-02T23:31:04.050011Z { "v": 1, "run_at": "2026-09-02T23:31:04.866937481Z", "attempt": 1, "delay_ms": 816, "next_attempt": 2 }
14	23:31:04.888	Task ready to run`TASK_SCHEDULED`	validate	·	attempt=1 The task's dependencies are satisfied, so it entered the queue. Event TASK_SCHEDULED Sequence 14 Task validate Worker · Occurred at 2026-09-02T23:31:04.888795Z { "v": 1, "attempt": 1, "scheduled_at": "2026-09-02T23:31:04.888795832Z" }
15	23:31:04.890	Task claimed by a worker`TASK_LEASED`	validate	db3c6da3	attempt=2 One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number. Event TASK_LEASED Sequence 15 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.890548Z { "v": 1, "attempt": 2, "lease_expires_at": "2026-09-02T23:31:34.890548215Z" }
16	23:31:04.893	Task running`TASK_STARTED`	validate	db3c6da3	attempt=2 The handler began executing. Each attempt writes one of these. Event TASK_STARTED Sequence 16 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.893384Z { "v": 1, "attempt": 2, "handler": "validate_rows" }
17	23:31:04.895	Failure injected on purpose`FAULT_INJECTED`	validate	·	fault_type=http-error ReLab degraded the real system here. This is the break, and everything after it is the recovery. Event FAULT_INJECTED Sequence 17 Task validate Worker · Occurred at 2026-09-02T23:31:04.895061Z { "v": 1, "draw": 0, "seed": 7, "params": { "status": 503 }, "scenario": "upstream-down-throughout", "fault_type": "http-error", "fault_point": "after-task-start" }
18	23:31:04.896	Task failed`TASK_FAILED`	validate	db3c6da3	attempt=2 error=injected http-error fault: upstream returned 503 duration_ms=0 The attempt returned an error. Whether it is retried depends on attempts remaining. Event TASK_FAILED Sequence 18 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:04.896821Z { "v": 1, "error": "injected http-error fault: upstream returned 503", "attempt": 2, "retryable": true, "duration_ms": 0 }
19	23:31:04.896	Retry scheduled`TASK_RETRY_SCHEDULED`	validate	·	attempt=2 next_attempt=3 delay_ms=2070 The task will be offered again after a backoff delay. Event TASK_RETRY_SCHEDULED Sequence 19 Task validate Worker · Occurred at 2026-09-02T23:31:04.896821Z { "v": 1, "run_at": "2026-09-02T23:31:06.967797438Z", "attempt": 2, "delay_ms": 2070, "next_attempt": 3 }
20	23:31:06.981	Task ready to run`TASK_SCHEDULED`	validate	·	attempt=2 The task's dependencies are satisfied, so it entered the queue. Event TASK_SCHEDULED Sequence 20 Task validate Worker · Occurred at 2026-09-02T23:31:06.981373Z { "v": 1, "attempt": 2, "scheduled_at": "2026-09-02T23:31:06.981373147Z" }
21	23:31:06.983	Task claimed by a worker`TASK_LEASED`	validate	db3c6da3	attempt=3 One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number. Event TASK_LEASED Sequence 21 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:06.983002Z { "v": 1, "attempt": 3, "lease_expires_at": "2026-09-02T23:31:36.98300296Z" }
22	23:31:06.987	Task running`TASK_STARTED`	validate	db3c6da3	attempt=3 The handler began executing. Each attempt writes one of these. Event TASK_STARTED Sequence 22 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:06.987892Z { "v": 1, "attempt": 3, "handler": "validate_rows" }
23	23:31:06.989	Failure injected on purpose`FAULT_INJECTED`	validate	·	fault_type=http-error ReLab degraded the real system here. This is the break, and everything after it is the recovery. Event FAULT_INJECTED Sequence 23 Task validate Worker · Occurred at 2026-09-02T23:31:06.989598Z { "v": 1, "draw": 0, "seed": 7, "params": { "status": 503 }, "scenario": "upstream-down-throughout", "fault_type": "http-error", "fault_point": "after-task-start" }
24	23:31:06.990	Task failed`TASK_FAILED`	validate	db3c6da3	attempt=3 error=injected http-error fault: upstream returned 503 duration_ms=0 The attempt returned an error. Whether it is retried depends on attempts remaining. Event TASK_FAILED Sequence 24 Task validate Worker db3c6da3-e415-4bab-9cca-4f681543a791 Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "error": "injected http-error fault: upstream returned 503", "attempt": 3, "retryable": false, "duration_ms": 0 }
25	23:31:06.990	Task gave up`TASK_DEAD_LETTERED`	validate	·	reason=attempts exhausted: injected http-error fault: upstream r… The task exhausted its attempts and will not be tried again. Event TASK_DEAD_LETTERED Sequence 25 Task validate Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "reason": "attempts exhausted: injected http-error fault: upstream returned 503", "attempts": 3 }
26	23:31:06.990	Task gave up`TASK_DEAD_LETTERED`	analyze	·	reason=upstream task validate did not succeed: The task exhausted its attempts and will not be tried again. Event TASK_DEAD_LETTERED Sequence 26 Task analyze Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "reason": "upstream task validate did not succeed: ", "attempts": 0 }
27	23:31:06.990	Task gave up`TASK_DEAD_LETTERED`	report	·	reason=upstream task validate did not succeed: The task exhausted its attempts and will not be tried again. Event TASK_DEAD_LETTERED Sequence 27 Task report Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "reason": "upstream task validate did not succeed: ", "attempts": 0 }
28	23:31:06.990	Workflow failed`RUN_FAILED`	·	·	reason=tasks dead-lettered detail=3 of 4 tasks did not succeed A task ran out of attempts and the run could not continue. Event RUN_FAILED Sequence 28 Task · Worker · Occurred at 2026-09-02T23:31:06.990518Z { "v": 1, "detail": "3 of 4 tasks did not succeed", "reason": "tasks dead-lettered" }