Run
3 of 4 tasks did not succeed
This run was disrupted and did not finish
3 tasks exhausted its attempts and was dead-lettered. All of it is in the 28 events below.
What happened
The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.
- Run started#6 23:31:04.036
RUN_STARTEDA worker picked up the first task in the run.
- Failure injected on purpose validate#11 23:31:04.049
FAULT_INJECTEDReLab degraded the real system here. This is the break, and everything after it is the recovery.
- Retry scheduled validate#13 23:31:04.050
TASK_RETRY_SCHEDULEDThe task will be offered again after a backoff delay.
- Task running validate#16 23:31:04.893
TASK_STARTEDThe handler began executing. Each attempt writes one of these.
- Task gave up validate#25 23:31:06.990
TASK_DEAD_LETTEREDThe task exhausted its attempts and will not be tried again.
- Workflow failed#28 23:31:06.990
RUN_FAILEDA task ran out of attempts and the run could not continue.
What the journal proves
Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.
Tasks
| Status | Task | Attempt | Worker | Error |
|---|---|---|---|---|
| DEAD | analyze | 0/3 | · | · |
| SUCCEEDED | import | 1/3 | db3c6da3 | · |
| DEAD | report | 0/3 | · | · |
| DEAD | validate | 3/3 | · | injected http-error fault: upstream returned 503 |
Timeline
The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.
| Seq | Time | What happened | Task | Worker | Technical detail |
|---|---|---|---|---|---|
| 2 | 23:31:04.020 | Task ready to runTASK_SCHEDULED | import | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"scheduled_at": "2026-09-02T23:31:04.020500779Z"
} |
| 4 | 23:31:04.028 | Task claimed by a workerTASK_LEASED | import | db3c6da3 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:34.028667542Z"
} |
| 5 | 23:31:04.036 | Task runningTASK_STARTED | import | db3c6da3 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "import_csv"
} |
| 7 | 23:31:04.039 | Task finishedTASK_SUCCEEDED | import | db3c6da3 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"rows": 1000
},
"attempt": 1,
"artifacts": [
{
"name": "imported.csv",
"size": 9,
"sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64",
"content_type": "text/csv"
}
],
"duration_ms": 0
} |
| 8 | 23:31:04.039 | Task ready to runTASK_SCHEDULED | validate | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"import"
],
"scheduled_at": "2026-09-02T23:31:04.039478796Z"
} |
| 9 | 23:31:04.044 | Task claimed by a workerTASK_LEASED | validate | db3c6da3 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:34.044949701Z"
} |
| 10 | 23:31:04.047 | Task runningTASK_STARTED | validate | db3c6da3 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "validate_rows"
} |
| 12 | 23:31:04.050 | Task failedTASK_FAILED | validate | db3c6da3 | attempt=1 error=injected http-error fault: upstream returned 503 duration_ms=0The attempt returned an error. Whether it is retried depends on attempts remaining.
{
"v": 1,
"error": "injected http-error fault: upstream returned 503",
"attempt": 1,
"retryable": true,
"duration_ms": 0
} |
| 13 | 23:31:04.050 | Retry scheduledTASK_RETRY_SCHEDULED | validate | · | attempt=1 next_attempt=2 delay_ms=816The task will be offered again after a backoff delay.
{
"v": 1,
"run_at": "2026-09-02T23:31:04.866937481Z",
"attempt": 1,
"delay_ms": 816,
"next_attempt": 2
} |
| 14 | 23:31:04.888 | Task ready to runTASK_SCHEDULED | validate | · | attempt=1The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 1,
"scheduled_at": "2026-09-02T23:31:04.888795832Z"
} |
| 15 | 23:31:04.890 | Task claimed by a workerTASK_LEASED | validate | db3c6da3 | attempt=2One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 2,
"lease_expires_at": "2026-09-02T23:31:34.890548215Z"
} |
| 16 | 23:31:04.893 | Task runningTASK_STARTED | validate | db3c6da3 | attempt=2The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 2,
"handler": "validate_rows"
} |
| 18 | 23:31:04.896 | Task failedTASK_FAILED | validate | db3c6da3 | attempt=2 error=injected http-error fault: upstream returned 503 duration_ms=0The attempt returned an error. Whether it is retried depends on attempts remaining.
{
"v": 1,
"error": "injected http-error fault: upstream returned 503",
"attempt": 2,
"retryable": true,
"duration_ms": 0
} |
| 19 | 23:31:04.896 | Retry scheduledTASK_RETRY_SCHEDULED | validate | · | attempt=2 next_attempt=3 delay_ms=2070The task will be offered again after a backoff delay.
{
"v": 1,
"run_at": "2026-09-02T23:31:06.967797438Z",
"attempt": 2,
"delay_ms": 2070,
"next_attempt": 3
} |
| 20 | 23:31:06.981 | Task ready to runTASK_SCHEDULED | validate | · | attempt=2The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 2,
"scheduled_at": "2026-09-02T23:31:06.981373147Z"
} |
| 21 | 23:31:06.983 | Task claimed by a workerTASK_LEASED | validate | db3c6da3 | attempt=3One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 3,
"lease_expires_at": "2026-09-02T23:31:36.98300296Z"
} |
| 22 | 23:31:06.987 | Task runningTASK_STARTED | validate | db3c6da3 | attempt=3The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 3,
"handler": "validate_rows"
} |
| 24 | 23:31:06.990 | Task failedTASK_FAILED | validate | db3c6da3 | attempt=3 error=injected http-error fault: upstream returned 503 duration_ms=0The attempt returned an error. Whether it is retried depends on attempts remaining.
{
"v": 1,
"error": "injected http-error fault: upstream returned 503",
"attempt": 3,
"retryable": false,
"duration_ms": 0
} |
| 25 | 23:31:06.990 | Task gave upTASK_DEAD_LETTERED | validate | · | reason=attempts exhausted: injected http-error fault: upstream r…The task exhausted its attempts and will not be tried again.
{
"v": 1,
"reason": "attempts exhausted: injected http-error fault: upstream returned 503",
"attempts": 3
} |
| 26 | 23:31:06.990 | Task gave upTASK_DEAD_LETTERED | analyze | · | reason=upstream task validate did not succeed:The task exhausted its attempts and will not be tried again.
{
"v": 1,
"reason": "upstream task validate did not succeed: ",
"attempts": 0
} |
| 27 | 23:31:06.990 | Task gave upTASK_DEAD_LETTERED | report | · | reason=upstream task validate did not succeed:The task exhausted its attempts and will not be tried again.
{
"v": 1,
"reason": "upstream task validate did not succeed: ",
"attempts": 0
} |