Run
This run was disrupted and finished anyway
1 worker holding work for this run went away. 1 lease expired and the work came back through the reaper. All of it is in the 26 events below.
What happened
The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.
- Run started#6 23:31:07.106
RUN_STARTEDA worker picked up the first task in the run.
- Failure injected on purpose analyze#15 23:31:07.513
FAULT_INJECTEDReLab degraded the real system here. This is the break, and everything after it is the recovery.
- Worker declared gone#16 23:31:09.086
WORKER_LOSTThe holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.
- Worker stopped responding analyze#17 23:31:09.285
TASK_LEASE_EXPIREDNobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.
- Task returned to the queue analyze#18 23:31:09.285
TASK_REQUEUEDThe work a vanished worker was holding became claimable again.
- Task running analyze#20 23:31:09.327
TASK_STARTEDThe handler began executing. Each attempt writes one of these.
- Task finished analyze#21 23:31:09.329
TASK_SUCCEEDEDThe handler returned without an error and the result was recorded.
- Workflow completed#26 23:31:09.360
RUN_SUCCEEDEDEvery task finished. This event is the run's last: a finished run's story cannot change.
What the journal proves
Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.
Tasks
| Status | Task | Attempt | Worker | Error |
|---|---|---|---|---|
| SUCCEEDED | analyze | 2/3 | 423bf373 | · |
| SUCCEEDED | import | 1/3 | 6f14b10a | · |
| SUCCEEDED | report | 1/3 | 63a9b177 | · |
| SUCCEEDED | validate | 1/3 | 423bf373 | · |
Timeline
The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.
| Seq | Time | What happened | Task | Worker | Technical detail |
|---|---|---|---|---|---|
| 1 | 23:31:07.078 | Run createdRUN_CREATED | · | · | no payload fieldsThe workflow was accepted and its tasks were written down.
{
"v": 1,
"seed": 42,
"scenario_hash": "3afd1ab3b6ec911b0bb45b0a8543ef2d6ff2988385893d98e6f0232d05dfa75c",
"scenario_name": "worker-crash-during-analyze",
"workflow_name": "data-pipeline",
"definition_hash": "d0c1b1539e630e362b4c1acb18f66e38edd787006e2efc1435bfa3bb17bb2e97",
"workflow_version": 1
} |
| 2 | 23:31:07.078 | Task ready to runTASK_SCHEDULED | import | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"scheduled_at": "2026-09-02T23:31:07.07842945Z"
} |
| 3 | 23:31:07.078 | Run queuedRUN_QUEUED | · | · | task_count=4The run's first tasks became available for a worker to claim.
{
"v": 1,
"task_count": 4
} |
| 4 | 23:31:07.098 | Task claimed by a workerTASK_LEASED | import | 6f14b10a | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:09.098040682Z"
} |
| 5 | 23:31:07.106 | Task runningTASK_STARTED | import | 6f14b10a | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "import_csv"
} |
| 6 | 23:31:07.106 | Run startedRUN_STARTED | · | · | no payload fieldsA worker picked up the first task in the run.
{
"v": 1
} |
| 7 | 23:31:07.110 | Task finishedTASK_SUCCEEDED | import | 6f14b10a | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"rows": 1000
},
"attempt": 1,
"artifacts": [
{
"name": "imported.csv",
"size": 9,
"sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64",
"content_type": "text/csv"
}
],
"duration_ms": 0
} |
| 8 | 23:31:07.110 | Task ready to runTASK_SCHEDULED | validate | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"import"
],
"scheduled_at": "2026-09-02T23:31:07.110009437Z"
} |
| 9 | 23:31:07.301 | Task claimed by a workerTASK_LEASED | validate | 423bf373 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:09.30115289Z"
} |
| 10 | 23:31:07.311 | Task runningTASK_STARTED | validate | 423bf373 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "validate_rows"
} |
| 11 | 23:31:07.314 | Task finishedTASK_SUCCEEDED | validate | 423bf373 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"valid": 994,
"invalid": 6
},
"attempt": 1,
"duration_ms": 0
} |
| 12 | 23:31:07.314 | Task ready to runTASK_SCHEDULED | analyze | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"validate"
],
"scheduled_at": "2026-09-02T23:31:07.314923929Z"
} |
| 13 | 23:31:07.507 | Task claimed by a workerTASK_LEASED | analyze | 6f14b10a | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:09.507566948Z"
} |
| 14 | 23:31:07.511 | Task runningTASK_STARTED | analyze | 6f14b10a | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "analyze"
} |
| 15 | 23:31:07.513 | Failure injected on purposeFAULT_INJECTED | analyze | · | fault_type=worker-crashReLab degraded the real system here. This is the break, and everything after it is the recovery.
{
"v": 1,
"draw": 0,
"seed": 42,
"scenario": "worker-crash-during-analyze",
"fault_type": "worker-crash",
"fault_point": "after-task-start"
} |
| 16 | 23:31:09.086 | Worker declared goneWORKER_LOST | · | 6f14b10a | leases_released=1 missed_beats=5The holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.
{
"v": 1,
"missed_beats": 5,
"leases_released": 1
} |
| 17 | 23:31:09.285 | Worker stopped respondingTASK_LEASE_EXPIRED | analyze | 6f14b10a | attempt=1Nobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.
{
"v": 1,
"attempt": 1,
"lease_expired_at": "2026-09-02T23:31:09.086405Z"
} |
| 18 | 23:31:09.285 | Task returned to the queueTASK_REQUEUED | analyze | · | attempt=1 next_attempt=2 reason=lease expiredThe work a vanished worker was holding became claimable again.
{
"v": 1,
"reason": "lease expired",
"attempt": 1,
"next_attempt": 2
} |
| 19 | 23:31:09.324 | Task claimed by a workerTASK_LEASED | analyze | 423bf373 | attempt=2One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 2,
"lease_expires_at": "2026-09-02T23:31:11.32420136Z"
} |
| 20 | 23:31:09.327 | Task runningTASK_STARTED | analyze | 423bf373 | attempt=2The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 2,
"handler": "analyze"
} |
| 21 | 23:31:09.329 | Task finishedTASK_SUCCEEDED | analyze | 423bf373 | attempt=2 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"mean": 497,
"count": 994
},
"attempt": 2,
"artifacts": [
{
"name": "analysis.json",
"size": 24,
"sha256": "3a6711131b05a37b630c267665849696566e3b0a7488e5bde0b0f118786ada7d",
"content_type": "application/json"
}
],
"duration_ms": 0
} |
| 22 | 23:31:09.329 | Task ready to runTASK_SCHEDULED | report | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"analyze"
],
"scheduled_at": "2026-09-02T23:31:09.329378906Z"
} |
| 23 | 23:31:09.348 | Task claimed by a workerTASK_LEASED | report | 63a9b177 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:11.348169533Z"
} |
| 24 | 23:31:09.357 | Task runningTASK_STARTED | report | 63a9b177 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "generate_report"
} |
| 25 | 23:31:09.360 | Task finishedTASK_SUCCEEDED | report | 63a9b177 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"bytes": 26
},
"attempt": 1,
"artifacts": [
{
"name": "report.txt",
"size": 26,
"sha256": "0178113cce88afc0b9d891a8762ff3b1f324d67484b30144349884d62353a3cd",
"content_type": "text/plain"
}
],
"duration_ms": 0
} |
| 26 | 23:31:09.360 | Workflow completedRUN_SUCCEEDED | · | · | tasks_succeeded=4Every task finished. This event is the run's last: a finished run's story cannot change.
{
"v": 1,
"tasks_succeeded": 4
} |