Run
This run was disrupted and finished anyway
1 worker holding work for this run went away. 1 lease expired and the work came back through the reaper. 1 side effect already performed was suppressed by the idempotency ledger, so the retry did not repeat work that had already happened. All of it is in the 23 events below.
What happened
The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.
- Run started#6 23:31:09.439
RUN_STARTEDA worker picked up the first task in the run.
- Failure injected on purpose charge#11 23:31:09.648
FAULT_INJECTEDReLab degraded the real system here. This is the break, and everything after it is the recovery.
- Worker declared gone#12 23:31:11.020
WORKER_LOSTThe holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.
- Worker stopped responding charge#13 23:31:11.220
TASK_LEASE_EXPIREDNobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.
- Task returned to the queue charge#14 23:31:11.220
TASK_REQUEUEDThe work a vanished worker was holding became claimable again.
- Task running charge#16 23:31:11.255
TASK_STARTEDThe handler began executing. Each attempt writes one of these.
- Duplicate effect prevented charge#17 23:31:11.257
SIDE_EFFECT_SKIPPEDThe retry asked to perform an effect already recorded under the same key, so it was not performed a second time.
- Task finished charge#18 23:31:11.259
TASK_SUCCEEDEDThe handler returned without an error and the result was recorded.
- Workflow completed#23 23:31:11.299
RUN_SUCCEEDEDEvery task finished. This event is the run's last: a finished run's story cannot change.
What the journal proves
Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.
Tasks
| Status | Task | Attempt | Worker | Error |
|---|---|---|---|---|
| SUCCEEDED | charge | 2/3 | 611a59e4 | · |
| SUCCEEDED | import | 1/3 | 611a59e4 | · |
| SUCCEEDED | report | 1/3 | 305c1862 | · |
Timeline
The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.
| Seq | Time | What happened | Task | Worker | Technical detail |
|---|---|---|---|---|---|
| 2 | 23:31:09.411 | Task ready to runTASK_SCHEDULED | import | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"scheduled_at": "2026-09-02T23:31:09.411212807Z"
} |
| 4 | 23:31:09.431 | Task claimed by a workerTASK_LEASED | import | 611a59e4 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:11.431167845Z"
} |
| 5 | 23:31:09.439 | Task runningTASK_STARTED | import | 611a59e4 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "import_csv"
} |
| 7 | 23:31:09.443 | Task finishedTASK_SUCCEEDED | import | 611a59e4 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"rows": 1000
},
"attempt": 1,
"artifacts": [
{
"name": "imported.csv",
"size": 9,
"sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64",
"content_type": "text/csv"
}
],
"duration_ms": 0
} |
| 8 | 23:31:09.443 | Task ready to runTASK_SCHEDULED | charge | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"import"
],
"scheduled_at": "2026-09-02T23:31:09.443160569Z"
} |
| 9 | 23:31:09.634 | Task claimed by a workerTASK_LEASED | charge | ec633076 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:11.634095745Z"
} |
| 10 | 23:31:09.643 | Task runningTASK_STARTED | charge | ec633076 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "charge_once"
} |
| 13 | 23:31:11.220 | Worker stopped respondingTASK_LEASE_EXPIRED | charge | ec633076 | attempt=1Nobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.
{
"v": 1,
"attempt": 1,
"lease_expired_at": "2026-09-02T23:31:11.020985Z"
} |
| 14 | 23:31:11.220 | Task returned to the queueTASK_REQUEUED | charge | · | attempt=1 next_attempt=2 reason=lease expiredThe work a vanished worker was holding became claimable again.
{
"v": 1,
"reason": "lease expired",
"attempt": 1,
"next_attempt": 2
} |
| 15 | 23:31:11.251 | Task claimed by a workerTASK_LEASED | charge | 611a59e4 | attempt=2One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 2,
"lease_expires_at": "2026-09-02T23:31:13.251555072Z"
} |
| 16 | 23:31:11.255 | Task runningTASK_STARTED | charge | 611a59e4 | attempt=2The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 2,
"handler": "charge_once"
} |
| 18 | 23:31:11.259 | Task finishedTASK_SUCCEEDED | charge | 611a59e4 | attempt=2 duration_ms=2The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"effect": {
"charged": true,
"by_attempt": 1
},
"attempt": 2
},
"attempt": 2,
"duration_ms": 2
} |
| 19 | 23:31:11.259 | Task ready to runTASK_SCHEDULED | report | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"charge"
],
"scheduled_at": "2026-09-02T23:31:11.25946313Z"
} |
| 20 | 23:31:11.285 | Task claimed by a workerTASK_LEASED | report | 305c1862 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:13.28504691Z"
} |
| 21 | 23:31:11.295 | Task runningTASK_STARTED | report | 305c1862 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "summarize"
} |
| 22 | 23:31:11.299 | Task finishedTASK_SUCCEEDED | report | 305c1862 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"inputs": 1
},
"attempt": 1,
"artifacts": [
{
"name": "summary.json",
"size": 65,
"sha256": "6daf054bc88db6a09bffab080e3e33c0d9b070208a7517141a96c34ebc8938e3",
"content_type": "application/json"
}
],
"duration_ms": 0
} |