Run
This run was disrupted and finished anyway
1 side effect already performed was suppressed by the idempotency ledger, so the retry did not repeat work that had already happened. All of it is in the 18 events below.
What happened
The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.
- Run started#6 23:31:07.030
RUN_STARTEDA worker picked up the first task in the run.
- Failure injected on purpose charge#13 23:31:07.047
FAULT_INJECTEDReLab degraded the real system here. This is the break, and everything after it is the recovery.
- Duplicate effect prevented charge#14 23:31:07.048
SIDE_EFFECT_SKIPPEDThe retry asked to perform an effect already recorded under the same key, so it was not performed a second time.
- Task running report#16 23:31:07.052
TASK_STARTEDThe handler began executing. Each attempt writes one of these.
- Task finished report#17 23:31:07.054
TASK_SUCCEEDEDThe handler returned without an error and the result was recorded.
- Workflow completed#18 23:31:07.054
RUN_SUCCEEDEDEvery task finished. This event is the run's last: a finished run's story cannot change.
What the journal proves
Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.
Tasks
| Status | Task | Attempt | Worker | Error |
|---|---|---|---|---|
| SUCCEEDED | charge | 1/3 | 60026864 | · |
| SUCCEEDED | import | 1/3 | 60026864 | · |
| SUCCEEDED | report | 1/3 | 60026864 | · |
Timeline
The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.
| Seq | Time | What happened | Task | Worker | Technical detail |
|---|---|---|---|---|---|
| 2 | 23:31:07.015 | Task ready to runTASK_SCHEDULED | import | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"scheduled_at": "2026-09-02T23:31:07.015775166Z"
} |
| 4 | 23:31:07.024 | Task claimed by a workerTASK_LEASED | import | 60026864 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:09.02422958Z"
} |
| 5 | 23:31:07.030 | Task runningTASK_STARTED | import | 60026864 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "import_csv"
} |
| 7 | 23:31:07.033 | Task finishedTASK_SUCCEEDED | import | 60026864 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"rows": 1000
},
"attempt": 1,
"artifacts": [
{
"name": "imported.csv",
"size": 9,
"sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64",
"content_type": "text/csv"
}
],
"duration_ms": 0
} |
| 8 | 23:31:07.033 | Task ready to runTASK_SCHEDULED | charge | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"import"
],
"scheduled_at": "2026-09-02T23:31:07.033676176Z"
} |
| 9 | 23:31:07.038 | Task claimed by a workerTASK_LEASED | charge | 60026864 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:09.038887823Z"
} |
| 10 | 23:31:07.041 | Task runningTASK_STARTED | charge | 60026864 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "charge_once"
} |
| 11 | 23:31:07.044 | Task finishedTASK_SUCCEEDED | charge | 60026864 | attempt=1 duration_ms=1The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"effect": {
"charged": true,
"by_attempt": 1
},
"attempt": 1
},
"attempt": 1,
"duration_ms": 1
} |
| 12 | 23:31:07.044 | Task ready to runTASK_SCHEDULED | report | · | attempt=0The task's dependencies are satisfied, so it entered the queue.
{
"v": 1,
"attempt": 0,
"depends_on": [
"charge"
],
"scheduled_at": "2026-09-02T23:31:07.044385533Z"
} |
| 15 | 23:31:07.050 | Task claimed by a workerTASK_LEASED | report | 60026864 | attempt=1One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.
{
"v": 1,
"attempt": 1,
"lease_expires_at": "2026-09-02T23:31:09.050833659Z"
} |
| 16 | 23:31:07.052 | Task runningTASK_STARTED | report | 60026864 | attempt=1The handler began executing. Each attempt writes one of these.
{
"v": 1,
"attempt": 1,
"handler": "summarize"
} |
| 17 | 23:31:07.054 | Task finishedTASK_SUCCEEDED | report | 60026864 | attempt=1 duration_ms=0The handler returned without an error and the result was recorded.
{
"v": 1,
"output": {
"inputs": 1
},
"attempt": 1,
"artifacts": [
{
"name": "summary.json",
"size": 65,
"sha256": "54cb487a3dd596a1d7337436435d9ff485ee285dc6e727a8eda83ff26eb2bc2b",
"content_type": "application/json"
}
],
"duration_ms": 0
} |