ReLab

Break it. Watch it recover.

← all runs

Run

Status
SUCCEEDED
Workflow
data-pipeline v1
Run id
e1a8ee6c-5bb4-442a-8570-5aceb2ce1065
Seed
3024093237604488000
Duration
47ms

What the journal proves

4
Attempts
1
Workers involved
0
Faults injected
0
Leases expired
0
Tasks requeued
0
Retries scheduled
0
Workers lost
0
Effects suppressed
0
Dead-lettered
·
Recovery

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Each task in the run, with the attempt it reached and the worker that last held it.
StatusTaskAttemptWorkerError
SUCCEEDEDanalyze1/345cf93e9·
SUCCEEDEDimport1/345cf93e9·
SUCCEEDEDreport1/345cf93e9·
SUCCEEDEDvalidate1/345cf93e9·

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.

20 of 20 recorded events, oldest first.
SeqTimeWhat happenedTaskWorkerTechnical detail
123:31:03.950Run createdRUN_CREATED··
no payload fields

The workflow was accepted and its tasks were written down.

Event
RUN_CREATED
Sequence
1
Task
·
Worker
·
Occurred at
2026-09-02T23:31:03.950342Z
{
  "v": 1,
  "seed": 3024093237604488000,
  "workflow_name": "data-pipeline",
  "definition_hash": "d0c1b1539e630e362b4c1acb18f66e38edd787006e2efc1435bfa3bb17bb2e97",
  "workflow_version": 1
}
223:31:03.950Task ready to runTASK_SCHEDULEDimport·
attempt=0

The task's dependencies are satisfied, so it entered the queue.

Event
TASK_SCHEDULED
Sequence
2
Task
import
Worker
·
Occurred at
2026-09-02T23:31:03.950342Z
{
  "v": 1,
  "attempt": 0,
  "scheduled_at": "2026-09-02T23:31:03.950342999Z"
}
323:31:03.950Run queuedRUN_QUEUED··
task_count=4

The run's first tasks became available for a worker to claim.

Event
RUN_QUEUED
Sequence
3
Task
·
Worker
·
Occurred at
2026-09-02T23:31:03.950342Z
{
  "v": 1,
  "task_count": 4
}
423:31:03.959Task claimed by a workerTASK_LEASEDimport45cf93e9
attempt=1

One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.

Event
TASK_LEASED
Sequence
4
Task
import
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.95949Z
{
  "v": 1,
  "attempt": 1,
  "lease_expires_at": "2026-09-02T23:31:33.959490742Z"
}
523:31:03.966Task runningTASK_STARTEDimport45cf93e9
attempt=1

The handler began executing. Each attempt writes one of these.

Event
TASK_STARTED
Sequence
5
Task
import
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.966637Z
{
  "v": 1,
  "attempt": 1,
  "handler": "import_csv"
}
623:31:03.966Run startedRUN_STARTED··
no payload fields

A worker picked up the first task in the run.

Event
RUN_STARTED
Sequence
6
Task
·
Worker
·
Occurred at
2026-09-02T23:31:03.966637Z
{
  "v": 1
}
723:31:03.970Task finishedTASK_SUCCEEDEDimport45cf93e9
attempt=1 duration_ms=0

The handler returned without an error and the result was recorded.

Event
TASK_SUCCEEDED
Sequence
7
Task
import
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.970614Z
{
  "v": 1,
  "output": {
    "rows": 1000
  },
  "attempt": 1,
  "artifacts": [
    {
      "name": "imported.csv",
      "size": 9,
      "sha256": "6eb31300aed462386f9f52f25d0155b3ca1a9ebce4c811950777282d69a81c64",
      "content_type": "text/csv"
    }
  ],
  "duration_ms": 0
}
823:31:03.970Task ready to runTASK_SCHEDULEDvalidate·
attempt=0

The task's dependencies are satisfied, so it entered the queue.

Event
TASK_SCHEDULED
Sequence
8
Task
validate
Worker
·
Occurred at
2026-09-02T23:31:03.970614Z
{
  "v": 1,
  "attempt": 0,
  "depends_on": [
    "import"
  ],
  "scheduled_at": "2026-09-02T23:31:03.97061459Z"
}
923:31:03.975Task claimed by a workerTASK_LEASEDvalidate45cf93e9
attempt=1

One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.

Event
TASK_LEASED
Sequence
9
Task
validate
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.975859Z
{
  "v": 1,
  "attempt": 1,
  "lease_expires_at": "2026-09-02T23:31:33.975859719Z"
}
1023:31:03.978Task runningTASK_STARTEDvalidate45cf93e9
attempt=1

The handler began executing. Each attempt writes one of these.

Event
TASK_STARTED
Sequence
10
Task
validate
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.978675Z
{
  "v": 1,
  "attempt": 1,
  "handler": "validate_rows"
}
1123:31:03.980Task finishedTASK_SUCCEEDEDvalidate45cf93e9
attempt=1 duration_ms=0

The handler returned without an error and the result was recorded.

Event
TASK_SUCCEEDED
Sequence
11
Task
validate
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.980691Z
{
  "v": 1,
  "output": {
    "valid": 994,
    "invalid": 6
  },
  "attempt": 1,
  "duration_ms": 0
}
1223:31:03.980Task ready to runTASK_SCHEDULEDanalyze·
attempt=0

The task's dependencies are satisfied, so it entered the queue.

Event
TASK_SCHEDULED
Sequence
12
Task
analyze
Worker
·
Occurred at
2026-09-02T23:31:03.980691Z
{
  "v": 1,
  "attempt": 0,
  "depends_on": [
    "validate"
  ],
  "scheduled_at": "2026-09-02T23:31:03.980691259Z"
}
1323:31:03.984Task claimed by a workerTASK_LEASEDanalyze45cf93e9
attempt=1

One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.

Event
TASK_LEASED
Sequence
13
Task
analyze
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.984276Z
{
  "v": 1,
  "attempt": 1,
  "lease_expires_at": "2026-09-02T23:31:33.984276147Z"
}
1423:31:03.987Task runningTASK_STARTEDanalyze45cf93e9
attempt=1

The handler began executing. Each attempt writes one of these.

Event
TASK_STARTED
Sequence
14
Task
analyze
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.987255Z
{
  "v": 1,
  "attempt": 1,
  "handler": "analyze"
}
1523:31:03.988Task finishedTASK_SUCCEEDEDanalyze45cf93e9
attempt=1 duration_ms=0

The handler returned without an error and the result was recorded.

Event
TASK_SUCCEEDED
Sequence
15
Task
analyze
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.988825Z
{
  "v": 1,
  "output": {
    "mean": 497,
    "count": 994
  },
  "attempt": 1,
  "artifacts": [
    {
      "name": "analysis.json",
      "size": 24,
      "sha256": "3a6711131b05a37b630c267665849696566e3b0a7488e5bde0b0f118786ada7d",
      "content_type": "application/json"
    }
  ],
  "duration_ms": 0
}
1623:31:03.988Task ready to runTASK_SCHEDULEDreport·
attempt=0

The task's dependencies are satisfied, so it entered the queue.

Event
TASK_SCHEDULED
Sequence
16
Task
report
Worker
·
Occurred at
2026-09-02T23:31:03.988825Z
{
  "v": 1,
  "attempt": 0,
  "depends_on": [
    "analyze"
  ],
  "scheduled_at": "2026-09-02T23:31:03.988825514Z"
}
1723:31:03.992Task claimed by a workerTASK_LEASEDreport45cf93e9
attempt=1

One worker holds the task for a bounded time and renews that hold while it works. The hold is on this attempt: if it expires while that worker is still running, a second worker may take the task under a new attempt number.

Event
TASK_LEASED
Sequence
17
Task
report
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.992839Z
{
  "v": 1,
  "attempt": 1,
  "lease_expires_at": "2026-09-02T23:31:33.992839772Z"
}
1823:31:03.995Task runningTASK_STARTEDreport45cf93e9
attempt=1

The handler began executing. Each attempt writes one of these.

Event
TASK_STARTED
Sequence
18
Task
report
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.995444Z
{
  "v": 1,
  "attempt": 1,
  "handler": "generate_report"
}
1923:31:03.997Task finishedTASK_SUCCEEDEDreport45cf93e9
attempt=1 duration_ms=0

The handler returned without an error and the result was recorded.

Event
TASK_SUCCEEDED
Sequence
19
Task
report
Worker
45cf93e9-d37c-4e2a-bb12-e45538a98745
Occurred at
2026-09-02T23:31:03.997252Z
{
  "v": 1,
  "output": {
    "bytes": 26
  },
  "attempt": 1,
  "artifacts": [
    {
      "name": "report.txt",
      "size": 26,
      "sha256": "0178113cce88afc0b9d891a8762ff3b1f324d67484b30144349884d62353a3cd",
      "content_type": "text/plain"
    }
  ],
  "duration_ms": 0
}
2023:31:03.997Workflow completedRUN_SUCCEEDED··
tasks_succeeded=4

Every task finished. This event is the run's last: a finished run's story cannot change.

Event
RUN_SUCCEEDED
Sequence
20
Task
·
Worker
·
Occurred at
2026-09-02T23:31:03.997252Z
{
  "v": 1,
  "tasks_succeeded": 4
}
RECORDINGfive real runs exported from real PostgreSQL by scripts/record-demo.sh, crashes delivered by SIGKILLrelab 241c136-dirty, recorded