ReLab

← all runs

Run

Status

SUCCEEDED

Workflow

data-pipeline v1

Run id

bb06d758-2500-4193-81ca-51fcbf0a51da

Seed

Scenario

worker-crash-during-analyze

Duration

2.28s

This run was disrupted and finished anyway

1 worker holding work for this run went away. 1 lease expired and the work came back through the reaper. All of it is in the 26 events below.

What happened

The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.

Run started#6 23:31:07.106
RUN_STARTED
A worker picked up the first task in the run.
Failure injected on purpose analyze#15 23:31:07.513
FAULT_INJECTED
ReLab degraded the real system here. This is the break, and everything after it is the recovery.
Worker declared gone#16 23:31:09.086
WORKER_LOST
The holder of this run's task is gone and its leases are released. Usually that is five missed heartbeats; a worker that shuts down deliberately while holding work writes the same event, because what the run experienced is the same either way.
Worker stopped responding analyze#17 23:31:09.285
TASK_LEASE_EXPIRED
Nobody renewed the hold on this task, so another process concluded the holder is gone. This is the mechanism that also works when a machine loses power.
Task returned to the queue analyze#18 23:31:09.285
TASK_REQUEUED
The work a vanished worker was holding became claimable again.
Task running analyze#20 23:31:09.327
TASK_STARTED
The handler began executing. Each attempt writes one of these.
Task finished analyze#21 23:31:09.329
TASK_SUCCEEDED
The handler returned without an error and the result was recorded.
Workflow completed#26 23:31:09.360
RUN_SUCCEEDED
Every task finished. This event is the run's last: a finished run's story cannot change.

What the journal proves

Attempts

Workers involved

Faults injected

Leases expired

Tasks requeued

Retries scheduled

Workers lost

Effects suppressed

Dead-lettered

1.85s

Recovery

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Each task in the run, with the attempt it reached and the worker that last held it.
Status	Task	Attempt	Worker	Error
SUCCEEDED	analyze	2/3	423bf373	·
SUCCEEDED	import	1/3	6f14b10a	·
SUCCEEDED	report	1/3	63a9b177	·
SUCCEEDED	validate	1/3	423bf373	·

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.

No effect events in this run. That is a fact about the run, not a missing page: nothing of that kind was recorded.