ReLab

Run

3 of 4 tasks did not succeed

What happened

The milestones of this run, in the order they were recorded. Each one is a real event; the full sequence is in the timeline below.

1. Run started#6 23:31:04.036
   RUN_STARTED

   A worker picked up the first task in the run.

2. Failure injected on purpose validate#11 23:31:04.049
   FAULT_INJECTED

   ReLab degraded the real system here. This is the break, and everything after it is the recovery.

3. Retry scheduled validate#13 23:31:04.050
   TASK_RETRY_SCHEDULED

   The task will be offered again after a backoff delay.

4. Task running validate#16 23:31:04.893
   TASK_STARTED

   The handler began executing. Each attempt writes one of these.

5. Task gave up validate#25 23:31:06.990
   TASK_DEAD_LETTERED

   The task exhausted its attempts and will not be tried again.

6. Workflow failed#28 23:31:06.990
   RUN_FAILED

   A task ran out of attempts and the run could not continue.

What the journal proves

Every number above is a count of events in this run’s journal. Recovery is measured from the first fault, lease expiry, task failure or lost worker to the run completing, the same interval relab test asserts on.

Tasks

Timeline

The run’s complete recorded history, in sequence order. This is the same journal relab replay reduces. Filtering hides rows from this view; it never changes the sequence numbers, so a gap in them is still a gap.

{
  "v": 1,
  "draw": 0,
  "seed": 7,
  "params": {
    "status": 503
  },
  "scenario": "upstream-down-throughout",
  "fault_type": "http-error",
  "fault_point": "after-task-start"
}

{
  "v": 1,
  "draw": 0,
  "seed": 7,
  "params": {
    "status": 503
  },
  "scenario": "upstream-down-throughout",
  "fault_type": "http-error",
  "fault_point": "after-task-start"
}

{
  "v": 1,
  "draw": 0,
  "seed": 7,
  "params": {
    "status": 503
  },
  "scenario": "upstream-down-throughout",
  "fault_type": "http-error",
  "fault_point": "after-task-start"
}