# Lifecycles

> The phases a Job, Pipeline or Sweep moves through, what moves it, and how spec.state and conditions relate.

Source: https://nodus-platform-site.pages.dev/docs/concepts/lifecycles/
Build revision: 211ad9f836655b1c3a2668c4693e442471f28614

Jobs, Pipelines and Sweeps share one lifecycle: they run until they finish. Each shows where it is in `status.phase`, why in `status.reason` and `status.message`, and the details in `status.conditions`. You steer it with `spec.state`.

## Phases

|Phase|Meaning|Billed|
|-|-|-|
|`Queued`|Admitted; waiting for capacity that fits and for a funded hold|No|
|`Provisioning`|Capacity acquired; the image, source, inputs and any saved state are being prepared|Yes|
|`Running`|The command is running|Yes|
|`Recovering`|The capacity was lost; the work is moving to new capacity|For new capacity, once acquired|
|`Suspending`|State is being saved and compute released|Yes|
|`Suspended`|Paused with its state saved; no compute is held|No compute|
|`Cancelling`|Stopping and releasing compute|Until released|
|`Succeeded`|Finished; outputs collected|No|
|`Failed`|Finished without success; `status.reason` says why|No|
|`Cancelled`|Stopped by `spec.state: Cancelled` or a delete|No|

`Succeeded`, `Failed` and `Cancelled` are final: once there, the phase never changes and nothing more is billed.

## Transitions

|From|To|When|
|-|-|-|
|(new)|`Queued`|The object is admitted|
|`Queued`|`Provisioning`|Capacity is placed and acquired|
|`Queued`|`Failed`|Nothing fits within `placement.queueTimeout` (`CapacityUnavailable`)|
|`Provisioning`|`Running`|The command starts|
|`Provisioning`|`Failed`|The container cannot start (`LaunchFailed`, `ImagePullFailed`)|
|`Running`|`Recovering`|The capacity is lost|
|`Recovering`|`Running`|The work restarts on new capacity|
|`Recovering`|`Failed`|Recovery limits are used up (`RecoveryLimitExceeded`, `NoProgress`, `RestoreFailed`)|
|`Running`|`Succeeded`|The work is done and its outputs are collected|
|`Running`|`Failed`|The command failed more than `backoffLimit` allows, or `timeout` elapsed|
|`Queued`, `Provisioning`, `Running`, `Recovering`|`Suspending`|`spec.state: Suspended`, credits or a budget ran out, or `maxCostUSD` was reached|
|`Suspending`|`Suspended`|State is saved and compute released|
|`Suspending`|`Running`|A suspend you asked for could not save state (`SuspendFailed`)|
|`Suspending`|`Failed`|Saving state failed permanently or timed out|
|`Suspended`|`Queued`|Resumed, and funds cover a new hold|
|any unfinished phase|`Cancelling`|`spec.state: Cancelled`, or the object is deleted|
|`Cancelling`|`Cancelled`|Compute is released and the final charge posted|

## spec.state

`spec.state` is what you want; `status.phase` is what is happening. `nodus suspend`, `nodus resume` and `nodus cancel` set it, and so can a manifest:

|`spec.state`|Effect|
|-|-|
|`Running` (default)|Run, or resume from `Suspended`|
|`Suspended`|Save state, release compute and stop billing|
|`Cancelled`|Stop for good|

A suspend caused by money (credits, a budget or `maxCostUSD`) leaves `spec.state` as it is and resumes on its own once the work is funded again or the cap is raised. Time spent suspended does not count against `timeout`.

## Conditions

|Condition|Meaning|
|-|-|
|`Admitted`|The object passed admission|
|`Scheduled`|Capacity is placed; while false, its message says what it is waiting for|
|`Funded`|Credits and budgets cover the work; false while it is stopped for money|
|`Ready`|The command is running|
|`Checkpointed`|The latest state save succeeded; false with `CheckpointFailed` when it did not|
|`Suspended`|The work is suspended; `SuspendFailed` when a suspend could not save state|
|`OutputsCommitted`|Every declared output is collected|
|`SinksLoaded`|Every output sink has loaded into its table|

Wait on a phase or a condition from the CLI:

Terminal window

```console
$ nodus wait job/train --for=jsonpath='{.status.phase}'=Succeeded --timeout 2h
$ nodus wait job/train --for=condition=OutputsCommitted
```

## Pipelines and Sweeps

A Pipeline or a Sweep takes its phase from its child Jobs. It is `Queued` until its first Job exists, `Running` while any runs, `Suspended` when every unfinished Job is suspended, and final once every Job is final: `Succeeded` if all succeeded, otherwise `Failed` with reason `StageFailed` (Pipeline) or `CellsFailed` (Sweep). Its `spec.state` passes to every unfinished child, and its `maxCostUSD` caps the spend of all its children together.