List or watch jobs in every project
const url = 'https://example.com/apis/nodus.dev/v1/jobs';const options = {method: 'GET'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request GET \ --url https://example.com/apis/nodus.dev/v1/jobsParameters
Section titled “ Parameters ”Query Parameters
Section titled “ Query Parameters ”Responses
Section titled “ Responses ”OK
object
Job is a run-to-completion container with checkpoints, recovery from preemption, Indexed completions, multi-node gangs (Beta), a cost estimate before launch and exact prepaid billing. Every index runs as Attempts.
object
object
object
object
object
object
object
JobSpec describes a Job. Everything is immutable after create except parallelism, backoffLimit, timeout, maxCostUSD (raise only), recovery.maxAttempts, recovery.checkpoint.interval and retainAfterFinish, ttlSecondsAfterFinished and state.
object
ActiveDeadlineSeconds is accepted as an alias of timeout on write and folded into it.
BackoffLimit is how many customer failures (non-zero exits, OOM) across all indexes are retried before the Job fails (0–100).
CompleteByTime is a finish-by hint for placement and checkpoint cadence. It never stops the Job.
Completions is how many indexes must succeed (1–10000). Each index gets NODUS_INDEX and JOB_COMPLETION_INDEX.
Distributed runs the Job as a gang of nodes that start together and restart together (Beta).
object
GPUsPerNode is 1, 2, 4 or 8 and equals resources.gpu.count when both are set.
Launcher is Plain (env contract only), Torchrun, Ray, Verl, Accelerate or Deepspeed.
MaxAssemblyRetries is how many times a timed-out assembly is re-placed (0–5, default 2).
Network is Colocated (one provider and region), Regional (one region class) or Global.
Nodes is the gang size (2–16; the beta caps it at 8). Exactly one of nodes and totalGPUs.
StartupTimeout is the gang assembly budget (5m–60m, default 15m).
TotalGPUs is resolved into status.topology at admission (2–64).
Transport is Direct (private or direct paths only) or Auto (relayed paths allowed).
EnvVar is one environment variable: a literal value or a Secret key.
object
Name matches ^[A-Za-z_][A-Za-z0-9_]*$.
Value is the literal value.
EnvVarSource names where an env value comes from.
object
SecretKeySelector selects one key of a Secret.
object
Key is the key within it.
Name is the Secret.
ExpectedDuration is the declared runtime the estimate uses first.
InitCommand is a preflight run before the command.
object
Command is the argv.
Timeout is at most 30m.
Inputs are materialized read-only under /nodus/inputs/
JobInput is one input: exactly one of volume, output, bucket, url and fromWorkspace. It is mounted at /nodus/inputs/
object
BucketInput is an object in customer storage.
object
Connection is an S3 Connection that can read it.
SHA256 is checked at end of stream when set.
URI is s3://bucket/key.
StageInput names an output of a Pipeline stage.
object
Output is the output name.
Stage is the producing stage.
WorkspaceInput mounts a Workspace home.
object
Name is the Workspace.
Path is a subdirectory of the home.
Name matches ^[A-Za-z][A-Za-z0-9_]{0,63}$ and is unique among inputs.
OutputInput names a committed output of another Job.
object
Index selects an index of an Indexed Job (default 0).
Job is the producing Job.
Name is the output.
URLInput is a file downloaded over HTTPS.
object
SHA256 is the expected digest.
URL is https:// only.
VolumeInput names a Volume revision.
object
Name is the Volume.
Revision pins a revision.
MaxCostUSD caps what the Job may spend. Reaching it suspends the Job; it can only be raised.
Network sets egress for the container.
object
Egress is the outbound network policy.
object
Allow lists DNS names or *.suffix patterns (AllowList only; at most 128).
Policy is Deny, AllowList or Open.
Presets add curated host lists such as python-packages or huggingface.
Outputs are files or directories collected when an index succeeds (at most 32). Everything under /nodus/outputs is also collected as the output named “outputs”.
JobOutput is a declared output.
object
FromRanks is Zero (rank 0 only) or All (every rank under rank-
Name matches ^[a-z0-9._-]{1,64}$; the nodus. prefix is reserved.
Path is an absolute file or directory path; directories are archived as tar.zst.
OutputSink loads an output into a customer database table.
object
Connection is a Postgres, Neon or Supabase Connection.
Mode is Append (default) or Replace.
Table is the destination table.
Parallelism is how many indexes run at once (1 to min(completions, 256)).
Placement constrains where work may run.
object
Dedicated is reserved; only true is accepted.
Interruptible is Never, Allow (the scheduler decides by expected cost) or Prefer.
MaxRateUSDPerHour filters out offerings above this customer rate.
Offerings restricts placement to these Offerings.
Pool is a BYOC Pool in the project.
Profile is Balanced, Cost or Speed.
QueueTimeout is how long the Job may wait for capacity before failing with CapacityUnavailable (default 24h; 0s waits forever).
Regions are region classes (us, eu, …); empty means any.
Recovery says how progress survives lost capacity.
object
CheckpointPolicy configures checkpoints.
object
Format is Snapshot, or Dcp for torch.distributed.checkpoint (the default for distributed Jobs).
Integration is Auto, None or HFTrainer.
Interval is auto or a duration from 1m to 6h.
MaxSize is checked before upload (default 1Ti, at most 2Ti).
Paths are saved together in one checkpoint (at most 64; default /nodus/state).
RetainAfterFinish keeps checkpoints this long after the Job finishes (default 168h).
Continuity is Checkpointed (restore the latest checkpoint), Restartable (progress cursor) or Ephemeral.
MaxAttempts is how many infrastructure replacements each index may use (1–32; gangs count restarts).
MinProgressDuration is the run time that counts as progress (default 2m).
OnInterruption is Resume, or Fail to fail instead of recovering or suspending.
Resources are capacity floors. For GPU work the offering’s host shape applies if larger.
object
CPU is the vCPU floor.
Disk is the ephemeral disk floor.
GPURequest asks for accelerators. A family (H100) matches any of its variants and never another family.
object
Count is the GPUs per node: 1, 2, 4 or 8.
Exact pins the exact variant instead of matching the family.
Interconnect is Any or NVLink.
MinMemory is the per-GPU memory floor.
Type lists 1–8 accelerator ids or families from the catalog.
Memory is the host memory floor.
Nodes above 1 is shorthand for spec.distributed.nodes and is stored in that form.
ScaledownWindow keeps the capacity warm for reuse this long after the Job ends (0s–20m).
SecretMount mounts one Secret, optionally pinned to a version. The bare name is accepted on write.
object
Name is the Secret.
Version pins a version; unset pins the latest at admission.
Sidecar is a helper container with native-sidecar semantics.
object
Command is the sidecar’s argv.
Env adds environment variables.
EnvVar is one environment variable: a literal value or a Secret key.
object
Name matches ^[A-Za-z_][A-Za-z0-9_]*$.
Value is the literal value.
EnvVarSource names where an env value comes from.
object
SecretKeySelector selects one key of a Secret.
object
Key is the key within it.
Name is the Secret.
Image defaults to the Job’s image.
Name is unique within the Job.
StartupTimeout bounds how long the sidecar may take to become ready.
Source is the code for the container: exactly one of git and blob.
object
Blob is sha256:
GitSource is a repository checkout.
object
Path is the checkout path (default workingDir).
Ref is a branch, tag or commit.
Repo is owner/name or an https:// URL.
State is the desired state: Running (the default), Suspended, or the cancel state that stops the Job for good.
Timeout is the wall-clock limit counted from the first Provisioning, excluding time spent Suspended. The Job then fails with DeadlineExceeded.
TTLSecondsAfterFinished deletes the Job, its outputs and checkpoints this long after it finishes.
VolumeMount mounts a Volume.
object
MountPath is the absolute mount point; system paths are refused.
ReadOnly mounts it read-only.
Revision pins a revision; only with readOnly.
Volume names the Volume.
JobStatus is what Nodus observed about a Job.
object
Active is the number of indexes with a live attempt.
AttemptCounts counts attempts of one slot.
object
Current is the current attempt’s number, from 1.
Max is recovery.maxAttempts.
Total is how many attempts have run.
CheckpointStatus is the latest committed checkpoint.
object
Interval is the cadence in use.
RestoredFromSeq is the checkpoint the current attempt restored.
Seq is the checkpoint sequence number.
Size is its size in bytes.
Time is when it committed.
CompletedIndexes lists the succeeded indexes as compressed ranges, for example 0-3,5.
CompletionTime is when the Job reached a terminal phase.
Conditions are the latest observations.
object
Cost is status.cost on every kind that spends money. It is a display copy of the ledger: amounts are decimal USD strings, and nothing is charged from these fields.
object
CostSegments splits a charge by billing segment.
object
BootUSD is the charge from the start of billing until the program started.
CoveredByNodus lists the segments (Boot, Running, Restore, Teardown) with time Nodus paid for instead of charging it, such as the start and teardown of a run that failed because of Nodus.
RestoreUSD is the charge for bringing a replacement up after capacity was lost.
RunningUSD is the charge while the program ran.
TeardownUSD is the charge from stop until deletion was confirmed, with the billing increment.
Final is true once the final charge has posted.
FundedUntilTime is when the current holds stop paying for the object.
HeldUSD is what is currently held for it.
LimitUSD is the object’s maxCostUSD, if it has one.
RateUSDPerHour is the sum of the hourly rates of its capacity that is still billing.
TotalUSD is what has been charged for this object so far.
Estimate is returned in status.estimate by a dry-run (?dryRun=All) and on create for every compute kind (ADR-031). Amounts are decimal USD strings; the ETag of the dry-run response binds a later create to it.
object
AssemblyBoundUSD is the most a multi-node Job can cost if its members never assemble.
BlockingReasons say why a real create would fail now, with the amounts and a fix.
BlockingReason says why a create would fail now.
object
Code is the error code the create would return, for example InsufficientCredits.
Fix says what to do.
Message states the reason with the amounts involved.
ConcurrentHoldUSD is the total of the holds for every slot that starts at once.
USDBand is a median and a 90th-percentile amount in USD.
object
P50 is the median amount.
P90 is the 90th-percentile amount.
GangHoldUSD is the total of the member holds of a multi-node Job.
HoldUSD is the first hold that must fit your balance, every matching Budget and the object’s own cap.
MinimumChargeUSD is the billing increment rounding charged once per allocation.
USDBand is a median and a 90th-percentile amount in USD.
object
P50 is the median amount.
P90 is the 90th-percentile amount.
PlacementPreview is the customer-safe description of the capacity a create would use.
object
Explain is the scheduler’s explanation, the same shape as Attempt status.placement.explain.
Interruptible reports whether the capacity can be reclaimed.
Offering is the offering id, for example h100-sxm-80g-x8-us.
RegionClass is the region class of the offering.
WarmReuse reports whether warm capacity would be reused.
PricebookVersion is the price book the estimate used.
RateUSDPerHour is the hourly rate of the chosen capacity.
StartupEstimate is the expected time to start.
object
DurationBand is a median and a 90th-percentile duration, as Go duration strings.
object
P50 is the median duration.
P90 is the 90th-percentile duration.
DurationBand is a median and a 90th-percentile duration, as Go duration strings.
object
P50 is the median duration.
P90 is the 90th-percentile duration.
Topology is the resolved multi-node topology of a gang Job.
ValidUntil is when the prices this estimate used stop being current. A create bound by If-Match is admitted after it too, and never placed above rateUSDPerHour plus 10 %.
Warnings are non-blocking notes, for example ColdStart or InterruptibleSelected.
EstimateWarning is a non-blocking note on an estimate.
object
Code is the CamelCase warning code.
Message explains the warning.
Failed is the number of customer failures (non-zero exits, OOM) across all indexes.
Fix is the reason’s documented fix.
GangStatus is the current epoch of a gang Job, in offering keys only.
object
Epoch is the gang epoch.
Members are the ranks of this epoch.
GangMember is one rank of a gang epoch.
object
Attempt is the rank’s attempt id.
Offering is the rank’s offering key.
Path is Private, Public, Direct or Relayed.
Phase is the rank attempt’s phase.
Rank is 0..size-1.
RegionClass is the rank’s region class.
Phase is Queued, Assembling, Meshing, Starting, Running, Recovering, Stopping, Releasing or Done.
GangProbe is a measured network path.
object
BusGbps is the measured all-reduce bus bandwidth.
MeasuredTime is when it was measured.
Path is the measured path.
RTTMs is the measured round-trip time.
Ready is how many ranks registered, meshed and passed the probe.
Rendezvous is rank 0’s advertised address.
Restarts counts gang restarts.
Size is the number of ranks.
Started is how many ranks started the command.
Transport is the weakest pair’s path: Private, Public, Direct or Relayed.
HandledRequests records the last handled *-requested-at annotation per action.
object
ImageStatus is the pinned image.
object
Digest is the pinned digest.
Reference is the image as written.
User is the user the image’s config runs its process as (root when it names none). Only an image that runs as root is placed on offerings that start the Nodus node agent inside the image itself.
Inputs are the pinned inputs.
InputStatus is a pinned input.
object
Name is the input.
SHA256 is the verified digest.
VolumeRevision is the pinned Volume revision.
InputsReadyTime is when declared inputs finished materializing.
Links are related pages, for example W&B runs.
Link is a related page.
object
Kind names the link, for example wandb.
URL is the page.
Message is a scrubbed explanation (at most three log lines for a customer exit).
ObservedGeneration is the spec generation the status reflects.
Outputs are the committed outputs.
OutputStatus is a committed output.
object
Index is the Job index that produced it.
Name is the declared output.
Path is where it was collected from.
Rank is the gang rank that produced it (fromRanks All).
SHA256 is the content digest.
SinkStatus is the state of one sink load.
object
Message explains a failure.
Phase is Pending, Loading, Loaded or Failed.
Rows is how many rows were loaded.
Size is the size in bytes.
Phase is the run-to-completion phase.
JobPlacement is where the current attempt runs, in customer terms.
object
Interruptible is true on reclaimable capacity.
Offering is the offering key, for example h100-sxm-80g-x1-us.
RateUSDPerHour is the frozen customer rate.
RegionClass is the region class.
WarmReuse is true when warm capacity was reused.
Progress is what the program reported.
object
Metrics are the latest scalar metrics, as decimal strings.
object
Stage is a domain stage name.
ProgressUnits counts work units.
object
Completed is units done.
Total is units planned, when known.
Reason says why the Job failed, stopped for good or is suspended.
SecretVersionIDs identify the pinned versions, which every attempt receives: a Secret deleted and recreated under the same name is a different Secret.
object
SecretVersions are pinned at admission; replacements reuse them.
object
SourceStatus is the resolved code.
object
Blob is the uploaded blob.
Commit is the resolved git commit.
StartTime is the first Running.
StopByTime is the funded edge while Funded is False: a money-driven suspend checkpoints until then and stops without the checkpoint once it passes (ADR-086).
StorageStatus is the bytes a Job owns.
object
BilledBytes is the share above the free inclusion.
Bytes is checkpoints plus outputs.
Succeeded is the number of indexes that succeeded.
SuspendFailures counts the failed checkpoints of the current customer suspend. After the first try and two retries the Job keeps running with SuspendFailed; the count resets when spec.state leaves Suspended.
SuspendedDuration is the time spent Suspended since timeoutStartTime; spec.timeout does not count it.
TimeoutStartTime is the first Provisioning, where the spec.timeout clock starts.
TimeoutTime is when spec.timeout fires: timeoutStartTime plus the current spec.timeout plus suspendedDuration, so an edit to spec.timeout moves it.
Topology is the resolved shape of a gang Job; it never changes across epochs.
object
Accelerator is the resolved accelerator id.
GPUsPerNode is the GPUs on each node.
Network is the locality the gang was placed under.
Nodes is the gang size.
ResolvedFrom is Nodes or TotalGPUs.
TotalGPUs is nodes × gpusPerNode.
UnitMetrics summarize per-unit timings.
object
Completed is units done.
CostPerUnitUSD is cost divided by units.
P50Ms is the median unit time.
P95Ms is the 95th percentile unit time.
object
object
Example generated
{ "apiVersion": "example", "items": [ { "apiVersion": "example", "kind": "example", "metadata": { "annotations": { "additionalProperty": "example" }, "creationTimestamp": "example", "deletionGracePeriodSeconds": 1, "deletionTimestamp": "example", "finalizers": [ "example" ], "generateName": "example", "generation": 1, "labels": { "additionalProperty": "example" }, "managedFields": [ { "apiVersion": "example", "fieldsType": "example", "fieldsV1": { "additionalProperty": "example" }, "manager": "example", "operation": "example", "subresource": "example", "time": "example" } ], "name": "example", "namespace": "example", "ownerReferences": [ { "apiVersion": "example", "blockOwnerDeletion": true, "controller": true, "kind": "example", "name": "example", "uid": "example" } ], "resourceVersion": "example", "selfLink": "example", "uid": "example" }, "spec": { "activeDeadlineSeconds": 1, "args": [ "example" ], "backoffLimit": 1, "command": [ "example" ], "completeByTime": "example", "completions": 1, "connections": [ "example" ], "distributed": { "gpusPerNode": 1, "launcher": "example", "maxAssemblyRetries": 1, "network": "example", "nodes": 1, "startupTimeout": "example", "totalGPUs": 1, "transport": "example" }, "env": [ { "name": "example", "value": "example", "valueFrom": { "secretKeyRef": { "key": "example", "name": "example" } } } ], "expectedDuration": "example", "image": "example", "imagePullSecrets": [ "example" ], "imageRef": "example", "initCommand": { "command": [ "example" ], "timeout": "example" }, "inputs": [ { "bucket": { "connection": "example", "sha256": "example", "uri": "example" }, "fromStage": { "output": "example", "stage": "example" }, "fromWorkspace": { "name": "example", "path": "example" }, "name": "example", "output": { "index": 1, "job": "example", "name": "example" }, "url": { "sha256": "example", "url": "example" }, "volume": { "name": "example", "revision": 1 } } ], "maxCostUSD": "example", "network": { "egress": { "allow": [ "example" ], "policy": "example", "presets": [ "example" ] } }, "outputs": [ { "fromRanks": "example", "name": "example", "path": "example", "sink": { "connection": "example", "mode": "example", "table": "example" } } ], "parallelism": 1, "placement": { "dedicated": true, "interruptible": "example", "maxRateUSDPerHour": "example", "offerings": [ "example" ], "pool": "example", "profile": "example", "queueTimeout": "example", "regions": [ "example" ] }, "recovery": { "checkpoint": { "format": "example", "integration": "example", "interval": "example", "maxSize": "example", "paths": [ "example" ], "retainAfterFinish": "example" }, "continuity": "example", "maxAttempts": 1, "minProgressDuration": "example", "onInterruption": "example" }, "resources": { "cpu": "example", "disk": "example", "gpu": { "count": 1, "exact": true, "interconnect": "example", "minMemory": "example", "type": [ "example" ] }, "memory": "example", "nodes": 1 }, "scaledownWindow": "example", "secrets": [ { "name": "example", "version": 1 } ], "serviceAccountName": "example", "sidecars": [ { "command": [ "example" ], "env": [ { "name": "example", "value": "example", "valueFrom": { "secretKeyRef": { "key": "example", "name": "example" } } } ], "image": "example", "name": "example", "startupTimeout": "example" } ], "source": { "blob": "example", "git": { "path": "example", "ref": "example", "repo": "example" } }, "state": "example", "timeout": "example", "ttlSecondsAfterFinished": 1, "volumes": [ { "mountPath": "example", "readOnly": true, "revision": 1, "volume": "example" } ], "workingDir": "example" }, "status": { "active": 1, "attempts": { "current": 1, "max": 1, "total": 1 }, "checkpoint": { "interval": "example", "restoredFromSeq": 1, "seq": 1, "size": 1, "time": "example" }, "completedIndexes": "example", "completionTime": "example", "conditions": [ { "lastTransitionTime": "example", "message": "example", "observedGeneration": 1, "reason": "example", "status": "example", "type": "example" } ], "cost": { "bySegment": { "bootUSD": "example", "coveredByNodus": [ "example" ], "restoreUSD": "example", "runningUSD": "example", "teardownUSD": "example" }, "final": true, "fundedUntilTime": "example", "heldUSD": "example", "limitUSD": "example", "rateUSDPerHour": "example", "totalUSD": "example" }, "estimate": { "assemblyBoundUSD": "example", "blockingReasons": [ { "code": "example", "fix": "example", "message": "example" } ], "concurrentHoldUSD": "example", "costUSD": { "p50": "example", "p90": "example" }, "gangHoldUSD": "example", "holdUSD": "example", "minimumChargeUSD": "example", "overheadUSD": { "p50": "example", "p90": "example" }, "placementPreview": { "explain": "example", "interruptible": true, "offering": "example", "regionClass": "example", "warmReuse": true }, "pricebookVersion": "example", "rateUSDPerHour": "example", "startup": { "cold": { "p50": "example", "p90": "example" }, "warm": { "p50": "example", "p90": "example" } }, "topology": "example", "validUntil": "example", "warnings": [ { "code": "example", "message": "example" } ] }, "failed": 1, "fix": "example", "gang": { "epoch": 1, "members": [ { "attempt": "example", "offering": "example", "path": "example", "phase": "example", "rank": 1, "regionClass": "example" } ], "phase": "example", "probe": { "busGbps": "example", "measuredTime": "example", "path": "example", "rttMs": "example" }, "ready": 1, "rendezvous": "example", "restarts": 1, "size": 1, "started": 1, "transport": "example" }, "handledRequests": { "additionalProperty": "example" }, "image": { "digest": "example", "reference": "example", "user": "example" }, "inputs": [ { "name": "example", "sha256": "example", "volumeRevision": 1 } ], "inputsReadyTime": "example", "links": [ { "kind": "example", "url": "example" } ], "message": "example", "observedGeneration": 1, "outputs": [ { "index": 1, "name": "example", "path": "example", "rank": 1, "sha256": "example", "sink": { "message": "example", "phase": "example", "rows": 1 }, "size": 1 } ], "phase": "example", "placement": { "interruptible": true, "offering": "example", "rateUSDPerHour": "example", "regionClass": "example", "warmReuse": true }, "progress": { "metrics": { "additionalProperty": "example" }, "stage": "example", "units": { "completed": 1, "total": 1 } }, "reason": "example", "secretVersionIDs": { "additionalProperty": "example" }, "secretVersions": { "additionalProperty": 1 }, "source": { "blob": "example", "commit": "example" }, "startTime": "example", "stopByTime": "example", "storage": { "billedBytes": 1, "bytes": 1 }, "succeeded": 1, "suspendFailures": 1, "suspendedDuration": "example", "timeoutStartTime": "example", "timeoutTime": "example", "topology": { "accelerator": "example", "gpusPerNode": 1, "network": "example", "nodes": 1, "resolvedFrom": "example", "totalGPUs": 1 }, "unitMetrics": { "completed": 1, "costPerUnitUSD": "example", "p50Ms": 1, "p95Ms": 1 } } } ], "kind": "example", "metadata": { "continue": "example", "remainingItemCount": 1, "resourceVersion": "example", "selfLink": "example", "shardInfo": { "selector": "example" } }}default
Section titled “ default ”An error: a metav1.Status whose reason is a registered code.
Status is the error body of every API response: a Kubernetes metav1.Status (so kubectl and client-go understand it) plus three top-level extensions that those clients ignore (ADR-028).
object
object
object
Docs is the URL of the code’s docs page.
Fix says what to do next, for example a CLI command or the field to change.
object
object
RequestID identifies the request in logs and support tickets.
Example generated
{ "apiVersion": "example", "code": 1, "details": { "causes": [ { "field": "example", "message": "example", "reason": "example" } ], "group": "example", "kind": "example", "name": "example", "retryAfterSeconds": 1, "uid": "example" }, "docs": "example", "fix": "example", "kind": "example", "message": "example", "metadata": { "continue": "example", "remainingItemCount": 1, "resourceVersion": "example", "selfLink": "example", "shardInfo": { "selector": "example" } }, "reason": "example", "requestId": "example", "status": "example"}