Skip to content

nodus.job

View Markdown

Job: a run-to-completion container with checkpoints, recovery and outputs (resources.md §3.1).

distributed=nodus.Distributed(...) makes it a multi-node gang (Beta, ADR-112); logs and exec then take a rank.

class Distributed(nodes: int | None = None, total_gpus: int | None = None, gpus_per_node: int | None = None, launcher: str | None = None, network: str | None = None, transport: str | None = None, startup_timeout: Any = None) -> None

A gang: nodes or total_gpus; launcher Plain, Torchrun, Ray, Verl; network and transport.

Type: int | None

Type: str | None

Type: str | None

Type: int | None

spec() -> Obj

Type: Any

Type: int | None

Type: str | None

class Job(obj: Obj) -> None
attempts() -> list[View]
cancel() -> None
create(*, name: str | None = None, image: Any = None, command: list[str] | None = None, args: list[str] | None = None, source: str | os.PathLike[str] | Mapping[str, Any] | None = None, gpu: Any = None, cpu: Any = None, memory: Any = None, disk: Any = None, env: dict[str, str] | None = None, secrets: list[Any] | None = None, volumes: Mapping[str, Any] | None = None, workdir: str | None = None, network: _spec.Egress | None = None, checkpoint: Any = None, timeout: Any = None, expected_duration: Any = None, interruptible: Any = None, region: Any = None, profile: Any = None, max_cost: Any = None, completions: int | None = None, parallelism: int | None = None, distributed: Distributed | None = None, outputs: Mapping[str, str] | None = None, labels: dict[str, str] | None = None, allow_large_source: bool = False, project: str | None = None) -> _Job
estimate() -> View

The Job’s status.estimate: cost p50/p90, start ETA, hold (and topology and gang hold for gangs).

exec(*command: str, pty: bool = False, rank: int | None = None, index: int | None = None) -> _Process

Type: _Files

from_name(name: str, project: str | None = None) -> _Job
logs(follow: bool = False, rank: int | str | None = None, **params: Any) -> AsyncIterator[str]

Log lines; for a gang, rank=n reads one rank and rank="all" merges them with [r<n>] prefixes.

Type: str

Type: _Outputs

job.outputs["model"].download("./model").

resume() -> None
run(**kwargs: Any) -> _Job

Create the Job and return its handle (the same as create); .wait() blocks until it finishes.

status() -> View
suspend() -> None
wait(timeout: float | None = None) -> View

Block until the Job finishes; raises JobFailed with the exit code and log tail when it fails.

class Output(parent: Any, name: str, kind: str = 'Job') -> None
download(path: str | os.PathLike[str], index: int | None = None) -> Path

Save the output at path: the sha256 from X-Nodus-SHA256 is verified, then the file is renamed in.

Type: str

class Outputs(parent: Any, kind: str = 'Job') -> None

Declared and collected outputs of a Job, TrainingJob or AgentRun: outputs["name"].download(path) for one, outputs.download(dir) for all of them, outputs.list().

download(path: str | os.PathLike[str], prefix: str = '') -> Path

Save every output named under prefix (all by default) into the directory path, each verified, at its name below prefix: outputs.download("./adapter", prefix="adapter/").

list() -> list[Obj]