Skip to content

nodus.checkpoint

View Markdown

The checkpoint handshake and gang checkpoints (resources.md §8, distributed-training.md §10.3).

Nodus owns checkpoint cadence and storage; programs write their own files into nodus.state_dir(). Before a snapshot the node sends checkpoint.request: on_request(callback) saves, then ack() tells the node the files are consistent. Restoring files never restores process memory: a restarted program loads what it saved.

ack(seq: str | None = None) -> None

Tell the node the checkpoint files are complete for request seq (the latest one by default).

handle(message: dict[str, Any]) -> None

Process one message from the events socket.

on_request(callback: Callable[[], None]) -> None

Call callback (then ack()) whenever the node asks for a checkpoint; a no-op outside Nodus.

requested() -> bool

True while a checkpoint request is pending; poll it at safe points in a training loop.

subscribe() -> None

Listen for checkpoint requests without a callback: poll requested() and call ack() after saving.