# Container runtime contract > The directories, environment variables and sockets your code sees inside every Nodus container. Source: https://nodus-platform-site.pages.dev/docs/concepts/nodusd-container-contract/ Build revision: 211ad9f836655b1c3a2668c4693e442471f28614 Every Job, Sandbox and Function worker runs your image in an isolated container. The same contract holds on every offering: your code can rely on the paths and variables below wherever Nodus places it. ## Directories |Path|Variable|What it is for| |-|-|-| |`/nodus/state`|`NODUS_STATE_DIR`|Recovery state. Write your model, optimizer and progress files here; Nodus checkpoints this directory and restores it on the next attempt.| |`/nodus/inputs/`|`NODUS_INPUT_`|Each declared input, read-only, downloaded before your command starts. The variable names the file (a URL or one object) or the directory (an object prefix, such as an earlier stage’s outputs). An input may set its own `path` instead.| |`/nodus/outputs`|`NODUS_OUTPUT_DIR`|Results. Every regular file here is uploaded when your command exits with code 0, with its SHA-256, and appears as the `outputs` output. Symbolic links are not collected.| |`/dev/shm`||Shared memory for NCCL between GPUs and for data-loader workers: half the container’s memory limit, or half the machine’s memory without one. It counts against the memory limit.| |`/etc/nodus/hostfile`||Multi-node Jobs (Beta): one `
slots=` line per node in rank order, for DeepSpeed and MPI tools.| |`/run/secrets//`||Secret values as read-only files (mode `0400`) on a memory-backed filesystem, so they never reach a disk. The environment also carries each value, named by its key.| |`/run/nodus/events.sock`|`NODUS_EVENTS_SOCKET`|Progress, metrics and the checkpoint handshake (below).| |`/run/nodus/api.sock`|`NODUS_RUNTIME_SOCKET`|The Nodus API and model calls, authenticated as your Job, Sandbox or Function (below).| `NODUS_CHECKPOINT_DIR` is kept as another name for `NODUS_STATE_DIR` for existing programs. ## Environment variables |Variable|Set for|Meaning| |-|-|-| |`NODUS_ATTEMPT`|Every container|The attempt id. A retried or recovered run gets a new attempt.| |`NODUS_JOB`, `NODUS_INDEX`, `JOB_COMPLETION_INDEX`|Jobs|The Job and, for indexed Jobs, this index.| |`NODUS_RESTORED`|Recovered attempts|`1` when `/nodus/state` was restored from a checkpoint before your command started.| |`NODUS_CURSOR_COMPLETED`, `NODUS_CURSOR_TOTAL`|Restartable Jobs|The progress cursor your program last reported.| |`NODUS_PARAM_`|Sweep cells|The cell’s parameters.| |`PET_NPROC_PER_NODE`|Single-node GPU Jobs|The GPU count, which `torchrun` reads as its `--nproc-per-node` default, so `torchrun train.py` starts one process per GPU. Your own value or flag wins.| Restoring a checkpoint restores files, never process memory: your program starts from the beginning and reads its own progress files from `/nodus/state`. Nodus never adds resume flags to your command. ## Network Jobs, Functions and Workspaces reach the public Internet by default; Sandboxes and Agents have no outbound network unless their spec opens it. Either way a container never reaches private addresses (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), cloud metadata services, other containers or the machine it runs on, and has no IPv6. `localhost` always works inside the container. `/etc/resolv.conf` points at public resolvers, and `/etc/hosts` resolves `localhost`. With an allow list (`egress: AllowList`) the container has no direct route out. `HTTPS_PROXY` and `HTTP_PROXY` point at a proxy on the container’s loopback (`http://127.0.0.1:3128`) that reaches only the listed hosts, on ports 80 and 443. A `*.example.com` entry allows every subdomain of `example.com`. Most HTTP clients (`pip`, `npm`, `curl`, `git` over HTTPS, the Python and Node SDKs) use the proxy on their own. A listed name that resolves to a private address is still refused. CPU Sandboxes run under gVisor, whose `localhost` belongs to the sandbox alone, so there the proxy and the inference proxy listen on the sandbox’s gateway address instead of `127.0.0.1`. Read the address from `HTTPS_PROXY` and `OPENAI_BASE_URL` rather than writing `127.0.0.1` into your code; `NO_PROXY` already covers it. On a multi-node Job whose nodes share a private network, each rank’s `eth0` lists the node’s private address first, so NCCL, Gloo and `torchrun` advertise an address the other ranks reach. Ranks reach each other only on the rendezvous and NCCL ports, whatever the Job’s outbound setting. ## GPUs A container on a GPU offering sees exactly the GPUs assigned to it, numbered from 0 as CUDA and PyTorch see them, with the NVIDIA driver libraries mounted read-only. It never sees another container’s GPUs. Use `nvidia-smi` or `torch.cuda.device_count()` to check what you have; do not set `CUDA_VISIBLE_DEVICES` yourself. ## Sidecars and the init command Sidecars start before your command, in the order you list them, in the same container: they share its files, network, environment and logs. Each must answer its readiness probe (an HTTP `GET` of its path, any status below 400, or a TCP connection to its port) within 15 minutes before the next starts, and they stop when your command exits. A sidecar runs from your container’s image. `initCommand` runs after the sidecars and before your command, for preflight checks such as imports, free disk or the GPU count. Your command starts only if it exits with code 0; any other code fails the attempt with that code, and running past its timeout (30 minutes unless you set one) fails it with code 124. Both run inside your billed time. ## Commands you run in a container Commands you start in a running container (`nodus exec`, Processes) run with the same environment, secrets and directories as your main command, plus any variables you add. Their output is streamed to you and, for Processes, kept with your logs with secret values masked. A command started from a terminal session stops when you disconnect; a Process keeps running until it exits, you cancel it, its timeout passes or the container stops. Port forwarding and preview URLs reach a server listening on the container’s `localhost`, whatever its outbound network setting. In a CPU Sandbox they reach the sandbox’s own address instead, so listen on all interfaces (`0.0.0.0`); the same holds for a sidecar’s readiness port. File operations (`nodus cp`, the console Files tab) read, write and watch paths as your container sees them, including `/nodus/state` and `/nodus/outputs`, with the permissions of your container’s user: files written this way belong to that user and appear only once complete. `/proc`, `/sys` and `/dev` are not available to them. ## The events socket The socket speaks newline-delimited JSON, one object of at most 16 KiB per line. Send telemetry as objects with a `type` and, to make retries safe, a unique `id`: ```json {"id": "evt-10", "type": "progress", "completed": 120, "total": 1000} {"id": "evt-11", "type": "log.metrics", "step": 1200, "loss": 0.41} ``` When a program cannot reach the socket, it can print the same object on standard output after the prefix `nodus.event `. The line also stays in your logs. A program that sends no `log.metrics` events still gets its training metrics charted: Nodus reads loss, learning rate, epoch, step and `eval_*` values from Hugging Face Trainer and PyTorch Lightning log lines until the program sends a `log.metrics` event of its own. With `recovery.checkpoint.integration: HFTrainer`, the Trainer helper package is mounted at `/.nodus/python` and put first on `PYTHONPATH`, so its `sitecustomize` registers the Nodus callback in images without the Nodus SDK. ### Checkpoint handshake Nodus decides when to checkpoint. To save a consistent state first, send `{"type": "checkpoint.subscribe"}` once. Before each checkpoint Nodus writes a request on the connection: ```json {"type": "checkpoint.request", "requestId": "ck-1790000000000", "seq": 1790000000000, "urgent": true} ``` Finish writing your files to `/nodus/state`, then answer with `{"type": "checkpoint.ready", "requestId": "ck-1790000000000"}`. `urgent` means the capacity is about to go away: save quickly. A program that never subscribes is checkpointed without being asked, so write your state files atomically (write to a temporary name, then rename). A checkpoint of an empty state directory never replaces an earlier checkpoint that had files in it. ## The API socket `/run/nodus/api.sock` is HTTP over a Unix socket. Requests under `/apis/nodus.dev/` reach the Nodus API, and requests under `/v1/` (OpenAI- and Anthropic-compatible routes) reach Nodus inference. Both carry the service account token of the Job, Sandbox or Function the container belongs to: Nodus adds it to each request, so the token is never in your environment or files, and any `Authorization` header you send is replaced. Terminal window ```sh curl --unix-socket "$NODUS_RUNTIME_SOCKET" http://nodus/apis/nodus.dev/v1/... ``` With the inference proxy enabled, the same model routes are also served on `http://127.0.0.1:7777` inside the container (on the gateway address in a CPU Sandbox), even with outbound network off, and `OPENAI_BASE_URL`, `ANTHROPIC_BASE_URL`, `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are set for it. The keys are placeholders, so SDKs and tools that take a base URL work unchanged. That port serves nothing but model calls. ## Signals and exit codes Your command runs under a small init process (`tini`, at `/.nodus/bin`) that forwards signals to your command’s process group and reaps finished child processes, so your command does not need to be written as PID 1. A stop sends `SIGTERM` to your command, then `SIGKILL` after the stop grace period (30 s unless your spec sets another). Exit code 0 completes the attempt; any other code, or a signal, fails it with your exit code and the last lines of your logs, with secret values masked.