Changelog2026-10-02Improved

Every GPU of a machine for torchrun

• A Job with several GPUs on one machine (--gpu A6000:2, --gpu H100:8) sets PET_NPROC_PER_NODE to the GPU count, so torchrun train.py starts one process per GPU. A --nproc-per-node you pass yourself wins. See Use every GPU of one machine.
• /dev/shm in a Job’s container is half its memory limit, or half the machine’s memory, instead of 64 MiB, so NCCL between GPUs and PyTorch data-loader workers no longer run out of shared memory.