Scaling and warm workers

A Function’s scaling bounds its worker pool. Nodus sizes the pool from the calls that are waiting and running.

Setting	Default	What it does
min_workers	0	Workers kept warm even when idle. They are billed.
max_workers	10	The most workers the Function ever has, 1 to 1,000.
scaledown_window	1m	How long an idle worker above min_workers stays before it is released, up to 20 minutes.
target_concurrency	1	Calls one worker runs at once, 1 to 1,000. Use more than 1 for I/O-bound functions.

@app.function(cpu=2, memory="4Gi", min_workers=1, max_workers=20, scaledown_window="5m", target_concurrency=4)
def embed(text: str) -> list[float]:
    ...

A pool with target_concurrency=4 holds one worker for every four calls that are waiting or running, and never more than max_workers. A worker leaves only after it has been idle for the whole scaledown_window, so a burst that comes back inside the window finds its workers still there.

Cold and warm starts

A call that lands on a warm worker with a free slot starts at once. A call that needs a new worker waits for the worker to place, pull its image and start. Nodus shows both before you run anything:

print(embed.estimate("hello"))   # expected cost, cold and warm start times, and the hold

$ nodus get function embed -o jsonpath={.status.estimate}

status.estimate.startup holds the cold and warm bands, and status.estimate.rateUSDPerHour the worker’s rate. While workers are starting, the Function’s Ready condition says WorkersStarting. With no workers and nothing queued it says ScaledToZero, and with an image that is still building it says ImagePending.

Classes: set up once per worker

A class keeps its state for the life of a worker. @nodus.enter() methods run once when the worker starts, before its first call. @nodus.exit() methods run once when the worker drains. Methods marked @nodus.method() get .remote(), .map() and .spawn().

examples/python/classes/app.py

"""A class whose model loads once per worker, then serves many calls.

Run it with `nodus run examples/python/classes/app.py`.
"""

import nodus

app = nodus.App("classes")


@app.cls(cpu=2, memory="4Gi", scaledown_window="5m", max_cost=1)
class Greeter:
    @nodus.enter()
    def load(self) -> None:
        # Runs once when a worker starts, before its first call: load weights or open connections here.
        self.greeting = "hello"

    @nodus.method()
    def greet(self, name: str) -> str:
        return f"{self.greeting}, {name}"

    @nodus.exit()
    def close(self) -> None:
        self.greeting = ""


@app.local_entrypoint()
def main() -> None:
    greeter = Greeter()
    print(greeter.greet.remote("Ada"))
    print(list(greeter.greet.map(["Grace", "Linus"])))

Load models and open connections in enter, so a call pays for them once per worker instead of once per call. A failing exit hook is logged and does not stop the others.

Changing a Function

Editing scaling takes effect on the next reconcile without restarting any worker. Changing the code, image, Python version or resources rolls the workers once: new workers start, and the old ones finish their calls and leave. nodus restart rolls them without a change.

A Function whose image is still building keeps the workers it already has and keeps serving calls until the new image is ready.