Autoscaling

Overview

Every endpoint kind — @app.web_endpoint, @app.web_server, @app.asgi_app — autoscales its replica count from live traffic. By default the signal is per-replica concurrency against max_concurrent_inputs: the more calls in flight relative to that ceiling, the more replicas AI Fabric runs, between min_replicas and max_replicas.

bladekit.Autoscale, passed as autoscale= on any of the three endpoint decorators, is autoscaling v2: four additive knobs layered on top of that default — target_inputs (a lower, latency-oriented concurrency target), buffer_replicas (warm spares while the endpoint is active), vllm_target_waiting (a curated preset that scales on vLLM's own scheduler queue instead of raw concurrency), and scaledown_window (the stabilization window before surplus replicas are shed). Every knob is optional, and none of them replace the guardrails: min/max clamp, scale-up cooldown, and the circuit breaker always win, however the fleet size a policy proposes.

Replica bounds & concurrency

min_replicas, max_replicas, max_concurrent_inputs, scale_down_idle_seconds, max_queue_depth and reserved_replicas are keyword-only arguments accepted — with identical validation — by all three endpoint decorators:

@app.web_endpoint(   # same kwargs on @app.web_server / @app.asgi_app
    *,
    min_replicas: Optional[int] = None,
    max_replicas: Optional[int] = None,
    max_concurrent_inputs: Optional[int] = None,
    scale_down_idle_seconds: Optional[int] = None,
    max_queue_depth: Optional[int] = None,
    reserved_replicas: Optional[int] = None,
    ...
)

min_replicas Optional[int]Default: None → 0

Floor replica count, 0–50. 0 (the default) allows scale-to-zero. Cannot exceed max_replicas — else InvalidArgumentError (AUTOSCALE_LIMIT_INVALID).

max_replicas Optional[int]Default: None → 1

Ceiling replica count, 1–50. Left unset, the endpoint is capped at a single replica — see the note above.

max_concurrent_inputs Optional[int]Default: None → 1

Per-replica admission ceiling for concurrent calls, 1–100. Also settable via @app.concurrent(max_inputs=...); if both are given on the same function, this endpoint-level value wins.

scale_down_idle_seconds Optional[int]Default: None → 300

Seconds without pending or running work before scale-down, 30–3600. It governs both scale-down clocks: the descent to min_replicas (including scale-to-zero), and the floor of the stabilization window that sheds surplus replicas above the demand target (override the latter alone with Autoscale.scaledown_window).

max_queue_depth Optional[int]Default: None → 0 (unbounded)

Ingress queue bound, 0–10000. Past this many requests waiting for a replica slot, new ones are rejected immediately with 503 ENDPOINT_QUEUE_FULL + Retry-After instead of waiting out a cold start. 0/unset = unbounded (the historical zero-rejection behavior). Covers the platform-mediated path only — requests batching inside a warm web_server are the server's own backlog. Admitted requests carry X-Forge-Queue-Position and, when it can be estimated, X-Forge-Queue-Eta-Ms.

reserved_replicas Optional[int]Default: None → 0 (no floor)

Hybrid-capacity always-on floor — see Hybrid capacity below. Must be between 0 and the endpoint's effective max_replicas.

bladekit.Autoscale

bladekit.Autoscale is a frozen dataclass passed as autoscale= to an endpoint decorator. It carries the autoscaling-v2 policy that merges with the default concurrency autoscaler — it never overrides min_replicas / max_replicas / the guardrails.

@dataclass(frozen=True)
class Autoscale:
    target_inputs: Optional[int] = None
    buffer_replicas: Optional[int] = None
    vllm_target_waiting: Optional[int] = None
    scaledown_window: Optional[int] = None

target_inputs Optional[int]Default: None

Per-replica concurrency the autoscaler aims for, 1–100 — lower than max_inputs (from @app.concurrent), which stays the admission ceiling replicas may burst to while new capacity boots. Scale before saturation instead of at it.

buffer_replicas Optional[int]Default: None

Warm spare replicas kept only while the endpoint is active, 0–10. An idle endpoint still settles down to min_replicas — the buffer doesn't change scale-to-zero behavior.

vllm_target_waiting Optional[int]Default: None

The curated vLLM preset, 1–1000 — scale on vLLM's own scheduler queue-depth metric instead of raw concurrency, so continuous-batching pressure is visible to the autoscaler. The platform owns the metrics plumbing: no exporter config, no PromQL. **@app.web_server endpoints only.**

scaledown_window Optional[int]Default: None

Stabilization window (seconds, 30–3600) of the proportional scale-down: how long the endpoint must be continuously observed needing fewer replicas before a surplus one is shed. Unset = the larger of the server default and scale_down_idle_seconds. Never affects scale-up, the hard max_replicas cap, or scale-to-zero. Requires a control plane of 2026-08-11 or newer — older ones reject the unknown policy field with SCALING_POLICY_INVALID.

@app.concurrent & target_inputs

@app.concurrent sets the per-replica HTTP concurrency ceiling on the underlying function, and can declare target_inputs as an alternative to putting it on bladekit.Autoscale.

@app.concurrent(
    *,
    max_inputs: Optional[int] = None,
    max_concurrent_inputs: Optional[int] = None,
    target_inputs: Optional[int] = None,
)

max_inputs Optional[int]

Admission ceiling per replica, 1–100. Required — either this or max_concurrent_inputs must be given, else InvalidArgumentError (INVALID_ARGUMENT).

max_concurrent_inputs Optional[int]

Alias for max_inputs; takes precedence if both are passed.

target_inputs Optional[int]Default: None

Same meaning as Autoscale.target_inputs (1–100), declared at the function level instead. Cannot exceed max_inputs — checked immediately when the decorator is applied (SCALING_POLICY_INVALID).

Hybrid capacity: reserved_replicas

A function's gpu_class (@app.function(gpu_class="spot"|"reserved")) picks its capacity class: **spot is the cheap, interruptible default; reserved** is non-interruptible and guarantees no preemption. reserved_replicas, set on the endpoint decorator, turns this into a hybrid policy: it keeps that many replicas always-on on non-interruptible capacity as a floor, while the endpoint bursts up to max_replicas on its regular (typically interruptible) capacity to absorb load spikes.

Example: hybrid vLLM endpoint

A web_server endpoint with a warm non-interruptible floor, interruptible burst capacity, and the vLLM queue-depth preset:

import bladekit

app = bladekit.app("llama-server")
vllm_image = bladekit.Image.from_catalog("fabric/vllm")

@app.web_server(
    startup_timeout=600,
    min_replicas=1,
    max_replicas=6,
    reserved_replicas=1,          # 1 always-on replica on non-interruptible capacity
    autoscale=bladekit.Autoscale(
        target_inputs=24,
        buffer_replicas=1,
        vllm_target_waiting=4,    # aim for <=4 requests waiting per replica
    ),
)
@app.concurrent(max_inputs=32)
@app.function(gpu="RTX-A4500", image=vllm_image, gpu_class="spot")
def serve():
    import os, subprocess

    port = os.environ["FABRIC_SERVER_PORT"]
    subprocess.Popen(
        ["vllm", "serve", "meta-llama/Llama-3.1-8B-Instruct", "--port", port],
    )

Related reference