Web endpoints

@app.web_endpoint

Expose a Python function through HTTP call routing: each incoming request becomes one invocation of the handler. Usable alone (it registers a function with default resources) or **stacked with @app.function / @app.concurrent in either order** — decorator order is not significant.

Signature

@app.web_endpoint(
    *,
    path: Optional[str] = None,
    methods: Optional[Sequence[str]] = None,
    name: Optional[str] = None,
    min_replicas: Optional[int] = None,
    max_replicas: Optional[int] = None,
    max_concurrent_inputs: Optional[int] = None,
    scale_down_idle_seconds: Optional[int] = None,
    max_queue_depth: Optional[int] = None,
    reserved_replicas: Optional[int] = None,
    public: bool = False,
    autoscale: Optional[bladekit.Autoscale] = None,
) -> Function

path Optional[str]Default: /<function name>: Route path. Must start with / and be at most 1024 characters, else raises InvalidArgumentError.
methods Optional[Sequence[str]]Default: ("POST",): Accepted HTTP methods. Upper-cased and validated against GET, POST, PUT, PATCH, DELETE, OPTIONS, HEAD; must be unique, non-empty, and at most 8 entries.
name Optional[str]Default: None: Endpoint name in the deployment manifest. Defaults to the function's name.
min_replicas Optional[int]Default: None: Replica floor (0–50). 0 allows scale-to-zero when idle. Wire default is 0.
max_replicas Optional[int]Default: None: Replica ceiling (1–50). Wire default is 1. Must be >= min_replicas.
max_concurrent_inputs Optional[int]Default: None: Per-replica in-flight HTTP call ceiling (1–100). Usually set with @app.concurrent instead; wire default is 1.
scale_down_idle_seconds Optional[int]Default: None: Idle seconds before scale-down (30–3600). Wire default is 300. Governs both the descent to min_replicas and the floor of the surplus-shedding stabilization window.
max_queue_depth Optional[int]Default: None: Ingress queue bound (0–10000). Past this many requests waiting for a replica slot, new ones get an immediate 503 ENDPOINT_QUEUE_FULL + Retry-After instead of waiting. Wire default is 0 = unbounded (the historical behavior).
reserved_replicas Optional[int]Default: None: Always-on floor served on non-interruptible capacity (0–max_replicas).
public boolDefault: False: Endpoints are private by default — the /serve ingress requires a valid user token. public=True opts out (no auth), so the URL opens directly.
autoscale Optional[bladekit.Autoscale]Default: None: Autoscaling-v2 policy. vllm_target_waiting inside it is only valid on @app.web_server and is rejected here at decorator time.

The handler receives a single argument: the parsed JSON body (when the request is application/json with a JSON object body) — faithful to Modal's ergonomics, so item["prompt"] works directly. The argument is a dict subclass that also carries the full HTTP envelope as attributes: item.method, item.path, item.query_string, item.headers, item.body (raw bytes), item.json, item.endpoint_id, item.app_id, item.deployment_id, plus item.request for the whole envelope as a dict.

Minimal handler

import bladekit

app = bladekit.app("api")

@app.web_endpoint(methods=["POST"])
def predict(item):
    prompt = item["prompt"]          # item IS the decoded JSON body
    return {"echo": prompt}

Stacked with @app.function (GPU, custom resources)

@app.web_endpoint(methods=["POST"], max_replicas=4)
@app.function(gpu="RTX-A4500", image=bladekit.Image.from_catalog("fabric/torch"))
def classify(item):
    ...

@app.web_server

Serve a long-lived HTTP server running inside the container rather than a handler called per request. The decorated function is invoked once at container boot to start the server — typically a non-blocking subprocess.Popen(["vllm", "serve", …]). AI Fabric probes the loopback until it accepts connections (up to startup_timeout), then forwards every request whose path matches the endpoint's path prefix — regardless of method, since the server owns its own routing. This is the path that serves vLLM and other OpenAI-compatible servers with continuous batching.

Signature

@app.web_server(
    *,
    port: Optional[int] = None,
    startup_timeout: int = 300,
    path: str = "/",
    name: Optional[str] = None,
    min_replicas: Optional[int] = None,
    max_replicas: Optional[int] = None,
    max_concurrent_inputs: Optional[int] = None,
    scale_down_idle_seconds: Optional[int] = None,
    max_queue_depth: Optional[int] = None,
    reserved_replicas: Optional[int] = None,
    public: bool = False,
    autoscale: Optional[bladekit.Autoscale] = None,
    snapshot: bool = False,
) -> Function

port Optional[int]Default: None: Passing a fixed port is rejected (InvalidArgumentError). Each container is assigned a node-unique loopback port via FABRIC_SERVER_PORT; your server must bind $FABRIC_SERVER_PORT.
startup_timeout intDefault: 300: Seconds the runner probes the loopback for readiness (10–1800). The replica fails if the server never answers within the budget.
path strDefault: "/": Path prefix the endpoint owns (all sub-paths, every method). / captures the whole app's URL space.
name Optional[str]Default: None: Endpoint name; defaults to the function's name.
snapshot boolDefault: False: Opt into GPU checkpoint/restore (checkpoint a warm replica at drain, restore on scale-up, fall back to a normal boot on error). Valid only on a single-GPU web_server (gpu_count == 1); otherwise raises InvalidArgumentError at deploy build time.

min_replicas, max_replicas, max_concurrent_inputs, scale_down_idle_seconds, max_queue_depth, reserved_replicas, public and autoscale behave exactly as on @app.web_endpoint. Combine with @app.concurrent(max_inputs=N) so multiple requests are forwarded concurrently — this is what enables server-side continuous batching (e.g. vLLM).

vLLM server with continuous batching

import os, subprocess

@app.web_server(startup_timeout=600)
@app.concurrent(max_inputs=32)
@app.function(gpu="RTX-A4500", image=bladekit.Image.from_catalog("fabric/vllm"))
def serve():
    port = os.environ["FABRIC_SERVER_PORT"]     # bind the assigned port
    subprocess.Popen(
        ["vllm", "serve", "meta-llama/Llama-3.1-8B-Instruct", "--port", port],
    )

Important

A local-server replica is marked ready (routable) only after it reports serving-ready, not at container RUNNING, so requests never hit a server that isn't listening yet. SSE streams end to end: when the upstream replies with text/event-stream (e.g. vLLM's OpenAI-compatible API with stream=true), the body is not buffered: each chunk is flushed to the client as the server emits it. WebSocket works end to end, from any state: warm traffic upgrades through the serve-proxy (a transparent reverse proxy relaying frames bidirectionally; the Moshi voice-chat tutorial streams Opus audio both ways over one), and a handshake against a scaled-to-zero app wakes a replica inside the handshake and then connects, with no client retry. Open sessions count as live load: the autoscaler never scales an endpoint to zero under them (and prefers session-free replicas when shedding). One session is bounded to 24h by default (FABRIC_MAX_SESSION_SECONDS); timeout_seconds gates only the handshake, never the established session. Deploys and drains still close sessions after the grace, so clients should reconnect.

@app.asgi_app

Mount an ASGI app (FastAPI, Starlette, …) as an endpoint. The decorated function returns the ASGI app object; at boot the runner allocates a loopback port, starts uvicorn on the returned app, and forwards each request to it. uvicorn must be present in the image. The endpoint captures every method under path. Stackable with @app.function / @app.concurrent like @app.web_endpoint.

Signature

@app.asgi_app(
    *,
    path: str = "/",
    startup_timeout: int = 300,
    name: Optional[str] = None,
    min_replicas: Optional[int] = None,
    max_replicas: Optional[int] = None,
    max_concurrent_inputs: Optional[int] = None,
    scale_down_idle_seconds: Optional[int] = None,
    max_queue_depth: Optional[int] = None,
    reserved_replicas: Optional[int] = None,
    public: bool = False,
    autoscale: Optional[bladekit.Autoscale] = None,
) -> Function

path strDefault: "/": Path prefix the ASGI app owns (all sub-paths, every method).
startup_timeout intDefault: 300: Seconds the runner probes uvicorn's loopback for readiness (10–1800) before marking the replica ready.
name Optional[str]Default: None: Endpoint name; defaults to the function's name.

min_replicas, max_replicas, max_concurrent_inputs, scale_down_idle_seconds, max_queue_depth, reserved_replicas, public and autoscale behave exactly as on @app.web_endpoint. Set public=True for an app that serves a browser UI (or WebRTC signaling) so the URL opens without a token.

FastAPI app

@app.asgi_app(public=True)
@app.function(image=bladekit.Image.from_catalog("fabric/base"))
def web():
    from fastapi import FastAPI

    api = FastAPI()

    @api.get("/health")
    def health():
        return {"ok": True}

    return api

@app.concurrent

Set the per-replica HTTP concurrency for an endpoint. max_inputs is the admission ceiling per replica; the optional target_inputs is what the autoscaler aims for (autoscaling v2): the fleet is sized to hold roughly target_inputs in-flight calls per replica, and the margin up to max_inputs absorbs bursts while new replicas boot. Stacks with the endpoint decorators and @app.function in either order.

Signature

@app.concurrent(
    *,
    max_inputs: Optional[int] = None,
    max_concurrent_inputs: Optional[int] = None,
    target_inputs: Optional[int] = None,
) -> Function

max_inputs Optional[int]Default: None: Per-replica admission ceiling (1–100). Required unless max_concurrent_inputs is given.
max_concurrent_inputs Optional[int]Default: None: Alias for max_inputs — exactly one of the two must be provided, else raises InvalidArgumentError.
target_inputs Optional[int]Default: None: In-flight calls per replica the autoscaler targets (1–100). Cannot exceed max_inputs.

Pick target from latency budget, max from resource limits

@app.web_endpoint(methods=["POST"])
@app.concurrent(max_inputs=32, target_inputs=24)
@app.function(gpu="RTX-A4500", image=bladekit.Image.from_catalog("fabric/torch"))
def infer(item):
    ...

Endpoints on @app.cls methods

@app.web_endpoint, @app.web_server and @app.asgi_app can decorate a **method of an @app.cls class. This is fully supported: the endpoint method inherits the class's image and resources** (gpu, cpu, memory, timeout, idle_timeout, volumes, secrets, gpu_class, best_effort) for anything the method does not set itself — exactly like a plain @app.method. The runner instantiates the class once per container and routes requests to the method, so @bladekit.enter() model-load hooks run before the first call.

web_endpoint on a class method

@app.cls(gpu="RTX-A4500", image=bladekit.Image.from_catalog("fabric/torch"))
class Classifier:
    @bladekit.enter()
    def load(self):
        self.model = load_model()      # once per container

    @app.web_endpoint(methods=["POST"])
    def predict(self, item):
        return {"label": self.model(item["text"])}

Functions & calls →Core API Autoscaling →Serving & autoscaling Images →Images & storage Configuration →CLI & configuration

@app.web_endpoint

@app.web_server

@app.asgi_app

@app.concurrent

Endpoints on @app.cls methods

Related reference