Developer docs / AI Fabric tutorials

Serve vLLM with continuous batching

  • GPU
  • Web endpoint
  • Concurrency
  • Image DSL

Take it for a spin

The whole program in one file, ready to run.

vllm_openai.py
import bladekit

app = bladekit.app("vllm-openai")

# vLLM ships its own OpenAI-compatible server; we just need it in the image.
# The bare debian_slim image has no CUDA toolchain (nvcc), so vLLM's flashinfer
# sampler (which JIT-compiles a kernel on first use) crashes the engine. Turn it
# off (the Torch sampler is the fallback) and add ffmpeg for torchcodec.
# Since vLLM 0.26 nothing flashinfer ships precompiled any more, so this escape
# and --enforce-eager below are what keep a toolchain-less image working. The
# alternative: start from bladekit.Image.from_catalog("fabric/vllm"), which already
# carries nvcc + ninja: it needs neither escape, runs CUDA graphs for more
# throughput, and skips this image's ~13 min first build.
vllm_image = (
    bladekit.Image.debian_slim(python_version="3.12")
    .apt_install("ffmpeg")
    .pip_install("vllm")
    .env({"VLLM_USE_FLASHINFER_SAMPLER": "0"})
)

# @app.web_server: the decorated function is called ONCE at container boot.
# It must start a long-lived HTTP server on the port AI Fabric assigns and return
# immediately (subprocess.Popen is non-blocking). AI Fabric gives each container a
# unique loopback port in $FABRIC_SERVER_PORT, probes it until it accepts
# connections (up to startup_timeout), then forwards every web request to it.
# The server owns its own routing, so the endpoint captures all paths under "/"
# (including /v1/chat/completions and /v1/models).
#
# @app.concurrent(max_inputs=32) lets up to 32 requests be forwarded at once;
# vLLM fuses them with continuous batching for real throughput.
@app.web_server(startup_timeout=600, min_replicas=1)
@app.concurrent(max_inputs=32)
# timeout= here caps ONE request (generous, for long generations). It never
# limits the service itself: a deployed endpoint lives indefinitely under the
# autoscaler (min/max_replicas, idle scale-down); there is no wall clock.
@app.function(
    gpu="RTX-A4500",
    image=vllm_image,
    timeout=3600,
    # Llama-3.1 is a gated HF repo, so inject a token via a Secret.
    secrets=[bladekit.Secret.from_name("hf-token")],
)
def serve():
    import os
    import subprocess

    # Bind the port AI Fabric assigned this container (never hard-code it: co-located
    # replicas each get a distinct port).
    port = os.environ["FABRIC_SERVER_PORT"]
    subprocess.Popen(
        # --enforce-eager: skip CUDA-graph capture, which crashes on sm_86 (A4500).
        "vllm serve meta-llama/Llama-3.1-8B-Instruct"
        f" --port {port} --max-model-len 8192 --enforce-eager",
        shell=True,
    )

Run it

  1. Save as vllm_openai.py. Deploy: blade deploy vllm_openai.py.
  2. The replica is only marked ready AFTER vLLM's server answers on the loopback, so requests never hit a cold server. startup_timeout=600 gives the weights time to load, and min_replicas=1 keeps one replica warm: an 8B model's ~5 min cold load is longer than the gateway's request wake window, so a scale-to-zero first request would otherwise time out.
  3. Call it with the official OpenAI SDK. vLLM serves the exact OpenAI schema: from openai import OpenAI client = OpenAI( base_url="https://vllm-openai.apps.run.blade.sh/v1", api_key="<FABRIC_TOKEN>") r = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hi"}]) print(r.choices[0].message.content)
  4. Why web_server over a single @app.web_endpoint handler? The handler runs one call at a time and you'd reimplement batching yourself. vLLM's server already does continuous batching; @app.concurrent lets AI Fabric forward many requests into it concurrently.
  5. Note: responses are buffered (non-streaming) for now, so call with stream=False, the SDK default. SSE token streaming is on the roadmap.

Concepts used: Endpoint, Concurrency, Replica, Cold start, Autoscaling

Contact the team

Loading the form…