Developer docs / AI Fabric tutorials
Serve vLLM with continuous batching
- GPU
- Web endpoint
- Concurrency
- Image DSL
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("vllm-openai")
# vLLM ships its own OpenAI-compatible server; we just need it in the image.
# The bare debian_slim image has no CUDA toolchain (nvcc), so vLLM's flashinfer
# sampler (which JIT-compiles a kernel on first use) crashes the engine. Turn it
# off (the Torch sampler is the fallback) and add ffmpeg for torchcodec.
# Since vLLM 0.26 nothing flashinfer ships precompiled any more, so this escape
# and --enforce-eager below are what keep a toolchain-less image working. The
# alternative: start from bladekit.Image.from_catalog("fabric/vllm"), which already
# carries nvcc + ninja: it needs neither escape, runs CUDA graphs for more
# throughput, and skips this image's ~13 min first build.
vllm_image = (
bladekit.Image.debian_slim(python_version="3.12")
.apt_install("ffmpeg")
.pip_install("vllm")
.env({"VLLM_USE_FLASHINFER_SAMPLER": "0"})
)
# @app.web_server: the decorated function is called ONCE at container boot.
# It must start a long-lived HTTP server on the port AI Fabric assigns and return
# immediately (subprocess.Popen is non-blocking). AI Fabric gives each container a
# unique loopback port in $FABRIC_SERVER_PORT, probes it until it accepts
# connections (up to startup_timeout), then forwards every web request to it.
# The server owns its own routing, so the endpoint captures all paths under "/"
# (including /v1/chat/completions and /v1/models).
#
# @app.concurrent(max_inputs=32) lets up to 32 requests be forwarded at once;
# vLLM fuses them with continuous batching for real throughput.
@app.web_server(startup_timeout=600, min_replicas=1)
@app.concurrent(max_inputs=32)
# timeout= here caps ONE request (generous, for long generations). It never
# limits the service itself: a deployed endpoint lives indefinitely under the
# autoscaler (min/max_replicas, idle scale-down); there is no wall clock.
@app.function(
gpu="RTX-A4500",
image=vllm_image,
timeout=3600,
# Llama-3.1 is a gated HF repo, so inject a token via a Secret.
secrets=[bladekit.Secret.from_name("hf-token")],
)
def serve():
import os
import subprocess
# Bind the port AI Fabric assigned this container (never hard-code it: co-located
# replicas each get a distinct port).
port = os.environ["FABRIC_SERVER_PORT"]
subprocess.Popen(
# --enforce-eager: skip CUDA-graph capture, which crashes on sm_86 (A4500).
"vllm serve meta-llama/Llama-3.1-8B-Instruct"
f" --port {port} --max-model-len 8192 --enforce-eager",
shell=True,
)Run it
- Save as
vllm_openai.py. Deploy:blade deploy vllm_openai.py. - The replica is only marked ready AFTER vLLM's server answers on the loopback, so requests never hit a cold server.
startup_timeout=600gives the weights time to load, andmin_replicas=1keeps one replica warm: an 8B model's ~5 min cold load is longer than the gateway's request wake window, so a scale-to-zero first request would otherwise time out. - Call it with the official OpenAI SDK. vLLM serves the exact OpenAI schema: from openai import OpenAI client = OpenAI( base_url="https://vllm-openai.apps.run.blade.sh/v1", api_key="<FABRIC_TOKEN>") r = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hi"}]) print(r.choices[0].message.content)
- Why
web_serverover a single@app.web_endpointhandler? The handler runs one call at a time and you'd reimplement batching yourself. vLLM's server already does continuous batching;@app.concurrentlets AI Fabric forward many requests into it concurrently. - Note: responses are buffered (non-streaming) for now, so call with
stream=False, the SDK default. SSE token streaming is on the roadmap.
Concepts used: Endpoint, Concurrency, Replica, Cold start, Autoscaling