Developer docs / AI Fabric tutorials
Hybrid capacity: non-interruptible floor + interruptible burst
- GPU
- Web endpoint
- Concurrency
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("vllm-hybrid")
# Same recipe as the vllm-openai-server tutorial: disable the flashinfer sampler
# (no nvcc in debian_slim) and add ffmpeg. Qwen2.5-3B is NOT gated, so no token.
# Mandatory since vLLM 0.26, which stopped shipping precompiled flashinfer
# kernels. Or start from bladekit.Image.from_catalog("fabric/vllm"), which has the
# toolchain and needs no escape at all.
vllm_image = (
bladekit.Image.debian_slim(python_version="3.12")
.apt_install("ffmpeg")
.pip_install("vllm")
.env({"VLLM_USE_FLASHINFER_SAMPLER": "0"})
)
# Hybrid capacity:
# gpu_class="spot" → the BURST class (interruptible, cheap)
# reserved_replicas=2 → 2 replicas always kept NON-INTERRUPTIBLE (never preempted)
# min_replicas=2 → keep the non-interruptible floor up even with zero traffic
# max_replicas=8 → burst up to 6 extra replicas, all interruptible
#
# AI Fabric fills the non-interruptible floor first, then overflows to interruptible
# capacity. If an interruptible replica is preempted, the non-interruptible floor is
# unaffected, so your baseline keeps serving.
@app.web_server(
startup_timeout=600,
min_replicas=2,
max_replicas=8,
reserved_replicas=2,
)
@app.function(
image=vllm_image,
gpu="RTX-A4500",
gpu_class="spot",
)
@app.concurrent(max_inputs=32)
def serve():
import os
import subprocess
port = os.environ["FABRIC_SERVER_PORT"]
subprocess.Popen(
# --enforce-eager: skip CUDA-graph capture (crashes on sm_86 / A4500).
"vllm serve Qwen/Qwen2.5-3B-Instruct"
f" --port {port} --max-model-len 8192 --enforce-eager",
shell=True,
)Run it
- Deploy:
blade deploy vllm_hybrid.py. The firstreserved_replicas(2) replicas land on non-interruptible capacity; anything the autoscaler adds above that goes to interruptible capacity. - Inspect the split on the app page (
/apps/vllm-hybrid): the endpoint shows a non-interruptible badge, and each replica's run detail shows its capacity class (non-interruptible vs interruptible). - Tune it live without redeploying. In the UI, go to Apps → your app → Edit the replica bounds (min/max/non-interruptible). Or use the CLI:
blade app scale vllm-hybrid --min 2 --max 12 --reserved 3. The autoscaler applies the new floor within seconds. - Why hybrid? Pure interruptible capacity is cheapest but a preemption can take your last replica down; pure non-interruptible is safe but you pay the premium on every burst. Hybrid keeps a guaranteed baseline on non-interruptible capacity and pays the interruptible discount on the variable part.
- Keep
min_replicas >= reserved_replicasso the non-interruptible floor is always running. The floor self-heals: if a non-interruptible replica dies, the autoscaler respawns it on non-interruptible capacity on the next tick. - Requires non-interruptible capacity in the fleet (nodes provisioned with
--capacity-class reserved). Without any non-interruptible node, the floor replicas stay pending until non-interruptible capacity exists; the interruptible burst still runs.
Concepts used: Autoscaling, Replica, Endpoint, Cold start, Concurrency