Developer docs / AI Fabric tutorials

Hybrid capacity: non-interruptible floor + interruptible burst

  • GPU
  • Web endpoint
  • Concurrency

Take it for a spin

The whole program in one file, ready to run.

vllm_hybrid.py
import bladekit

app = bladekit.app("vllm-hybrid")

# Same recipe as the vllm-openai-server tutorial: disable the flashinfer sampler
# (no nvcc in debian_slim) and add ffmpeg. Qwen2.5-3B is NOT gated, so no token.
# Mandatory since vLLM 0.26, which stopped shipping precompiled flashinfer
# kernels. Or start from bladekit.Image.from_catalog("fabric/vllm"), which has the
# toolchain and needs no escape at all.
vllm_image = (
    bladekit.Image.debian_slim(python_version="3.12")
    .apt_install("ffmpeg")
    .pip_install("vllm")
    .env({"VLLM_USE_FLASHINFER_SAMPLER": "0"})
)

# Hybrid capacity:
#   gpu_class="spot"        → the BURST class (interruptible, cheap)
#   reserved_replicas=2     → 2 replicas always kept NON-INTERRUPTIBLE (never preempted)
#   min_replicas=2          → keep the non-interruptible floor up even with zero traffic
#   max_replicas=8          → burst up to 6 extra replicas, all interruptible
#
# AI Fabric fills the non-interruptible floor first, then overflows to interruptible
# capacity. If an interruptible replica is preempted, the non-interruptible floor is
# unaffected, so your baseline keeps serving.
@app.web_server(
    startup_timeout=600,
    min_replicas=2,
    max_replicas=8,
    reserved_replicas=2,
)
@app.function(
    image=vllm_image,
    gpu="RTX-A4500",
    gpu_class="spot",
)
@app.concurrent(max_inputs=32)
def serve():
    import os
    import subprocess

    port = os.environ["FABRIC_SERVER_PORT"]
    subprocess.Popen(
        # --enforce-eager: skip CUDA-graph capture (crashes on sm_86 / A4500).
        "vllm serve Qwen/Qwen2.5-3B-Instruct"
        f" --port {port} --max-model-len 8192 --enforce-eager",
        shell=True,
    )

Run it

  1. Deploy: blade deploy vllm_hybrid.py. The first reserved_replicas (2) replicas land on non-interruptible capacity; anything the autoscaler adds above that goes to interruptible capacity.
  2. Inspect the split on the app page (/apps/vllm-hybrid): the endpoint shows a non-interruptible badge, and each replica's run detail shows its capacity class (non-interruptible vs interruptible).
  3. Tune it live without redeploying. In the UI, go to Apps → your app → Edit the replica bounds (min/max/non-interruptible). Or use the CLI: blade app scale vllm-hybrid --min 2 --max 12 --reserved 3. The autoscaler applies the new floor within seconds.
  4. Why hybrid? Pure interruptible capacity is cheapest but a preemption can take your last replica down; pure non-interruptible is safe but you pay the premium on every burst. Hybrid keeps a guaranteed baseline on non-interruptible capacity and pays the interruptible discount on the variable part.
  5. Keep min_replicas >= reserved_replicas so the non-interruptible floor is always running. The floor self-heals: if a non-interruptible replica dies, the autoscaler respawns it on non-interruptible capacity on the next tick.
  6. Requires non-interruptible capacity in the fleet (nodes provisioned with --capacity-class reserved). Without any non-interruptible node, the floor replicas stay pending until non-interruptible capacity exists; the interruptible burst still runs.

Concepts used: Autoscaling, Replica, Endpoint, Cold start, Concurrency

Contact the team

Loading the form…