Developer docs / AI Fabric tutorials

Tensor-parallel LLM on 8× RTX-A4500

  • GPU
  • Web endpoint
  • Concurrency
  • Volume

Take it for a spin

The whole program in one file, ready to run.

llm_tp.py
import bladekit

app = bladekit.app("llm-tensor-parallel")

# Model weights persist here across deploys; the ~62 GB download happens once.
weights = bladekit.Volume.from_name("llm-weights-tp", create_if_missing=True)

# Proven vLLM recipe for this fleet: bare debian_slim has no CUDA toolchain
# (nvcc), so disable the flashinfer JIT sampler; ffmpeg is needed by torchcodec.
# Since vLLM 0.26 there are no precompiled flashinfer kernels left to fall back
# on, so that escape is required, unless you start from
# bladekit.Image.from_catalog("fabric/vllm"), which ships nvcc + ninja.
# vLLM pins its own torch: never re-install torch on top of it.
# NCCL_P2P_DISABLE=1: RTX-A4500s are PCIe-only (no NVLink), so skip NCCL's
# peer-to-peer probing and route collectives through host memory.
vllm_image = (
    bladekit.Image.debian_slim(python_version="3.12")
    .apt_install("ffmpeg")
    .pip_install("vllm")
    .env(
        {
            "VLLM_USE_FLASHINFER_SAMPLER": "0",
            "HF_HOME": "/weights",
            "NCCL_P2P_DISABLE": "1",
        }
    )
)

# gpu="RTX-A4500:8": ONE replica owns all 8 GPUs of the node. Qwen2.5-32B in
# bf16 is ~62 GB of weights, far beyond a single 20 GB card, but a comfortable
# ~8 GB per GPU (plus ~8.5 GiB of KV cache each) when vLLM shards it with
# tensor parallelism.
#
# min_replicas=1 keeps the replica warm: the multi-minute download + shard load
# is far longer than the gateway's request wake window, so a scale-to-zero
# first request would time out. startup_timeout=1800 (the platform max) covers
# the one-time cold boot; the replica is only marked ready once vLLM answers.
@app.web_server(startup_timeout=1800, min_replicas=1)
# Up to 32 requests are forwarded at once; vLLM continuous-batches them
# across all 8 GPUs.
@app.concurrent(max_inputs=32)
@app.function(
    gpu="RTX-A4500:8",
    image=vllm_image,
    timeout=3600,  # per-request cap; a deployed endpoint has no wall clock
    volumes={"/weights": weights},
    # Qwen2.5 is not gated, but a token avoids HF rate limits on a 62 GB pull.
    secrets=[bladekit.Secret.from_name("hf-token")],
)
def serve():
    import os
    import subprocess

    # Bind the port AI Fabric assigned this container (never hard-code it).
    port = os.environ["FABRIC_SERVER_PORT"]
    subprocess.Popen(
        # Pre-download the weights ONCE before starting vLLM. With tp=8 all
        # 8 worker processes would otherwise hit the Hub simultaneously,
        # enough to get workers killed mid-download and crash engine startup.
        "hf download Qwen/Qwen2.5-32B-Instruct && "
        # --enforce-eager: skip CUDA-graph capture, which crashes on sm_86.
        "vllm serve Qwen/Qwen2.5-32B-Instruct"
        f" --port {port} --tensor-parallel-size 8"
        " --max-model-len 8192 --gpu-memory-utilization 0.90"
        " --enforce-eager",
        shell=True,
    )

Run it

  1. Save as llm_tp.py. Deploy: blade deploy llm_tp.py.
  2. gpu="RTX-A4500:8" gives ONE replica all 8 GPUs of the node; vLLM shards Qwen2.5-32B with --tensor-parallel-size 8: about 8 GB of weights plus ~8.5 GiB of KV cache per GPU.
  3. The weights are pre-downloaded with hf download BEFORE vllm serve starts: with tp=8, all 8 worker processes would otherwise hit the Hub at once, enough to get workers killed mid-download and crash engine startup.
  4. First boot takes ~15 min (one-time ~62 GB download to the Volume, then shard load + engine init). Later boots skip the download, and min_replicas=1 keeps the replica warm, so warm completions answer in ~1-3 s.
  5. Call it with the OpenAI SDK or curl: POST <app-url>/v1/chat/completions with Authorization: Bearer <FABRIC_TOKEN>. @app.concurrent(max_inputs=32) lets vLLM continuous-batch concurrent requests across all 8 GPUs.
  6. Redeploy gotcha on a single 8-GPU node: the old replica holds all 8 GPUs while the new one waits pending. Run blade app stop llm-tensor-parallel first, then deploy.

Concepts used: GPU, Replica, Cold start, Endpoint

Contact the team

Loading the form…