Developer docs / AI Fabric tutorials
Tensor-parallel LLM on 8× RTX-A4500
- GPU
- Web endpoint
- Concurrency
- Volume
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("llm-tensor-parallel")
# Model weights persist here across deploys; the ~62 GB download happens once.
weights = bladekit.Volume.from_name("llm-weights-tp", create_if_missing=True)
# Proven vLLM recipe for this fleet: bare debian_slim has no CUDA toolchain
# (nvcc), so disable the flashinfer JIT sampler; ffmpeg is needed by torchcodec.
# Since vLLM 0.26 there are no precompiled flashinfer kernels left to fall back
# on, so that escape is required, unless you start from
# bladekit.Image.from_catalog("fabric/vllm"), which ships nvcc + ninja.
# vLLM pins its own torch: never re-install torch on top of it.
# NCCL_P2P_DISABLE=1: RTX-A4500s are PCIe-only (no NVLink), so skip NCCL's
# peer-to-peer probing and route collectives through host memory.
vllm_image = (
bladekit.Image.debian_slim(python_version="3.12")
.apt_install("ffmpeg")
.pip_install("vllm")
.env(
{
"VLLM_USE_FLASHINFER_SAMPLER": "0",
"HF_HOME": "/weights",
"NCCL_P2P_DISABLE": "1",
}
)
)
# gpu="RTX-A4500:8": ONE replica owns all 8 GPUs of the node. Qwen2.5-32B in
# bf16 is ~62 GB of weights, far beyond a single 20 GB card, but a comfortable
# ~8 GB per GPU (plus ~8.5 GiB of KV cache each) when vLLM shards it with
# tensor parallelism.
#
# min_replicas=1 keeps the replica warm: the multi-minute download + shard load
# is far longer than the gateway's request wake window, so a scale-to-zero
# first request would time out. startup_timeout=1800 (the platform max) covers
# the one-time cold boot; the replica is only marked ready once vLLM answers.
@app.web_server(startup_timeout=1800, min_replicas=1)
# Up to 32 requests are forwarded at once; vLLM continuous-batches them
# across all 8 GPUs.
@app.concurrent(max_inputs=32)
@app.function(
gpu="RTX-A4500:8",
image=vllm_image,
timeout=3600, # per-request cap; a deployed endpoint has no wall clock
volumes={"/weights": weights},
# Qwen2.5 is not gated, but a token avoids HF rate limits on a 62 GB pull.
secrets=[bladekit.Secret.from_name("hf-token")],
)
def serve():
import os
import subprocess
# Bind the port AI Fabric assigned this container (never hard-code it).
port = os.environ["FABRIC_SERVER_PORT"]
subprocess.Popen(
# Pre-download the weights ONCE before starting vLLM. With tp=8 all
# 8 worker processes would otherwise hit the Hub simultaneously,
# enough to get workers killed mid-download and crash engine startup.
"hf download Qwen/Qwen2.5-32B-Instruct && "
# --enforce-eager: skip CUDA-graph capture, which crashes on sm_86.
"vllm serve Qwen/Qwen2.5-32B-Instruct"
f" --port {port} --tensor-parallel-size 8"
" --max-model-len 8192 --gpu-memory-utilization 0.90"
" --enforce-eager",
shell=True,
)Run it
- Save as
llm_tp.py. Deploy:blade deploy llm_tp.py. gpu="RTX-A4500:8"gives ONE replica all 8 GPUs of the node; vLLM shards Qwen2.5-32B with--tensor-parallel-size 8: about 8 GB of weights plus ~8.5 GiB of KV cache per GPU.- The weights are pre-downloaded with
hf downloadBEFOREvllm servestarts: withtp=8, all 8 worker processes would otherwise hit the Hub at once, enough to get workers killed mid-download and crash engine startup. - First boot takes ~15 min (one-time ~62 GB download to the Volume, then shard load + engine init). Later boots skip the download, and
min_replicas=1keeps the replica warm, so warm completions answer in ~1-3 s. - Call it with the OpenAI SDK or curl: POST
<app-url>/v1/chat/completionswithAuthorization: Bearer <FABRIC_TOKEN>.@app.concurrent(max_inputs=32)lets vLLM continuous-batch concurrent requests across all 8 GPUs. - Redeploy gotcha on a single 8-GPU node: the old replica holds all 8 GPUs while the new one waits pending. Run
blade app stop llm-tensor-parallelfirst, then deploy.
Concepts used: GPU, Replica, Cold start, Endpoint