Developer docs / AI Fabric tutorials
Low-latency serving with SGLang
- GPU
- Class
- Concurrency
- Web endpoint
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("llm-sglang")
# The AI Fabric catalog ships a ready-made SGLang serving image: sglang, torch and
# the CUDA libraries are preinstalled, so there is nothing to pip install (and
# no Rust toolchain needed to build sglang's native extensions).
# The image keeps its Python packages in the forge user's site-packages while
# server containers run as root, so point PYTHONPATH at it.
image = bladekit.Image.from_catalog("fabric/sglang").env(
{"PYTHONPATH": "/usr/local/lib/python3.12/site-packages"}
)
MODEL = "Qwen/Qwen2.5-1.5B-Instruct" # ungated, no HF token required
# @app.web_server: the function is called ONCE at container boot. It must start
# a long-lived HTTP server on the port AI Fabric assigns ($FABRIC_SERVER_PORT) and
# return immediately (subprocess.Popen is non-blocking). AI Fabric probes the port
# until the server answers (up to startup_timeout), then forwards every request
# to it, SGLang's OpenAI-compatible routes (/v1/chat/completions, /v1/models)
# included.
#
# @app.concurrent(max_inputs=32): up to 32 requests are forwarded at once, and
# SGLang fuses them with continuous batching + RadixAttention prefix caching, so
# shared prefixes (e.g. a common system prompt) are computed once.
#
# min_replicas=1 keeps one replica warm: the model load takes longer than the
# gateway's request wake window, so a scale-to-zero first request would 503.
@app.web_server(startup_timeout=900, min_replicas=1)
@app.concurrent(max_inputs=32)
# timeout= caps ONE request; a deployed endpoint itself lives indefinitely
# under the autoscaler.
@app.function(gpu="RTX-A4500", image=image, timeout=3600)
def serve():
import os
import subprocess
# Bind the port AI Fabric assigned this container; never hard-code it.
port = os.environ["FABRIC_SERVER_PORT"]
subprocess.Popen(
f"python -m sglang.launch_server --model-path {MODEL}"
f" --host 0.0.0.0 --port {port}"
# The RTX-A4500 is sm_86: SGLang's default flashinfer attention backend
# targets newer GPUs, but the portable Triton backend runs everywhere.
" --attention-backend triton",
shell=True,
)Run it
- Deploy:
blade deploy llm_sglang.py. - The
fabric/sglangcatalog image ships sglang prebuilt: nopip install, and no Rust toolchain needed for its native extensions (outlines_core). - One image gotcha handled in the snippet: server containers run as root while the image keeps its packages in the forge user's
site-packages, hence thePYTHONPATHenv pointing at/home/forge/.local/.... - The A4500 is
sm_86: SGLang's default flashinfer attention backend targets newer GPUs, so the snippet selects the portable Triton backend (--attention-backend triton). - SGLang serves the OpenAI schema: POST
<app-url>/v1/chat/completionswithAuthorization: Bearer <FABRIC_TOKEN>. RadixAttention gives you prefix caching: shared system prompts are computed once.
Concepts used: Cold start, Replica, Endpoint, Autoscaling, Scale-to-zero