Developer docs / AI Fabric tutorials

Low-latency serving with SGLang

  • GPU
  • Class
  • Concurrency
  • Web endpoint

Take it for a spin

The whole program in one file, ready to run.

llm_sglang.py
import bladekit

app = bladekit.app("llm-sglang")

# The AI Fabric catalog ships a ready-made SGLang serving image: sglang, torch and
# the CUDA libraries are preinstalled, so there is nothing to pip install (and
# no Rust toolchain needed to build sglang's native extensions).
# The image keeps its Python packages in the forge user's site-packages while
# server containers run as root, so point PYTHONPATH at it.
image = bladekit.Image.from_catalog("fabric/sglang").env(
    {"PYTHONPATH": "/usr/local/lib/python3.12/site-packages"}
)

MODEL = "Qwen/Qwen2.5-1.5B-Instruct"  # ungated, no HF token required

# @app.web_server: the function is called ONCE at container boot. It must start
# a long-lived HTTP server on the port AI Fabric assigns ($FABRIC_SERVER_PORT) and
# return immediately (subprocess.Popen is non-blocking). AI Fabric probes the port
# until the server answers (up to startup_timeout), then forwards every request
# to it, SGLang's OpenAI-compatible routes (/v1/chat/completions, /v1/models)
# included.
#
# @app.concurrent(max_inputs=32): up to 32 requests are forwarded at once, and
# SGLang fuses them with continuous batching + RadixAttention prefix caching, so
# shared prefixes (e.g. a common system prompt) are computed once.
#
# min_replicas=1 keeps one replica warm: the model load takes longer than the
# gateway's request wake window, so a scale-to-zero first request would 503.
@app.web_server(startup_timeout=900, min_replicas=1)
@app.concurrent(max_inputs=32)
# timeout= caps ONE request; a deployed endpoint itself lives indefinitely
# under the autoscaler.
@app.function(gpu="RTX-A4500", image=image, timeout=3600)
def serve():
    import os
    import subprocess

    # Bind the port AI Fabric assigned this container; never hard-code it.
    port = os.environ["FABRIC_SERVER_PORT"]
    subprocess.Popen(
        f"python -m sglang.launch_server --model-path {MODEL}"
        f" --host 0.0.0.0 --port {port}"
        # The RTX-A4500 is sm_86: SGLang's default flashinfer attention backend
        # targets newer GPUs, but the portable Triton backend runs everywhere.
        " --attention-backend triton",
        shell=True,
    )

Run it

  1. Deploy: blade deploy llm_sglang.py.
  2. The fabric/sglang catalog image ships sglang prebuilt: no pip install, and no Rust toolchain needed for its native extensions (outlines_core).
  3. One image gotcha handled in the snippet: server containers run as root while the image keeps its packages in the forge user's site-packages, hence the PYTHONPATH env pointing at /home/forge/.local/....
  4. The A4500 is sm_86: SGLang's default flashinfer attention backend targets newer GPUs, so the snippet selects the portable Triton backend (--attention-backend triton).
  5. SGLang serves the OpenAI schema: POST <app-url>/v1/chat/completions with Authorization: Bearer <FABRIC_TOKEN>. RadixAttention gives you prefix caching: shared system prompts are computed once.

Concepts used: Cold start, Replica, Endpoint, Autoscaling, Scale-to-zero

Contact the team

Loading the form…