Developer docs / AI Fabric tutorials
Stateful LLM service with @app.cls
- GPU
- Web endpoint
- Class
- Concurrency
The fastest way to serve a model is to load it once and reuse it for every request, but a module-level global is an awkward place to keep that state. A Python class is the natural fit: the weights live on self, loaded in __init__, and every method gets them for free. In this tutorial you'll wrap Qwen 2.5 (1.5B) in an AI Fabric class so the model loads exactly once per replica and a single GPU can answer many requests at the same time.
Two decorators do the work. @app.cls turns the class into a GPU-backed service whose __init__ runs when a replica starts, and @app.concurrent(max_inputs=16) lets one replica handle up to 16 simultaneous requests instead of serializing them. The result is cleaner than a global and gets far more out of each GPU.
Before you start
You need the AI Fabric CLI installed and authenticated: run blade setup once. There's nothing to build locally: the class runs on fabric/torch, a catalog image that already ships CUDA, PyTorch and transformers, and asks for a single RTX A4500.
Create the app
Every AI Fabric program starts with an App, the namespace your functions and classes live under. Create a file called llm_cls.py and declare it:
import bladekit
app = bladekit.app("llm-cls") Load the model once in __init__
Decorate the class with @app.cls to request the GPU and image, and @app.concurrent(max_inputs=16) so one replica can fan out across many requests. AI Fabric runs __init__ once when the replica boots. That's where you download Qwen 2.5 and move it onto the GPU, then stash the tokenizer and model on self:
# fabric/torch catalog image: CUDA + PyTorch + transformers already installed.
@app.cls(gpu="RTX-A4500", image="fabric/torch", timeout=600)
@app.concurrent(max_inputs=16)
class LLMService:
def __init__(self):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "Qwen/Qwen2.5-1.5B-Instruct"
self.tok = AutoTokenizer.from_pretrained(name)
self.model = AutoModelForCausalLM.from_pretrained(
name, torch_dtype=torch.float16, device_map="cuda"
)Add a web endpoint method
A method decorated with @app.web_endpoint becomes an HTTP handler that runs on a live replica, so it skips straight to inference, since self.tok and self.model are already loaded. Format the prompt with the model's chat template, generate, and decode only the new tokens so the caller gets the reply, not their own question echoed back:
@app.web_endpoint(methods=["POST"])
def generate(self, item: dict) -> dict:
messages = [{"role": "user", "content": item.get("prompt", "")}]
chat = self.tok.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = self.tok(chat, return_tensors="pt").to("cuda")
out = self.model.generate(**inputs, max_new_tokens=256, do_sample=False)
n_in = inputs["input_ids"].shape[1]
return {"text": self.tok.decode(out[0][n_in:], skip_special_tokens=True)}Because the method lives on the same replica that loaded the model, there's no per-request reload. With max_inputs=16, up to 16 HTTP requests share that one replica and one resident copy of the weights at the same time.
Deploy it
Deploy the app to get a stable HTTPS endpoint. AI Fabric starts one replica and runs __init__, loading the model into GPU VRAM:
blade deploy llm_cls.pyEvery call to generate reuses that already-loaded model. When the replica goes idle AI Fabric scales it to zero to stop the GPU bill, and the next request cold-starts a fresh replica, running __init__ once more before it serves. The trade-off: pay for the GPU only while there's traffic, at the cost of a one-time cold start when traffic returns.
Take it for a spin
Everything above in one file, ready to run.
import bladekit
app = bladekit.app("llm-cls")
# fabric/torch catalog image: CUDA + PyTorch + transformers already installed.
@app.cls(gpu="RTX-A4500", image="fabric/torch", timeout=600)
@app.concurrent(max_inputs=16)
class LLMService:
def __init__(self):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "Qwen/Qwen2.5-1.5B-Instruct"
self.tok = AutoTokenizer.from_pretrained(name)
self.model = AutoModelForCausalLM.from_pretrained(
name, torch_dtype=torch.float16, device_map="cuda"
)
@app.web_endpoint(methods=["POST"])
def generate(self, item: dict) -> dict:
messages = [{"role": "user", "content": item.get("prompt", "")}]
chat = self.tok.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = self.tok(chat, return_tensors="pt").to("cuda")
out = self.model.generate(**inputs, max_new_tokens=256, do_sample=False)
n_in = inputs["input_ids"].shape[1]
return {"text": self.tok.decode(out[0][n_in:], skip_special_tokens=True)}Concepts used: Cold start, Replica, Endpoint, Scale-to-zero