Developer docs / AI Fabric tutorials
Parametrized classes: one definition, many instances
- GPU
- Class
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("parametrized-functions")
# fabric/torch catalog image: CUDA + PyTorch + transformers already installed.
@app.cls(gpu="RTX-A4500", image="fabric/torch", timeout=600)
class LLM:
# Typed parameters: one class definition, one instance per distinct value.
# A parametrized class defines NO __init__; the values are set for you.
model_name: str = bladekit.parameter(default="Qwen/Qwen2.5-1.5B-Instruct")
temperature: float = bladekit.parameter(default=0.7)
@bladekit.enter()
def load(self):
# Each instance reads its own bound parameters.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
self.tok = AutoTokenizer.from_pretrained(self.model_name)
self.model = AutoModelForCausalLM.from_pretrained(
self.model_name, torch_dtype=torch.float16, device_map="cuda"
)
@app.method()
def generate(self, prompt: str) -> str:
messages = [{"role": "user", "content": prompt}]
chat = self.tok.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = self.tok(chat, return_tensors="pt").to("cuda")
out = self.model.generate(
**inputs, max_new_tokens=256,
do_sample=self.temperature > 0, temperature=self.temperature,
)
n_in = inputs["input_ids"].shape[1]
return self.tok.decode(out[0][n_in:], skip_special_tokens=True)
@app.local_entrypoint()
def main():
# Two parametrized instances → two independent container pools, one per model.
small = LLM(model_name="Qwen/Qwen2.5-1.5B-Instruct")
large = LLM(model_name="meta-llama/Llama-3.1-8B-Instruct", temperature=0.2)
print("small:", small.generate.remote("Say hi in one word."))
print("large:", large.generate.remote("Say hi in one word."))Run it
- Save as
parametrized_functions.pyand runblade run parametrized_functions.py. - Declare parameters as class attributes with
bladekit.parameter(). The value type comes from the annotation (str,int,float,bool,bytes); a parameter without a default is required at call time. A parametrized class defines no__init__: AI Fabric sets the values for you. - Instantiate with keyword arguments:
LLM(model_name=...). Each distinct set of values is a separate instance with its own run and its own container pool. - The bound values travel with every call, so AI Fabric routes each call to the matching container and
@bladekit.enterreadsself.model_name/self.temperatureoff the instance. - One class definition now serves as many models as you have parameter combinations, with no copy-pasted classes, no per-model boilerplate.
Concepts used: Replica, Cold start, Scale-to-zero, Endpoint