Developer docs / AI Fabric tutorials

Parametrized classes: one definition, many instances

  • GPU
  • Class

Take it for a spin

The whole program in one file, ready to run.

parametrized_functions.py
import bladekit

app = bladekit.app("parametrized-functions")

# fabric/torch catalog image: CUDA + PyTorch + transformers already installed.
@app.cls(gpu="RTX-A4500", image="fabric/torch", timeout=600)
class LLM:
    # Typed parameters: one class definition, one instance per distinct value.
    # A parametrized class defines NO __init__; the values are set for you.
    model_name: str = bladekit.parameter(default="Qwen/Qwen2.5-1.5B-Instruct")
    temperature: float = bladekit.parameter(default=0.7)

    @bladekit.enter()
    def load(self):
        # Each instance reads its own bound parameters.
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer
        self.tok = AutoTokenizer.from_pretrained(self.model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            self.model_name, torch_dtype=torch.float16, device_map="cuda"
        )

    @app.method()
    def generate(self, prompt: str) -> str:
        messages = [{"role": "user", "content": prompt}]
        chat = self.tok.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=True
        )
        inputs = self.tok(chat, return_tensors="pt").to("cuda")
        out = self.model.generate(
            **inputs, max_new_tokens=256,
            do_sample=self.temperature > 0, temperature=self.temperature,
        )
        n_in = inputs["input_ids"].shape[1]
        return self.tok.decode(out[0][n_in:], skip_special_tokens=True)


@app.local_entrypoint()
def main():
    # Two parametrized instances → two independent container pools, one per model.
    small = LLM(model_name="Qwen/Qwen2.5-1.5B-Instruct")
    large = LLM(model_name="meta-llama/Llama-3.1-8B-Instruct", temperature=0.2)
    print("small:", small.generate.remote("Say hi in one word."))
    print("large:", large.generate.remote("Say hi in one word."))

Run it

  1. Save as parametrized_functions.py and run blade run parametrized_functions.py.
  2. Declare parameters as class attributes with bladekit.parameter(). The value type comes from the annotation (str, int, float, bool, bytes); a parameter without a default is required at call time. A parametrized class defines no __init__: AI Fabric sets the values for you.
  3. Instantiate with keyword arguments: LLM(model_name=...). Each distinct set of values is a separate instance with its own run and its own container pool.
  4. The bound values travel with every call, so AI Fabric routes each call to the matching container and @bladekit.enter reads self.model_name / self.temperature off the instance.
  5. One class definition now serves as many models as you have parameter combinations, with no copy-pasted classes, no per-model boilerplate.

Concepts used: Replica, Cold start, Scale-to-zero, Endpoint

Contact the team

Loading the form…