Developer docs / AI Fabric tutorials
OpenAI-compatible LLM web API
- GPU
- Web endpoint
- Concurrency
The openai SDK is everywhere: in scripts, agent frameworks, and half the tools your team already runs. Your own GPU model can speak the OpenAI API: the exact Chat Completions schema at the exact path the SDK calls. Then every one of those tools works against your hardware with a one-line change.
In this tutorial you'll serve a drop-in OpenAI Chat Completions API backed by your own GPU. It takes a custom endpoint path plus the precise response shape the SDK expects. Callers point base_url at AI Fabric and pass their AI Fabric token as the api_key.
Before you start
You need the AI Fabric CLI installed and authenticated: run blade setup once. The endpoint runs on a single RTX A4500 and uses fabric/torch, the catalog image that already ships CUDA, PyTorch and transformers, so there's no image to build.
Create the app
Create openai_llm.py. Set a default image on the app so each function inherits it, and pick an ungated model. Qwen2.5-1.5B-Instruct needs no Hugging Face access token and swaps cleanly for any HF causal LM. The _state dict will hold the loaded model and tokenizer across requests:
import bladekit
# A drop-in OpenAI Chat Completions API, backed by your own GPU.
# fabric/torch is a catalog image (PyTorch + transformers already installed).
app = bladekit.app("openai-llm", image="fabric/torch")
MODEL = "Qwen/Qwen2.5-1.5B-Instruct" # ungated; swap for any HF causal LM
_state = {}Load the model once
Wrap the model load in a helper that populates _state only on the first call. The first request through a container loads the weights onto the GPU; every later request finds them already there and skips straight to inference. It's the same load-once-reuse-many pattern that makes warm serving fast:
def _engine():
if "model" not in _state:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
_state["tok"] = AutoTokenizer.from_pretrained(MODEL)
_state["model"] = AutoModelForCausalLM.from_pretrained(
MODEL, torch_dtype=torch.float16, device_map="cuda")
return _stateExpose it at the OpenAI path
@app.web_endpoint(path="/v1/chat/completions") makes the endpoint answer exactly where the OpenAI SDK looks: base_url + /chat/completions. min_replicas=1 keeps one replica warm so the first real request never pays a cold start, and @app.concurrent(max_inputs=4) lets one GPU serve several requests at once. Read the incoming messages (falling back to a prompt field), or default to a friendly hello:
# The custom path is the trick: the endpoint answers exactly where the
# OpenAI SDK looks (base_url + "/chat/completions").
@app.web_endpoint(path="/v1/chat/completions", methods=["POST"],
min_replicas=1, max_replicas=2)
@app.function(gpu="RTX-A4500")
@app.concurrent(max_inputs=4)
def chat_completions(item: dict) -> dict:
import time, uuid, torch
s = _engine()
messages = item.get("messages") or [
{"role": "user", "content": str(item.get("prompt", "Hello!"))}]Generate the reply
Apply the model's chat template, move the tokens onto the GPU, and generate. Slicing out[0][n_in:] drops the prompt tokens so you decode only the model's reply, and tracking n_in and n_total gives you the token counts the OpenAI schema reports back:
enc = s["tok"].apply_chat_template(
messages, add_generation_prompt=True,
return_tensors="pt", return_dict=True)
enc = {k: v.to("cuda") for k, v in enc.items()}
n_in = int(enc["input_ids"].shape[1])
with torch.no_grad():
out = s["model"].generate(
**enc, max_new_tokens=int(item.get("max_tokens", 256)),
do_sample=False, pad_token_id=s["tok"].eos_token_id)
text = s["tok"].decode(out[0][n_in:], skip_special_tokens=True)
n_total = int(out.shape[1])Return the exact OpenAI schema
This is what makes the SDK accept your endpoint: return the precise Chat Completions envelope: an id, the chat.completion object type, a created timestamp, a choices array whose message carries the assistant's text, and a usage block with the token counts. Match this shape and the official client parses it without a shim:
return {
"id": "chatcmpl-" + uuid.uuid4().hex[:24],
"object": "chat.completion",
"created": int(time.time()),
"model": item.get("model", MODEL),
"choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": text}}],
"usage": {"prompt_tokens": n_in,
"completion_tokens": n_total - n_in,
"total_tokens": n_total},
}Deploy it
Deploy the app to get a stable HTTPS URL. With min_replicas=1 the model loads once and stays warm, so the first real request doesn't cold-start:
blade deploy openai_llm.pyThe endpoint lives at https://openai-llm.apps.run.blade.sh/v1/chat/completions. That is the path the OpenAI SDK calls when base_url ends in /v1.
Call it with the OpenAI SDK
No shim, no custom client: point the official SDK at AI Fabric. Your **AI Fabric token is the api_key** (AI Fabric auth is Authorization: Bearer, which is exactly how the SDK sends the key):
from openai import OpenAI
client = OpenAI(
base_url="https://openai-llm.apps.run.blade.sh/v1",
api_key="<FABRIC_TOKEN>")
r = client.chat.completions.create(
model="qwen",
messages=[{"role": "user", "content": "What is 2+2?"}])
print(r.choices[0].message.content) # -> "4"Or hit it directly with curl:
curl -X POST https://openai-llm.apps.run.blade.sh/v1/chat/completions -H 'Authorization: Bearer <FABRIC_TOKEN>' -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"Hi"}]}'You now have an OpenAI-compatible API on your own GPU. To scale up the model, change MODEL (e.g. Qwen/Qwen2.5-7B-Instruct) and add memory=49152 to @app.function, since bigger weights need more host RAM during load.
Take it for a spin
Everything above in one file, ready to run.
import bladekit
# A drop-in OpenAI Chat Completions API, backed by your own GPU.
# fabric/torch is a catalog image (PyTorch + transformers already installed).
app = bladekit.app("openai-llm", image="fabric/torch")
MODEL = "Qwen/Qwen2.5-1.5B-Instruct" # ungated; swap for any HF causal LM
_state = {}
def _engine():
if "model" not in _state:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
_state["tok"] = AutoTokenizer.from_pretrained(MODEL)
_state["model"] = AutoModelForCausalLM.from_pretrained(
MODEL, torch_dtype=torch.float16, device_map="cuda")
return _state
# The custom path is the trick: the endpoint answers exactly where the
# OpenAI SDK looks (base_url + "/chat/completions").
@app.web_endpoint(path="/v1/chat/completions", methods=["POST"],
min_replicas=1, max_replicas=2)
@app.function(gpu="RTX-A4500")
@app.concurrent(max_inputs=4)
def chat_completions(item: dict) -> dict:
import time, uuid, torch
s = _engine()
messages = item.get("messages") or [
{"role": "user", "content": str(item.get("prompt", "Hello!"))}]
enc = s["tok"].apply_chat_template(
messages, add_generation_prompt=True,
return_tensors="pt", return_dict=True)
enc = {k: v.to("cuda") for k, v in enc.items()}
n_in = int(enc["input_ids"].shape[1])
with torch.no_grad():
out = s["model"].generate(
**enc, max_new_tokens=int(item.get("max_tokens", 256)),
do_sample=False, pad_token_id=s["tok"].eos_token_id)
text = s["tok"].decode(out[0][n_in:], skip_special_tokens=True)
n_total = int(out.shape[1])
return {
"id": "chatcmpl-" + uuid.uuid4().hex[:24],
"object": "chat.completion",
"created": int(time.time()),
"model": item.get("model", MODEL),
"choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": text}}],
"usage": {"prompt_tokens": n_in,
"completion_tokens": n_total - n_in,
"total_tokens": n_total},
}Concepts used: Endpoint, Autoscaling, Replica, Concurrency, Cold start