Developer docs / AI Fabric tutorials

OpenAI-compatible LLM web API

  • GPU
  • Web endpoint
  • Concurrency

The openai SDK is everywhere: in scripts, agent frameworks, and half the tools your team already runs. Your own GPU model can speak the OpenAI API: the exact Chat Completions schema at the exact path the SDK calls. Then every one of those tools works against your hardware with a one-line change.

In this tutorial you'll serve a drop-in OpenAI Chat Completions API backed by your own GPU. It takes a custom endpoint path plus the precise response shape the SDK expects. Callers point base_url at AI Fabric and pass their AI Fabric token as the api_key.

Before you start

You need the AI Fabric CLI installed and authenticated: run blade setup once. The endpoint runs on a single RTX A4500 and uses fabric/torch, the catalog image that already ships CUDA, PyTorch and transformers, so there's no image to build.

Create the app

Create openai_llm.py. Set a default image on the app so each function inherits it, and pick an ungated model. Qwen2.5-1.5B-Instruct needs no Hugging Face access token and swaps cleanly for any HF causal LM. The _state dict will hold the loaded model and tokenizer across requests:

openai_llm.py
import bladekit

# A drop-in OpenAI Chat Completions API, backed by your own GPU.
# fabric/torch is a catalog image (PyTorch + transformers already installed).
app = bladekit.app("openai-llm", image="fabric/torch")

MODEL = "Qwen/Qwen2.5-1.5B-Instruct"  # ungated; swap for any HF causal LM
_state = {}

Load the model once

Wrap the model load in a helper that populates _state only on the first call. The first request through a container loads the weights onto the GPU; every later request finds them already there and skips straight to inference. It's the same load-once-reuse-many pattern that makes warm serving fast:

Python
def _engine():
    if "model" not in _state:
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer
        _state["tok"] = AutoTokenizer.from_pretrained(MODEL)
        _state["model"] = AutoModelForCausalLM.from_pretrained(
            MODEL, torch_dtype=torch.float16, device_map="cuda")
    return _state

Expose it at the OpenAI path

@app.web_endpoint(path="/v1/chat/completions") makes the endpoint answer exactly where the OpenAI SDK looks: base_url + /chat/completions. min_replicas=1 keeps one replica warm so the first real request never pays a cold start, and @app.concurrent(max_inputs=4) lets one GPU serve several requests at once. Read the incoming messages (falling back to a prompt field), or default to a friendly hello:

Python
# The custom path is the trick: the endpoint answers exactly where the
# OpenAI SDK looks (base_url + "/chat/completions").
@app.web_endpoint(path="/v1/chat/completions", methods=["POST"],
                  min_replicas=1, max_replicas=2)
@app.function(gpu="RTX-A4500")
@app.concurrent(max_inputs=4)
def chat_completions(item: dict) -> dict:
    import time, uuid, torch
    s = _engine()
    messages = item.get("messages") or [
        {"role": "user", "content": str(item.get("prompt", "Hello!"))}]

Generate the reply

Apply the model's chat template, move the tokens onto the GPU, and generate. Slicing out[0][n_in:] drops the prompt tokens so you decode only the model's reply, and tracking n_in and n_total gives you the token counts the OpenAI schema reports back:

Python
    enc = s["tok"].apply_chat_template(
        messages, add_generation_prompt=True,
        return_tensors="pt", return_dict=True)
    enc = {k: v.to("cuda") for k, v in enc.items()}
    n_in = int(enc["input_ids"].shape[1])

    with torch.no_grad():
        out = s["model"].generate(
            **enc, max_new_tokens=int(item.get("max_tokens", 256)),
            do_sample=False, pad_token_id=s["tok"].eos_token_id)
    text = s["tok"].decode(out[0][n_in:], skip_special_tokens=True)
    n_total = int(out.shape[1])

Return the exact OpenAI schema

This is what makes the SDK accept your endpoint: return the precise Chat Completions envelope: an id, the chat.completion object type, a created timestamp, a choices array whose message carries the assistant's text, and a usage block with the token counts. Match this shape and the official client parses it without a shim:

Python
    return {
        "id": "chatcmpl-" + uuid.uuid4().hex[:24],
        "object": "chat.completion",
        "created": int(time.time()),
        "model": item.get("model", MODEL),
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": text}}],
        "usage": {"prompt_tokens": n_in,
                  "completion_tokens": n_total - n_in,
                  "total_tokens": n_total},
    }

Deploy it

Deploy the app to get a stable HTTPS URL. With min_replicas=1 the model loads once and stays warm, so the first real request doesn't cold-start:

Terminal
blade deploy openai_llm.py

The endpoint lives at https://openai-llm.apps.run.blade.sh/v1/chat/completions. That is the path the OpenAI SDK calls when base_url ends in /v1.

Call it with the OpenAI SDK

No shim, no custom client: point the official SDK at AI Fabric. Your **AI Fabric token is the api_key** (AI Fabric auth is Authorization: Bearer, which is exactly how the SDK sends the key):

Python
from openai import OpenAI
client = OpenAI(
    base_url="https://openai-llm.apps.run.blade.sh/v1",
    api_key="<FABRIC_TOKEN>")
r = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": "What is 2+2?"}])
print(r.choices[0].message.content)  # -> "4"

Or hit it directly with curl:

Terminal
curl -X POST https://openai-llm.apps.run.blade.sh/v1/chat/completions -H 'Authorization: Bearer <FABRIC_TOKEN>' -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"Hi"}]}'

You now have an OpenAI-compatible API on your own GPU. To scale up the model, change MODEL (e.g. Qwen/Qwen2.5-7B-Instruct) and add memory=49152 to @app.function, since bigger weights need more host RAM during load.

Take it for a spin

Everything above in one file, ready to run.

openai_llm.py
import bladekit

# A drop-in OpenAI Chat Completions API, backed by your own GPU.
# fabric/torch is a catalog image (PyTorch + transformers already installed).
app = bladekit.app("openai-llm", image="fabric/torch")

MODEL = "Qwen/Qwen2.5-1.5B-Instruct"  # ungated; swap for any HF causal LM
_state = {}

def _engine():
    if "model" not in _state:
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer
        _state["tok"] = AutoTokenizer.from_pretrained(MODEL)
        _state["model"] = AutoModelForCausalLM.from_pretrained(
            MODEL, torch_dtype=torch.float16, device_map="cuda")
    return _state

# The custom path is the trick: the endpoint answers exactly where the
# OpenAI SDK looks (base_url + "/chat/completions").
@app.web_endpoint(path="/v1/chat/completions", methods=["POST"],
                  min_replicas=1, max_replicas=2)
@app.function(gpu="RTX-A4500")
@app.concurrent(max_inputs=4)
def chat_completions(item: dict) -> dict:
    import time, uuid, torch
    s = _engine()
    messages = item.get("messages") or [
        {"role": "user", "content": str(item.get("prompt", "Hello!"))}]

    enc = s["tok"].apply_chat_template(
        messages, add_generation_prompt=True,
        return_tensors="pt", return_dict=True)
    enc = {k: v.to("cuda") for k, v in enc.items()}
    n_in = int(enc["input_ids"].shape[1])

    with torch.no_grad():
        out = s["model"].generate(
            **enc, max_new_tokens=int(item.get("max_tokens", 256)),
            do_sample=False, pad_token_id=s["tok"].eos_token_id)
    text = s["tok"].decode(out[0][n_in:], skip_special_tokens=True)
    n_total = int(out.shape[1])

    return {
        "id": "chatcmpl-" + uuid.uuid4().hex[:24],
        "object": "chat.completion",
        "created": int(time.time()),
        "model": item.get("model", MODEL),
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": text}}],
        "usage": {"prompt_tokens": n_in,
                  "completion_tokens": n_total - n_in,
                  "total_tokens": n_total},
    }

Concepts used: Endpoint, Autoscaling, Replica, Concurrency, Cold start

Contact the team

Loading the form…