Developer docs / AI Fabric tutorials
Zero-shot text classification on GPU
- GPU
- Batch / map
- Image DSL
- Volume
Most text classifiers need a labelled training set and a fine-tuning run before they can tell a billing complaint from a shipping question. Zero-shot classification skips all of that: a natural-language-inference model like BART-large-MNLI scores each text against label names it has never been trained on, deciding purely from the words. Change your taxonomy and you change a list of strings, with no retraining.
In this tutorial you'll classify ~200 support-style messages into six labels with **facebook/bart-large-mnli** on RTX A4500s, fanned out with .map(). Each worker also times a CPU baseline, so the speedup the script prints (~104x here) is measured. The dataset is built in-process, so there's nothing to download.
Before you start
You need the AI Fabric CLI installed and authenticated: run blade setup once. The classifier runs on a single RTX A4500. We extend fabric/torch (the catalog base: CUDA + PyTorch + Python 3.12) with transformers and sentencepiece, and cache the ~1.6 GB model on a Volume so later cold starts skip the download.
Create the app, image, and labels
Create zeroshot_classify.py. Declare a Volume for the weight cache, build the image on the catalog base with HF_HOME pointed at the Volume, and pin the model. Then define the candidate labels. These are the entire "training", short noun phrases the model scores each text against on the fly:
import bladekit
app = bladekit.app("zeroshot-classify")
# Cache HF model weights across runs (downloaded once from the public hub, then
# reused forever) so later cold starts skip the ~1.6 GB download.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)
# fabric/torch is the AI Fabric catalog base (CUDA + PyTorch + Python 3.12). We add
# transformers on top: pure PyPI, no git URLs, no torchvision needed (the
# zero-shot text pipeline is text-only). No Dockerfile, no manual setup.
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("transformers", "sentencepiece")
.env({"HF_HOME": "/hf-cache"})
)
MODEL_ID = "facebook/bart-large-mnli" # ~1.6 GB, public on the HF hub
# The candidate labels. Zero-shot means the model never saw these during the
# task; it scores each text against each label name on the fly. Swap in your
# own labels (any short noun phrases) without retraining anything.
CANDIDATE_LABELS = [
"billing",
"technical issue",
"account access",
"feature request",
"shipping",
"cancellation",
]Build a self-contained dataset
Generate the corpus in-process from a set of realistic support-message templates, tagging each with a ticket number to keep them distinct. As with the other batch tutorials, you'd swap this for any list of strings in real use:
def build_dataset(n: int = 200) -> list[str]:
"""A small, self-contained dataset of short support-style messages.
Replace this with any list of strings, e.g. a HuggingFace dataset slice:
from datasets import load_dataset
rows = load_dataset("ag_news", split="train[:200]")["text"]
"""
templates = [
"I was charged twice on my last invoice, please refund the extra amount.",
"The app crashes every time I open the dashboard on my phone.",
"I forgot my password and the reset email never arrives.",
"It would be great if you could add a dark mode to the settings.",
"My package still hasn't arrived and the tracking link is broken.",
"I want to cancel my subscription before the next renewal date.",
"Why is my credit card being declined when the details are correct?",
"The export button does nothing when I click it in the browser.",
"I can't log in, it says my account is locked after one attempt.",
"Please consider supporting CSV export in addition to PDF.",
"When will my order ship? I placed it five days ago.",
"How do I delete my account permanently and remove my data?",
]
docs = []
i = 0
while len(docs) < n:
base = templates[i % len(templates)]
docs.append(f"{base} (ticket {len(docs)})")
i += 1
return docsClassify a chunk and measure the speedup
The per-chunk function builds a zero-shot-classification pipeline on the GPU (device=0), runs the whole chunk against the candidate labels, and takes the top-ranked label as each text's prediction. The pipeline returns labels sorted by score, so index 0 is the winner:
@app.function(
gpu="RTX-A4500",
image=image,
timeout=900,
volumes={"/hf-cache": hf_cache},
)
def classify_chunk(texts: list[str]) -> dict:
"""Classify one chunk of texts on the GPU. Loads the model once per container.
Returns the predicted label for each text plus timing for both GPU and a
small CPU baseline, so the caller can report a real speedup.
"""
import time
import torch
from transformers import pipeline
# device=0 -> first CUDA GPU. Loaded once per container; subsequent chunks
# routed to this hot container reuse the model (no reload, no re-download).
gpu_clf = pipeline(
"zero-shot-classification",
model=MODEL_ID,
device=0,
)
# --- GPU timing (the real workload over the whole chunk) ---
torch.cuda.synchronize()
t0 = time.perf_counter()
gpu_out = gpu_clf(texts, candidate_labels=CANDIDATE_LABELS, batch_size=8)
torch.cuda.synchronize()
gpu_seconds = time.perf_counter() - t0
# The pipeline ranks labels by score; the top one (index 0) is the prediction.
labels = [row["labels"][0] for row in gpu_out]Running the entire chunk on the CPU only to get a baseline would be needlessly slow, so instead time a small slice and extrapolate to a per-text rate. The function returns the predictions plus the GPU time and the projected CPU time for the whole chunk:
# --- CPU baseline on a SMALL slice of the same chunk, for an honest number.
# (Running the full chunk on CPU would be needlessly slow; we extrapolate.)
sample = texts[: min(4, len(texts))]
cpu_clf = pipeline(
"zero-shot-classification",
model=MODEL_ID,
device=-1, # -1 = CPU
)
t0 = time.perf_counter()
cpu_clf(sample, candidate_labels=CANDIDATE_LABELS, batch_size=8)
cpu_seconds_per_text = (time.perf_counter() - t0) / max(len(sample), 1)
return {
"labels": labels,
"n": len(texts),
"gpu_seconds": gpu_seconds,
# Projected CPU time for the whole chunk, from the per-text CPU rate.
"cpu_seconds_est": cpu_seconds_per_text * len(texts),
}Fan out and collect predictions
In the entrypoint, slice the dataset into chunks of 50 and fan them out with .map(): one GPU container per chunk, all at once. Results arrive in input order, so the labels line up with docs, and you sum the GPU and projected-CPU timings as the chunks return:
@app.local_entrypoint()
def main():
from collections import Counter
docs = build_dataset(200)
chunk_size = 50
chunks = [docs[i : i + chunk_size] for i in range(0, len(docs), chunk_size)]
print(
f"Zero-shot classifying {len(docs)} texts into {len(CANDIDATE_LABELS)} "
f"labels, in {len(chunks)} chunks across the GPU fleet..."
)
all_labels: list[str] = []
total_gpu = 0.0
total_cpu_est = 0.0
# .map() fans the chunks out: one GPU container per chunk, all in parallel.
# Results arrive in input order, so labels line up with `docs`.
for res in classify_chunk.map(chunks, return_exceptions=True):
if isinstance(res, Exception):
print(f" chunk failed: {res}")
continue
all_labels.extend(res["labels"])
total_gpu += res["gpu_seconds"]
total_cpu_est += res["cpu_seconds_est"]Finally, tally the label distribution into a little text histogram and report the measured speedup (the summed projected CPU time over the summed GPU time):
if not all_labels:
print("No predictions produced. Check the run logs above.")
return
print(f"\nClassified {len(all_labels)} texts.")
# --- label distribution ---
dist = Counter(all_labels)
print("\n--- label distribution ---")
width = max(len(lbl) for lbl in dist)
for label, count in dist.most_common():
bar = "#" * round(40 * count / len(all_labels))
print(f" {label:<{width}} {count:4d} {bar}")
# --- timing ---
speedup = (total_cpu_est / total_gpu) if total_gpu > 0 else float("nan")
print("\n--- timing (summed over all chunks) ---")
print(f" GPU classify time: {total_gpu:7.2f}s")
print(f" CPU classify time (est):{total_cpu_est:7.2f}s")
print(f" speedup: {speedup:6.1f}x faster on GPU")
print(
"\nAI Fabric bills each container per started minute: this whole job used a few GPU-"
"minutes across the fleet, and nothing while idle."
)Run it
Run the program with the AI Fabric CLI:
blade run zeroshot_classify.pyEach chunk gets its own GPU container, the BART-large-MNLI weights cache on the Volume, and the labels come back in order. You'll see a histogram of how the 200 tickets fell across the six labels, followed by the measured speedup (~104x here). To re-target the classifier for a different domain, edit CANDIDATE_LABELS. No training data, no fine-tuning.
Take it for a spin
Everything above in one file, ready to run.
"""Zero-shot text classification on the GPU fleet, fanned out with .map().
Run with: blade run zeroshot_classify.py
What it does
------------
- Builds a small, self-contained dataset (~200 short texts) in-process, with no files
to download, no setup. Each text is a customer-support-style message.
- Classifies every text into a fixed set of candidate labels with
facebook/bart-large-mnli (the transformers "zero-shot-classification"
pipeline). No training, no fine-tuning: the model decides labels it has never
been told about, purely from their names.
- Splits the dataset into chunks and uses `.map()` to fan the chunks out across
the GPU fleet: many RTX-A4500 containers classify in parallel, results come
back in input order.
- Each GPU worker also classifies its first few texts on the CPU once, so the
script prints a real measured GPU-vs-CPU speedup, not a hand-waved number.
The headline: a couple hundred texts labelled in seconds on GPU vs minutes on
CPU, and AI Fabric bills each container per started minute, so the job costs a few GPU-minutes.
bart-large-mnli is ~1.6 GB and fits comfortably in the A4500's 20 GB of VRAM.
"""
import bladekit
app = bladekit.app("zeroshot-classify")
# Cache HF model weights across runs (downloaded once from the public hub, then
# reused forever) so later cold starts skip the ~1.6 GB download.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)
# fabric/torch is the AI Fabric catalog base (CUDA + PyTorch + Python 3.12). We add
# transformers on top: pure PyPI, no git URLs, no torchvision needed (the
# zero-shot text pipeline is text-only). No Dockerfile, no manual setup.
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("transformers", "sentencepiece")
.env({"HF_HOME": "/hf-cache"})
)
MODEL_ID = "facebook/bart-large-mnli" # ~1.6 GB, public on the HF hub
# The candidate labels. Zero-shot means the model never saw these during the
# task; it scores each text against each label name on the fly. Swap in your
# own labels (any short noun phrases) without retraining anything.
CANDIDATE_LABELS = [
"billing",
"technical issue",
"account access",
"feature request",
"shipping",
"cancellation",
]
def build_dataset(n: int = 200) -> list[str]:
"""A small, self-contained dataset of short support-style messages.
Replace this with any list of strings, e.g. a HuggingFace dataset slice:
from datasets import load_dataset
rows = load_dataset("ag_news", split="train[:200]")["text"]
"""
templates = [
"I was charged twice on my last invoice, please refund the extra amount.",
"The app crashes every time I open the dashboard on my phone.",
"I forgot my password and the reset email never arrives.",
"It would be great if you could add a dark mode to the settings.",
"My package still hasn't arrived and the tracking link is broken.",
"I want to cancel my subscription before the next renewal date.",
"Why is my credit card being declined when the details are correct?",
"The export button does nothing when I click it in the browser.",
"I can't log in, it says my account is locked after one attempt.",
"Please consider supporting CSV export in addition to PDF.",
"When will my order ship? I placed it five days ago.",
"How do I delete my account permanently and remove my data?",
]
docs = []
i = 0
while len(docs) < n:
base = templates[i % len(templates)]
docs.append(f"{base} (ticket {len(docs)})")
i += 1
return docs
@app.function(
gpu="RTX-A4500",
image=image,
timeout=900,
volumes={"/hf-cache": hf_cache},
)
def classify_chunk(texts: list[str]) -> dict:
"""Classify one chunk of texts on the GPU. Loads the model once per container.
Returns the predicted label for each text plus timing for both GPU and a
small CPU baseline, so the caller can report a real speedup.
"""
import time
import torch
from transformers import pipeline
# device=0 -> first CUDA GPU. Loaded once per container; subsequent chunks
# routed to this hot container reuse the model (no reload, no re-download).
gpu_clf = pipeline(
"zero-shot-classification",
model=MODEL_ID,
device=0,
)
# --- GPU timing (the real workload over the whole chunk) ---
torch.cuda.synchronize()
t0 = time.perf_counter()
gpu_out = gpu_clf(texts, candidate_labels=CANDIDATE_LABELS, batch_size=8)
torch.cuda.synchronize()
gpu_seconds = time.perf_counter() - t0
# The pipeline ranks labels by score; the top one (index 0) is the prediction.
labels = [row["labels"][0] for row in gpu_out]
# --- CPU baseline on a SMALL slice of the same chunk, for an honest number.
# (Running the full chunk on CPU would be needlessly slow; we extrapolate.)
sample = texts[: min(4, len(texts))]
cpu_clf = pipeline(
"zero-shot-classification",
model=MODEL_ID,
device=-1, # -1 = CPU
)
t0 = time.perf_counter()
cpu_clf(sample, candidate_labels=CANDIDATE_LABELS, batch_size=8)
cpu_seconds_per_text = (time.perf_counter() - t0) / max(len(sample), 1)
return {
"labels": labels,
"n": len(texts),
"gpu_seconds": gpu_seconds,
# Projected CPU time for the whole chunk, from the per-text CPU rate.
"cpu_seconds_est": cpu_seconds_per_text * len(texts),
}
@app.local_entrypoint()
def main():
from collections import Counter
docs = build_dataset(200)
chunk_size = 50
chunks = [docs[i : i + chunk_size] for i in range(0, len(docs), chunk_size)]
print(
f"Zero-shot classifying {len(docs)} texts into {len(CANDIDATE_LABELS)} "
f"labels, in {len(chunks)} chunks across the GPU fleet..."
)
all_labels: list[str] = []
total_gpu = 0.0
total_cpu_est = 0.0
# .map() fans the chunks out: one GPU container per chunk, all in parallel.
# Results arrive in input order, so labels line up with `docs`.
for res in classify_chunk.map(chunks, return_exceptions=True):
if isinstance(res, Exception):
print(f" chunk failed: {res}")
continue
all_labels.extend(res["labels"])
total_gpu += res["gpu_seconds"]
total_cpu_est += res["cpu_seconds_est"]
if not all_labels:
print("No predictions produced. Check the run logs above.")
return
print(f"\nClassified {len(all_labels)} texts.")
# --- label distribution ---
dist = Counter(all_labels)
print("\n--- label distribution ---")
width = max(len(lbl) for lbl in dist)
for label, count in dist.most_common():
bar = "#" * round(40 * count / len(all_labels))
print(f" {label:<{width}} {count:4d} {bar}")
# --- timing ---
speedup = (total_cpu_est / total_gpu) if total_gpu > 0 else float("nan")
print("\n--- timing (summed over all chunks) ---")
print(f" GPU classify time: {total_gpu:7.2f}s")
print(f" CPU classify time (est):{total_cpu_est:7.2f}s")
print(f" speedup: {speedup:6.1f}x faster on GPU")
print(
"\nAI Fabric bills each container per started minute: this whole job used a few GPU-"
"minutes across the fleet, and nothing while idle."
)Concepts used: GPU, Run, Volume