Developer docs / AI Fabric tutorials
Batch translation on GPU (NLLB-200)
- GPU
- Batch / map
- Image DSL
- Volume
Translating one sentence is easy. Translating a hundred thousand of them is where the machine matters: a translation model that crawls on a CPU flies on a GPU, and the question is how to push the whole batch through without writing a job queue yourself. The answer is to chop the work into chunks and let the platform fan them out across many GPUs at once.
In this tutorial you'll translate a batch of English sentences into French with Meta's NLLB-200 on a GPU, fanning the chunks out with .map(). Each worker also times the same chunk on a CPU, so the script prints a measured speedup instead of a marketing number. This exact snippet clocked in at 52.4x faster than CPU.
Before you start
You need the AI Fabric CLI installed and authenticated. Run blade setup once. The job runs on a single RTX A4500 per chunk and builds on fabric/torch, the catalog image that already ships CUDA, PyTorch and Python 3.12, so there's no Dockerfile to write.
Create the app and cache the weights
Create a file called translate_batch.py. The NLLB weights are ~2.4 GB, and you don't want to re-download them on every run, so attach a Volume and the download happens once, ever. Every later run mounts the same Volume and starts instantly:
import bladekit
app = bladekit.app("batch-translation")
# Cache the HF weights on a Volume so the ~2.4 GB download happens once, ever.
# Every later run mounts the same Volume and starts instantly.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)Build the image
Start from fabric/torch and add only what NLLB needs on top: transformers and sentencepiece (its tokenizer requires it). Point HF_HOME at the mounted Volume so the Hugging Face cache lands there. NLLB is a pure text seq2seq model, so there's no torchvision and no multi-GB rebuild:
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("transformers==4.46.3", "sentencepiece")
.env({"HF_HOME": "/hf-cache"})
)
MODEL_ID = "facebook/nllb-200-distilled-600M" # ~2.4 GB, public on the HF hub
# NLLB uses BCP-47-ish "FLORES-200" language codes. A few handy ones:
SRC_LANG = "eng_Latn" # English
TGT_LANGS = {
"French": "fra_Latn",
"Spanish": "spa_Latn",
"German": "deu_Latn",
}Make some sentences to translate
So the tutorial is self-contained, generate a list of short English sentences in-process, with no files to download and no setup. In a real job you'd drop in your own list of strings (the docstring shows a Hugging Face dataset slice):
def build_sentences(n: int = 100) -> list[str]:
"""A small, self-contained set of short English sentences to translate.
Replace this with any list of strings, e.g. a HuggingFace dataset slice:
from datasets import load_dataset
rows = load_dataset("ag_news", split="train[:100]")["text"]
"""
subjects = [
"The serverless platform", "Our embedding model", "This dataset",
"The transformer network", "The vector index", "The retrieval pipeline",
"The fleet scheduler", "A batch of documents", "The cold-start path",
"Per-minute billing", "The translation model", "The GPU container",
]
predicates = [
"scales out across many GPUs.",
"runs much faster on a GPU.",
"is cached for the next run.",
"fans work out with one call.",
"fits in twenty gigabytes of memory.",
"returns results in order.",
"only costs the minutes it runs.",
"starts almost instantly.",
]
out = []
i = 0
while len(out) < n:
s = subjects[i % len(subjects)]
p = predicates[(i // len(subjects)) % len(predicates)]
out.append(f"{s} {p}")
i += 1
return out[:n]Translate a chunk on the GPU
This is the function that runs remotely: one RTX A4500 per chunk, with the Volume mounted at /hf-cache. It loads the tokenizer, forces decoding into the target language by setting forced_bos_token_id, and runs the same chunk twice: once on the GPU (the real workload) and once on the CPU (the baseline). The model loads once per container, so later chunks routed to a hot container reuse it with no reload:
@app.function(gpu="RTX-A4500", image=image, timeout=900, volumes={"/hf-cache": hf_cache})
def translate_chunk(texts: list[str], tgt_lang: str = "fra_Latn") -> dict:
"""Translate one chunk of sentences on the GPU. Loads the model once per
container; later chunks routed to this hot container reuse it (no reload).
Returns the translations plus timing for both GPU and a one-shot CPU
baseline so the caller can report a real, measured speedup.
"""
import time
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(MODEL_ID, src_lang=SRC_LANG)
bos = tok.convert_tokens_to_ids(tgt_lang) # force decoding into target lang
def run(model, device: str) -> list[str]:
enc = tok(texts, return_tensors="pt", padding=True, truncation=True,
max_length=128).to(device)
with torch.no_grad():
gen = model.generate(**enc, forced_bos_token_id=bos, max_length=128)
return tok.batch_decode(gen, skip_special_tokens=True)
# --- GPU timing (the real workload) ---
gpu_model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).to("cuda").eval()
torch.cuda.synchronize()
t0 = time.perf_counter()
translations = run(gpu_model, "cuda")
torch.cuda.synchronize()
gpu_seconds = time.perf_counter() - t0
# --- CPU baseline on the SAME chunk, for an honest speedup number ---
cpu_model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).to("cpu").eval()
t0 = time.perf_counter()
run(cpu_model, "cpu")
cpu_seconds = time.perf_counter() - t0
return {
"translations": translations,
"n": len(texts),
"gpu_seconds": gpu_seconds,
"cpu_seconds": cpu_seconds,
} Fan the chunks out with .map()
The local_entrypoint runs on your machine. Split the sentences into chunks of 25, then call translate_chunk.map(chunks), and AI Fabric ships one chunk per GPU container and runs them all in parallel. Results arrive in input order, so you can zip them straight back to your sentences, and return_exceptions=True means one bad chunk doesn't sink the run:
@app.local_entrypoint()
def main():
sentences = build_sentences(100)
target_name, target_code = "French", TGT_LANGS["French"]
chunk_size = 25
chunks = [sentences[i : i + chunk_size]
for i in range(0, len(sentences), chunk_size)]
print(f"Translating {len(sentences)} English sentences to {target_name} "
f"in {len(chunks)} chunks across the GPU fleet...")
# .map() passes each item as the SOLE positional arg, so bind the target
# language with a default on the function (translate_chunk(texts, tgt_lang)).
# All chunks share the same target here, so .map() over the chunks is enough.
all_translations: list[str] = []
total_gpu = 0.0
total_cpu = 0.0
# .map() fans the chunks out: one GPU container per chunk, all in parallel.
# Results arrive in input order, so we can zip them back to `sentences`.
for res in translate_chunk.map(chunks, return_exceptions=True):
if isinstance(res, Exception):
print(f" chunk failed: {res}")
continue
all_translations.extend(res["translations"])
total_gpu += res["gpu_seconds"]
total_cpu += res["cpu_seconds"]Because .map() passes each item as the sole positional argument, the target language is bound by the tgt_lang default on translate_chunk. Every chunk shares the same target, so mapping over the chunks alone is enough.
Report the measured speedup
Finally, print a few sample translations and the summed GPU-vs-CPU timing. Because every worker timed both devices on the same chunk, the speedup is a number you measured, not one you guessed:
if not all_translations:
print("No translations produced. Check the run logs above.")
return
print(f"\nTranslated {len(all_translations)} sentences to {target_name}.")
print("\n--- sample translations ---")
for en, fr in list(zip(sentences, all_translations))[:6]:
print(f" EN: {en}")
print(f" {target_name[:2].upper()}: {fr}\n")
speedup = (total_cpu / total_gpu) if total_gpu > 0 else float("nan")
print("--- timing (summed over all chunks) ---")
print(f" GPU translate time: {total_gpu:6.2f}s")
print(f" CPU translate time: {total_cpu:6.2f}s")
print(f" speedup: {speedup:5.1f}x faster on GPU")
print("\nAI Fabric bills each container per started minute: this whole job used a few "
"GPU-minutes across the fleet, and nothing while idle.")Run it
Run the program with the AI Fabric CLI:
blade run translate_batch.pyThe first run downloads the NLLB weights into the Volume; every run after that mounts the cache and skips straight to translating. When it finishes you'll see sample translations and the timing summary printed in your terminal:
speedup: 52.4x faster on GPUA hundred sentences translated in seconds, across the fleet, billed per started minute. You pay a few GPU-minutes and nothing more. Point build_sentences at your own list of strings and the same .map() pattern scales to as many documents as you have.
Take it for a spin
Everything above in one file, ready to run.
"""Batch machine translation on the GPU fleet, fanned out with .map().
Run with: blade run batch_translation.py
What it does
------------
- Builds a list of ~100 short English sentences in-process, no files to
download, no setup. Swap in any list of strings (e.g. a HF dataset slice).
- Translates every sentence to French (and, optionally, more languages) with
facebook/nllb-200-distilled-600M on a RTX-A4500 GPU. The model (~2.4 GB)
downloads from the public HF hub on the first run, is cached on a Volume,
and fits comfortably in 20 GB of VRAM.
- Splits the sentences into chunks and uses `.map()` to fan the chunks out
across the GPU fleet. Many containers translate in parallel, results come
back in the original input order.
- Each worker also translates its chunk on the CPU once, so the script can
print a real measured GPU-vs-CPU speedup, not a hand-waved number.
The headline: ~100 sentences translated in seconds on GPU vs minutes on CPU,
and AI Fabric bills each container per started minute, so the job costs a few GPU-minutes.
"""
import bladekit
app = bladekit.app("batch-translation")
# Cache the HF weights on a Volume so the ~2.4 GB download happens once, ever.
# Every later run mounts the same Volume and starts instantly.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)
# fabric/torch is the AI Fabric catalog base (CUDA + PyTorch + Python 3.12). We add
# transformers + sentencepiece (NLLB's tokenizer needs it) on top. NLLB is a
# pure text seq2seq model, so no torchvision is required. No Dockerfile.
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("transformers==4.46.3", "sentencepiece")
.env({"HF_HOME": "/hf-cache"})
)
MODEL_ID = "facebook/nllb-200-distilled-600M" # ~2.4 GB, public on the HF hub
# NLLB uses BCP-47-ish "FLORES-200" language codes. A few handy ones:
SRC_LANG = "eng_Latn" # English
TGT_LANGS = {
"French": "fra_Latn",
"Spanish": "spa_Latn",
"German": "deu_Latn",
}
def build_sentences(n: int = 100) -> list[str]:
"""A small, self-contained set of short English sentences to translate.
Replace this with any list of strings, e.g. a HuggingFace dataset slice:
from datasets import load_dataset
rows = load_dataset("ag_news", split="train[:100]")["text"]
"""
subjects = [
"The serverless platform", "Our embedding model", "This dataset",
"The transformer network", "The vector index", "The retrieval pipeline",
"The fleet scheduler", "A batch of documents", "The cold-start path",
"Per-minute billing", "The translation model", "The GPU container",
]
predicates = [
"scales out across many GPUs.",
"runs much faster on a GPU.",
"is cached for the next run.",
"fans work out with one call.",
"fits in twenty gigabytes of memory.",
"returns results in order.",
"only costs the minutes it runs.",
"starts almost instantly.",
]
out = []
i = 0
while len(out) < n:
s = subjects[i % len(subjects)]
p = predicates[(i // len(subjects)) % len(predicates)]
out.append(f"{s} {p}")
i += 1
return out[:n]
@app.function(gpu="RTX-A4500", image=image, timeout=900, volumes={"/hf-cache": hf_cache})
def translate_chunk(texts: list[str], tgt_lang: str = "fra_Latn") -> dict:
"""Translate one chunk of sentences on the GPU. Loads the model once per
container; later chunks routed to this hot container reuse it (no reload).
Returns the translations plus timing for both GPU and a one-shot CPU
baseline so the caller can report a real, measured speedup.
"""
import time
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(MODEL_ID, src_lang=SRC_LANG)
bos = tok.convert_tokens_to_ids(tgt_lang) # force decoding into target lang
def run(model, device: str) -> list[str]:
enc = tok(texts, return_tensors="pt", padding=True, truncation=True,
max_length=128).to(device)
with torch.no_grad():
gen = model.generate(**enc, forced_bos_token_id=bos, max_length=128)
return tok.batch_decode(gen, skip_special_tokens=True)
# --- GPU timing (the real workload) ---
gpu_model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).to("cuda").eval()
torch.cuda.synchronize()
t0 = time.perf_counter()
translations = run(gpu_model, "cuda")
torch.cuda.synchronize()
gpu_seconds = time.perf_counter() - t0
# --- CPU baseline on the SAME chunk, for an honest speedup number ---
cpu_model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).to("cpu").eval()
t0 = time.perf_counter()
run(cpu_model, "cpu")
cpu_seconds = time.perf_counter() - t0
return {
"translations": translations,
"n": len(texts),
"gpu_seconds": gpu_seconds,
"cpu_seconds": cpu_seconds,
}
@app.local_entrypoint()
def main():
sentences = build_sentences(100)
target_name, target_code = "French", TGT_LANGS["French"]
chunk_size = 25
chunks = [sentences[i : i + chunk_size]
for i in range(0, len(sentences), chunk_size)]
print(f"Translating {len(sentences)} English sentences to {target_name} "
f"in {len(chunks)} chunks across the GPU fleet...")
# .map() passes each item as the SOLE positional arg, so bind the target
# language with a default on the function (translate_chunk(texts, tgt_lang)).
# All chunks share the same target here, so .map() over the chunks is enough.
all_translations: list[str] = []
total_gpu = 0.0
total_cpu = 0.0
# .map() fans the chunks out: one GPU container per chunk, all in parallel.
# Results arrive in input order, so we can zip them back to `sentences`.
for res in translate_chunk.map(chunks, return_exceptions=True):
if isinstance(res, Exception):
print(f" chunk failed: {res}")
continue
all_translations.extend(res["translations"])
total_gpu += res["gpu_seconds"]
total_cpu += res["cpu_seconds"]
if not all_translations:
print("No translations produced. Check the run logs above.")
return
print(f"\nTranslated {len(all_translations)} sentences to {target_name}.")
print("\n--- sample translations ---")
for en, fr in list(zip(sentences, all_translations))[:6]:
print(f" EN: {en}")
print(f" {target_name[:2].upper()}: {fr}\n")
speedup = (total_cpu / total_gpu) if total_gpu > 0 else float("nan")
print("--- timing (summed over all chunks) ---")
print(f" GPU translate time: {total_gpu:6.2f}s")
print(f" CPU translate time: {total_cpu:6.2f}s")
print(f" speedup: {speedup:5.1f}x faster on GPU")
print("\nAI Fabric bills each container per started minute: this whole job used a few "
"GPU-minutes across the fleet, and nothing while idle.")Concepts used: GPU, Run, Volume