Developer docs / AI Fabric tutorials

Batch summarization on GPU (BART)

  • GPU
  • Batch / map
  • Image DSL
  • Volume

Summarizing one article is a quick LLM call. Summarizing a backlog of thousands (support threads, transcripts, news archives) is a throughput problem, and that's where a GPU earns its keep. The pattern is the same one that scales any batch job: split the work into chunks and let AI Fabric fan them out across many GPUs at once.

In this tutorial you'll summarize a batch of text passages with BART-large-CNN on a GPU, fanned out with .map(). Each worker times one passage on a CPU too, so the script extrapolates a measured per-document baseline to the whole batch. This exact snippet came out ~93x faster than CPU per document.

Before you start

You need the AI Fabric CLI installed and authenticated (run blade setup once). Each chunk runs on a single RTX A4500 and builds on fabric/torch, the catalog image that already ships CUDA, PyTorch and Python 3.12, so there's nothing to build locally.

Create the app and cache the weights

Create a file called summarize_batch.py. BART-large-CNN is ~1.6 GB; attach a Volume so those weights download once from the public Hugging Face hub and later runs skip the download entirely:

summarize_batch.py
import bladekit

app = bladekit.app("batch-summarize")

# Cache HF weights on a Volume so they download once, not on every run.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)

Build the image

Start from fabric/torch and add transformers and sentencepiece on top, with no Dockerfile and no manual setup. Setting HF_HOME to /hf-cache points the Hugging Face hub cache at the mounted Volume:

Python
image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("transformers==4.46.3", "sentencepiece")
    .env({"HF_HOME": "/hf-cache"})
)

MODEL_ID = "facebook/bart-large-cnn"  # ~1.6 GB, public on the HF hub, text-only

Make some passages to summarize

So the tutorial stands on its own, generate ~50 medium-length passages in-process, so there are no files to download. Each passage is a few sentences of filler, long enough to be worth summarizing. In production you'd swap this for your own list of strings (the docstring shows a cnn_dailymail slice):

Python
def build_passages(n: int = 50) -> list[str]:
    """~50 medium-length passages, self-contained with no files to download.

    Replace this with any list of strings, e.g. a HuggingFace dataset slice:
        from datasets import load_dataset
        rows = load_dataset("cnn_dailymail", "3.0.0", split="test[:50]")["article"]
    """
    topics = [
        ("serverless GPU computing", "renting GPUs by the second"),
        ("vector databases", "storing dense embeddings for fast search"),
        ("transformer architectures", "the attention mechanism"),
        ("container cold starts", "the time to a first response"),
        ("distributed training", "splitting a model across many devices"),
        ("model quantization", "shrinking weights to lower precision"),
        ("retrieval-augmented generation", "grounding answers in documents"),
        ("autoscaling schedulers", "matching capacity to demand"),
        ("mixed-precision math", "using 16-bit floats on the GPU"),
        ("batch inference", "processing many inputs at once"),
    ]
    passages: list[str] = []
    i = 0
    while len(passages) < n:
        subject, gist = topics[i % len(topics)]
        # A few sentences of filler so each passage is worth summarizing.
        passage = (
            f"This passage discusses {subject}, a topic that has grown rapidly "
            f"in importance over the past few years. At its core, {subject} is "
            f"concerned with {gist}, which sounds simple but has many subtle "
            f"engineering trade-offs. Practitioners care about it because it "
            f"directly affects cost, latency, and reliability at scale. Early "
            f"systems handled {subject} poorly, wasting resources and "
            f"frustrating users, but modern platforms have made it far more "
            f"approachable. Today, a small team can adopt {subject} in an "
            f"afternoon and see real benefits, whereas a decade ago it required "
            f"a dedicated infrastructure group. This is passage number "
            f"{len(passages)} in the batch."
        )
        passages.append(passage)
        i += 1
    return passages

Summarize a chunk on the GPU

This is the remote function: one RTX A4500 per chunk, Volume mounted at /hf-cache. It builds a transformers summarization pipeline on cuda:0 in fp16, runs the whole chunk, then times a single passage on the CPU as a baseline. The model loads once per container, so later chunks routed to a hot container reuse it with no reload or re-download:

Python
@app.function(
    gpu="RTX-A4500",
    image=image,
    volumes={"/hf-cache": hf_cache},
    timeout=900,
)
def summarize_chunk(passages: list[str]) -> dict:
    """Summarize one chunk of passages on the GPU. Loads the model once.

    Returns the summaries plus timing for both GPU and a one-shot CPU baseline
    so the caller can report a real speedup.
    """
    import time

    import torch
    from transformers import pipeline

    # Load once per container; subsequent chunks routed to this hot container
    # reuse the already-loaded model (no reload, no re-download).
    gpu_summarizer = pipeline(
        "summarization",
        model=MODEL_ID,
        device=0,  # cuda:0
        torch_dtype=torch.float16,
    )

    def run(summarizer, texts):
        out = summarizer(
            texts,
            max_length=60,
            min_length=15,
            truncation=True,
            do_sample=False,
            batch_size=8,
        )
        return [o["summary_text"].strip() for o in out]

With the GPU pipeline ready, time the real workload, and only the workload. Wrapping the call in torch.cuda.synchronize() makes the timing honest: CUDA is asynchronous, so without it you'd clock the launch, not the actual generation. Then run one passage on a CPU pipeline to get a per-document baseline (BART on CPU is slow, so timing a single doc and scaling beats making the whole job wait):

Python
    # --- GPU timing (the real workload) ---
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    summaries = run(gpu_summarizer, passages)
    torch.cuda.synchronize()
    gpu_seconds = time.perf_counter() - t0

    # --- CPU baseline on ONE passage, for an honest per-doc speedup number ---
    # (BART on CPU is slow, so we time a single passage and scale, rather than
    # making the whole job wait on the CPU.)
    cpu_summarizer = pipeline(
        "summarization",
        model=MODEL_ID,
        device=-1,  # cpu
    )
    t0 = time.perf_counter()
    run(cpu_summarizer, passages[:1])
    cpu_seconds_per_doc = time.perf_counter() - t0

    return {
        "summaries": summaries,
        "n": len(passages),
        "gpu_seconds": gpu_seconds,
        "cpu_seconds_per_doc": cpu_seconds_per_doc,
    }

Fan the chunks out with .map()

The local_entrypoint runs on your machine. Split the passages into chunks of 10 and call summarize_chunk.map(chunks): AI Fabric runs one chunk per GPU container in parallel, returns results in input order, and return_exceptions=True keeps a single failed chunk from cancelling the rest. As results stream back, accumulate the summaries and the timing:

Python
@app.local_entrypoint()
def main():
    passages = build_passages(50)
    chunk_size = 10
    chunks = [
        passages[i : i + chunk_size] for i in range(0, len(passages), chunk_size)
    ]
    print(
        f"Summarizing {len(passages)} passages in {len(chunks)} chunks "
        f"across the GPU fleet..."
    )

    all_summaries: list[str] = []
    total_gpu = 0.0
    total_docs = 0
    cpu_per_doc_samples: list[float] = []

    # .map() fans the chunks out: one GPU container per chunk, all in parallel.
    # Results arrive in input order, so we can zip them back to `passages`.
    for res in summarize_chunk.map(chunks, return_exceptions=True):
        if isinstance(res, Exception):
            print(f"  chunk failed: {res}")
            continue
        all_summaries.extend(res["summaries"])
        total_gpu += res["gpu_seconds"]
        total_docs += res["n"]
        cpu_per_doc_samples.append(res["cpu_seconds_per_doc"])

Report the estimated speedup

Print a few sample summaries, then turn the per-document CPU baselines into an estimate for the whole batch. Averaging the per-doc samples and multiplying by the document count gives an honest CPU estimate to compare against the real GPU time:

Python
    if not all_summaries:
        print("No summaries produced. Check the run logs above.")
        return

    print(f"\nSummarized {len(all_summaries)} passages.")
    print("\n--- sample summaries ---")
    for i in (0, len(all_summaries) // 2, len(all_summaries) - 1):
        print(f"\n[passage {i}]")
        print(f"  original : {passages[i][:90]}...")
        print(f"  summary  : {all_summaries[i]}")

    # Estimate the CPU cost for the full batch from the per-doc baseline.
    avg_cpu_per_doc = sum(cpu_per_doc_samples) / len(cpu_per_doc_samples)
    est_cpu_total = avg_cpu_per_doc * total_docs
    speedup = (est_cpu_total / total_gpu) if total_gpu > 0 else float("nan")

    print("\n--- timing ---")
    print(f"  GPU summarize time (summed over chunks): {total_gpu:6.2f}s")
    print(f"  CPU baseline:                            {avg_cpu_per_doc:6.2f}s / doc")
    print(f"  estimated CPU time for all {total_docs} docs:     {est_cpu_total:6.2f}s")
    print(f"  speedup:                                 {speedup:5.1f}x faster on GPU")
    print(
        "\nAI Fabric bills each container per started minute: this whole job used a few "
        "GPU-minutes across the fleet, and nothing while idle."
    )

Run it

Run the program with the AI Fabric CLI:

Terminal
blade run summarize_batch.py

The first run downloads the BART weights into the Volume; later runs mount the cache and start summarizing immediately. When it finishes you'll see sample summaries and the timing block printed in your terminal:

Output
  speedup:                                 93.0x faster on GPU

Fifty documents summarized in seconds, billed per started minute: you pay a few GPU-minutes across the fleet and nothing more. Swap build_passages for your own documents (long articles, transcripts, support threads) and the same .map() pattern scales straight up.

Take it for a spin

Everything above in one file, ready to run.

summarize_batch.py
"""Batch document summarization on the GPU fleet, fanned out with .map().

Run with:  blade run batch_summarize.py

What it does
------------
- Builds ~50 medium-length text passages in-process (no files to download, no
  setup). Swap in any list of strings, e.g. a HuggingFace dataset slice.
- Summarizes every passage with facebook/bart-large-cnn (~1.6 GB) on an
  RTX-A4500 GPU. The weights download once from the public HF hub and are
  cached on a Volume, so later runs skip the download. The model fits
  comfortably in 20 GB of VRAM.
- Splits the passages into chunks and uses `.map()` to fan the chunks out
  across the GPU fleet: many containers summarize in parallel, results come
  back in input order.
- Each GPU worker also summarizes its first passage on the CPU once, so the
  script can print a real measured GPU-vs-CPU speedup, not a hand-waved number.

The headline: ~50 documents summarized in seconds on GPU vs minutes on CPU,
and AI Fabric bills each container per started minute, so the job costs a few GPU-minutes.
"""

import bladekit

app = bladekit.app("batch-summarize")

# Cache HF weights on a Volume so they download once, not on every run.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)

# fabric/torch is the AI Fabric catalog base (CUDA + PyTorch + Python 3.12). We add
# transformers on top; no Dockerfile, no manual setup. HF_HOME points the hub
# cache at the mounted Volume.
image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("transformers==4.46.3", "sentencepiece")
    .env({"HF_HOME": "/hf-cache"})
)

MODEL_ID = "facebook/bart-large-cnn"  # ~1.6 GB, public on the HF hub, text-only


def build_passages(n: int = 50) -> list[str]:
    """~50 medium-length passages, self-contained, with no files to download.

    Replace this with any list of strings, e.g. a HuggingFace dataset slice:
        from datasets import load_dataset
        rows = load_dataset("cnn_dailymail", "3.0.0", split="test[:50]")["article"]
    """
    topics = [
        ("serverless GPU computing", "renting GPUs by the second"),
        ("vector databases", "storing dense embeddings for fast search"),
        ("transformer architectures", "the attention mechanism"),
        ("container cold starts", "the time to a first response"),
        ("distributed training", "splitting a model across many devices"),
        ("model quantization", "shrinking weights to lower precision"),
        ("retrieval-augmented generation", "grounding answers in documents"),
        ("autoscaling schedulers", "matching capacity to demand"),
        ("mixed-precision math", "using 16-bit floats on the GPU"),
        ("batch inference", "processing many inputs at once"),
    ]
    passages: list[str] = []
    i = 0
    while len(passages) < n:
        subject, gist = topics[i % len(topics)]
        # A few sentences of filler so each passage is worth summarizing.
        passage = (
            f"This passage discusses {subject}, a topic that has grown rapidly "
            f"in importance over the past few years. At its core, {subject} is "
            f"concerned with {gist}, which sounds simple but has many subtle "
            f"engineering trade-offs. Practitioners care about it because it "
            f"directly affects cost, latency, and reliability at scale. Early "
            f"systems handled {subject} poorly, wasting resources and "
            f"frustrating users, but modern platforms have made it far more "
            f"approachable. Today, a small team can adopt {subject} in an "
            f"afternoon and see real benefits, whereas a decade ago it required "
            f"a dedicated infrastructure group. This is passage number "
            f"{len(passages)} in the batch."
        )
        passages.append(passage)
        i += 1
    return passages


@app.function(
    gpu="RTX-A4500",
    image=image,
    volumes={"/hf-cache": hf_cache},
    timeout=900,
)
def summarize_chunk(passages: list[str]) -> dict:
    """Summarize one chunk of passages on the GPU. Loads the model once.

    Returns the summaries plus timing for both GPU and a one-shot CPU baseline
    so the caller can report a real speedup.
    """
    import time

    import torch
    from transformers import pipeline

    # Load once per container; subsequent chunks routed to this hot container
    # reuse the already-loaded model (no reload, no re-download).
    gpu_summarizer = pipeline(
        "summarization",
        model=MODEL_ID,
        device=0,  # cuda:0
        torch_dtype=torch.float16,
    )

    def run(summarizer, texts):
        out = summarizer(
            texts,
            max_length=60,
            min_length=15,
            truncation=True,
            do_sample=False,
            batch_size=8,
        )
        return [o["summary_text"].strip() for o in out]

    # --- GPU timing (the real workload) ---
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    summaries = run(gpu_summarizer, passages)
    torch.cuda.synchronize()
    gpu_seconds = time.perf_counter() - t0

    # --- CPU baseline on ONE passage, for an honest per-doc speedup number ---
    # (BART on CPU is slow, so we time a single passage and scale, rather than
    # making the whole job wait on the CPU.)
    cpu_summarizer = pipeline(
        "summarization",
        model=MODEL_ID,
        device=-1,  # cpu
    )
    t0 = time.perf_counter()
    run(cpu_summarizer, passages[:1])
    cpu_seconds_per_doc = time.perf_counter() - t0

    return {
        "summaries": summaries,
        "n": len(passages),
        "gpu_seconds": gpu_seconds,
        "cpu_seconds_per_doc": cpu_seconds_per_doc,
    }


@app.local_entrypoint()
def main():
    passages = build_passages(50)
    chunk_size = 10
    chunks = [
        passages[i : i + chunk_size] for i in range(0, len(passages), chunk_size)
    ]
    print(
        f"Summarizing {len(passages)} passages in {len(chunks)} chunks "
        f"across the GPU fleet..."
    )

    all_summaries: list[str] = []
    total_gpu = 0.0
    total_docs = 0
    cpu_per_doc_samples: list[float] = []

    # .map() fans the chunks out: one GPU container per chunk, all in parallel.
    # Results arrive in input order, so we can zip them back to `passages`.
    for res in summarize_chunk.map(chunks, return_exceptions=True):
        if isinstance(res, Exception):
            print(f"  chunk failed: {res}")
            continue
        all_summaries.extend(res["summaries"])
        total_gpu += res["gpu_seconds"]
        total_docs += res["n"]
        cpu_per_doc_samples.append(res["cpu_seconds_per_doc"])

    if not all_summaries:
        print("No summaries produced. Check the run logs above.")
        return

    print(f"\nSummarized {len(all_summaries)} passages.")
    print("\n--- sample summaries ---")
    for i in (0, len(all_summaries) // 2, len(all_summaries) - 1):
        print(f"\n[passage {i}]")
        print(f"  original : {passages[i][:90]}...")
        print(f"  summary  : {all_summaries[i]}")

    # Estimate the CPU cost for the full batch from the per-doc baseline.
    avg_cpu_per_doc = sum(cpu_per_doc_samples) / len(cpu_per_doc_samples)
    est_cpu_total = avg_cpu_per_doc * total_docs
    speedup = (est_cpu_total / total_gpu) if total_gpu > 0 else float("nan")

    print("\n--- timing ---")
    print(f"  GPU summarize time (summed over chunks): {total_gpu:6.2f}s")
    print(f"  CPU baseline:                            {avg_cpu_per_doc:6.2f}s / doc")
    print(f"  estimated CPU time for all {total_docs} docs:     {est_cpu_total:6.2f}s")
    print(f"  speedup:                                 {speedup:5.1f}x faster on GPU")
    print(
        "\nAI Fabric bills each container per started minute: this whole job used a few "
        "GPU-minutes across the fleet, and nothing while idle."
    )

Concepts used: GPU, Run, Volume

Contact the team

Loading the form…