Developer docs / AI Fabric tutorials
High-throughput Whisper with batching
- GPU
- Batch / map
- Concurrency
- Image DSL
The simple way to transcribe a lot of audio is to give every file its own GPU container. It's clean and parallel, but you pay for a whole GPU per file even when each one barely keeps the card busy. When your clips are small or your fleet is finite, the cheaper win is the opposite move: pack many files onto one GPU and keep it saturated.
In this tutorial you'll push Whisper throughput two ways at once. faster-whisper (built on CTranslate2) runs the model far faster than the original and batches chunks within a file, while @app.concurrent lets several files be processed on the same warm replica at the same time. The result is a transcriber that gets a lot of work out of a single RTX A4500.
Before you start
Make sure the AI Fabric CLI is installed and authenticated (blade setup). The function requests one RTX A4500, and the image builds on the fabric/torch catalog base, so CUDA and PyTorch are already in place; you only add the faster-whisper stack and ffmpeg.
Create the app
Create whisper_highperf.py and declare the app:
import bladekit
app = bladekit.app("whisper-highperf")Build the image
Layer faster-whisper and numpy onto the fabric/torch base, plus ffmpeg for decoding. faster-whisper reimplements Whisper on CTranslate2, which is where the speedup comes from:
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("torch", "faster-whisper", "numpy")
.apt_install("ffmpeg")
)Define a concurrent GPU function
Stack two decorators. @app.function asks for the RTX A4500 and the image; @app.concurrent(max_inputs=8) tells AI Fabric that one replica can handle up to 8 inputs at once. Without it, a replica would process a single file at a time and the GPU would sit half-idle between files:
_model = None
@app.function(gpu="RTX-A4500", image=image, timeout=600)
@app.concurrent(max_inputs=8)
def transcribe_batch(audio_bytes: bytes) -> str:
global _model
import tempfile
from faster_whisper import WhisperModel, BatchedInferencePipelineThat _model global is the warm cache: it's loaded once and shared by every concurrent call on the replica. Combining concurrency with a single shared model is the whole point: you load the weights once and keep many requests flowing through them.
Load the batched pipeline
Load the model on first use only. Wrapping WhisperModel in a BatchedInferencePipeline enables intra-file batching, splitting one audio file into chunks and running them through the GPU together. Note that the batch_size argument belongs on the pipeline's transcribe, not on a plain WhisperModel:
if _model is None:
# Batched pipeline = intra-file batching (the batch_size arg lives here,
# not on a plain WhisperModel).
_model = BatchedInferencePipeline(
WhisperModel("large-v3", device="cuda", compute_type="float16"))Transcribe with batching
Write the incoming bytes to a temp file so ffmpeg can decode it, then transcribe. beam_size=5 improves accuracy and batch_size=16 runs chunks of the file through the GPU together. faster-whisper yields segments, so join their text into one string:
with tempfile.NamedTemporaryFile(suffix=".mp3", delete=False) as f:
f.write(audio_bytes)
tmp = f.name
segments, _ = _model.transcribe(tmp, beam_size=5, batch_size=16)
return " ".join(s.text for s in segments)Two layers of batching work together: batch_size packs chunks of a single file onto the GPU, while max_inputs=8 lets up to eight different files share the same warm replica. The card stays busy on both axes.
Map files onto the replica
The local_entrypoint reads every clip in audio/ and calls transcribe_batch.map(...). Because the function is concurrent, those inputs don't each spin up a fresh container. Up to 8 of them ride a single warm replica at once:
@app.local_entrypoint()
def main():
import pathlib
files = list(pathlib.Path("audio/").glob("*.mp3"))
audio_list = [f.read_bytes() for f in files]
results = list(transcribe_batch.map(audio_list, return_exceptions=True))
for path, text in zip(files, results):
status = "ERR" if isinstance(text, Exception) else "OK"
print(f"[{status}] {path.name}: {str(text)[:80]}")As with any .map(), results come back in input order and return_exceptions=True keeps one bad file from sinking the batch. It shows up as an ERR line while everything else stays OK.
Run it
Put your MP3 files in an audio/ folder next to the script and run it:
blade run whisper_highperf.pyThe first file pays a cold start while the replica boots and the model loads; after that the warm large-v3 pipeline chews through the rest, with up to eight files in flight on the one GPU. You get the throughput of a batch job without paying for a GPU per file.
Take it for a spin
Everything above in one file, ready to run.
import bladekit
app = bladekit.app("whisper-highperf")
# fabric/torch already ships torch, so don't reinstall it. Add faster-whisper +
# datasets/soundfile so the script can fetch sample clips itself.
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("faster-whisper", "numpy", "datasets", "soundfile")
.apt_install("ffmpeg")
)
@app.function(image=image, timeout=300)
def sample_clips(n: int = 8) -> list[bytes]:
"""Return a handful of short, stable public clips as WAV bytes, so the
tutorial is self-contained (no audio folder to supply)."""
import io, soundfile as sf
from datasets import load_dataset, Audio
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
# decode=False avoids datasets' torchcodec backend; soundfile reads the FLAC.
ds = ds.cast_column("audio", Audio(decode=False))
clips = []
for i in range(min(n, len(ds))):
arr, sr = sf.read(io.BytesIO(ds[i]["audio"]["bytes"]))
buf = io.BytesIO()
sf.write(buf, arr, sr, format="WAV")
clips.append(buf.getvalue())
return clips
_model = None
@app.function(gpu="RTX-A4500", image=image, timeout=600)
@app.concurrent(max_inputs=8)
def transcribe_batch(audio_bytes: bytes) -> str:
global _model
import tempfile
from faster_whisper import WhisperModel, BatchedInferencePipeline
if _model is None:
# Batched pipeline = intra-file batching (the batch_size arg lives here,
# not on a plain WhisperModel).
_model = BatchedInferencePipeline(
WhisperModel("large-v3", device="cuda", compute_type="float16"))
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
f.write(audio_bytes)
tmp = f.name
segments, _ = _model.transcribe(tmp, beam_size=5, batch_size=16)
return " ".join(s.text for s in segments)
@app.local_entrypoint()
def main():
clips = sample_clips.remote()
results = list(transcribe_batch.map(clips, return_exceptions=True))
for i, text in enumerate(results):
status = "ERR" if isinstance(text, Exception) else "OK"
print(f"[{status}] clip {i}: {str(text)[:80]}")Concepts used: Cold start, GPU, Run