Developer docs / AI Fabric tutorials
Dynamic batching with @bladekit.batched
- GPU
- Batch / map
- Concurrency
- Web endpoint
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("dynamic-batching")
# One GPU forward pass over N prompts is far cheaper than N passes of one prompt.
# @bladekit.batched collects concurrent single-item calls into a batch, invokes the
# function ONCE with the whole batch, then routes each result back to its caller.
@app.function(gpu="RTX-A4500", image="fabric/torch")
@app.concurrent(max_inputs=64) # admit up to 64 in-flight calls per replica
@bladekit.batched(max_batch_size=32, wait_ms=50)
def embed(prompts: list[str]) -> list[list[float]]:
from sentence_transformers import SentenceTransformer
# Loaded once per container, reused across every batch.
global _model
try:
_model
except NameError:
_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cuda")
# 'prompts' is the collected batch: return exactly one vector per prompt.
return _model.encode(prompts, batch_size=len(prompts)).tolist()
@app.local_entrypoint()
def main():
# Each caller asks for ONE embedding; AI Fabric fuses concurrent calls into
# batches of up to 32 (or whatever arrived within 50 ms) under the hood.
handles = [embed.spawn(f"Document {i}") for i in range(100)]
vecs = [h.get() for h in handles]
print(f"Embedded {len(vecs)} prompts, dim={len(vecs[0])}")Run it
- Save as
dynamic_batching.py. Write the function to take a LIST of prompts and return a LIST of results, one per input. @bladekit.batched(max_batch_size=32, wait_ms=50)collects individual calls: a batch dispatches as soon as it hits 32 inputs OR 50 ms elapse after its first input.- Callers still invoke one item at a time (
embed.remote("hi")orembed.spawn("hi")) and get one result back. The batching is invisible to them. - Stack it under
@app.concurrentso a replica admits enough in-flight calls to actually fill a batch. Distinct from@app.concurrentalone, which handles each request independently rather than fusing them. - Tune the trade-off: a larger
max_batch_sizeraises throughput; a smallerwait_mslowers tail latency for the first caller in a batch.
Concepts used: GPU, Run, Cold start