Developer docs / AI Fabric tutorials

Dynamic batching with @bladekit.batched

  • GPU
  • Batch / map
  • Concurrency
  • Web endpoint

Take it for a spin

The whole program in one file, ready to run.

dynamic_batching.py
import bladekit

app = bladekit.app("dynamic-batching")

# One GPU forward pass over N prompts is far cheaper than N passes of one prompt.
# @bladekit.batched collects concurrent single-item calls into a batch, invokes the
# function ONCE with the whole batch, then routes each result back to its caller.
@app.function(gpu="RTX-A4500", image="fabric/torch")
@app.concurrent(max_inputs=64)          # admit up to 64 in-flight calls per replica
@bladekit.batched(max_batch_size=32, wait_ms=50)
def embed(prompts: list[str]) -> list[list[float]]:
    from sentence_transformers import SentenceTransformer

    # Loaded once per container, reused across every batch.
    global _model
    try:
        _model
    except NameError:
        _model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cuda")

    # 'prompts' is the collected batch: return exactly one vector per prompt.
    return _model.encode(prompts, batch_size=len(prompts)).tolist()

@app.local_entrypoint()
def main():
    # Each caller asks for ONE embedding; AI Fabric fuses concurrent calls into
    # batches of up to 32 (or whatever arrived within 50 ms) under the hood.
    handles = [embed.spawn(f"Document {i}") for i in range(100)]
    vecs = [h.get() for h in handles]
    print(f"Embedded {len(vecs)} prompts, dim={len(vecs[0])}")

Run it

  1. Save as dynamic_batching.py. Write the function to take a LIST of prompts and return a LIST of results, one per input.
  2. @bladekit.batched(max_batch_size=32, wait_ms=50) collects individual calls: a batch dispatches as soon as it hits 32 inputs OR 50 ms elapse after its first input.
  3. Callers still invoke one item at a time (embed.remote("hi") or embed.spawn("hi")) and get one result back. The batching is invisible to them.
  4. Stack it under @app.concurrent so a replica admits enough in-flight calls to actually fill a batch. Distinct from @app.concurrent alone, which handles each request independently rather than fusing them.
  5. Tune the trade-off: a larger max_batch_size raises throughput; a smaller wait_ms lowers tail latency for the first caller in a batch.

Concepts used: GPU, Run, Cold start

Contact the team

Loading the form…