Developer docs / AI Fabric tutorials
Warm pools: cutting sandbox cold-start
- Sandbox
- Image DSL
Take it for a spin
The whole program in one file, ready to run.
import statistics
import time
from collections import deque
import bladekit
app = bladekit.app("sandbox-warm-pools")
# CPU-only image: no GPU needed for this benchmark.
image = bladekit.Image.from_catalog("fabric/base")
class WarmPool:
"""A small client-side pool of pre-created, running sandboxes.
checkout() pops an already-running sandbox, so no create() round-trip on
the hot path. checkin_and_replenish() terminates the used sandbox and
immediately creates a replacement so the pool stays at its target size.
NOTE: this is a client-side pattern, not a server-side primitive. AI Fabric's
Sandbox.create() has no min_warm=N / pre-warm knob today; the pool lives
in this process's memory. A real deployment would run this inside a
long-lived @app.function (or a small service) so many callers share the
same warm sandboxes.
"""
def __init__(self, size: int, **create_kwargs):
self._create_kwargs = create_kwargs
self._pool: deque[bladekit.Sandbox] = deque()
for _ in range(size):
self._pool.append(bladekit.Sandbox.create(**create_kwargs))
def checkout(self) -> bladekit.Sandbox:
"""Hand out an already-running sandbox. No create() call here."""
return self._pool.popleft()
def checkin_and_replenish(self, sb: bladekit.Sandbox) -> None:
"""Retire a used sandbox and top the pool back up to size."""
sb.terminate()
self._pool.append(bladekit.Sandbox.create(**self._create_kwargs))
def drain(self) -> None:
while self._pool:
self._pool.popleft().terminate()
def _time_cold_create() -> float:
t0 = time.monotonic()
sb = bladekit.Sandbox.create(image=image, timeout=60)
elapsed = time.monotonic() - t0
sb.terminate()
return elapsed
def _time_warm_checkout(pool: WarmPool) -> float:
t0 = time.monotonic()
sb = pool.checkout()
elapsed = time.monotonic() - t0
pool.checkin_and_replenish(sb)
return elapsed
@app.local_entrypoint()
def main():
n_trials = 3
cold_times = [_time_cold_create() for _ in range(n_trials)]
pool = WarmPool(n_trials, image=image, timeout=60)
warm_times = [_time_warm_checkout(pool) for _ in range(n_trials)]
pool.drain()
cold_avg = statistics.mean(cold_times)
warm_avg = statistics.mean(warm_times)
print(f"cold create avg: {cold_avg:.3f}s")
print(f"warm checkout avg: {warm_avg:.4f}s")
print(f"speedup: {cold_avg / warm_avg:.0f}x")Run it
- Save as
sandbox_warm_pools.py. Runblade run sandbox_warm_pools.py. - AI Fabric does NOT have a built-in warm-pool primitive today:
bladekit.Sandbox.create()has nomin_warm=Nargument, and the control plane has no concept of a pre-warmed sandbox reserve. This tutorial shows the client-side pattern that gets you the same effect. WarmPool.__init__eagerly callsSandbox.create()sizetimes up front, absorbing the create latency before any request needs a sandbox.checkout()is a plaindeque.popleft(): no network round-trip, no scheduling wait.checkin_and_replenish()terminates the used sandbox and immediately creates its replacement so the pool is back at target size for the next caller.- Live measurement on the
runscpool: coldSandbox.create()averaged ~0.96 s (image already built/cached, CPU-onlyfabric/base). Warmcheckout()from a pre-filled pool averaged under 0.1 ms, a ~15,000-150,000x reduction, because the entire scheduling + container-start path is paid ahead of time instead of on the request path. - In production, run the
WarmPoolinside a long-lived@app.function(or a small always-on service) so many callers share the same warm sandboxes, and size the pool to your expected concurrent burst. checkin_and_replenish()still pays the create latency, but off the critical path. Pick a replenishment strategy (synchronous like here, or a background thread) based on whether your workload can tolerate a temporarily smaller pool under bursty checkout.
Concepts used: Cold start, Run