Developer docs / AI Fabric tutorials

Warm pools: cutting sandbox cold-start

  • Sandbox
  • Image DSL

Take it for a spin

The whole program in one file, ready to run.

sandbox_warm_pools.py
import statistics
import time
from collections import deque

import bladekit

app = bladekit.app("sandbox-warm-pools")

# CPU-only image: no GPU needed for this benchmark.
image = bladekit.Image.from_catalog("fabric/base")


class WarmPool:
    """A small client-side pool of pre-created, running sandboxes.

    checkout() pops an already-running sandbox, so no create() round-trip on
    the hot path. checkin_and_replenish() terminates the used sandbox and
    immediately creates a replacement so the pool stays at its target size.

    NOTE: this is a client-side pattern, not a server-side primitive. AI Fabric's
    Sandbox.create() has no min_warm=N / pre-warm knob today; the pool lives
    in this process's memory. A real deployment would run this inside a
    long-lived @app.function (or a small service) so many callers share the
    same warm sandboxes.
    """

    def __init__(self, size: int, **create_kwargs):
        self._create_kwargs = create_kwargs
        self._pool: deque[bladekit.Sandbox] = deque()
        for _ in range(size):
            self._pool.append(bladekit.Sandbox.create(**create_kwargs))

    def checkout(self) -> bladekit.Sandbox:
        """Hand out an already-running sandbox. No create() call here."""
        return self._pool.popleft()

    def checkin_and_replenish(self, sb: bladekit.Sandbox) -> None:
        """Retire a used sandbox and top the pool back up to size."""
        sb.terminate()
        self._pool.append(bladekit.Sandbox.create(**self._create_kwargs))

    def drain(self) -> None:
        while self._pool:
            self._pool.popleft().terminate()


def _time_cold_create() -> float:
    t0 = time.monotonic()
    sb = bladekit.Sandbox.create(image=image, timeout=60)
    elapsed = time.monotonic() - t0
    sb.terminate()
    return elapsed


def _time_warm_checkout(pool: WarmPool) -> float:
    t0 = time.monotonic()
    sb = pool.checkout()
    elapsed = time.monotonic() - t0
    pool.checkin_and_replenish(sb)
    return elapsed


@app.local_entrypoint()
def main():
    n_trials = 3

    cold_times = [_time_cold_create() for _ in range(n_trials)]

    pool = WarmPool(n_trials, image=image, timeout=60)
    warm_times = [_time_warm_checkout(pool) for _ in range(n_trials)]
    pool.drain()

    cold_avg = statistics.mean(cold_times)
    warm_avg = statistics.mean(warm_times)
    print(f"cold create avg: {cold_avg:.3f}s")
    print(f"warm checkout avg: {warm_avg:.4f}s")
    print(f"speedup: {cold_avg / warm_avg:.0f}x")

Run it

  1. Save as sandbox_warm_pools.py. Run blade run sandbox_warm_pools.py.
  2. AI Fabric does NOT have a built-in warm-pool primitive today: bladekit.Sandbox.create() has no min_warm=N argument, and the control plane has no concept of a pre-warmed sandbox reserve. This tutorial shows the client-side pattern that gets you the same effect.
  3. WarmPool.__init__ eagerly calls Sandbox.create() size times up front, absorbing the create latency before any request needs a sandbox.
  4. checkout() is a plain deque.popleft(): no network round-trip, no scheduling wait. checkin_and_replenish() terminates the used sandbox and immediately creates its replacement so the pool is back at target size for the next caller.
  5. Live measurement on the runsc pool: cold Sandbox.create() averaged ~0.96 s (image already built/cached, CPU-only fabric/base). Warm checkout() from a pre-filled pool averaged under 0.1 ms, a ~15,000-150,000x reduction, because the entire scheduling + container-start path is paid ahead of time instead of on the request path.
  6. In production, run the WarmPool inside a long-lived @app.function (or a small always-on service) so many callers share the same warm sandboxes, and size the pool to your expected concurrent burst.
  7. checkin_and_replenish() still pays the create latency, but off the critical path. Pick a replenishment strategy (synchronous like here, or a background thread) based on whether your workload can tolerate a temporarily smaller pool under bursty checkout.

Concepts used: Cold start, Run

Contact the team

Loading the form…