Developer docs / AI Fabric tutorials
Hyperparameter grid search with early stopping
- GPU
- Batch / map
- Volume
- Image DSL
Hyperparameter tuning is embarrassingly parallel: each combination of learning rate and batch size is an independent experiment that doesn't need to talk to the others. Run them one after another on a single GPU and you wait for the sum of all of them; run them at the same time, each on its own GPU, and you wait only for the slowest one. That second option is what fan-out is for.
In this tutorial you'll sweep a small grid of learning rates and batch sizes by fine-tuning BERT on SST-2, launching every experiment in parallel with .starmap(). Each run reports its validation accuracy, so you can pick the winner once they all return.
Before you start
Install and authenticate the AI Fabric CLI with blade setup. Each experiment requests its own RTX A4500 and builds on the fabric/torch catalog image (CUDA + PyTorch preinstalled), with scikit-learn and the training stack layered on top.
Create the app and image
Create hp_grid_search.py. Declare the app, a Volume to hold each run's checkpoints, and an image that adds the training libraries on top of fabric/torch:
import bladekit
app = bladekit.app("hp-grid-search")
checkpoints = bladekit.Volume.from_name("hp-search-checkpoints", create_if_missing=True)
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("torch", "transformers", "datasets", "accelerate", "scikit-learn")
)Write one experiment as a function
Fan-out starts with making a single experiment a plain function, then let AI Fabric run many copies of it at once. This one takes the hyperparameters as arguments (lr, batch_size, and a run_name), loads BERT and a small slice of the SST-2 sentiment dataset, and tokenizes it:
@app.function(
gpu="RTX-A4500",
image=image,
timeout=3600,
volumes={"/checkpoints": checkpoints},
)
def train_experiment(lr: float, batch_size: int, run_name: str) -> dict:
import numpy as np
from transformers import (
AutoModelForSequenceClassification,
AutoTokenizer,
TrainingArguments,
Trainer,
)
from datasets import load_dataset
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
raw = load_dataset("nyu-mll/glue", "sst2")
def tokenize(b):
return tok(b["sentence"], truncation=True, padding="max_length", max_length=64)
train_ds = raw["train"].select(range(1000)).map(tokenize, batched=True)
eval_ds = raw["validation"].select(range(200)).map(tokenize, batched=True)
model = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=2
)Score each run and keep the best checkpoint
Define compute_metrics so each run reports accuracy. The early-stopping contract in TrainingArguments is load_best_model_at_end=True together with matching eval_strategy and save_strategy and a metric_for_best_model. That combination tells the Trainer to keep the highest-accuracy checkpoint rather than the last one. Train, evaluate, and return the metrics tagged with this run's config:
def compute_metrics(p):
preds = np.argmax(p.predictions, axis=1)
return {"accuracy": float((preds == p.label_ids).mean())}
# load_best_model_at_end requires save_strategy == eval_strategy and a
# metric_for_best_model; that is the best-checkpoint / early-stopping contract.
args = TrainingArguments(
output_dir=f"/checkpoints/{run_name}",
learning_rate=lr,
per_device_train_batch_size=batch_size,
num_train_epochs=1,
eval_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="accuracy",
report_to=[],
)
trainer = Trainer(
model=model,
args=args,
train_dataset=train_ds,
eval_dataset=eval_ds,
compute_metrics=compute_metrics,
)
trainer.train()
metrics = trainer.evaluate()
return {"run_name": run_name, "lr": lr, "batch_size": batch_size, **metrics} Fan out the grid with .starmap()
Build the grid as a list of (lr, batch_size, run_name) tuples: three learning rates times two batch sizes is six experiments. .starmap() is like .map() but unpacks each tuple as separate arguments, so every tuple becomes one train_experiment(lr, batch_size, run_name) call running on its own GPU. return_exceptions=True means a single crashed run doesn't sink the rest:
@app.local_entrypoint()
def main():
# Grid: 3 learning rates × 2 batch sizes = 6 experiments, fanned out with .starmap().
grid = [
(lr, bs, f"run-lr{lr}-bs{bs}")
for lr in [1e-4, 5e-5, 1e-5]
for bs in [16, 32]
]
results = list(train_experiment.starmap(grid, return_exceptions=True))
valid = [r for r in results if not isinstance(r, Exception)]
if not valid:
print("All experiments failed:", results)
return
best = max(valid, key=lambda r: r.get("eval_accuracy", 0))
print(
f"Best: lr={best['lr']}, bs={best['batch_size']}, "
f"acc={best.get('eval_accuracy', 0):.4f} ({len(valid)}/{len(grid)} ran)"
)Run it
Launch the whole sweep with one command:
blade run hp_grid_search.pyAll six experiments run at once, each on its own RTX A4500, so the total wall time is roughly the time of the slowest experiment rather than the sum of all six. When they return, the entrypoint sorts by eval_accuracy and prints the winning combination. Add more learning rates or batch sizes and AI Fabric auto-provisions more containers to keep them parallel.
Take it for a spin
Everything above in one file, ready to run.
import bladekit
app = bladekit.app("hp-grid-search")
checkpoints = bladekit.Volume.from_name("hp-search-checkpoints", create_if_missing=True)
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("torch", "transformers", "datasets", "accelerate", "scikit-learn")
)
@app.function(
gpu="RTX-A4500",
image=image,
timeout=3600,
volumes={"/checkpoints": checkpoints},
)
def train_experiment(lr: float, batch_size: int, run_name: str) -> dict:
import numpy as np
from transformers import (
AutoModelForSequenceClassification,
AutoTokenizer,
TrainingArguments,
Trainer,
)
from datasets import load_dataset
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
raw = load_dataset("nyu-mll/glue", "sst2")
def tokenize(b):
return tok(b["sentence"], truncation=True, padding="max_length", max_length=64)
train_ds = raw["train"].select(range(1000)).map(tokenize, batched=True)
eval_ds = raw["validation"].select(range(200)).map(tokenize, batched=True)
model = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=2
)
def compute_metrics(p):
preds = np.argmax(p.predictions, axis=1)
return {"accuracy": float((preds == p.label_ids).mean())}
# load_best_model_at_end requires save_strategy == eval_strategy and a
# metric_for_best_model; that is the best-checkpoint / early-stopping contract.
args = TrainingArguments(
output_dir=f"/checkpoints/{run_name}",
learning_rate=lr,
per_device_train_batch_size=batch_size,
num_train_epochs=1,
eval_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="accuracy",
report_to=[],
)
trainer = Trainer(
model=model,
args=args,
train_dataset=train_ds,
eval_dataset=eval_ds,
compute_metrics=compute_metrics,
)
trainer.train()
metrics = trainer.evaluate()
return {"run_name": run_name, "lr": lr, "batch_size": batch_size, **metrics}
@app.local_entrypoint()
def main():
# Grid: 3 learning rates × 2 batch sizes = 6 experiments, fanned out with .starmap().
grid = [
(lr, bs, f"run-lr{lr}-bs{bs}")
for lr in [1e-4, 5e-5, 1e-5]
for bs in [16, 32]
]
results = list(train_experiment.starmap(grid, return_exceptions=True))
valid = [r for r in results if not isinstance(r, Exception)]
if not valid:
print("All experiments failed:", results)
return
best = max(valid, key=lambda r: r.get("eval_accuracy", 0))
print(
f"Best: lr={best['lr']}, bs={best['batch_size']}, "
f"acc={best.get('eval_accuracy', 0):.4f} ({len(valid)}/{len(grid)} ran)"
)Concepts used: GPU, Run