Developer docs / AI Fabric tutorials

Resumable long training run

  • GPU
  • Volume
  • Image DSL

A training run that takes hours can lose its machine before it finishes. On cheap, interruptible capacity the node can be reclaimed at any moment, and without a safety net you lose everything and start from zero. The fix is checkpoint often, persist the checkpoints somewhere durable, and resume from the last one, so an interruption costs you minutes, not hours.

In this tutorial you'll run a multi-hour language-model training job that survives preemption. Checkpoints land on an AI Fabric Volume that outlives any single container, and AI Fabric's retry policy re-queues the run automatically. You write a normal Hugging Face Trainer loop; AI Fabric handles the durability.

Before you start

You need the AI Fabric CLI installed and authenticated: run blade setup once. The job requests a single RTX A4500, and it builds on fabric/torch, the catalog image that already ships CUDA and PyTorch, so there's no Docker work to do.

Create the app and a checkpoint Volume

Create a file called resumable_training.py. The key piece is the Volume: it's storage that persists across runs, so a checkpoint written by one container is still there when a retry spins up a new one. create_if_missing=True makes the first run create it for you.

resumable_training.py
import bladekit

app = bladekit.app("resumable-training")

checkpoints = bladekit.Volume.from_name("training-checkpoints", create_if_missing=True)

Then start from fabric/torch and add the training libraries on top. Because the heavy CUDA + PyTorch base is already built, this layer is quick:

Python
image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("torch", "transformers", "datasets", "accelerate")
)

Declare the function with retries and a mounted Volume

The decorator is where durability is configured. Mount the Volume at /checkpoints so the function can write to it, set a generous timeout for the long run, and ask for retries=3 so AI Fabric re-queues the job if the node is reclaimed. priority=0 runs it as preemptible (the cheapest tier), which is the situation checkpointing protects you from.

Python
@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=86400,   # 24 h max runtime
    volumes={"/checkpoints": checkpoints},
    retries=3,       # auto-resume from checkpoint on preemption
    priority=0,  # lowest priority = run as preemptible to save cost
)
def train_model(config: dict) -> str:
    import os, torch
    from transformers import (
        AutoModelForCausalLM, AutoTokenizer,
        TrainingArguments, Trainer, DataCollatorForLanguageModeling
    )
    from datasets import load_dataset

Load the model and dataset

The body is ordinary transformers. Read the model name and a per-run output directory from the config dict. Pointing output_dir inside the Volume is what keeps checkpoints durable. Load GPT-2, set its pad token, and pull in WikiText:

Python
    model_name = config.get("model", "gpt2")
    output_dir = "/checkpoints/" + config.get("run_name", "run1")

    model = AutoModelForCausalLM.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token

    dataset = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="train")

    def tokenize(examples):
        return tokenizer(examples["text"], truncation=True, max_length=512)

    dataset = dataset.map(tokenize, batched=True, remove_columns=["text"])

Train with frequent checkpoints

TrainingArguments is where the resumption contract lives. save_steps=200 writes a checkpoint every 200 steps to the Volume, and resume_from_checkpoint=True tells the Trainer to look in output_dir for an existing checkpoint and continue from it. On a fresh run there's nothing to resume; on a retry, that directory is already populated.

Python
    args = TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=config.get("epochs", 5),
        per_device_train_batch_size=8,
        save_steps=200,
        resume_from_checkpoint=True,  # key for resumption
        logging_steps=50,
    )
    collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)
    Trainer(model=model, args=args, train_dataset=dataset, data_collator=collator).train()
    return output_dir

Kick it off from your machine

A local_entrypoint runs on your laptop. Calling train_model.remote(...) ships the config to the GPU container, runs the job there, and returns the final checkpoint path:

Python
@app.local_entrypoint()
def main():
    out = train_model.remote({"model": "gpt2", "run_name": "gpt2-wiki-run1", "epochs": 3})
    print(f"Training complete. Checkpoint at: {out}")

Run it

Launch the run with the AI Fabric CLI:

Terminal
blade run resumable_training.py

Checkpoints accumulate in the training-checkpoints Volume every 200 steps. If the run is preempted (say the machine is reclaimed), retries=3 means AI Fabric re-queues it up to three times, and each retry picks up the latest checkpoint instead of restarting from scratch. When training finishes, mount the same Volume in a serving function to deploy the model.

Take it for a spin

Everything above in one file, ready to run.

resumable_training.py
import bladekit

app = bladekit.app("resumable-training")

checkpoints = bladekit.Volume.from_name("training-checkpoints", create_if_missing=True)

image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("torch", "transformers", "datasets", "accelerate")
)

@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=86400,   # 24 h max runtime
    volumes={"/checkpoints": checkpoints},
    retries=3,       # auto-resume from checkpoint on preemption
    priority=0,  # lowest priority = run as preemptible to save cost
)
def train_model(config: dict) -> str:
    import os, torch
    from transformers import (
        AutoModelForCausalLM, AutoTokenizer,
        TrainingArguments, Trainer, DataCollatorForLanguageModeling
    )
    from datasets import load_dataset

    model_name = config.get("model", "gpt2")
    output_dir = "/checkpoints/" + config.get("run_name", "run1")

    model = AutoModelForCausalLM.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token

    dataset = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="train")

    def tokenize(examples):
        return tokenizer(examples["text"], truncation=True, max_length=512)

    dataset = dataset.map(tokenize, batched=True, remove_columns=["text"])

    args = TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=config.get("epochs", 5),
        per_device_train_batch_size=8,
        save_steps=200,
        resume_from_checkpoint=True,  # key for resumption
        logging_steps=50,
    )
    collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)
    Trainer(model=model, args=args, train_dataset=dataset, data_collator=collator).train()
    return output_dir

@app.local_entrypoint()
def main():
    out = train_model.remote({"model": "gpt2", "run_name": "gpt2-wiki-run1", "epochs": 3})
    print(f"Training complete. Checkpoint at: {out}")

Concepts used: GPU, Run

Contact the team

Loading the form…