Developer docs / AI Fabric tutorials

Fine-tune an LLM with Unsloth

  • GPU
  • Volume
  • Image DSL

Take it for a spin

The whole program in one file, ready to run.

finetune_unsloth.py
import bladekit

app = bladekit.app("finetune-unsloth")

checkpoints = bladekit.Volume.from_name("unsloth-checkpoints", create_if_missing=True)

image = (
    bladekit.Image.debian_slim(python_version="3.12")
    # Unsloth JIT-compiles Triton kernels at the first training step,
    # which needs a full C/C++ toolchain in the image.
    .apt_install("build-essential")
    .pip_install(
        "torch", "unsloth", "trl", "transformers",
        "datasets", "accelerate", "peft", "bitsandbytes"
    )
)

@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=7200,
    volumes={"/checkpoints": checkpoints},
)
def finetune(model_name: str = "unsloth/Llama-3.2-1B-Instruct-bnb-4bit") -> str:
    from unsloth import FastLanguageModel
    from trl import SFTConfig, SFTTrainer
    from datasets import load_dataset

    # Pre-quantized 4-bit weights: quick to download and fits
    # comfortably in the RTX-A4500's VRAM.
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=model_name,
        max_seq_length=2048,
        load_in_4bit=True,
    )
    model = FastLanguageModel.get_peft_model(
        model,
        r=16,
        target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
        lora_alpha=16,
    )

    dataset = load_dataset("tatsu-lab/alpaca", split="train[:2000]")

    trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=dataset,
        args=SFTConfig(
            dataset_text_field="text",
            output_dir="/checkpoints/unsloth-run",
            max_steps=60,
            per_device_train_batch_size=2,
            logging_steps=1,
            # Save once at the end: mid-training checkpoint saves and
            # multiprocess tokenization both hit unsloth/trl pickling issues.
            save_strategy="no",
            dataset_num_proc=1,
            report_to="none",
        ),
    )
    trainer.train()

    # Persist the LoRA adapter (~13 MB) to the Volume.
    out_dir = "/checkpoints/unsloth-run/final"
    model.save_pretrained(out_dir)
    tokenizer.save_pretrained(out_dir)
    return out_dir

@app.local_entrypoint()
def main():
    path = finetune.remote()
    print(f"Adapter saved to: {path}")

Run it

  1. Save as finetune_unsloth.py. Run blade run finetune_unsloth.py.
  2. build-essential in the image is the key fix: Unsloth JIT-compiles its Triton kernels at the first training step and needs a C/C++ toolchain (the bare debian_slim and fabric/torch images ship none).
  3. The pre-quantized 4-bit base model (unsloth/Llama-3.2-1B-Instruct-bnb-4bit) downloads fast and fits the A4500 with lots of headroom; only ~0.3% of parameters (the LoRA) are trained.
  4. The LoRA adapter (~13 MB) is saved once to the Volume when training finishes. Mid-training checkpoint saves currently trip an unsloth/trl pickling issue, hence save_strategy="no" and dataset_num_proc=1.
  5. Scale up: raise max_steps, swap the base model, or point at your own dataset.

Concepts used: GPU, Image build

Contact the team

Loading the form…