Developer docs / AI Fabric tutorials

GRPO + TRL code generation RL

  • GPU
  • Volume
  • Image DSL

Supervised fine-tuning teaches a model to imitate examples; reinforcement learning teaches it to maximize a reward you define. For code generation that works well, because "is this code good?" is something you can check (does it parse, does it run, does it pass a test?) rather than something you have to hand-label. GRPO (Group Relative Policy Optimization) turns that check into a training signal.

In this tutorial you'll fine-tune Qwen 2.5 (1.5B) for code generation using TRL's GRPOTrainer, with a reward function that scores whether the generated code is syntactically valid. Checkpoints are written to a Volume so you can resume and so you can load the best one for evaluation afterwards.

Before you start

Install and authenticate the AI Fabric CLI with blade setup. The run uses a single RTX A4500 and builds on fabric/torch, the catalog image with CUDA and PyTorch already baked in, so you only layer the RL libraries on top.

Create the app and image

Create grpo_trl_code.py. Declare the app, a Volume for checkpoints, and an image that adds TRL (the library that provides GRPOTrainer) plus the usual training stack on top of fabric/torch:

grpo_trl_code.py
import bladekit

app = bladekit.app("grpo-trl-code")

checkpoints = bladekit.Volume.from_name("grpo-trl-checkpoints", create_if_missing=True)

image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("torch", "trl>=0.9.0", "transformers", "datasets", "accelerate", "peft")
)

Define the reward function

The reward function is the objective. GRPO samples several completions per prompt and uses their relative rewards to nudge the model toward better ones. Here the signal is deliberately simple: parse each completion with Python's ast module and give 1.0 if it's syntactically valid, 0.0 if not. It receives the batch of completions and returns one float per completion.

Python
def code_reward(completions: list[str], **kwargs) -> list[float]:
    """Reward: 1.0 if completion compiles and passes basic test, else 0.0."""
    import ast
    rewards = []
    for code in completions:
        try:
            ast.parse(code)
            rewards.append(1.0)
        except SyntaxError:
            rewards.append(0.0)
    return rewards

Declare the training function

Wrap the training in an @app.function that requests the GPU, mounts the checkpoint Volume, and allows one retry. Inside, load the tokenizer and model and pull a slice of HumanEval (a standard code-generation benchmark), mapping each example into a prompt and a reference:

Python
@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=14400,
    volumes={"/checkpoints": checkpoints},
    retries=1,
)
def train_grpo_code(
    model_name: str = "Qwen/Qwen2.5-1.5B-Instruct",
) -> str:
    from trl import GRPOTrainer, GRPOConfig
    from transformers import AutoModelForCausalLM, AutoTokenizer
    from datasets import load_dataset

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name)

    dataset = load_dataset("openai/openai_humaneval", split="test[:100]")
    dataset = dataset.map(lambda x: {"prompt": x["prompt"], "reference": x["canonical_solution"]})

Wire up GRPOTrainer and train

GRPOConfig carries the hyperparameters. The one to watch is num_generations: GRPO samples N completions per prompt, and your per_device_train_batch_size must be divisible by it. Pass your code_reward to reward_funcs so the trainer scores each generation, then call train():

Python
    config = GRPOConfig(
        output_dir="/checkpoints/grpo-code-run",
        num_train_epochs=2,
        per_device_train_batch_size=4,
        num_generations=2,   # GRPO samples N completions/prompt; batch must divide it
        save_steps=50,
    )
    trainer = GRPOTrainer(
        model=model,
        processing_class=tokenizer,   # TRL renamed tokenizer -> processing_class
        reward_funcs=code_reward,
        args=config,
        train_dataset=dataset,
    )
    trainer.train()
    return "/checkpoints/grpo-code-run"

Launch it

The local_entrypoint runs on your machine and ships the work to the GPU with .remote():

Python
@app.local_entrypoint()
def main():
    out = train_grpo_code.remote()
    print(f"Training complete: {out}")

Run it with the AI Fabric CLI:

Terminal
blade run grpo_trl_code.py

GRPO trains against your code_reward, checkpointing to the Volume every 50 steps. When it finishes, load the best checkpoint from the grpo-trl-checkpoints Volume for final evaluation. When you're ready for a sharper objective, replace the syntax check with execution-based testing that actually runs each completion against unit tests.

Take it for a spin

Everything above in one file, ready to run.

grpo_trl_code.py
import bladekit

app = bladekit.app("grpo-trl-code")

checkpoints = bladekit.Volume.from_name("grpo-trl-checkpoints", create_if_missing=True)

image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("torch", "trl>=0.9.0", "transformers", "datasets", "accelerate", "peft")
)

def code_reward(completions: list[str], **kwargs) -> list[float]:
    """Reward: 1.0 if completion compiles and passes basic test, else 0.0."""
    import ast
    rewards = []
    for code in completions:
        try:
            ast.parse(code)
            rewards.append(1.0)
        except SyntaxError:
            rewards.append(0.0)
    return rewards

@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=14400,
    volumes={"/checkpoints": checkpoints},
    retries=1,
)
def train_grpo_code(
    model_name: str = "Qwen/Qwen2.5-1.5B-Instruct",
) -> str:
    from trl import GRPOTrainer, GRPOConfig
    from transformers import AutoModelForCausalLM, AutoTokenizer
    from datasets import load_dataset

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name)

    dataset = load_dataset("openai/openai_humaneval", split="test[:100]")
    dataset = dataset.map(lambda x: {"prompt": x["prompt"], "reference": x["canonical_solution"]})

    config = GRPOConfig(
        output_dir="/checkpoints/grpo-code-run",
        num_train_epochs=2,
        per_device_train_batch_size=4,
        num_generations=2,   # GRPO samples N completions/prompt; batch must divide it
        save_steps=50,
    )
    trainer = GRPOTrainer(
        model=model,
        processing_class=tokenizer,   # TRL renamed tokenizer -> processing_class
        reward_funcs=code_reward,
        args=config,
        train_dataset=dataset,
    )
    trainer.train()
    return "/checkpoints/grpo-code-run"

@app.local_entrypoint()
def main():
    out = train_grpo_code.remote()
    print(f"Training complete: {out}")

Concepts used: GPU, Image build

Contact the team

Loading the form…