Developer docs / AI Fabric tutorials
GRPO + TRL code generation RL
- GPU
- Volume
- Image DSL
Supervised fine-tuning teaches a model to imitate examples; reinforcement learning teaches it to maximize a reward you define. For code generation that works well, because "is this code good?" is something you can check (does it parse, does it run, does it pass a test?) rather than something you have to hand-label. GRPO (Group Relative Policy Optimization) turns that check into a training signal.
In this tutorial you'll fine-tune Qwen 2.5 (1.5B) for code generation using TRL's GRPOTrainer, with a reward function that scores whether the generated code is syntactically valid. Checkpoints are written to a Volume so you can resume and so you can load the best one for evaluation afterwards.
Before you start
Install and authenticate the AI Fabric CLI with blade setup. The run uses a single RTX A4500 and builds on fabric/torch, the catalog image with CUDA and PyTorch already baked in, so you only layer the RL libraries on top.
Create the app and image
Create grpo_trl_code.py. Declare the app, a Volume for checkpoints, and an image that adds TRL (the library that provides GRPOTrainer) plus the usual training stack on top of fabric/torch:
import bladekit
app = bladekit.app("grpo-trl-code")
checkpoints = bladekit.Volume.from_name("grpo-trl-checkpoints", create_if_missing=True)
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("torch", "trl>=0.9.0", "transformers", "datasets", "accelerate", "peft")
)Define the reward function
The reward function is the objective. GRPO samples several completions per prompt and uses their relative rewards to nudge the model toward better ones. Here the signal is deliberately simple: parse each completion with Python's ast module and give 1.0 if it's syntactically valid, 0.0 if not. It receives the batch of completions and returns one float per completion.
def code_reward(completions: list[str], **kwargs) -> list[float]:
"""Reward: 1.0 if completion compiles and passes basic test, else 0.0."""
import ast
rewards = []
for code in completions:
try:
ast.parse(code)
rewards.append(1.0)
except SyntaxError:
rewards.append(0.0)
return rewardsDeclare the training function
Wrap the training in an @app.function that requests the GPU, mounts the checkpoint Volume, and allows one retry. Inside, load the tokenizer and model and pull a slice of HumanEval (a standard code-generation benchmark), mapping each example into a prompt and a reference:
@app.function(
gpu="RTX-A4500",
image=image,
timeout=14400,
volumes={"/checkpoints": checkpoints},
retries=1,
)
def train_grpo_code(
model_name: str = "Qwen/Qwen2.5-1.5B-Instruct",
) -> str:
from trl import GRPOTrainer, GRPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
dataset = load_dataset("openai/openai_humaneval", split="test[:100]")
dataset = dataset.map(lambda x: {"prompt": x["prompt"], "reference": x["canonical_solution"]}) Wire up GRPOTrainer and train
GRPOConfig carries the hyperparameters. The one to watch is num_generations: GRPO samples N completions per prompt, and your per_device_train_batch_size must be divisible by it. Pass your code_reward to reward_funcs so the trainer scores each generation, then call train():
config = GRPOConfig(
output_dir="/checkpoints/grpo-code-run",
num_train_epochs=2,
per_device_train_batch_size=4,
num_generations=2, # GRPO samples N completions/prompt; batch must divide it
save_steps=50,
)
trainer = GRPOTrainer(
model=model,
processing_class=tokenizer, # TRL renamed tokenizer -> processing_class
reward_funcs=code_reward,
args=config,
train_dataset=dataset,
)
trainer.train()
return "/checkpoints/grpo-code-run"Launch it
The local_entrypoint runs on your machine and ships the work to the GPU with .remote():
@app.local_entrypoint()
def main():
out = train_grpo_code.remote()
print(f"Training complete: {out}")Run it with the AI Fabric CLI:
blade run grpo_trl_code.pyGRPO trains against your code_reward, checkpointing to the Volume every 50 steps. When it finishes, load the best checkpoint from the grpo-trl-checkpoints Volume for final evaluation. When you're ready for a sharper objective, replace the syntax check with execution-based testing that actually runs each completion against unit tests.
Take it for a spin
Everything above in one file, ready to run.
import bladekit
app = bladekit.app("grpo-trl-code")
checkpoints = bladekit.Volume.from_name("grpo-trl-checkpoints", create_if_missing=True)
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("torch", "trl>=0.9.0", "transformers", "datasets", "accelerate", "peft")
)
def code_reward(completions: list[str], **kwargs) -> list[float]:
"""Reward: 1.0 if completion compiles and passes basic test, else 0.0."""
import ast
rewards = []
for code in completions:
try:
ast.parse(code)
rewards.append(1.0)
except SyntaxError:
rewards.append(0.0)
return rewards
@app.function(
gpu="RTX-A4500",
image=image,
timeout=14400,
volumes={"/checkpoints": checkpoints},
retries=1,
)
def train_grpo_code(
model_name: str = "Qwen/Qwen2.5-1.5B-Instruct",
) -> str:
from trl import GRPOTrainer, GRPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
dataset = load_dataset("openai/openai_humaneval", split="test[:100]")
dataset = dataset.map(lambda x: {"prompt": x["prompt"], "reference": x["canonical_solution"]})
config = GRPOConfig(
output_dir="/checkpoints/grpo-code-run",
num_train_epochs=2,
per_device_train_batch_size=4,
num_generations=2, # GRPO samples N completions/prompt; batch must divide it
save_steps=50,
)
trainer = GRPOTrainer(
model=model,
processing_class=tokenizer, # TRL renamed tokenizer -> processing_class
reward_funcs=code_reward,
args=config,
train_dataset=dataset,
)
trainer.train()
return "/checkpoints/grpo-code-run"
@app.local_entrypoint()
def main():
out = train_grpo_code.remote()
print(f"Training complete: {out}")Concepts used: GPU, Image build