---
name: "trl-reinforcement-learning-ppo-dpo"
description: "Align language models with human preferences using HuggingFace TRL DPO trainer."
metadata:
  author: "HuggingFace"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "train dpo with trl"
  - "align model with dpotrainer"
  - "rlhf preference training"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# TRL Direct Preference Optimization (DPO) & PPO

Aligns models with human feedback using HuggingFace TRL: DPOTrainer, PPOTrainer, SFTTrainer, AutoModelForCausalLMWithValueHead, and reference model log-probability scoring.

## When to Invoke
Invoke this skill whenever you need to:
- train dpo with trl
- align model with dpotrainer
- rlhf preference training

## Execution SOP & Rules
1. Prepare preference datasets containing prompt, chosen, and rejected string columns.
2. Initialize trainer with DPOTrainer(model=model, ref_model=ref_model, beta=0.1, ...).
3. Monitor train/rewards/accuracies and train/loss/dpo metrics to detect policy collapse.
4. Save aligned model weights and verify reductions in harmful or hallucinated outputs.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.