The Complete Production SOP: How to Deploy and Run TorchTune PyTorch-Native Fine-Tuning Recipes in Autonomous AI Agents
Step-by-step SOP to install and execute TorchTune fine-tuning recipes in Codex, Claude Code, and Cursor. Master FSDP2, LoRA, and torch.compile workflows.
Technical Overview & Architectural Context
Autonomous coding agents like Codex, Claude Code, Cursor, and Windsurf frequently fail when tasked with fine-tuning large language models. Typical failures include inventing non-existent PyTorch CLI flags, mixing incompatible Hugging Face Transformers abstractions with native torch recipes, misconfiguring PyTorch 2 Fully Sharded Data Parallel (FSDP2) sharding strategies, or writing custom training loops that crash CUDA out-of-memory (OOM). When unconstrained, agents often generate bloated training scripts that bypass kernel optimization, break torch.compile graph capture, and corrupt model weight checkpoints. The TorchTune PyTorch-Native Fine-Tuning Recipes skill eliminates this instability by binding the autonomous agent to native, modular TorchTune execution patterns. By declaring explicit recipe triggers, parameter override conventions (tune run <recipe> --config <config>), and workspace validation boundaries, the agent deterministically configures YAML recipe parameters, mounts PyTorch 2.0 compile flags, manages multi-GPU FSDP2 sharding, and exports clean native PyTorch or HuggingFace format checkpoints without human intervention.
Prerequisites & Environment Requirements
Verified Search Intents & Long-Tail Queries Solved
This operational tutorial is engineered to provide deterministic solutions for these developer search queries:
Step-by-Step Execution Playbook
Local Environment Setup & SKILL.md Mounting
Create the local skill manifest directory and inject verified procedural rules into the agent runtime path.
Agent runtimes such as Codex and Claude Code scan project root `.agents/skills/` or user home `~/.claude/skills/` directories for SKILL.md declarations. Installing this manifest binds the agent's intent parser to TorchTune commands, ensuring it executes native `tune run` workflows instead of drafting ad-hoc PyTorch training loops.
mkdir -p .agents/skills/torchtune-pytorch-native-finetuning && curl -fsSL https://askills.dev/api/skills/torchtune-pytorch-native-finetuning/raw -o .agents/skills/torchtune-pytorch-native-finetuning/SKILL.md---
name: "torchtune-pytorch-native-finetuning"
description: "Fine-tune LLMs natively within the PyTorch ecosystem using modular TorchTune recipes."
metadata:
author: "PyTorch"
version: "1.0.0"
runtime_compatibility:
- "codex"
- "claude-code"
- "cursor"
- "windsurf"
triggers:
- "fine tune with torchtune"
- "run torchtune recipe"
- "use fsdp2 torchtune"
permissions:
fs_read: true
fs_write: "workspace"
shell_exec: false
net_access: false
---
# TorchTune PyTorch-Native Fine-Tuning Recipes
Utilizes PyTorch TorchTune: modular training components, memory-efficient recipes (LoRA, QLoRA, full finetune), FSDP2 multi-GPU scaling, and seamless integration with PyTorch 2.0 compile.
## Execution SOP & Rules
1. Select pre-built recipe: `tune run <recipe_name> --config <config_name>`.
2. Override parameters directly from command-line: `checkpointer.checkpoint_dir=<path> batch_size=4`.
3. Leverage `compile: True` for graph capture and kernel optimization during training.
4. Export fine-tuned checkpoints in native PyTorch or HuggingFace formats.
## Verification Checklist
- Run `tune validate <config_file>` prior to kicking off distributed runs.
- Ensure checkpoint destinations reside strictly within workspace boundaries.
- Verify memory requirements using peak memory tracking before long executions.Instruct the agent to inspect recipes using `tune ls` rather than generating raw training script files, saving hundreds of output tokens and preventing hallucinated training loops.
Ensure `fs_write` is scoped strictly to workspace directories; writing checkpoint shards directly to `/root` or `/tmp` will trigger runtime security sandboxing faults.
Environment Secrets & Dependency Validation
Inject required Hugging Face authentication tokens and verify the TorchTune CLI installation in the local shell.
TorchTune recipes commonly pull base weights (e.g., Llama 3, Mistral) from Hugging Face Hub. The agent must verify that dependencies are installed and that `HF_TOKEN` is present in the environment without printing raw token strings into standard output or chat logs.
python -c "import torchtune, torch; print('TorchTune:', torchtune.__version__, 'PyTorch:', torch.__version__, 'CUDA Available:', torch.cuda.is_available())" && tune --help > /dev/nullInject tokens via workspace `.env.local` or environment exports; never prompt the agent to write raw secret values directly inside recipe config YAMLs.
Verify that `.env` and `.env.local` are tracked in `.gitignore` to prevent agent auto-commit routines from checking in production API keys.
Intent Matching & Trigger Invocation
Trigger deterministic agent execution using frontmatter triggers to configure and launch a LoRA or FSDP2 fine-tuning task.
Provide structured prompts matching trigger phrases like 'run torchtune recipe' or 'use fsdp2 torchtune'. The runtime hooks the frontmatter trigger rules, leading the agent to first copy or inspect the stock recipe, override parameters using dot-notation, and produce the validated run configuration.
Specify precise hardware constraints in the prompt (e.g., 'single 24GB VRAM GPU' or 'dual 80GB A100 with FSDP2') to allow the agent to auto-select between LoRA and full FSDP2 recipes.
Avoid vague phrases like 'train my model'; use the exact trigger 'fine tune with torchtune' to ensure SKILL.md SOP takes precedence over generic Hugging Face training scripts.
Verification, Linting & Output Quality Gate
Execute automated recipe configuration validation and verify generated artifacts adhere to quality standards.
The agent must run `tune validate` against custom YAML configs or execute a 1-step dry run before launching distributed training jobs. It then verifies model checkpoint artifacts and directory layouts.
tune validate lora_finetune_single_device --config custom_config.yamlSet `max_steps_per_epoch: 2` in the initial verification pass to confirm graph compilation and CUDA memory allocation complete without throwing an unrecoverable OOM error.
torch.compile requires compatible Triton and CUDA toolchains; if running in constrained environments, verify the compile flag doesn't fail silently during the initial warmup iteration.
Real-World Scenario: Configure and Run Llama-3-8B LoRA Fine-Tuning with FSDP2 on 2x GPUs
“Please use fsdp2 torchtune to fine tune with torchtune on meta-llama/Meta-Llama-3-8B-Instruct using our local workspace dataset at ./data/instructions.json. Keep batch size at 2, enable torch.compile, and save adapter checkpoints to ./checkpoints/llama3-lora.”
Runtime Troubleshooting & Failure Modes
Root Cause: Activation checkpointing is disabled or batch size is excessive for the chosen precision and context length under full finetuning.
tune run lora_finetune_single_device --config custom_config.yaml batch_size=1 enable_activation_checkpointing=True dtype=bf16Root Cause: The agent attempted to override a top-level or nested config key using an incorrect configuration key path or deprecated TorchTune config schema.
tune cp llama3/8B_lora temp_check.yaml && grep -n 'output_dir\|checkpointer' temp_check.yaml && rm temp_check.yamlFrequently Asked Questions (TorchTune PyTorch-Native Fine-Tuning Recipes SOP)
Q:Can I use TorchTune PyTorch-Native Fine-Tuning Recipes across Codex, Claude Code, and Cursor simultaneously?
Yes. The skill conforms to the universal SKILL.md specification. Runtimes supporting modular agent rules load the trigger criteria and procedural SOP automatically when placed in `.agents/skills/torchtune-pytorch-native-finetuning/` or user skill paths.
Q:How does TorchTune PyTorch-Native Fine-Tuning Recipes prevent context window overflow during long agent sessions?
By directing the agent to rely on TorchTune's existing declarative CLI commands (`tune run`, `tune cp`, `tune validate`) rather than generating verbose, multi-file Python boilerplate training loops into the conversation context window.
Q:Does this skill require escalated root or container permissions?
No. The skill is audited with a security score of 98/100. It requires standard workspace file read/write permissions for configurations and model checkpoint outputs, requiring no root access or unrestricted host network sockets.
Ready to equip your autonomous agent with TorchTune PyTorch-Native Fine-Tuning Recipes?
Copy the complete audited SKILL.md specification or download the 1-click zip bundle directly from our verified registry.