---
name: "deepspeed-zero-distributed-training"
description: "Scale large model training across GPU clusters with DeepSpeed ZeRO-3 and CPU offloading."
metadata:
  author: "Microsoft"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "configure deepspeed zero-3"
  - "offload optimizer to cpu deepspeed"
  - "train large model deepspeed"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# DeepSpeed ZeRO-3 Memory Optimization

Configures distributed training with Microsoft DeepSpeed: ZeRO-1/2/3 stage partitioning, CPU/NVMe parameter offloading, activation checkpointing, and FP16/BF16 mixed precision.

## When to Invoke
Invoke this skill whenever you need to:
- configure deepspeed zero-3
- offload optimizer to cpu deepspeed
- train large model deepspeed

## Execution SOP & Rules
1. Define DeepSpeed configuration in ds_config.json with zero_optimization: { stage: 3 }.
2. Configure cpu_offload: { device: 'cpu', pin_memory: true } when GPU VRAM is constrained.
3. Launch training jobs using deepspeed --num_gpus=8 train.py --deepspeed ds_config.json.
4. Enable bfloat16: { enabled: true } on Ampere and Hopper GPUs for optimal numerical stability.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.