---
name: "megatron-lm-tensor-parallelism"
description: "Scale training of 100B+ frontier models with Megatron-LM tensor and pipeline parallelism."
metadata:
  author: "NVIDIA"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "configure megatron tensor parallelism"
  - "setup pipeline parallelism"
  - "train 100b model megatron"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Megatron-LM 3D Parallelism Tensor & Pipeline

Configures NVIDIA Megatron-LM: Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), sequence parallelism, and custom Transformer engine kernels on DGX SuperPODs.

## When to Invoke
Invoke this skill whenever you need to:
- configure megatron tensor parallelism
- setup pipeline parallelism
- train 100b model megatron

## Execution SOP & Rules
1. Set --tensor-model-parallel-size to match the number of GPUs within a single NVLink node.
2. Set --pipeline-model-parallel-size to partition transformer layers across interconnected nodes.
3. Enable --sequence-parallel to distribute activation memory across tensor parallel ranks.
4. Verify linear scaling efficiency and monitor TFLOPS per GPU using Megatron logging timers.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.