The Complete Production SOP: How to Deploy and Run Megatron-LM 3D Parallelism Tensor & Pipeline in Autonomous AI Agents

---
name: "megatron-lm-tensor-parallelism"
description: "Scale training of 100B+ frontier models with Megatron-LM tensor and pipeline parallelism."
metadata:
  author: "NVIDIA"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "configure megatron tensor parallelism"
  - "setup pipeline parallelism"
  - "train 100b model megatron"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Megatron-LM 3D Parallelism Tensor & Pipeline

Configures NVIDIA Megatron-LM: Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), sequence parallelism, and custom Transformer engine kernels on DGX SuperPODs.

## When to Invoke
Invoke this skill whenever you need to:
- configure megatron tensor parallelism
- setup pipeline parallelism
- train 100b model megatron

## Execution SOP & Rules
1. Set --tensor-model-parallel-size to match the number of GPUs within a single NVLink node (typically 8).
2. Set --pipeline-model-parallel-size such that Total_Layers % PP_Size == 0.
3. Always append --sequence-parallel when TP > 1 to distribute LayerNorm and Dropout activations.
4. Ensure (TP * PP * DP) == Total World Size (NNODES * GPUS_PER_NODE).
5. Configure --num-layers-per-virtual-pipeline-stage when interleaving 1F1B schedules.

#!/usr/bin/env bash
source .env.distributed
echo "Configuring 3D Parallelism for Cluster:"
echo "Total GPUs: $((NNODES * GPUS_PER_NODE))"
if [ $((WORLD_SIZE % (GPUS_PER_NODE))) -ne 0 ]; then
  echo "ERROR: Cluster world size must align with physical nodes."
  exit 1
fi

import sys

def verify_megatron_topology(tp, pp, dp, world_size, layers, hidden_size, num_heads):
    assert tp * pp * dp == world_size, f"Topology error: TP({tp}) * PP({pp}) * DP({dp}) != WORLD({world_size})"
    assert layers % pp == 0, f"Layers ({layers}) must be divisible by Pipeline Parallel size ({pp})"
    assert hidden_size % tp == 0, f"Hidden size ({hidden_size}) must be divisible by TP ({tp})"
    assert num_heads % tp == 0, f"Attention heads ({num_heads}) must be divisible by TP ({tp})"
    print("Megatron-LM 3D Parallelism configuration is mathematically valid.")

if __name__ == '__main__':
    verify_megatron_topology(tp=8, pp=4, dp=4, world_size=128, layers=80, hidden_size=8192, num_heads=64)