---
name: "bitsandbytes-8bit-4bit-quant"
description: "Load and train large language models in 4-bit and 8-bit precision with bitsandbytes."
metadata:
  author: "Tim Dettmers"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "load model in 4bit bitsandbytes"
  - "use nf4 quantization bnb"
  - "setup 8bit adamw optimizer"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# bitsandbytes 8-bit & 4-bit Matrix Kernels

Integrates bitsandbytes: BitsAndBytesConfig, bnb_4bit_compute_dtype, NF4 (NormalFloat4) quantization, double quantization, and 8-bit AdamW optimizer memory savings.

## When to Invoke
Invoke this skill whenever you need to:
- load model in 4bit bitsandbytes
- use nf4 quantization bnb
- setup 8bit adamw optimizer

## Execution SOP & Rules
1. Configure BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16).
2. Pass quantization config into AutoModelForCausalLM.from_pretrained(..., quantization_config=config).
3. Replace standard AdamW with bnb.optim.AdamW8bit in PyTorch training scripts.
4. Verify CUDA driver and PyTorch compatibility before deploying 4-bit matrix multiplication kernels.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.