The Complete Production SOP: How to Deploy and Run bitsandbytes 8-bit & 4-bit Matrix Kernels in Autonomous AI Agents

---
name: "bitsandbytes-8bit-4bit-quant"
description: "Load and train large language models in 4-bit and 8-bit precision with bitsandbytes."
metadata:
  author: "Tim Dettmers"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "load model in 4bit bitsandbytes"
  - "use nf4 quantization bnb"
  - "setup 8bit adamw optimizer"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# bitsandbytes 8-bit & 4-bit Matrix Kernels

Integrates bitsandbytes: BitsAndBytesConfig, bnb_4bit_compute_dtype, NF4 (NormalFloat4) quantization, double quantization, and 8-bit AdamW optimizer memory savings.

## Execution SOP & Rules
1. Configure BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16).
2. Pass quantization config into AutoModelForCausalLM.from_pretrained(..., quantization_config=config, device_map='auto').
3. Replace standard AdamW with bnb.optim.AdamW8bit in PyTorch training scripts.
4. Verify CUDA driver and PyTorch compatibility before deploying 4-bit matrix multiplication kernels.

import torch
import bitsandbytes as bnb

def test_bnb_installation():
    assert torch.cuda.is_available(), "CUDA is required for 4-bit/8-bit bitsandbytes kernels."
    major, minor = torch.cuda.get_device_capability()
    print(f"[INFO] Device Compute Capability: {major}.{minor}")
    
    # Test 8-bit linear matrix allocation
    linear = bnb.nn.Linear8bitLt(32, 64, has_fp16_weights=False).cuda()
    x = torch.randn(2, 32).cuda()
    out = linear(x)
    assert out.shape == (2, 64), "Linear8bitLt kernel computation failed."
    print("[SUCCESS] bitsandbytes 8-bit & 4-bit CUDA kernels loaded correctly.")

if __name__ == '__main__':
    test_bnb_installation()

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

def load_quantized_model(model_id: str):
    compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
    
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=compute_dtype
    )
    
    tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        quantization_config=bnb_config,
        device_map="auto",
        torch_dtype=compute_dtype,
        low_cpu_mem_usage=True
    )
    return model, tokenizer

import torch
import bitsandbytes as bnb
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

def test_4bit_model_and_8bit_optimizer():
    model_id = "facebook/opt-125m"
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16
    )
    model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map={ '': 0 })
    model = prepare_model_for_kbit_training(model)
    
    peft_config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
    model = get_peft_model(model, peft_config)
    
    optimizer = bnb.optim.AdamW8bit(model.parameters(), lr=2e-4)
    
    dummy_input = torch.tensor([[1, 2, 3, 4]], device="cuda:0")
    loss = model(dummy_input, labels=dummy_input).loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    
    assert not torch.isnan(loss), "Loss exploded to NaN during 4-bit backward pass!"