---
name: "llama-cpp-gguf-quantization"
description: "Quantize and run LLMs with 4-bit GGUF formats at peak speed on consumer hardware."
metadata:
  author: "Georgi Gerganov"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "quantize model to gguf"
  - "run llama.cpp server"
  - "use 4-bit k-quants llama.cpp"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# llama.cpp GGUF Quantization & Fast CPU Inference

Quantizes and runs language models with llama.cpp: converting HuggingFace weights to GGUF, applying k-quantization (Q4_K_M, Q5_K_M, Q8_0), Metal/CUDA acceleration, and llama-server.

## When to Invoke
Invoke this skill whenever you need to:
- quantize model to gguf
- run llama.cpp server
- use 4-bit k-quants llama.cpp

## Execution SOP & Rules
1. Convert weights using python convert_hf_to_gguf.py <model_dir>.
2. Quantize using ./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M.
3. Launch HTTP server with ./llama-server -m model-Q4_K_M.gguf -c 4096 -ngl 99.
4. Offload layers to GPU memory using -ngl flag to maximize tokens per second.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.