---
name: "exllamav2-fast-gptq-inference"
description: "Run local LLM inference at extreme speeds on modern GPUs using ExLlamaV2 and EXL2."
metadata:
  author: "turboderp"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "run exllamav2 inference"
  - "quantize model to exl2"
  - "enable speculative decoding exllamav2"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# ExLlamaV2 Ultra-Fast GPU EXL2 Quantization

Executes inference with ExLlamaV2: variable-bitrate EXL2 quantization, custom 4-bit CUDA kernels, speculative decoding, 4-bit/8-bit cache quantization, and extreme tokens/second throughput.

## When to Invoke
Invoke this skill whenever you need to:
- run exllamav2 inference
- quantize model to exl2
- enable speculative decoding exllamav2

## Execution SOP & Rules
1. Load models with ExLlamaV2(config) and allocate cache with ExLlamaV2Cache(model).
2. Quantize models with python convert.py -i <input_dir> -o <output_dir> -b 4.5.
3. Use ExLlamaV2GeneratorBase for streaming token generation in Python pipelines.
4. Utilize 4-bit or 8-bit KV cache modes to double available context window lengths.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.