K-Quants Guide¶

Understanding K-quantization formats.

What are K-Quants?¶

K-quants use "double quantization": 1. Quantize weights 2. Quantize the quantization parameters

Result: Better quality-to-size ratio

Available K-Quants¶

Q4_K_M (Recommended)¶

• Bits per weight: 4.8
• Size: ~40% of FP16
• Quality: Excellent for most uses
• Use case: Default choice

Q5_K_M¶

• Bits per weight: 5.7
• Size: ~48% of FP16
• Quality: Higher than Q4_K_M
• Use case: Quality-sensitive

Q6_K¶

• Bits per weight: 6.6
• Size: ~55% of FP16
• Quality: Near FP16
• Use case: Maximum quality

Q8_0¶

• Bits per weight: 8.5
• Size: ~70% of FP16
• Quality: Virtually lossless
• Use case: Research, validation

Size Comparison¶