AI Notes

1. Quantization

What is quantization? Simply put, it’s the process of reducing the precision of model weights. Instead of storing each weight as a 16-bit floating point number, we can store it as a 3-bit, 4-bit, 5-bit, or 8-bit integer. This makes the model smaller and faster to run. But with fewer bits comes less accuracy.

1.1. Quality

Table 1. Quality
Format Bits/Param Quality vs FP16 Speed Usage

FP16 (16-bit)

16

100%

Fastest (GPU)

This provides baseline quality against which everything else is measured.

Q8 (8-bit)

8

98-99%

Fast

In blind testing, most users cannot distinguish Q8 from FP16 for standard tasks. The 50% memory savings come with essentially no perceptible quality loss.

Q5 (5-bit)

5

95-97%

Fast

Balanced high‑accuracy

Q4 (4-bit)

4

90-95%

Very Fast

Quality loss is noticeable in direct comparison. Reserve Q4 for memory-constrained scenarios or when running larger models (30B at Q4) that wouldn’t fit at higher precision.

Q3 (3-bit)

3

80-90%

Very Fast

Unless experimenting or working with extremely constrained environments. Quality degradation at 3-bit becomes significant enough to impact practical utility for many tasks.

1.2. Ram Requirements

Table 2. Required GPU VRAM
Model Size FP16 (16‑bit) Q8 (8‑bit) Q5 (5‑bit) Q4 (4‑bit) Q3 (3‑bit)

7B

~14 GB

~7 GB

~4.5 GB

~3.5 GB

~2.6 GB

13B

~26 GB

~13 GB

~8.5 GB

~6.5 GB

~4.8 GB

33B

~66 GB

~33 GB

~22 GB

~16 GB

~12 GB

70B

~140 GB

~70 GB

~45 GB

~34 GB

~25 GB

110B

~220 GB

~110 GB

~72 GB

~54 GB

~40 GB

236B

~470 GB

~235 GB

~155 GB

~115 GB

~85 GB

405B

~810 GB

~405 GB

~265 GB

~200 GB

~150 GB

671B

~1.34 TB

~670 GB

~440 GB

~330 GB

~250 GB
Table 3. GPU with 24GB VRAM (Nvidia RTX 3090/4090)
Model Size FP16 Q8 Q5 Q4 Q3

7B

✅ (~14 GB)

✅ (~7 GB)

✅ (~4.5 GB)

✅ (~3.5 GB)

✅ (~2.6 GB)

13B

❌ (~26 GB)

✅ (~13 GB)

✅ (~8.5 GB)

✅ (~6.5 GB)

✅ (~4.8 GB)

33B

❌ (~66 GB)

❌ (~33 GB)

✅ (~22 GB Tight)

✅ (~16 GB)

✅ (~12 GB)

70B

❌ (~140 GB)

❌ (~70 GB)

❌ (~45 GB)

❌ (~34 GB)

❌ (~25 GB)
Table 4. GPU with 32GB VRAM (Nvidia RTX 5090)
Model Size FP16 Q8 Q5 Q4 Q3

7B

✅ (~14 GB)

✅ (~7 GB)

✅ (~4.5 GB)

✅ (~3.5 GB)

✅ (~2.6 GB)

13B

✅ (~26 GB)

✅ (~13 GB)

✅ (~8.5 GB)

✅ (~6.5 GB)

✅ (~4.8 GB)

33B

❌ (~66 GB)

❌ (~33 GB, borderline)

✅ (~22 GB)

✅ (~16 GB)

✅ (~12 GB)

70B

❌ (~140 GB)

❌ (~70 GB)

❌ (~45 GB)

❌ (~34 GB, borderline)

✅ (~25 GB)

110B

❌ (~220 GB)

❌ (~110 GB)

❌ (~72 GB)

❌ (~54 GB)

❌ (~40 GB)
Table 5. GPU with 96GB VRAM (Nvidia RTX 6000 Blackwell Edition)
Model Size FP16 Q8 Q5 Q4 Q3

7B

✅

✅

✅

✅

✅

13B

✅

✅

✅

✅

✅

33B

✅ (~66 GB)

✅ (~33 GB)

✅ (~22 GB)

✅ (~16 GB)

✅ (~12 GB)

70B

❌ (~140 GB)

✅ (~70 GB)

✅ (~45 GB)

✅ (~34 GB)

✅ (~25 GB)

110B

❌ (~220 GB)

❌ (~110 GB)

✅ (~72 GB)

✅ (~54 GB)

✅ (~40 GB)

236B

❌ (~470 GB)

❌ (~235 GB)

❌ (~155 GB)

❌ (~115 GB)

✅ (~85 GB)

405B

❌

❌

❌

❌

❌ (~150 GB)