[v0.12.2] Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5 8GB ยท b4rtaz/distributed-llama ยท Discussion #162
Model: deepseek_r1_distill_llama_8b_q40
Version: 0.12.2
Evaluation
Prediction
2 x Raspberry Pi 5 8GB
7.70 tok/s
3.54 tok/s
4 x Raspberry Pi 5 8GB
11.68 tok/s
6.43 tok/s
2 x Raspberry Pi 5 8GB
...
๐ถ P 278 ms S 288 kB R 522 kB First
๐ถ P 258 ms S 288 kB R 522 kB ,
๐ถ P 323 ms S 288 kB R 522 kB I
๐ถ P 275 ms S 288 kB R 522 kB need
๐ถ P 293 ms S 288 kB R 522 kB to
๐ถ P 269 ms S 288 kB R 522 kB understand
๐ถ P 281 ms S 288 kB R 522 kB what
Evaluation
nBatches:...
Read more at github.com