News Score: Score the News, Sort the News, Rewrite the Headlines

Who uses Google TPUs for inference in production?

I am really puzzled by TPUs. I've been reading everywhere that TPUs are powerful and a great alternative to NVIDIA.I have been playing with TPUs for a couple of months now, and to be honest I don't understand how can people use them in production for inference:- almost no resources online showing how to run modern generative models like Mistral, Yi 34B, etc. on TPUs - poor compatibility between JAX and Pytorch - very hard to understand the memory consumption of the TPU chips (no nvidia-smi equiv...

Read more at news.ycombinator.com

© News Score  score the news, sort the news, rewrite the headlines