Developer runs 3.16M-parameter language model entirely on-chip in $250 AMD FPGA in Wales, achieving 59,965 tokens per second by storing INT4 weights in UltraRAM—5,400x faster than board's ARM cores, 83x faster than RTX 3050 Ti laptop GPU.

Taalas-Style On-Chip Weights on a $250 FPGA: a Language Model at 60k tok/s

I was so impressed by the chatjimmy.ai demo by Taalas, I wanted to see what I could squeeze inside the The FPGA's reconfigurable logic. Same thing as 'PL'. Where the whole model runs here. of a $250 Field-Programmable Gate Array: a chip full of reconfigurable logic you wire into a custom digital circuit, instead of running software on a fixed CPU.. By not using the (4 GB available) The off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric. This shared controlle...