New C-based engine runs 2.78-trillion-parameter Kimi K3 AI model on 64GB consumer laptop by streaming experts from NVMe disk at 0.5 tokens/second, eliminating cloud API dependency.

GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

WASTE — Weight-Aware Streaming Tensor Engine Kimi K3 — 2.78 trillion parameters — running on a consumer laptop. $ waste run ~/models/k3.waste 'What is the capital of Italy?' waste: no --budget, using 46.24 GB of 64.00 GB (expert cache 17.56 GB) The capital of Italy is **Rome**. [16 tokens, 31.09 s, 0.51 tok/s | experts 3357 hit / 20195 miss = 14%] WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected e...