"Mamba-based Language Models Match or Outperform Transformers in Large-scale Training Studies: Hybrid Mamba-2 Model Tops Performance on 12 Standard Tasks and is 8x Faster, Unveils NVIDIA's Megatron-LM Project"

An Empirical Study of Mamba-based Language Models

Authors:Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, Bryan Catanzaro View PDF HTML (experimental) Abstract:Selective state-space models (SSMs) like Mamba overcome some of the shortcomings of Transformers, such as quadratic computational complexity with sequence length and large inference-time memory requirements fro...