GPT-2 (XL) has 1.5 billion parameters, and its parameters consume ~3GB of memory in 16-bit precision.However, one can hardly train it on a single GPU with 30GB of memory.That’s 10x the model’s memory, and you might wonder how that could be even possible.While the focus of this article is not LLM memory consumption (you can check this if you want to learn more about it though), the example was used to help you reflect on the unfathomable scale and memory requirements of LLMs.In fact, in the above...