Contents Memory Summary of GPU specs GPUs vs. TPUs at the chip level Quiz 1: GPU hardware At the node level Quiz 2: GPU nodes Beyond the node level Quiz 3: Beyond the node level Intra-node collectives Cross-node collectives Quiz 4: Collectives Data Parallelism Tensor Parallelism Expert Parallelism Pipeline Parallelism Examples TLDR of LLM scaling on GPUs Quiz 5: LLM rooflines Appendix A: How does this change with GB200? Appendix B: More ...