-
Notifications
You must be signed in to change notification settings - Fork 987
Home
Stella Biderman edited this page Jan 30, 2021
·
4 revisions
Welcome to the gpt-neox wiki!
The purpose of this wiki is to organize information about all the different terminology and ideas floating around in the DeepSpeed papers, how they connect to each other, what benefits they provide, and why we care about them.
- ZeRO
- ZeRO Stage 1 vs 2 vs 3
- Pipeline Parallelism
- Kernel Optimization
- Model Checkpointing
- Activation Checkpointing
- Adam
- 1-Bit Adam
- TCP
- Infiniband
- PCIE
- NVLINK
- MPI
- NCCL