ZeRO: memory optimizations toward training trillion parameter models.

Samyam Rajbhandari Jeff Rasley Olatunji Ruwase Yuxiong He

Published in: SC (2020)

Keyphrases

statistical models
training set
parameter estimation
experimental data
mathematical models
neural network
data mining
feature selection
supervised learning
online learning
maximum likelihood
complex systems
parameter values
linear model
training phase
structured prediction