PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices.

Kazuki Osawa Shigang Li Torsten Hoefler

Published in: MLSys (2023)

Keyphrases

language model
language modeling
fisher information
probabilistic model
n gram
document retrieval
retrieval model
test collection
information retrieval
mixture model
query expansion
positive definite
machine learning
training data
training set
context sensitive