LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelism.
Diandian GuPeng SunQinghao HuTing HuangXun ChenYingtong XiongGuoteng WangQiaoling ChenShangchun ZhaoJiarui FangYonggang WenTianwei ZhangXin JinXuanzhe LiuPublished in: CoRR (2024)
Keyphrases
- computationally efficient
- contextual information
- artificial neural networks
- object detection
- context aware
- information retrieval
- computer vision
- parallel processing
- parallel execution
- data sets
- massively parallel
- training algorithm
- human body
- training examples
- training set
- search algorithm
- machine learning
- neural network