LocMoE: A Low-overhead MoE for Large Language Model Training.
Jing LiZhijie SunXuan HeLi ZengYi LinEntong LiBinfan ZhengRongqian ZhaoXin ChenPublished in: CoRR (2024)
Keyphrases
- language model
- low overhead
- language modeling
- n gram
- document retrieval
- information retrieval
- high reliability
- retrieval model
- probabilistic model
- speech recognition
- statistical language models
- load balancing
- language modelling
- query expansion
- ad hoc information retrieval
- context sensitive
- mixture model
- test collection
- communication cost
- shared memory
- smoothing methods
- language model for information retrieval
- relevance model
- translation model
- query terms
- energy efficient
- word clouds
- low cost
- pairwise