LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World Knowledge in Language Model Alignment.
Shihan DouEnyu ZhouYan LiuSongyang GaoJun ZhaoWei ShenYuhao ZhouZhiheng XiXiao WangXiaoran FanShiliang PuJiang ZhuRui ZhengTao GuiQi ZhangXuanjing HuangPublished in: CoRR (2023)
Keyphrases
- language model
- world knowledge
- mixture model
- language modeling
- n gram
- bag of words
- noun phrases
- probabilistic model
- knowledge sources
- information retrieval
- background knowledge
- document retrieval
- query terms
- retrieval model
- query expansion
- pseudo relevance feedback
- natural language text
- vector space model
- test collection
- document representation
- knowledge base
- expectation maximization
- text documents
- image classification
- domain knowledge
- decision trees
- dirichlet distribution