SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining.
Chenglei SiZhengyan ZhangYingfa ChenFanchao QiXiaozhi WangZhiyuan LiuMaosong SunPublished in: CoRR (2021)
Keyphrases
- language model
- language modeling
- probabilistic model
- document retrieval
- n gram
- word segmentation
- information retrieval
- query expansion
- speech recognition
- retrieval model
- context sensitive
- language modelling
- test collection
- ad hoc information retrieval
- smoothing methods
- statistical language models
- translation model
- query terms
- mixture model
- vector space model
- pseudo relevance feedback
- language model for information retrieval
- relevance model
- text summarization
- information retrieval systems
- feature selection