Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages.
Tomasz LimisiewiczJirí BalharDavid MarecekPublished in: ACL (Findings) (2023)
Keyphrases
- language modeling
- cross lingual
- language model
- language independent
- n gram
- comparable corpora
- language specific
- character n grams
- retrieval model
- cross language
- out of vocabulary
- information retrieval
- query expansion
- probabilistic model
- translation model
- text classification
- parallel corpora
- linguistic resources
- word segmentation
- named entities
- parallel corpus
- multilingual retrieval
- statistical machine translation
- relevance model
- query translation
- query terms
- test collection
- improvements in retrieval effectiveness
- bayesian networks