CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages.
Thuat NguyenChien Van NguyenViet Dac LaiHieu ManNghia Trung NgoFranck DernoncourtRyan A. RossiThien Huu NguyenPublished in: LREC/COLING (2024)
Keyphrases
- language model
- language modeling
- cross lingual
- language independent
- n gram
- comparable corpora
- document retrieval
- cross language
- cross lingual information retrieval
- statistical language models
- speech recognition
- retrieval model
- probabilistic model
- statistical machine translation
- language modelling
- query expansion
- information retrieval
- translation model
- ad hoc information retrieval
- query terms
- vector space model
- test collection
- document ranking
- relevance model
- smoothing methods
- machine translation system
- language models for information retrieval
- word segmentation
- pseudo relevance feedback
- query translation
- context sensitive
- machine translation
- multiword
- parallel corpora
- text summarization
- chinese english
- cross language information retrieval
- bag of words
- digital libraries