CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages.
Thuat NguyenChien Van NguyenViet Dac LaiHieu ManNghia Trung NgoFranck DernoncourtRyan A. RossiThien Huu NguyenPublished in: CoRR (2023)
Keyphrases
- language model
- language modeling
- cross lingual
- language independent
- n gram
- comparable corpora
- information retrieval
- document retrieval
- speech recognition
- language modelling
- retrieval model
- probabilistic model
- cross lingual information retrieval
- query expansion
- context sensitive
- test collection
- cross language
- statistical machine translation
- vector space model
- language models for information retrieval
- translation model
- statistical language models
- ad hoc information retrieval
- relevance model
- digital libraries
- smoothing methods
- retrieval effectiveness
- machine translation
- document ranking
- query terms
- machine learning
- bilingual dictionaries
- query specific
- linguistic resources
- parallel corpora
- information extraction
- document level
- text retrieval