Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models.
Orevaoghene AhiaSachin KumarHila GonenJungo KasaiDavid R. MortensenNoah A. SmithYulia TsvetkovPublished in: CoRR (2023)
Keyphrases
- language model
- n gram
- language modeling
- character n grams
- language independent
- document retrieval
- cross lingual
- speech recognition
- information retrieval
- retrieval model
- probabilistic model
- language modelling
- context sensitive
- test collection
- query expansion
- statistical language models
- pseudo relevance feedback
- named entities
- relevance model
- translation model
- language models for information retrieval
- smoothing methods
- language model for information retrieval
- ad hoc information retrieval
- multiword
- feature selection
- term dependencies
- document length
- parallel corpora
- document ranking
- topic models