Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models.
Orevaoghene AhiaSachin KumarHila GonenJungo KasaiDavid R. MortensenNoah A. SmithYulia TsvetkovPublished in: EMNLP (2023)
Keyphrases
- language model
- n gram
- language modeling
- character n grams
- language independent
- document retrieval
- cross lingual
- probabilistic model
- language modelling
- retrieval model
- test collection
- statistical language models
- query expansion
- information retrieval
- speech recognition
- ad hoc information retrieval
- pseudo relevance feedback
- vector space model
- context sensitive
- translation model
- cross language retrieval
- language models for information retrieval
- smoothing methods
- relevance model
- text retrieval
- named entities
- query translation
- text summarization
- document ranking
- web documents
- okapi bm
- information extraction