You should evaluate your language model on marginal likelihood over tokenisations.
Kris CaoLaura RimellPublished in: EMNLP (1) (2021)
Keyphrases
- language model
- marginal likelihood
- model selection
- language modeling
- gaussian process
- probabilistic model
- information criterion
- approximate inference
- exponential family
- mixture model
- closed form
- speech recognition
- document retrieval
- information retrieval
- cross validation
- graphical models
- bayesian information criterion
- hyperparameters
- feature selection
- particle filter