A study on the evaluation of tokenizer performance in natural language processing.
Sanghyun ChooWonjoon KimPublished in: Appl. Artif. Intell. (2023)
Keyphrases
- natural language processing
- formal evaluation
- theoretical framework
- information extraction
- natural language
- empirical studies
- machine learning
- quantitative evaluation
- empirical analysis
- database
- case study
- website
- knowledge representation
- information systems
- artificial intelligence
- statistical analysis
- gold standard
- factors affecting