Assessors Agreement: A Case Study Across Assessor Type, Payment Levels, Query Variations and Relevance Dimensions.
João R. M. PalottiGuido ZucconJohannes Bernhardt-MelischnigAllan HanburyLorraine GoeuriotPublished in: CLEF (2016)
Keyphrases
- relevance judgments
- relevance assessments
- test collection
- relevance feedback
- retrieval systems
- average precision
- retrieval effectiveness
- relevance judgements
- user feedback
- document relevance
- web search engines
- learning to rank
- relevant documents
- evaluation metrics
- ranking functions
- document ranking
- information retrieval systems
- score distributions
- case study
- information retrieval
- relevance ranking
- relevant and non relevant documents
- response time
- result list
- implicit feedback
- relevance scores
- query processing
- retrieval model
- user intent
- xml retrieval
- credit card
- image retrieval
- query expansion
- electronic commerce
- search tasks
- evaluation measures
- information seeking
- data structure
- database
- user interaction
- retrieved documents
- user behavior