Crowdsourcing Document Relevance Assessment with Mechanical Turk.
Catherine GradyMatthew LeasePublished in: Mturk@HLT-NAACL (2010)
Keyphrases
- mechanical turk
- relevance assessments
- retrieval systems
- test collection
- information retrieval systems
- gold standard
- interactive information retrieval
- relevant documents
- relevance judgements
- information seeking
- document collections
- information retrieval
- evaluation metrics
- user feedback
- retrieval model
- relevance judgments
- relevance scores
- document relevance
- co occurrence
- xml retrieval
- multimedia
- search engine
- document retrieval
- document images
- web documents
- xml documents
- pairwise
- highly relevant documents