IRLbot: Scaling to 6 billion pages and beyond.

Hsin-Tsang Lee Derek Leonard Xiaoming Wang Dmitri Loguinov

Published in: ACM Trans. Web (2009)

Keyphrases

website
search engine
web pages
data objects
web users
keywords
web search
data sets
web applications
digital libraries
domain specific
artificial intelligence
machine learning
real time
web graph
link structure
anchor text
textual content
dynamically generated