网络搜索引擎原理002 crawling.pdf

architecture 3 Udacity Udacity is a totally new kind of learning experience. You learn by solving challenging problems and pursuing udacious projects with world-renowned university instructors (not by watching long, boring lectures). At Udacity, we put you, the student, at the center of the universe. 4 What any crawler must be ◆ Be Polite: Respect implicit and explicit politeness considerations for a website • Only crawl pages you’re allowed to • Respect robots.txt (more on this shortly) ◆ Be Robust: Be imm o spider traps and other malicious behavior from web servers 5 What any crawler should be ◆ Distributed ◆ Scalable ◆ Performance and Effeciency ◆ Quality ◆ Freshness ◆ Extensible 6 Crawling the Web ◼ W ges(hypertext ) • Few thousand characters long • Few or many Hyperlinks • Served through HTTP • Viewed at end using 'browsers' ◼ Crawler(spider, robot or bot) • To fetch the pages to the computer • At the computer  Automatic programs can yze hypertext s • Other names: ants, worms, bots 7 8 HTML basic ✓ HTML = HyperText Marku nguage ✓ Let the author ✓ specify layout and typeface ✓ embed diagrams ✓ create hyperlinks. ✓ expressed as an anchor tag with a HREF attribute ✓ HREF names another page using a