A Framework of Deep Web Crawler.pdfVIP

th Proceedings of the 27 Chinese Control Conference July 16-18, 2008, Kunming,Yunnan, China * A Framework of Deep Web Crawler 1 2 1 Xiang Peisu , Tian Ke , Huang Qinzhen 1. College of Electrical Information Engineering ,Southwest University for Nationalities, Chengdu Sichuan 610041, P.R.China E-mail: xiangpeisu@ 2. Technology Center, Chengdu Institute of ZTE Corporation, Chengdu Sichuan 610041, P.R.China E-mail: tian.ke@ Abstract: As an ever-increasing amount of information on the web today is available through search interfaces, users have to key in a set of keywords in order to access the pages from certain web sites, which are often referred to as the hidden web or the deep web. Since there is no static links to the hidden web pages, search engines cannot discover and index such pages. However, according to recent studies, the content provided by many hidden web sites is often of very high quality and can be extremely valuable to many users. How to build an effective hidden web crawler that can autonomously discover and download pages from the hidden web is studied. A framework of deep web crawler is provided and we propose novel techniques to handle the actual mechanics of crawling the deep Web. Experiment shows that these policies are effective. Key Words: Deep Web, Deep Web Crawler semantics of the element. Such descr