web crawler research paper pdf

Web Crawling

• First Online: 01 January 2011

Cite this chapter

• Bing Liu 2 &
• Filippo Menczer  

Part of the book series: Data-Centric Systems and Applications ((DCSA))

11k Accesses

4 Citations

Web crawlers, also known as spiders or robots, are programs that automatically download Web pages. Since information on the Web is scattered among billions of pages served by millions of servers around the globe, users who browse the Web can follow hyperlinks to access information, virtually moving from one page to the next. A crawler can visit many sites to collect information that can be analyzed and mined in a central location, either online (as it is downloaded) or off-line (after it is stored).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

Current Challenges in Web Crawling

A Study on Different Types of Web Crawlers

Large Scale Web Crawling and Distributed Search Engines: Techniques, Challenges, Current Trends, and Future Prospects

Bibliography.

Aggarwal, C., F. Al-Garawi, and P. Yu. Intelligent crawling on the World Wide Web with arbitrary predicates. In Proceedings of 10th Internaitonal Conference on World Wide Web (WWW-2001) , 2001.

Google Scholar  

Akavipat, R., L. Wu, and F. Menczer. Small world peer networks in distributed Web search. In Proceedings of Alternative Track Papers and Posters Proceedings of International Conference on World Wide Web , 2004.

Amento, B., L. Terveen, and W. Hill. Does “authority” mean quality? Predicting expert quality ratings of Web documents. In Proceedings of ACM SIGIR Conf. on Research and Development in Information Retrieval (SIGIR-2000) , 2000.

Arasu, A., J. Cho, H. Garcia-Molina, A. Paepcke, and S. Raghavan. Searching the web . ACM Transactions on Internet Technology (TOIT) , 2001, 1(1): p. 2–43.

Article   Google Scholar  

Bharat, K. and M. Henzinger. Improved algorithms for topic distillation in a hyperlinked environment. In Proceedings of ACM SIGIR Conf. on Research and Development in Information Retrieval (SIGIR-1998) , 1998.

Brin, S. and P. Lawrence. The anatomy of a large-scale hypertextual web search engine . Computer Networks , 1998, 30(1–7): p. 107–117.

Broder, A., R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, and J. Wiener. Graph structure in the web . Computer Networks , 2000, 33(1–6): p. 309–320.

Chakrabarti, S. Mining the Web: discovering knowledge from hypertext data . 2003: Morgan Kaufmann Publishers.

Chakrabarti, S., B. Dom, S. Kumar, P. Raghavan, S. Rajagopalan, A. Tomkins, D. Gibson, and J. Kleinberg. Mining the Web's link structure . Computer , 2002, 32(8): p. 60–67.

Chakrabarti, S., B. Dom, P. Raghavan, S. Rajagopalan, D. Gibson, and J. Kleinberg. Automatic resource compilation by analyzing hyperlink structure and associated text . Computer Networks , 1998, 30(1–7): p. 65–74.

Chakrabarti, S., M. Van den Berg, and B. Dom. Focused crawling: a new approach to topic-specific Web resource discovery . Computer Networks , 1999, 31(11–16): p. 1623–1640.

Chen, H., Y. Chung, M. Ramsey, and C. Yang. A smart itsy bitsy spider for the web . Journal of the American Society for Information Science , 1998, 49(7): p. 604–618.

Cho, J. and H. Garcia-Molina. The evolution of the web and implications for an incremental crawler. In Proceedings of International Conference on Very Large Data Bases (VLDB-2000) , 2000.

Cho, J., H. Garcia-Molina, and L. Page. Efficient crawling through URL ordering . Computer Networks , 1998, 30(1–7): p. 161–172.

Davison, B. Topical locality in the Web. In Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR-2000) , 2000.

De Bra, P. and R. Post. Information retrieval in the World-Wide Web: making client-based searching feasible . Computer Networks , 1994, 27(2): p. 183–192.

Degeratu, M., G. Pant, and F. Menczer. Latency-dependent fitness in evolutionary multithreaded web agents. In Proceedings of GECCO Workshop on Evolutionary Computation and Multi-Agent Systems , 2001.

Diligenti, M., F. Coetzee, S. Lawrence, C. Giles, and M. Gori. Focused crawling using context graphs. In Proceedings of International Conference on Very Large Data Bases (VLDB-2000) , 2000.

Eichmann, D. Ethical Web agents . Computer Networks and ISDN Systems , 1995, 28(1–2): p. 127–136.

Fetterly, D., M. Manasse, M. Najork, and J. Wiener. A large scale study of the evolution of Web pages . Software: Practice and Experience , 2004, 34(2): p. 213–237.

Gasparetti, F. and A. Micarelli. Swarm intelligence: Agents for adaptive web search. In Proceedings of European Conf. on Artificial Intelligence (ECAI- 2004) , 2004.

Henzinger, M., A. Heydon, M. Mitzenmacher, and M. Najork. Measuring index quality using random walks on the Web . Computer Networks , 1999, 31(11–16): p. 1291–1303.

Henzinger, M., A. Heydon, M. Mitzenmacher, and M. Najork. On nearuniform URL sampling . Computer Networks , 2000, 33(1–6): p. 295–308.

Hersovici, M., M. Jacovi, Y. Maarek, D. Pelleg, M. Shtalhaim, and S. Ur. The shark-search algorithm. An application: tailored Web site mapping . Computer Networks , 1998, 30(1–7): p. 317–326.

Heydon, A. and M. Najork. Mercator: A scalable, extensible web crawler . World Wide Web , 1999, 2(4): p. 219–229.

Jagatic, T., N. Johnson, M. Jakobsson, and F. Menczer. Social phishing . Communications of the ACM , 2007, 50(10): p. 94–100.

Kaelbling, L., M. Littman, and A. Moore. Reinforcement learning: A survey . Journal of Artificial Intelligence Research 1996, 4: p. 237–285.

Kleinberg, J. Authoritative sources in a hyperlinked environment . Journal of the ACM (JACM) , 1999, 46(5): p. 604–632.

Article   MATH   MathSciNet   Google Scholar  

Lawrence, S., L. Giles, and K. Bollacker. Digital libraries and autonomous citation indexing . Computer , 2002, 32(6): p. 67–71.

Lin, L. Self-improving reactive agents based on reinforcement learning, planning and teaching . Machine Learning , 1992, 8(3): p. 293–321.

Lu, J. and J. Callan. Content-based retrieval in hybrid peer-to-peer networks. In Proceedings of ACM International Conference on Information and knowledge management (CIKM-2003) , 2003.

Maguitman, A., F. Menczer, H. Roinestad, and A. Vespignani. Algorithmic detection of semantic similarity. In Proceedings of International Conference on World Wide Web (WWW-2005) , 2005.

McCallum, A., K. Nigam, J. Rennie, and K. Seymore. A machine learning approach to building domain-specific search engines. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI-1999) , 1999.

Menczer, F. ARACHNID: Adaptive retrieval agents choosing heuristic neighborhoods for information discovery. In Proceedings of International Conference on Machine Learning (ICML-1997) , 1997.

Menczer, F. Lexical and semantic clustering by web links . Journal of the American Society for Information Science and Technology , 2004, 55(14): p. 1261–1269.

Menczer, F. Mapping the semantics of web text and links . Internet Computing, IEEE , 2005, 9(3): p. 27–36.

Menczer, F. and R. Belew. Adaptive retrieval agents: Internalizing local

context and scaling up to the Web . Machine Learning , 2000, 39(2): p. 203–242.

Menczer, F., G. Pant, and P. Srinivasan. Topical web crawlers: Evaluating adaptive algorithms . ACM Transactions on Internet Technology (TOIT) , 2004, 4(4): p. 378–419.

Menczer, F., G. Pant, P. Srinivasan, and M. Ruiz. Evaluating topic-driven Web crawlers. In Proceedings of ACM SIGIR Conf. on Research and Development in Information Retrieval (SIGIR-2001) , 2001.

Micarelli, A. and F. Gasparetti. Adaptive focused crawling. In P. Brusilovsky, W. Nejdl, and A. Kobsa (eds.), Adaptive Web. , 2007: Springer-Verlag.

Najork, M. and J. Wiener. Breadth-first crawling yields high-quality pages. In Proceedings of International Conference on World Wide Web (WWW-2001) , 2001.

Ntoulas, A., J. Cho, and C. Olston. What's new on the web?: the evolution of the web from a search engine perspective. In Proceedings of International Conference on World Wide Web (WWW-2004) , 2004.

Pant, G. Deriving link-context from HTML tag tree. In Proceedings of ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery (DMKD’03) , 2003.

Pant, G., S. Bradshaw, and F. Menczer. Search engine-crawler symbiosis: Adapting to community interests . Research and AdvancedTechnology for Digital Libraries , 2004: p. 221–232.

Pant, G. and F. Menczer. MySpiders: Evolve your own intelligent Web crawlers . Autonomous Agents and Multi-Agent Systems , 2002, 5(2): p. 221–229.

Pant, G. and F. Menczer. Topical crawling for business intelligence . Research and Advanced Technology for Digital Libraries , 2004: p. 233–244.

Pant, G. and P. Srinivasan. Learning to crawl: Comparing classification schemes . ACM Transactions on Information Systems (TOIS) , 2005, 23(4): p. 430–462.

Pant, G., P. Srinivasan, and F. Menczer. Exploration versus exploitation in topic driven crawlers. In Proceedings of WWW-02 Workshop on Web Dynamics , 2002.

Pastor-Satorras, R. and A. Vespignani. Evolution and structure of the Internet: A statistical physics approach . 2004: Cambridge Univ Press.

Rennie, J. and A. McCallum. Using reinforcement learning to spider the web efficiently. In Proceedings of International Conference on Machine Learning (ICML-1999) , 1999.

Rijsbergen, C.v. Information Retrieval . 1979: Butterworths. Second edition.

Rumelhart, D., G. Hinton, and R. Williams. Learning internal representations by error propagation . D. Rumelhart and J. McClelland (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition , 1996.

Srinivasan, P., F. Menczer, and G. Pant. A general evaluation framework for topical crawlers . Information Retrieval , 2005, 8(3): p. 417–447.

Srinivasan, P., J. Mitchell, O. Bodenreider, G. Pant, F. Menczer, and P. Acd. Web crawling agents for retrieving biomedical information. In Proceedings of Workshop on Agents in Bioinformatics (NETTAB’02) , 2002.

Von Ahn, L., M. Blum, N. Hopper, and J. Langford. CAPTCHA: Using hard AI problems for security . Advances in Cryptology—EUROCRYPT-2003 , 2003: p. 646–646.

Witten, I., C. Nevill-Manning, and S. Cunningham. Building a digital library for computer science research: technical issues . Australian Computer Science Communications , 1996, 18 p. 534–542.

Wu, L., R. Akavipat, and F. Menczer. 6S: Distributing crawling and searching across Web peers. In Proceedings of IASTED Int. Conf. on Web Technologies, Applications, and Services , 2005.

Wu, L., R. Akavipat, and F. Menczer. Adaptive query routing in peer Web search. In Proceedings of International Conference on World Wide Web (WWW-2005) , 2005.

Download references

Author information

Authors and affiliations.

Department of Computer Science, University of Illinois, Chicago, 851 S. Morgan St., Chicago, IL, 60607-7053, USA

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Bing Liu .

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Liu, B., Menczer, F. (2011). Web Crawling. In: Web Data Mining. Data-Centric Systems and Applications. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-19460-3_8

Download citation

DOI : https://doi.org/10.1007/978-3-642-19460-3_8

Published : 15 April 2011

Publisher Name : Springer, Berlin, Heidelberg

Print ISBN : 978-3-642-19459-7

Online ISBN : 978-3-642-19460-3

eBook Packages : Computer Science Computer Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Web Crawler: Design And Implementation For Extracting Article-Like Contents

2020, Cybernetics and Physics

The World Wide Web is a large, wealthy, and accessible information system whose users are increasing rapidly nowadays. To retrieve information from the web as per users' requests, search engines are built to access web pages. As search engine systems play a significant role in cybernetics, telecommunication, and physics, many efforts were made to enhance their capacity. However, most of the data contained on the web are unmanaged, making it impossible to access the entire network at once by current search engine system mechanisms. Web Crawler, therefore, is a critical part of search engines to navigate and download full texts of the web pages. Web crawlers may also be applied to detect missing links and for community detection in complex networks and cybernetic systems. However, template-based crawling techniques could not handle the layout diversity of objects from web pages. In this paper, a web crawler module was designed and implemented, attempted to extract article-like contents from 495 websites. It uses a machine learning approach with visual cues, trivial HTML, and text-based features to filter out clutters. The outcomes are promising for extracting article-like contents from websites, contributing to the search engine systems development and future research gears towards proposing higher performance systems.

Related Papers

Bio-Inspired Computing for Information Retrieval Applications

SUBRATA PAUL

Computational biology and bio inspired techniques are part of a larger revolution that is increasing the processing, storage and retrieving of data in major way. This larger revolution is being driven by the generation and use of information in all forms and in enormous quantities and requires the development of intelligent systems for gathering, storing and accessing information. This chapter describes the concepts, design and implementation of a distributed web crawler that runs on a network of workstations and has been used for web information extraction. The crawler needs to scale (at least) several hundred pages per second, is resilient against system crashes and other events, and is capable to adapted various crawling applications. Further this chapter, focusses on various ways in which appropriate biological and bio inspired tools can be used to implement, automatically locate, understand, and extract online data independent of the source and also to make it available for Sem...

Proceedings of the 18th Panhellenic Conference on Informatics - PCI '14

P. Tsantilas

International Journal IJRITCC

PACIS 2006 Proceedings

Jon Patrick

Manpreet Sehgal

Diving into the World Wide Web for the purpose of fetching precious stones (relevant information) is a tedious task under the limitations of current diving equipments (Current Browsers). While a lot of work is being carried out to improve the quality of diving equipments, a related area of research is to devise a novel approach for mining. This paper describes a novel approach to extract the web data from the hidden websites so that it can be used as a free service to a user for a better and improved experience of searching relevant data. Through the proposed method, relevant data (Information) contained in the web pages of hidden websites is extracted by the crawler and stored in the local database so as to build a large repository of structured and indexed and ultimately relevant data. Such kind of extracted data has a potential to optimally satisfy the relevant Information starving end user.

Journal of Computer Science IJCSIS

Knowledge Discovery in database (KDD) is the non-trivial method of identifying legitimate, novel, potentially useful and in the long run comprehensible styles in massive statistics collections. But, with the growing quantity of records the complexity of statistics items increases as well. This turned into performed via Multi-example and multi-represented gadgets are important sorts of item illustration for complex gadgets. Multi-example items carries a fixed of item representations that everyone belong to the identical feature space. Multi-represented items are built as a tuple of characteristic representations in which every characteristic illustration belongs to an extraordinary characteristic space. The proposed paintings will be high accurate internet crawling, which used to extracts new unknown web sites from the WWW with excessive accuracy. We speak the subsequent classes of techniques: (1) sensible Crawling methods: those techniques research the relationship among the hyper-link shape/net page content material and the topic of the internet page. This learned information is applied which will manual the course of the crawl. (2) Collaborative Crawling methods: these techniques utilize the pattern of worldwide net accesses by means of person uses on the way to build the getting to know data. In many cases, person access styles include precious statistical patterns which cannot be inferred from in basic terms linkage data. We are able to additionally speak a few creative approaches of mixing one of a kind form of linkage-and user-focused strategies for you to enhance the effectiveness of the crawl.

International Journal of Scientific Research in Science, Engineering and Technology IJSRSET

As there is profound web development, there has been expanded enthusiasm for methods that help productively find profound web interfaces. Because of accessibility of inexhaustible information on web, seeking has a noteworthy effect. Ongoing examines place accentuation on the pertinence and strength of the information found, as the found examples closeness is a long way from the investigated. Notwithstanding their importance pages for any inquiry subject, the outcomes are colossal to be investigated. One issue of pursuit on the Web is that internet searchers return huge hit records with low accuracy. Clients need to filter applicable reports from insignificant ones by physically bringing and skimming pages. Another debilitating viewpoint is that URLs or entire pages are returned as list items. It is likely that the response to a client question is just part of the page. Recovering the entire page really leaves the errand of inquiry inside a page to Web clients. With these two viewpoints staying unaltered, Web clients won't be liberated from the substantial weight of perusing pages and finding required data, and data got from one pursuit will be characteristically constrained.

Izaura Xhumari

Websites are getting richer and richer with information in different formats. The data that such sites possess today goes through millions of terabytes of data, but not every information that is on the net is useful. To enable the most efficient internet browsing for the user, one methodology is to use web crawler. This study presents web crawler methodology, the first steps of development, how it works, the different types of web crawlers, the benefits of using and comparing their operating methods which are the advantages and disadvantages of each algorithm used by them.

Ee-Peng LIM

The huge amount of information available on the Web has attracted many research efforts into developing wrappers that extract data from webpages. However, as most of the systems for generating wrappers focus on extracting data at page-level, data extraction at site-level remains a manual or semi-automatic process. In this paper, we study the problem of extracting website skeleton, ie extracting the underlying hyperlink structure that is used to organize the content pages in a given website.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

International Journal of Computer Applications

Mohsen Asfia , Mir Mohsen Pedram

francois rousselot

iir publications

Thirugnana Sambanthan

Knowledge and Information Systems

Kostyantyn Shchekotykhin

IRJET Journal

Satinder Bal Gupta

INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARC

F M Javed Mehedi Shamrat

International Journal of Knowledge and Web Intelligence

Sangita Chaudhari

Proceedings of National Conference on Recent Trends in Parallel Computing (RTPC - 2014)

Mohd Shoaib , Mohammad Khan

Jan-Ming Ho

Marie-Francine Moens

Ganesh Puri

Sheba Gaikwad

Sudhir Aggarwal

Samiksha Bharne

Dr. Eda Baykan

Dhwanish Doshi

Dr_Aliea Sabir

International Journal of Applied Engineering Research

apoorv vikram singh , vikas soni , achyut mishra

RELATED TOPICS

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024