This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data.
From the beginning, a key motivation for designing Web crawlers has been to retrieve Web pages and add them or their representations to a local repository. Such a repository may then serve particular application needs such as those of a Web search engine. In its simplest form a crawler starts from a seed page and then uses the external links within it to attend to other pages.

Running a web crawler is a challenging task. There are tricky performance and reliability issues.
In order to scale to hundreds of millions of web pages, Google has a fast distributed crawling system. Each crawler keeps roughly 300 connections open at once. This is necessary to retrieve web pages at a fast enough pace. At peak speeds, the system can crawl over 100 web pages per second using four crawlers or we have been able to crawl up to tens of thousands of pages within a few minutes. This amounts to roughly 600K per second of data.

In order to fetch a Web page, we need an HTTP client which sends an HTTP request for a page and reads the response. The client needs to have timeouts to make sure that an unnecessary amount of time is not spent on slow servers or in reading large pages.

Once a page has been fetched, we need to parse its content to extract information that will feed and possibly guide the future path of the crawler. Parsing may imply simple hyperlink/URL extraction or it may involve the more complex process of tidying up the HTML content in order to analyze the HTML tag tree.

Post by : SEO Service provider company India