This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data.
From the beginning, a key motivation for designing Web crawlers has been to retrieve Web pages and add them or their representations to a local repository. Such a repository may then serve particular application needs such as those of a Web search engine. In its simplest form a crawler starts from a seed page and then uses the external links within it to attend to other pages.

Running a web crawler is a challenging task. There are tricky performance and reliability issues.
In order to scale to hundreds of millions of web pages, Google has a fast distributed crawling system. Each crawler keeps roughly 300 connections open at once. This is necessary to retrieve web pages at a fast enough pace. At peak speeds, the system can crawl over 100 web pages per second using four crawlers or we have been able to crawl up to tens of thousands of pages within a few minutes. This amounts to roughly 600K per second of data.

In order to fetch a Web page, we need an HTTP client which sends an HTTP request for a page and reads the response. The client needs to have timeouts to make sure that an unnecessary amount of time is not spent on slow servers or in reading large pages.

Once a page has been fetched, we need to parse its content to extract information that will feed and possibly guide the future path of the crawler. Parsing may imply simple hyperlink/URL extraction or it may involve the more complex process of tidying up the HTML content in order to analyze the HTML tag tree.

Post by : SEO Service provider company India

While link building (placing links to the site or from the site) and most importantly, writing useful content form the foundation of top search engine rankings.

Short and Descriptive URL: If you have got a three, four or five words in your URL that can be perfectly normal. As it gets a little longer, then it starts to look a little worse. Now, our algorithms typically will just weight those words less and just not give you as much credit.

URL’s are case sensitive: Most of the people use standard lower case URL’s. It is advisable to use lower case URL because if you are using URL’s in upper case and somebody links to your post with lower case URL will not found the page.

Use keywords in URL: URL gives an idea to the reader about web content. For example, if you search in a Google about ‘Top 10 ranking’, you will get number of results with different URL’s. Some of them are less appealing to users and having very complex URL structure. Reader cannot make it out easily about the content inside the article. It is the possibility that the URL may contain the objectionable content. So it is suggested to use keywords in the URL. The results with this URL structure drives more traffic as the user tends to click those URL’s.

Advantages of keywords in static URLs
• The URLs look nicer and will likely get clicked on more often.
• The URLs will provide better anchor text if people use the URLs as the link anchor text.

Avoid rewriting Dynamic URLs
In most cases rewriting Dynamic URLs, the translation can be messy, and if not handled correctly can lead to indexing problems.

Post: SEO Consultant India providing SEO Services