A Web crawler is a computer program that browses the World Wide Web. Web crawlers are also called ants, automatic indexers, bots, and worms or Web spider, Web robot, Wanderers.

This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. (more…)