A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages. Typically search engines use crawlers to build indexes.
Learn more
- Web crawler on Wikipedia
A web crawler is a program, often called a bot or robot, which systematically browses the Web to collect data from webpages. Typically search engines use crawlers to build indexes.