|
bot
Also called robots, spiders, and crawlers. Used by search engines to crawl the WWW and index sites/pages following the hyperlinks within the pages along the way. Well behaving bots will check for a file in the www directory of the site named robots.txt and adhere to the contents of this file. The robots.txt file can allow or block certain areas of the site or block certain “bots” from access completely.
|
|
