Managing Website
Started by
LuciferWillis
, Oct 19 2013 03:46 AM
#1
Posted 19 October 2013 - 03:46 AM
#2
Posted 19 October 2013 - 09:33 AM
Welcome to Geeks to Go!
You control the crawl rate and access of web robots or bots on your site using a special text file in the http root of your website called robots.txt. Robotstxt.org is a good resource: http://www.robotstxt.org/
You control the crawl rate and access of web robots or bots on your site using a special text file in the http root of your website called robots.txt. Robotstxt.org is a good resource: http://www.robotstxt.org/
Similar Topics
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users