Another Great Free Tool – Robots.txt Builder

Filed under: Information,web design — Tags:

The robots.txt file is a plain text file that you put in the root directory of your site which tells search engine robots where they can go, where they cannot go, how often to visit, what times to visit, where your sitemap is and more. This is a great file for website owners to have and is very much under used.

On my site I have a blank page. It is still called robots.txt so the search engines visit it first and that does two things for me. First it will keep (some of) them from loading my 404 error page, which comes up when you type in a page on my site that doesn’t exist and that saves me bandwidth, and second it tells the robots that they have free range to index all of the pages in my site. There are many “bad robots” which serve no useful purpose, including many “data scrapers” and other nefarious activities which you may want to consider banning. Most bad bots do not obey the Robots Exclusion Standard but a surprising number do.

Robots.txt Builder is a new free tool that will import your site structure, use your input to decide which parts of your site to use or not use and will create a valid robots.test file for you to use on your site. It’s a great free tool and is well worth checking out.


No Comments »

No comments yet.

RSS feed for comments on this post.

Leave a comment

first time commenters are moderated

Top Of Page