Manta SEO Solutions Blog

I discovered a little post this morning on Search Engine Roundtable about whether a robots.txt file is required for SEO?

The question was asked in a Search Engine Watch Forum, and I agree with both of the comments made at either references, a robots.txt file is not required for SEO at all, at least to a degree.

The search engines will find and crawl your site automatically and don’t require a robots.txt before they index or crawl your site. Adding a robots.txt file or not having one at all does not affect your site ranking one bit.

That is not to say the robots.txt file isn’t useful.

Firstly, should you not have a robots.txt, you’ll get many 404 errors shown in your statistics pages as search engines do look for this file to see which restrictions might be set in place for them via this file.
Secondly, the purpose of the robots.txt file is to control which pages the search engines should not crawl, which can be very useful for duplicate content and SEO purposes.

In addition, you can add crawl-delays for Yahoo to reduce the overall bandwidth the search engine is generating if visiting too frequently.

Also, if you want to specify a sitemaps location, you can.

As you can see, there are many useful uses for the robots.txt file and though it is not really required, it sure is useful.

Popularity: 66% [?]

• September 7th, 2007
• Search Engine Optimization