Comments on: Managing Robot’s Access To Your Website

By: Daniel Noll

Daniel Noll — Wed, 02 Dec 2009 03:39:40 +0000

@Nathan: Thanks a million. I came to the same solution a few minutes after I posted my comment, but I just wasn’t confident about it. Only after your reply, did I implement it.

By: Nathan Buggia

Nathan Buggia — Tue, 01 Dec 2009 06:34:36 +0000

@Daniel Noll, hey daniel, you’re right, you would need to incorporate both the * and the $ like this:

Disallow: /*/aaa/bbb/$

By: Daniel Noll

Daniel Noll — Sun, 29 Nov 2009 01:22:05 +0000

Terrifically thorough. But I have one question that I hope you might be able to clarify.

Say for example I wish to disallow any URL in my site that ends with: “/aaa/bbb/”
Would this entry sufficient:
Disallow: /aaa/bbb/

Or must I somehow use * and $ in the entry?

By: Godwin

Godwin — Tue, 17 Nov 2009 14:47:21 +0000

This is a very thorough report. I have bookmarked it and will reference to it. Good luck to your endeavors! :)

By: Andy Beard

Andy Beard — Sun, 01 Nov 2009 11:52:12 +0000

GoogleBot follows the most specific directive, ignoring all others. In the robots.txt file, if you specify a section for all user-agents (user-agent: *) and also declare a section for Googlebot (user-agent: Googlebot), Google will disregard all sections in the robots.txt file except the Googlebot section. This could potentially leave you exposing much more content to Google that you might have thought.

I have been seeing some evidence that Google can take the wildcard user-agent into consideration even when there is a specific declaration present.
Is this something new/official? It would make sense from a human logic perspective, but is a change to published material.

By: Gary Trotter

Gary Trotter — Thu, 22 Oct 2009 10:47:44 +0000

I have allsorts of problems with Google accessing URLs that return 404. Unfortunately the URLs that Google is accessing were erroneously generated by my website and I do not know exactly what they are. As a result of this I am looking to setup my robots.txt file to disallow access to everything and then allow access to named URLs. However I cannot figure out the syntax to allow access to my website’s homepage (www.wedding-favours-online.co.uk) having previously used the disallow command to disallow all access. Does this make any sense?

Thanks

gary

By: nathan

nathan — Sun, 04 Oct 2009 16:14:23 +0000

@Andy H – yes, you absolutely can. The search engine would download the page, process the links and then discard the page so that it would not show up in search results.

However, this may not be the best approach for handling site search results, because there are still an infinite number (or at least very large) number of pages that could be created through your site search. A better approach would be to make sure that your sitemap was added to your robots.txt file (using the Sitemap: auto-discovery directive). And you should ensure that your sitemap is high quality, e.g. all URLs should be in their canonical form, and it should not list links to pages that return 404, or 500 server errors.

By: Andy H

Andy H — Wed, 30 Sep 2009 08:15:23 +0000

Hmmm, not sure if this comment will get found, but I have a question:

Could you put a META Robots Tag on the results pages that said “NOINDEX, FOLLOW”?

Say you have a large site with a bunch of “Search Result” style pages leading to quality, unique asset pages. According to Webmaster guidelines, you wouldn’t want the results pages indexed, but you would want the asset pages indexed, would this tag work?

By: The Ultimate SEO Audit

The Ultimate SEO Audit — Mon, 24 Aug 2009 19:38:19 +0000

[...] 1. Make sure Meta robots.txt isn’t blocking the whole site. JaneAndRobot offers a great guide for managing Robot’s access to your website. [...]

By: Adrian Hemsley

Adrian Hemsley — Thu, 25 Jun 2009 10:27:32 +0000

That’s about as through explanation as I’ve seen! :-)