Disallow certain page directories but NOT that page itself

8
votes

Let's say, I have a dynamic page that creates URL's from user inputs. For example: www.XXXXXXX.com/browse <-------- (Browse being the page)

Every time user enters some query, it generates more pages. For example: www.XXXXXXX.com/browse/abcd <-------- (abcd being the new page)

Now, I want Google to do crawl this "browse" page but not the sub pages generated by it.

I'm thinking of adding this to my robots.txt page; "Disallow: /browse/"

Would that be the right thing to do? or will it also prevent the Googlebot from crawling the "browse" page? What should I do to get the optimal result?

1
robots.txt
I believe you can use the meta tag 'no index, no follow' on your inner pages, and just use 'no follow' on your main pages, so google will not 'go in' more...robotstxt.org/meta.html – Ziv Weissman
These sub pages are like in hundreds, I don't think I can add no follow to every page manually :/ – Raj Sandhu

1 Answers

4
votes

URL doesn't end with slash:

www.XXXXXXX.com/browse

Therefore this code should work:

User-agent: *
Disallow: /browse/