0
votes

I have various paginations on my site and I want to stop google and other search engines crawling the index of my paginations.

Example of a crawled page:

http://www.mydomain.com/explore/recently-updated/index/12

How can I, using robots.txt deny robots from crawling any pages that contain /index/ ?

2

2 Answers

3
votes
Disallow: /*?page=
Disallow: /*&page=
1
votes

I believe if you create your robots.txt file at your root web directory with the following:

User-agent: *
Disallow: /explore/recently-updated/index/

Give that a try, then use someone (probably google's) robot checker to ensure it is being stopped.