robots.txt

When you make a copy of your site for development/testing purpose, you don’t need search engines index your site as it will cuase duplicate content.

To disable indexing of a website by search engines, create a file with name robots.txt with following content.

Add a Comment

Your email address will not be published. Required fields are marked *