robots.txt
When you make a copy of your site for development/testing purpose, you don’t need search engines index your site as it will cuase duplicate content.
To disable indexing of a website by search engines, create a file with name robots.txt with following content.
|
1 2 |
User-agent: * Disallow: / |
To Allow all robots, use
|
1 2 |
User-agent: * Allow: / |
To specify sitemap
|
1 2 3 |
User-agent: * Allow: / Sitemap: https://serverok.in/sitemap.xml |
Crawl delay
|
1 2 3 4 |
User-agent: * Allow: / Sitemap: https://serverok.in/sitemap.xml Crawl-delay: 10 |
Crawl-delay set number of seconds between each page request. In this cause bot wait 10 seconds before indexing next page. It is supported by Bing and Yahoo.
Only allow search engines to index Home page. Deny indexing all other pages
|
1 2 3 |
user-agent: * Allow: /$ Disallow: / |