What is robots.txt file?

A “robots.txt” computer file informs or gives permission/allow/Disallow search applications whether they can spider areas of your website content/Pages. It functions like a entrance in your home which is necessary factor same as spiders.txt computer file allow spiders. To make effective this computer file you must produce this in the website owner resources (shown below how it appears) and duplicate this value in note pad which must be known as “robots.txt”, is placed in the main index of your website. If You may not want certain webpages of your website get indexed in a look for motor's serp's then Google Webmaster Tools has a helpful spiders.txt turbine to help you make this computer file.

Text seems to be when you make spiders value in website owner tools

User-agent: *

Disallow: /images/

Disallow: /search

All look for website optimization spiders must not accessibility and spider the articles under /images/ or any URL whose direction starts with /search.

Robots.txt for subdomain:

If your website comprise subdomain and you wish to have certain webpages get crawled/not indexed on a particular subdomain, its need to make a individual spiders.txt data declare that subdomain.

Writing Robots.txt in meta tags:

There are other ways to avoid articles showing in serp's, such as including “NOINDEX” to your spiders meta tag, using .htaccess to code secure internet directories, and using Google Webmaster Tools to eliminate articles that has already been indexed.

<meta name=”robots” content=”noindex”/>