When the #robots.txt just won't do
Robots Meta Tag
In addition to uploading a "robots.txt" into the root directory or perhaps as an alternative to using "robots.txt", the use of the "robots" meta tag is an option.
The "robots" meta tag looks similar to any meta tag and should be added between the head sections of your pages.
Here are a few examples:
1) This disallows both indexing and following of links by a crawler on that specific page:
<meta name="robots" content="noindex,nofollow" />
2) This disallows indexing of the page, but lets the crawler go on and follow/crawl links contained within it.
<meta name="robots" content="noindex,follow" />
3) This allows indexing of the page, but instructs the crawler to not crawl links contained within it:
<meta name="robots" content="index,nofollow" />
4) Finally, there is a shorthand way of declaring 1) above (don't index nor follow links on page):
<meta name="robots" content="none">Here are descriptions:
Posted by Marty A. Muse
