Meta Robots Tag
Let’s examine one of the key technical SEO tools — the Meta Robots Tag. This tag helps control how search engines index and display website pages, allowing webmasters to fine-tune robot behavior.
What is a Meta Robots Tag?
A Meta Robots Tag is an HTML tag placed in the <head> section of a webpage that provides instructions to search engine bots regarding:
- Whether the page can be indexed,
- Whether they should follow links on it,
- Whether the page should be displayed in search results.
This tag is an internal SEO control tool that allows you to manage search engine behavior without modifying the robots.txt file.
Syntax
Example of a standard meta robots tag:
html
<meta name=”robots” content=”index, follow”>
This example tells the search engine:
“You can index this page and follow all links placed on it.”
Main Meta Robots Directives
| Directive | Meaning | What it does |
| index | Index | Allows search engines to add the page to their index. |
| noindex | Do Not Index | Prevents the page from being added to the index. |
| follow | Follow Links | Allows bots to follow links and pass link equity (PageRank). |
| nofollow | Do Not Follow Links | Prevents bots from following links and does not pass link equity. |
| noarchive | No Cached Copy | Prevents showing a cached version of the page in search results. |
| nosnippet | No Snippet | Does not allow a snippet (page description) to be shown in search results. |
| noimageindex | Do Not Index Images | Prevents indexing images from the page. |
| notranslate | Do Not Translate | Prevents search engines from offering to translate the page. |
| max-snippet: [number] | Limit Snippet Length | Controls the number of characters in the search snippet. |
| max-image-preview: none / standard / large | Control Image Preview | Allows setting the size of image previews in search results. |
| max-video-preview: [number] | Set Video Preview Length | Controls how many seconds of video can be shown in search results. |
Usage Examples
- Allow Indexing and Link Following
html
<meta name=”robots” content=”index, follow”>
Used by default for public pages — articles, product pages, blog posts.
- Disallow Indexing, But Allow Link Following
html
<meta name=”robots” content=”noindex, follow”>
Suitable for pages that should not appear in search (e.g., cart, user account), but contain important internal links.
- Disallow Both Indexing and Link Following
html
<meta name=”robots” content=”noindex, nofollow”>
Completely closes the page to bots. Used rarely — e.g., for administrative or test pages.
- Only Disallow Cached Copies
html
<meta name=”robots” content=”noarchive”>
Useful for sites with frequently updated information (e.g., prices) where it’s important that users don’t see outdated cached versions.
- For Specific Search Engines
Instructions can be set for a particular search engine:
html
<meta name=”googlebot” content=”noindex”>
<meta name=”yandex” content=”nofollow”>
This allows independent control over different search engines’ behavior.
When to Use the Meta Robots Tag
Use it when you need to:
- Hide non-informative pages (login, cart, filters),
- Avoid duplicate content,
- Prevent passing link equity through certain links,
- Protect content from copying via cached versions.
Avoid using it:
- On pages that should rank in search results,
- As a replacement for proper robots.txt configuration — these tools work together, not as substitutes.
Difference Between Meta Robots and Robots.txt
| Criterion | Meta Robots Tag | Robots.txt File |
| Scope | Individual page level | Whole website level |
| Controls | Indexing of page content | Access to site files/directories |
| Precision | Allows individual page instructions | Sets global restrictions |
| Used for | HTML pages | All file types (images, PDF, CSS, etc.) |
The ideal approach is to use both tools:
- robots.txt regulates access,
- meta robots manages the indexing of specific pages.
Common Mistakes
- Conflicting Directives. For example, index, nofollow can lead to incorrect processing.
- Duplicate Tags. If multiple robots tags are specified, a search engine might only follow the first one.
- Using noindex on Important Pages. This will cause them to drop out of search results.
- Ignoring Caching. Without noarchive, outdated page versions may appear in search results.
Summary
The Meta Robots Tag is a powerful SEO tool that helps control how search engines index, display, and process website pages.
When configured correctly, it allows you to:
- Avoid duplicate content,
- Protect administrative sections,
- Improve the quality of site indexing.
Ideally, this tag should be configured individually for each page type — informational, technical, and commercial.
