Why does Yoast SEO output “index, follow” in the meta robots tag?
The meta robots tag gives search engines instructions on how you’d like them to crawl or index parts of your site. The Yoast SEO plugin allows you to easily set the meta robots tag for every post you write.
In the Advanced settings in the Yoast SEO meta box, you can determine if you want to allow search engines to show your post in the search results. You can also determine if search engines should follow links on the post (see image below).
If you set both of these to yes, which is the default, Yoast SEO outputs the following meta robots tag in the <head> of that page:
<meta name="robots" content="index, follow" />
<meta name="googlebot" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />
<meta name="bingbot" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />But if “index, follow” is the default, why explicitly add it?
Google and Bing have changed their handling of robots directives. The way in which these search engines now handle restrictions means it makes sense to include these defaults (“index, follow”).
Yoast SEO outputs three meta robots tags by default on each (public) page, with the following structure:
<meta name="robots" content="{{values}}" />
<meta name="googlebot" content="{{values}}, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />
<meta name="bingbot" content="{{values}}, max-snippet:-1, max-image-preview:large, max-video-preview:-1" />You can read more about how Yoast SEO handles meta robots here.
Get Yoast SEO Premium
Drive more traffic to your site. Use AI to save time doing SEO tasks. Make SEO easier. And get 24/7 support.
