Make your site easier to crawl for search engines

Why do you need the crawl optimization settings in Yoast SEO?

Getting every possible URL on your site crawled, indexed, and ranked by a search engine is nearly impossible. Every crawler has a limit to what it can do at any given time, so it’s best to focus on getting your most important URLs crawled. Unfortunately, a CMS like WordPress comes with a lot of baggage: URLs that it automatically adds to your site, unnecessary scripts, useless metadata, and much more. What if you could throw this all out?

Warning: Undefined array key “className” in /home/staging-yoast/staging-platform.yoast.com/versions/a4c25e841414f0e83231dded987decb991c04b7e/web/app/themes/yoast-theme/html_includes/blocks/block-ctba.php on line 20

Making it easier to get your essential URLs crawled

Crawling processes are very resource-intensive, and with a never-ending stream of new content to discover, search engines like Google have to be selective in what they can do. This means that search engines cannot come by all the time. So if they come, you better offer the most important URLs to them — the ones you want them to crawl! The crawl settings let you turn off certain types of URLs, scripts, and metadata that WordPress automatically adds to enhance crawling.

Making your site more sustainable

Crawling costs search engines a tremendous amount of energy. Search engines like Google operate tens of data centers, helping them to crawl, index, and rank information from around the world. All of these use incredible power to function, but the same goes for your site. The server you host your site on uses power, and every request to your site costs energy. So, to make the whole process more power-efficient, use the crawl settings to turn off stuff that has no reason to be crawled. Together, we can make your site — and the web — a lot greener!

Crawl settings help clean up your site

You can see the crawl settings as a way to easily clean up your site of unwanted overhead. For instance, you’ll also find options to clean up the internal site search of your site to prevent SEO spam attacks. And a way to clean up your permalinks by filtering campaign tracking and unregistered URL parameters, which leads to fewer duplicates found by search engines.