Yoast SEO reduces your site’s carbon footprint with crawl optimization

We’re very excited to be releasing Yoast SEO 20.4. With this release, we’re bringing our crawl optimization feature to Yoast SEO Free. With this feature, you can improve your SEO and reduce your carbon footprint with just a few clicks. This blog post will tell you about this feature and why we’ve brought it to Yoast SEO.

Before we explain this Yoast SEO feature, it’s good to start with a quick reminder of what crawling is. Search engines like Google or Bing use crawlers, also known as bots, to find your website, read it and save its content to their index. They go around the internet 24/7 to ensure the content saved in its index is as up-to-date as possible. Depending on the number of changes you make on your website and how important search engines deem your site, the crawler comes around more or less often.

That’s nice, but did you know crawlers do an incredible amount of unnecessary crawling?

Let’s reduce unnecessary crawling

As you can imagine, search engine crawlers don’t just visit your website but every single one they can find. The incredible number of websites out there keeps them quite busy. In fact, bots are responsible for around 30% of all web traffic. This uses lots of electricity, and a lot of that crawling isn’t necessary at all. This is where our crawl optimization feature comes in. With just a few simple changes, you can tell search engines like Google which pages or website elements they can skip — making it easier to visit the right pages on your website while reducing the energy wasted on unnecessary bot traffic.

The carbon footprint of your website

You might be wondering why we want to help you reduce the energy consumption of your website. Does it make that much of a difference? The answer is yes! Regardless of the size of your website, the fact is that your website has a carbon footprint. Internet usage and digital technology are two massive players in pollution and energy consumption.

Every interaction on your website results in electricity being used. For instance, when someone visits your website, their browser needs to make an HTTP request to your server, and that server needs to return the necessary information. On the other side, the browser also needs the power to process data and present the page to the visitor. The energy needed to complete these requests might be small, but it adds up when you consider all the interactions on your website. Similar to when a visitor lands on your site, crawlers or bots also make these requests to your server that cost energy. Considering the amount of bot traffic (30% of web traffic), reducing the number of irrelevant pages and other resources crawled by search engines is worth it.

Take control of what’s being crawled

The crawl optimization feature in Yoast SEO lets you turn off crawling for certain types of URLs, scripts, and metadata that WordPress automatically adds. This makes it possible to improve your SEO and reduce your carbon footprint with just a few clicks.

Check out this fun animation to get an idea of what this feature can do for your website:

The crawl optimization feature was already part of Yoast SEO Premium, but now we’re also bringing it to the free version of our plugin. We do this to make as much of an impact as possible. There are over 13 million Yoast SEO users, so if everyone’s website crawling is optimized, we can have an enormous impact!

How to use the crawl optimization feature

How do you get started with crawl optimization for your website? Just go to Yoast SEO > Settings > Advanced > Crawl optimization. Here you will find an overview of all the types of metadata, content formats, etc., that you can tell search engines not to crawl. You can use the toggles on the right to enable crawl optimization.

The crawl optimization settings in Yoast SEO 20.4 allow you to:

• Remove unwanted metadata: WordPress adds a lot of links and content to your site’s and HTTP headers. For most websites, you can safely disable these, making your site faster and more efficient.
• Disable unwanted content formats: For every post, page, and category on your site, WordPress creates multiple types of feeds; content formats designed to be consumed by crawlers and machines. But most of these are outdated, and many websites won’t need to support them. Disable the formats you’re not actively using to improve your site’s efficiency.
• Remove unused resources: WordPress loads countless resources, some of which your site might not need. Removing these can speed up your site and save energy if you’re not using them.
• Internal site search cleanup: Your internal site search can create many confusing URLs for search engines and can even be used by SEO spammers to attack your site. This feature identifies some common spam patterns and stops them in their tracks. Most sites will benefit from experimenting with these optimizations, even if your theme doesn’t have a search feature.
• Advanced: URL cleanup: Users and search engines may often request your URLs using query parameters, like ?color=red. These can help track, filter, and power advanced functionality – but they come at a performance and SEO ‘cost.’ Sites that don’t rely on URL parameters might benefit from these options. Important note: These are expert features, so ensure you know what you’re doing before removing the parameters.

Would you like to know more about using this feature and the separate toggles? Check out the help documentation on the Yoast SEO crawl optimization settings.

Update to Yoast SEO 20.4 now

That’s it for now. Make sure to update to Yoast SEO 20.4 and optimize your website’s crawling immediately! It’s not only better for your website, your site visitors, and search engines. It also has a positive impact on our environment. Especially when you realize how many we are, if all 13 million of us optimize the crawling on our website, we can reduce the amount of energy used by a ridiculous amount. So let’s start right now!