Use robots.txt and .htaccess rules to control crawling and indexing

Yoast SEO is set up to make crawlability easy out of the box. But, what if you have some specific technical challenges that need to be fixed with a robots.txt or .htaccess file? We cover those cases too! With the Yoast SEO tools, you can edit robots.txt and .htaccess files. By doing that, you can gain even more control over how your website gets crawled!

Why do I need to edit robots.txt or .htaccess files? 

While we provide this option in the Yoast SEO plugin, we recommend you use it only if you know what you are doing. In most cases, it is enough to block search engines from particular pages by adding meta robots tags or robots HTTP headers. However, we recognize that some cases may face complex technical challenges that require the editing of the robots.txt file. For those cases, we made editing robots.txt and .htaccess files possible. 

Rank with your high-quality pages  

On large or complex sites, search engines may sometimes limit the number of URLs they’re willing to crawl and consume. This limit is sometimes known as “crawl budget”. Based on your website’s size and reputation, they may decide to allocate more (or less) resources to crawl your pages. If you have some pages on your site that need some SEO work, you might want to block crawlers from accessing them. In that way, you give crawlers more room to access the pages you want to rank with.

Block access to the pages you don’t want crawlers to access 

There are also pages on your website that do not contribute to your ranking, but they are necessary for the functioning of your website. For example, you may have internal search results, admin pages, account pages, shopping carts, etc. You can block crawlers from accessing these pages with the robots.txt file. Doing that may improve your site’s technical SEO, and contribute to better rankings! 

Manage server settings

Being able to edit your .htaccess file lets you control some aspects of how your server responds to requests. You can manage redirects, file handling options, and much more. This can be helpful in advance setups, or where it’s difficult to make changes on the server itself.

How does it work? 

Depending on the server you use, you can edit your robots.txt or the .htaccess files in Yoast SEO > Tools > File editor. 

For more information see:

How to edit .htaccess files with Yoast SEO

How to edit robots.txt files with Yoast SEO