A much-improved internal linking tool in Yoast SEO Premium: What’s new?

Our users often cite the redirect manager as their favorite tool in Yoast SEO Premium, but the internal linking tool is also high on the list. For months, we worked hard to improve the internal linking tool making it smarter and faster. I sat down with our developers Natalia Shitova and Hans-Christiaan Braun, two of the developers involved, to help give you the lowdown on this new internal linking tool and the magic that happens behind the scenes. 

The internal linking tool in Yoast SEO Premium

For those not familiar with the internal linking tool, a short recap. This tool analyzes the content on your site to automatically give suggestions for links relevant to the post you are writing at the moment. It saves you a lot of time and maybe even your sanity. You don’t have to remember all those hundreds of posts you’ve written over the years. The internal linking tool surfaces them if relevant to the content you are writing now. After that, simply copy the link and paste it where you want it to appear. Easy peasy!

The initial version of the internal linking tool worked fine, but we saw a lot of potential enhancements. For one, we wanted to make it much faster. Plus, we wanted to make it more accurate and reliable. We noticed the tool didn’t always give the best answers. Sometimes, it’d miss a couple of posts or suggested less relevant ones. That’s just not an option for a tool people rely on, so we wanted to fix that. 

The new version in Yoast SEO Premium 14.7 won’t look too different, but the way it works has totally changed — for the better! 

Here’s the new internal linking tool in action:

Understanding content to suggest links

Internal linking is incredibly important and that’s one of the reasons we built the internal linking tool. For this to work well, the tool must know about all your content and understand it as well. The initial version of the internal linking tool was awesome, but it didn’t always give valuable suggestions. How come?

According to Natalia, looking for the most relevant posts to link to is a challenging task for both a human and a plugin. “You need to have a good understanding of what your current post is about. Plus, you need to keep a detailed overview of what all other posts on your website are about. In the new version of the internal linking tool, we attempted to make the analysis as intuitive as possible. We tried to make the plugin work in a way a human would — provided one with the perfect memory of a computer.”

How it was before

The initial version of the internal linking tool was one of the first WordPress SEO tools to automatically suggest relevant links based on your content. But how did that work?

Natalia explains: “The first version relied fully on topics of separate posts and did not take the site-wide context into consideration. The plugin would determine the most often used words and word combinations of all posts on the website — we call them “Prominent words” of the text — and then simply match those: the more words and word combinations matched between two texts, the more “similar” those texts were deemed and the better internal linking suggestion one was considered for the other.”

That sounds like a very logical process, but why wasn’t this approach optimal?

“The problem with this approach,” says Natalia, “is that it counts all words as equally valuable when it comes to determining the topic of the text, but that’s not the case.” 

“Let’s take yoast.com as an example. Almost every post that we write is about SEO, WordPress, Yoast plugins, website optimization. Are these words important when we want to describe the topic of the post on its own? Of course! Are they worth looking at if we want to determine if two posts from yoast.com are related? No. To give better internal linking suggestions, we need to focus on often-used words but those that are more specific to a group of blog posts. In the case of yoast.com, it’s better to look at words like “copywriting”, “site structure”, “structured data”, etc. This is what we did in the new version of the internal linking tool.” 

Old vs new: a quick comparison

A lot of the behind the scenes improvements are about understanding the content on your website better. We’ve optimized the way we collect words from posts. In the initial version, we used only the text of the article. The new version also looks at the title, meta description, taxonomies and eventually keyphrases. But the most important change is that we now consider the entire website as context when creating internal linking suggestions. 

Natalia explains how that works: “When we try to determine the topic of a post not only do we check how often you use a certain word in this text, but also how often you use that word in other posts. These two measures together help us see how specific this word is for this post and whether it is valuable to use it for internal linking suggestions: a few words that you use a lot in the current post and rarely elsewhere will likely capture the specific topic of the post very well.”

Improving the internal linking tools started with research and building a demo to prove that it would work. Natalia: “For this demo, we took ten yoast.com texts strongly related to each other — all of them covered the topic of branding. We also selected ten control texts that were not specifically related to each other or to the branding texts from the first group. We ran the texts through both old and new internal linking algorithms and visualised how likely the algorithms would suggest every two texts to each other. As you see on the plot below, the old algorithm did not do a very good job. Strongly related texts (Text 1 through Text 10) were only considered slightly better linking suggestions for each other by the algorithm [red triangle], than unrelated texts (Text 11 through Text 20) [blue square]. The new algorithm definitely preferred linking strongly related texts over unrelated texts.”

A note on the technical side of things

Our linguists are very apt at developing language-related tools. The team saw the new internal linking tool a nice challenge. Asked about the technical side of improving the internal linking tool, Hans-Christiaan noted: 

“Yoast SEO builds its own index of posts on your website based on the prominent words of these posts. We use the prominent words of the current post as a kind of search query to find a list of relevant posts from this index. To get this done, we used the tf/idf metric, which is often used by NLP applications for this sort of task. We knew it would work from a theoretical AI point of view, but it was challenging to get it up and running within a plugin that runs in someone’s browser — without slowing its performance. For this, our solution was to get data via indexables.”

The new internal linking tool is one of the first features built on top of our new indexables data gathering system introduced in Yoast SEO 14.0. But how does this work? 

Hans-Christiaan explains how the team used the indexables to gather data: “In a nutshell, indexables allow you to store information that you need in one place. You can then request it when you need it with no hassle. You can imagine that with the mechanism that we discussed above, we’re sending a lot of requests. If we store everything in one place and make it easy to request, we can make sure the entire algorithm works as fast as it possibly can. We noticed a 12x speed increase for the internal linking site-wide analysis on some testing environments. It’s hard to believe that before, it sometimes took our users hours to complete.”

In this post by Hans-Christiaan you can read more about the technical side of improving our internal link suggestions.

Improved internal linking: better and faster suggestions!

By rebuilding the internal linking feature using proven NLP processes and revolutionary technologies like indexables, we’ve created an essential tool that helps you quickly and surely build an awesome site structure. The tool provides you with great suggestions and does so instantly. All you have to do is index your content and write your post: you’ll see the suggestions appear immediately. Awesome, right?