The exciting new technology that is indexables

Yoast SEO 14.0 ushered in a new age for the Yoast SEO plugin. In it, we’ve rewritten our entire metadata engine and built an abstraction called indexables. We believe this rewrite will boost WordPress + Yoast SEO as an excellent platform for SEO for many years to come! Please let us walk you through the technological advancements and their meaning.

Think like a search engine

SEO has always been about thinking like a search engine. Search engines want to retrieve as much information as possible and use it to provide searchers with the best possible answers to their questions. This means we constantly have to ask ourselves; how does a search engine treat information? 

Information on the web is addressable via URLs. Anything with a URL could be discovered, scraped, indexed, and shown in the search results. WordPress has posts, pages, custom post types, categories, tags, custom taxonomies, different types of archives, special pages, and maybe even more types of content. Do you think a search engine like Google cares about that? It doesn’t. It just looks for things with a URL that can scrape and index.

Better information architecture for WordPress

From an SEO perspective, any type of page in WordPress is simply an indexable object. This is the basic intuition that has led to indexables. At its core, indexables is just a database table that contains metadata and URLs for all indexables on a site. The abstraction normalizes the information architecture for any type of page in WordPress and makes its metadata directly queryable. On top of that, we can now quickly and economically relate different indexable objects to each other and other things, such as links, redirects, attachments, and perhaps even schema markup.

This is a huge deal. Links, for example, are references from one information object with a URL (an indexable) to another with a URL (another indexable). We’ve been storing links for posts in WordPress for quite some time. We can now start doing that for any type of page in WordPress. And by relating them to indexables, we could create a graph of all information on a site inside WordPress. That would enable us to provide users with valuable insights about their site’s SEO.

Direct benefits of indexables

But the direct benefits are great. With Yoast SEO, we seem to be hitting massive performance gains, have dramatically reduced the cost of change for our metadata functionality, and can deliver much nicer and more stable APIs for third-party developers who want to integrate with us. Last but not least, Yoast SEO is now fully ready for headless WordPress. Let’s quickly go over these benefits and how they came about.

Performance gains

WordPress offers standard APIs to store metadata for content resources such as posts and terms. The post meta and term meta APIs implement an entity-attribute-value (EAV) model. To WordPress’s advantage, this is a very flexible and open model that allows any developer to easily add a custom field or piece of metadata to a post or category. However, this model also quickly becomes quite slow to query, especially on big sites with many custom fields. It is part of the reason all SEO plugins tend to slow down a website. They have a ton of metadata to output, all of which needs to be queried separately inefficiently.

Moving to a custom table, we move from EAV to a relational model for fetching SEO metadata, making it much easier and more efficient to query metadata for any page in WordPress. This advantage becomes especially big when you start relating objects to each other. An indexable is now directly mapped to a term or a post. We can now get all of our data in a single query, which makes any request much faster: another big performance gain we get from storing the URL with the indexable!Because of this, breadcrumbs can now be generated in a breeze, whereas in the past, this used to take quite some expensive calculations. In a future release, we also plan to generate our XML sitemaps straight from indexables. This will probably make our XML sitemaps the fastest and most reliable on the web, which is especially good news for large websites, for whom generating XML sitemaps has always been a pain.

Lower cost of change

In Yoast SEO 14.0, we completely rewrote our front-end code and structure. We’ve moved away from a PHP 5.2 compatible procedural style architecture to an object-oriented architecture that makes extended use of all features that become available with higher versions of PHP (5.6+). We’re starting to use namespaces, Symfony’s dependency injection container, and strict separation between pure PHP services and stateful objects like ORM models and value objects. 

These tools and strategies have helped us structure our code in a way that makes it much easier to reason about. The benefits of this are legion: 

• The code becomes much easier to debug.
• We can change code more easily because code concepts are better defined, and it’s clearer what part of the code is responsible for what.
• The consequences of code changes are easier to oversee, making unforeseen bugs less likely.
• The code becomes easy to unit test, which helps prevent future regressions even more.
• The time to change or fix something in the code drastically diminishes because of the above. This means we can iterate faster on new features and quickly fix bugs.

We’ve experienced these benefits firsthand. We’ve seen that our development team has had a much easier time understanding the functionality. They even came up with clever and critical questions about it because inconsistencies were much easier to spot. This has already resulted in discovering some small bugs and inconsistencies that we might’ve overlooked. We’ve also seen dramatic productivity improvements, with developers often reaching twice the productivity or more when working on the new code.

Better APIs through surfaces

Figuring out how to offer a consistent API to integrators has been a longstanding challenge. Any code executed within a platform like WordPress shares the same runtime with all other active plugins and themes. Our “public” interface has technically always been available to integrators. Any class, public method, or function could be invoked by third-party code. 

Now the question for us has always been: should you use those? We’ve never been fond of plugins relying on our public APIs outside our filters and actions. This would mean we’d have to keep those APIs backward compatible forever to prevent sites from breaking. That would slow us down and would increase the cost of change again. For WordPress, it makes more sense to treat all APIs as public. After all, it is the platform that is running all the code. But how about plugins? Treating any plugin like a standard library you control like any other dependency feels off.

We now have a very elegant pattern for offering developers a sustainable API contract with Yoast plugins. We call it surfaces, and we hope other plugin authors will also start adopting this pattern. A surface is an object we explicitly expose for third-party use and promise to keep backward compatible. 

To that end, we’ve introduced a global YoastSEO() function. It has direct access to our DI container and returns a top-level surface, which exposes other surfaces via magic getters (which might, in turn, expose more surfaces). Through surfaces, we can expose pure PHP services, repositories, and factories straight from the container or create more toned-down objects that expose only a subset of the behavior. While globally available, each surface is an object you can inject into your code, making it possible for integrators to keep their code decoupled from Yoast SEO. Go check out our surfaces!

Headless WordPress

Yoast SEO makes generating metadata output for any frontend request easy and efficient. Because of this, we’ve decided to add a simple metadata endpoint that will make it possible for site builders to use Yoast SEO on headless installs. You can now simply fetch the metadata for a page via the REST API and output it anywhere you like.

How to generate indexables for all your pages?

Yoast SEO will work best if it has an indexable stored for every page on your website. The plugin automatically generates them over time, which might not cover everything. So on top of that, we also provide tools to simultaneously generate indexables for the entire site. Here’s what we provide:

• Yoast SEO adds or updates an indexable whenever you save content or metadata.
• Yoast SEO generates and stores an indexable whenever a page is visited and doesn’t have an indexable yet. This only needs to be done once per page. After that, we’ll use the indexable, and the page load will be much faster. Any page that gets traffic will eventually have an indexable.
• We’ve added a tool to the Yoast SEO settings to allow site administrators to add indexables for their entire site at once.
• We’ve added a WP CLI command to allow site administrators to add indexables for their entire site via the command line.

A look into the future

What we’re seeing today is truly only the beginning. Believe it or not, the real benefits of indexables are in what we can build on top of it. We can now start delivering features that will give a whole new meaning to sitewide SEO. While not giving too much away here, the first thing we will build on top of this is a completely revised version of our internal linking algorithm. We’ll soon be able to make our internal linking suggestions sitewide and much more accurate by building a search engine into WordPress on top of indexables. Sounds a bit crazy, right? Yet, with indexables and some help from our computational linguists, we can make all this relatively simple. More on that later. Check out the API documentation and stay tuned for the upcoming Yoast SEO release and many more goodies!