Yoast SEO 14.0 was a release that needed quite a few patches so far. We’re sorry about that. In building Yoast SEO 14.0 we made a mistake: we bypassed wpdb. That caused issues and errors for some users when they tried to upgrade, which we’re very sorry about. This post explains the mistakes we made, and our reasoning. I hope it explains a bit of how this happens, and, for the developers among you, prevents you from making these same mistakes.
Let’s start with what we got right, which paves the way for an exciting future:
We started using a custom database table for our metadata. As posts tend to have a lot of SEO metadata, this sped up our frontend performance significantly.
To access that data, we started using an ORM (which stands for Object Relation Mapper). Think of that as a more convenient way of mapping the data you use in your application to the data in the database.
We didn’t remove the data from wp_postmeta, we just copied it to where we could access it most efficiently, but left it around (and even updated it there) for when other people relied on it being there.
The vast majority of websites upgraded with no problems, errors or issues.
Note that all the stuff described below was done because a portion of our users was affected. At the same time, the vast majority was not affected, and in fact, saw an immediate speed improvement on their sites.
Picking our ORM
In choosing our ORM we were severely limited as we had to support PHP 5.6. The problem here is that while WordPress has only recently increased its minimum requirements to PHP 5.6, the rest of the PHP world considers that version ancient history. We would have preferred to use Doctrine, but unfortunately, due to this PHP 5.6 requirement, this was unfeasible. Which is why we ended up with Idiorm. It’s not perfect, but it was still a lot less work than building our own. We also started using Ruckusing for database migrations. This allows us to easily set up and change our database schema.
Idiorm and Ruckusing both had a dependency though: they are built on top of PDO, a way of connecting to the database that is slightly different from the mysqli and mysql interfaces WordPress normally uses in wpdb. And this is where we made the wrong decision: we decided to open a second connection to the database, using PDO, bypassing wpdb.
Releasing 14.0
Timeline of events
As we discovered issues and edge-cases, our team worked tirelessly to diagnose, debug, and release patches. This meant that we launched multiple small updates, which resulted in some users having to (re)index their sites multiple times. We’re sorry about the inconvenience that caused.
We thought we were ready to release our 14.0 update. We’d tested this extensively, across multiple configurations and systems, and had asked hosts and other plugin developers all over the world to test along with us. We can’t always catch every obscure edge-case, but we weren’t really expecting a lot of problems. Well. We were wrong. Unfortunately, when you have the number of installs we have, even a problem that affects 0.1% of our users becomes an iceberg.
It turns out loads of people have very non-standard DB connections. Ranging from setups where DB_HOST or DB_CHARSET aren’t defined, to larger issues where they are running encrypted database connections, which of course our PDO solution didn’t know about.
We encountered lots and lots of small little issues. We could have fixed all that, potentially, but we’d be spending a lot of time doing that and new issues would inevitably keep popping up. So on Tuesday we made a very hard call. We released version 14.0.1 with fixes for all the things we could fix immediately, and then started working on the bigger and more complex challenges, which became version 14.0.2.
The fix, 14.0.2
So we started working on a better solution: still using Idiorm, but passing it all through wpdb. We’ve forked Idiorm and we’ve “semi”-forked Ruckusing: both now connect to the database through wpdb. This was quite a bit of work, but in hindsight, it wasn’t as bad as we thought it would be. Had we known we could do it “this easily”, we would have immediately made that step and used this approach – but this is new territory; there are few WordPress plugins which extend the platform in this way, at this scale, with this number of unknowns.
The results
The upside of not connecting through wpdb was that it had none of the wpdb overhead. On the other hand, we no longer have to open a connection. In our tests, we see mixed results, but overall, not a whole lot of impact. The added positive side effect is that we can once again inspect our queries through our favorite development plugin, Query Monitor.
Next problem: updates not working as they should
The next problem we ran into was a vexing one: some people were updating to Yoast SEO and they had all the new files, but they were reporting errors with files that no longer existed in the plugin. Note that this does not happen if you update through the WordPress admin. That was baffling. In the end, it turned out that the problem was that some of the WordPress site management tools simply copy files over the old files, instead of removing the directory entirely and replacing it with a directory that contains the new version. This caused errors, but it took a while to diagnose what was going on. That’s because this isn’t how modern hosting or plugin systems should behave, so it never crossed our mind that they would behave in that way. We got there with the kind help of Ipstenu and Otto in the WordPress forums team, though. Note that the upgrade process for plugins is dealt with by WordPress core, not by our plugin, so this was out of our hands. Regardless, we had to find a workaround.
Because of how Ruckusing, which I mentioned above, works, we included files in a directory to run migrations. If the old files in that directory didn’t get deleted, they’d reference classes that we had since removed, and everything breaks. To fix that, we simply renamed that directory, which prevents this from happening for now, and released 14.0.3, this afternoon.
wpdb bites back
Lazy loading
When you update to 14.0 or later, we add a row to our indexables table for every post, page, tag etc. on your site. We can either do this in one go, through our indexing process, or we do it lazily. Every time a URL for which there is no indexable is loaded, we create it, and store it. The next time you open that URL it can immediately take the data from our table and thus be faster.
And then came the final problem, on Thursday 30th April, which caused us to do yet another update. Turns out that when you wpdb::prepare a query which has null in it, wpdb escapes it into 0, which turns out to be a long standing issue in WordPress core. This is typically something we would have caught if we had not fixed this in a patch release, so had we made our decision to use wpdb sooner, we wouldn’t have caught this problem this late. Unfortunately, now it meant that we were storing the wrong values for noindex, nofollow and other robots meta tags in the wrong way. So… We had to do another patch release, 14.0.4 and re-index everybody’s sites to make sure we stored the data right. The re-indexing is possible because we left all the data in wp_postmeta, a decision I’m now very thankful we made.
Is it worth it?
After reading all this, you might think: is this worth it all? And despite the fact that I and a part of our dev team have been working incredibly hard the last 3 days, the answer is still a resounding: yes! We’re laying a foundation for a much better future and we’re making sites instantly faster. I hope we’ve caught all the major issues now, something I of course can’t guarantee, and once again, I’m sorry if we caused you trouble!
11 September 2024
Learn how to start your SEO journey the right way with our free webinar. Get practical tips and answers to all your questions in the live Q&A!
All Yoast SEO webinars »
13 Responses to Yoast SEO 14.0.x; or “Why you should never bypass wpdb”
13 Responses to Yoast SEO 14.0.x; or “Why you should never bypass wpdb”