Working around Joomla’s broken extensions updater

If you tried to update our software and got a 403, we have two workarounds for you.

If you are trying to install new versions of Akeeba Backup, Admin Tools or Akeeba Ticket System but getting a 403 error when you try to install the update here’s what to do.

Option 1. Re-enable the installer plugins

If you have installed Akeeba Backup 8.0.2 or later; Admin Tools 6.0.3 or later; or Akeeba Ticket System 4.0.2 or later: you can enable the following plugins:

Installer – Akeeba Backup Professional
Installer – Admin Tools Professional
Installer – Akeeba Ticket System Professional

This will allow you to update to the newest versions of our Professional versions.

Option 2. New version of FOF 4

Today we released FOF 4.0.5 which includes a workaround for Joomla’s broken updates. Thanks to some proactive and clever engineering we can run some custom code before you update our other extensions, just by installing FOF 4.0.5 and visiting the control panel page of our extensions. So here’s what to do:

Go to Extensions, Manage, Update and install the new FOF 4.0.5 package. If you don’t see it yet please wait for up to 24 hours.
DO NOT try to install the updates to our other extensions just yet.
Go to the Control Panel of each one of our Professional extensions you have installed: Akeeba Backup Professional, Admin Tools Professional and Akeeba Ticket System Professional.
If you are asked to enter your Download ID please follow the instructions on your screen.
You can now go back to Extensions, Manage, Update and install the updates.

How these solutions work (executive summary)

The installer plugins have been reworked in the most recent versions of our software so they do not cause problems in case we make a significant update to our frameworks AND Joomla or your server malfunctions in the course of attempting to install an update. They no longer need to be disabled for updates to work.

As for FOF 4.0.5, visiting the Control Panel page triggers some new code in FOF 4 which works around four different Joomla bugs by doing the following:

Re-applies the Download ID to Joomla’s #__update_sites table after Joomla removed it from that table (the only place the Joomla updater will look for the Download ID when fetching update information), either when applying the update site when you upgraded our software or when you clicked the Rebuild Update Sites button in Joomla’s Updates Sites page.
Retroactively applies the Download ID to any updates Joomla has already found for our extension by changing the respective records in the #__updates table (the only place Joomla will look for the Download ID when installing updates).
Clears the _system cache which prevents the previous two changes from having any effect on many sites.

About Joomla extensions update and its bugs (technical discussion)

We have been reporting the issues in the Joomla extensions updater since 2014 when we started using it. We have been reporting the bugs in the Joomla extensions installer for much longer.

Reporting this issues privately led nowhere. When we reported them in public on march 6th, 2021 we were met with open hostility. Worse yet, some people in the Joomla organisation seem to have launched a smear campaign against our company and me personally. They are falsely asserting that the update issues are caused by our own code.

In an effort to prove I am not an elephant and that I really know what I am doing I am doing two things today.

First, I am releasing the new version of FOF 4 which works around Joomla bugs. The code is open and so is the GitHub repository. Any developer worth their salt can see what I am doing and read the comments as to why.

Furthermore, I am providing a deep-dive technical write-up of the bugs in the Joomla extensions installer with links to the code commits, an explanation of what is going wrong and the solution for each of them.

Any developer reading the analysis below will get a much better understanding of how things work under the hood and will realise that all these bugs with a major impact to people’s ability to install update are for the most part quick 2-5 line fixes each. All of them can be solved over a long weekend. That’s really all it takes. Joomla is dragging its feet for a decade because they wouldn’t let someone (I had volunteered, several times!) spend a long weekend to address them. Isn’t that preposterous?

A birds-eye view of how Joomla extensions update works

Joomla added support for extension updates in Joomla 1.6. This was the work of Joomla co-founder Andrew Eddie and then-lead-architect Louis Landry in the first and last case where the Joomla project paid two developers to work on the CMS — it has been unpaid volunteers before and ever since. Their work was treated as something holy, not to be touched too much, which is what has caused chronic problems in the Joomla extensions updater.

But let’s take a step back and start from what happens when you install an extension manually, either for the first time or when upgrading it manually.

Joomla stores information about each installed extension (package, library, files package, component, module, plugin or template) in the #__extensions table. Note that #__ is a Joomla convention which means “the common table naming prefix as defined in your configuration.php”. We will be using it throughout.

Each extension can have one of more update sites. The update sites are XML files hosted publicly on the web. Their location is defined in the XML manifest file of each extension per https://docs.joomla.org/Manifest_files#Update_servers

When you install or update an extension (note that installing an update and installing an extension afresh uses the same code in Joomla), Joomla inserts or replaces records in the #__update_sites table for the extension it is installing. This is done through the extension/joomla plugin’s (plgExtensionJoomla) onExtensionAfterInstall event handler, see https://github.com/joomla/joomla-cms/commit/05cdddafe2eccb335a757080c4035a4419090fa9 By the way this is why disabling this plugin breaks extension updates and the reason why have a workaround in our software to install missing update sites manually (i.e. in case you unpublished this critically important plugin!).

When you tell Joomla to retrieve updates it runs Joomla\CMS\Updater\Updater::findUpdates see https://github.com/joomla/joomla-cms/commit/15a9e49c0f836960425bcfaa812bd94e9f1ba495 As you can see this loops through all of the #__update_sites records and runs getUpdateObjectsForSite. This method loads the XML file from the URL stored in the #__update_sites, finds any updates (newer version with minimum stability matching your preferences and which supports your Joomla version) and creates records in the #__updates site table.

This is the original way Joomla updates worked. What you may have noticed is that this doesn’t work with commercial extensions because Joomla is using the download URL from the update site XML file which is supposed to be common for all clients. Developers would have to publish a separate update site for each client so they can provide a custom download URL. Once you have more than a few hundred clients it becomes unrealistic. Developers either have to publish hundreds of static files on each release (meaning that each release takes HOURS instead of seconds) or they are essentially launching a Distributed Denial of Service attack against their own sites because hundreds to thousands of sites will be bombarding them with requests for a custom XML file.

Joomla extension updates, for commercial extensions

It was back in 2014, a couple months before Joomla 3.3, when I tried to contribute a solution to Joomla’s extensions updater not being fit for purpose for those of us who publish commercial extensions. I was given very strict constraints to work in. I was to make minimal changes and follow the existing way the code worked. This would lead to a subpar solution but I was fine with it, given Joomla’s release schedule at the time.

Joomla’s release schedule was based on a six month release, a Long Term Support version 3.5 coming six months after Joomla 3.3 and a new major version Joomla 4.0 six to nine months after that. I was explicitly told I would be allowed to rework the code and fix the architectural problems for Joomla 4 in 2015. This didn’t sound that bad at all.

Of course right after Joomla 3.3 was released the release schedule changed. Gone were the Short Term Support and Long Term Support release, gone was the plan to release Joomla 4. It would be Joomla 3 all the way and I would not be allowed to change the bad architecture I was forced to follow on the false promise it’d be a temporary solution for a year at most.

So, let’s see how this ended up working.

Most commercial extension developers need to append a special query string to the download URL of the extension to authenticate the user and determine whether they are allowed to download the extension installation package file. Therefore I added an extra_query column to the #__update_sites table.

Given the constraint for not making major changes, I could NOT include any GUI or developer interface (API) to manage the extra_query column in the #__update_sites table. Developers would have to do that in the post-installation code of their extensions installation or as part of their component configuration etc.

Since we have the duality of #__update_sites and #__updates I had to add the extra_query column to the #__updates table as well. When Joomla finds updates it copies over the extra_query data from #__update_sites to the #__updates record, see https://github.com/joomla/joomla-cms/commit/8880ed9d16bf97ced4543724b89cc5f33a2791d8

When you are asking Joomla to install an update it calls the Update model’s update() method, see https://github.com/joomla/joomla-cms/commit/15a9e49c0f836960425bcfaa812bd94e9f1ba495 This goes through the #__updates sites. For each record there it will read the publicly hosted update XML file again, then try to install it with the extra_query from the #__updates site. At this point Joomla calls the model’s install() method which combines the download URL in the update XML file with the extra_query to create a valid download URL for paid software. If the download URL is invalid (e.g. the extra_query information is missing or invalid) or your subscription has expire you get a 403 when trying to retrieve the update and updates fail to install.

This architecture has two obvious flaws:

There is no user accessible interface to change the extra_query directly in the #__update_sites. At this point in time there was no Update Sites page in Joomla and I was not allowed to create one due to the artificial constraint of not making “major” changes and the time constraints for making the Joomla 3.3 release (the plan communicated to me at the time was that Joomla would not allow any significant changes in the next version, 3.5, either).
The extra_query exists in two places. It’s not needed in the #__updates table. The Update model’s update() method should load BOTH the #__updates record AND the associated #__update_sites record, making management simple.

The first issue is solved in Joomla 4. I even contributed improvements to its implementation so it’s more user friendly as well.

The second issue remains unresolved.

If there is one thing in Joomla’s extensions update I would take responsibility for this is it. However, I will only take partial responsibility for it. I knew that the architecture was bad when I wrote the code. I had said so. I only contributed this bad architecture BECAUSE I was asked to follow the bad architecture dating back to 1.6 without making any major refactoring AND I was promised I would revisit it in a year. Unfortunately, the latter didn’t happen because Joomla once again reneged on a promise it had explicitly given me as soon as the term of the person who gave me the promise was up. I have been trying to convince people to change this bad architectural decision ever since.

That said, this architecture, however bad, did work for a while until more changes were made.

Other Joomla feature break the updates and how we work around them

I already mentioned that if you unpublish the Extension – Joomla plugin your extension updates will break since Joomla will no longer be installing the update sites in the #__update_sites table. There is nothing obvious about this artefact anywhere in the Joomla interface. We are aware of this issue and work around that in our software.

The Extension – Joomla plugin can add update sites of an extension on update when the location changes. When it does that it won’t copy over the extra_query from the existing update site. This means that you may end up with an update site that effectively doesn’t have the Download ID in it. Moreover, it doesn’t have a provision to remove obsolete update sites which means you might end up with two separate update sites for the same extension, only one of which would have its extra_query managed by the extension. We are aware of this issue and work around that in our software.

Joomla 3.6 added the Rebuild Update Sites button in the Update Sites page of Joomla (see https://github.com/joomla/joomla-cms/commit/350c8d4542a1d4198d0b5105254b99490021a580). Unfortunately, the way this is implemented it will completely replace existing #__update_sites records, removing their extra_query. The solution here is for that code to check if there is an extra_query and propagate it to the rebuilt update site. At the time of this writing there’s an open Pull Request by Phil Taylor to address this.

Why is this important? If you have unpublished the Extension – Joomla plugin you will see that no updates are found since there are no update site. You will, therefore, click on the Rebuild Update Sites button to fix this issue. By doing that you have reached a state where you’ve entered a Download ID in our extensions but their #__update_sites record have blank extra_query columns, therefore Joomla is unaware of the Download ID and the updates fail with a 403. The only thing we can do is always check if the update site has the correct extra_query, not just when you change your Download ID, which is exactly what the first change in FOF 4.0.5 does.

Remember that we have two tables, #__update_sites and #__updates, with an extra_query? This duality means that the Download ID (extra_query) Joomla sees when installing the update is what you had in the #__update_sites when it fetched the update information, NOT what it is when you try to install the updates. Even though we fix the #__update_sites’ extra_query column your updates would still fail because the #__updates record had the wrong extra_query. This is the second issue we address in FOF 4.0.5. We go through the #__updates records and fix their extra_query.

However, this is not all. Joomla has a “hidden” cache called _system which caches database queries. Among the other things it caches are the database queries to #__update_sites when fetching update information and the database queries to #__updates when you are trying to install updates. This can have two adverse effects.

First, you might have entered your Download ID correctly and the #__update_sites has the correct extra_query. However, when Joomla is trying to fetch update information it sees the cached query, reading the old extra_query from the #__update_sites, the one which was either blank (no Download ID) or had an invalid Download ID.

Second, even if the #__update_sites is correct, read correctly from the database and fetching the update information created the correct extra_query column in #__updates the hidden cache means that Joomla sees the out of date #__updates records with the invalid extra_query content, failing to install the update.

This is the third issue we addressed in FOF 4.0.5, clearing the _system cache if we made any change to the #__update_sites and #__updates tables.

This is a bug in Joomla, one which can be trivially fixed in two ways.

First, calling the Clear Cache in the Updates page (which calls https://github.com/joomla/joomla-cms/commit/46e21dcae4dc81b94a363a835c9ea0b9d0d936fa) should ALSO clear the _system cache.

Second, the Update model’s update() method should clear the _system cache BEFORE installing any updates. Right now it will only clear the _system cache if all of the updates you selected have been installed correctly. This is why trying to update our extensions might fail but if you first update a third party extension and THEN try to update our own extensions everything works. We are not doing anything wrong. It’s just that Joomla works in a nonsensical way.

How the first option (installer plugins) works

Instead of letting me fix the temporary solution from 2014, Joomla instead decided to implement yet another way to handle the URLs of paid extensions, allowing plugins to modify the download URL and any HTTP headers using the onInstallerBeforePackageDownload plugin event, see https://github.com/joomla/joomla-cms/commit/cce201af83e1816d05b9c54fdf566aec03b172c0

The problem is that the only information remitted by Joomla is the URL and the (empty) array of headers. You are not told which extension this is about.

This works great for developer who have exactly one paid extension — that’s most commercial extension developers — or very predictable download URLs. We have three paid extensions and their URLs were not predictable. We did fix the predictability of the URLs and tried to work back from the URL which extension they are about.

So a couple of years ago we did create installer plugins to work around Joomla losing the #__update_sites records.

Now, remember that the problem these plugins try to solve is that the #__updates and #__update_sites records may have an invalid or missing extra_query. So what we need to do is figure out how to detect the correct #__extensions table record for the extension being updated, gets its download ID and apply it to the URL.

Since this is a complicated process and we had to replicate that code across three different extensions we wrote some common code which was part of FOF 3 at the time. When you were trying to retrieve the update our installer plugin would be loaded, it would load FOF 3, let its code figure out the component of the package being updated, read its Download ID, fall back to the other two paid extensions of ours if the Download ID was missing and then apply it to the download URL passed to the plugin event. Phew!

This works great except when you run into the situation where Joomla half-updated the extension and you try to install the update again. In this case the plugin may be referring to FOF 3 but the new versions of our extensions (as of March 2nd, 2021) use FOF 4. This caused a PHP error. This error was dependent on a. Joomla having lost the extra_query column content from the #__update_sites table records for our software and b. not fully updating our software. Neither of these conditions happened during our week-long testing before release which is why many of your had a problem installing our updates.

Basically, you were caught out by a combination of Joomla bugs:

The #__update_sites record was screwed up because Joomla itself screwed it up
Joomla has a hidden _system cache you can’t clear unless you follow a very weird 18 step process (documented by us e.g. https://www.akeeba.com/documentation/akeeba-backup-documentation/update.html#dlid-change-after-update), which is a Joomla bug as we explained — at the very list purge updates should clear that cache as well!
Joomla not installing the update in full, which is a Joomla bug

We had not anticipated this combination of Joomla bugs. The next updates we published had code which explicitly anticipates for these bugs. The instructions we published on March 4th (https://www.akeeba.com/news/1744-advice-for-install-probs-with-march-2021-software-releases.html) told you to disable the installer plugins because there was still ONE case where the partial installation of our extensions could not be automatically dealt with.

Now, since you have unpublished the installer plugins you are exposed to the Joomla bugs these installer plugins were addressing. Hence the first option at the top of this article. We have rewritten the installer plugins to use only core Joomla code, without going through our FOF framework. This means that even if we were to upgrade our framework again or make major changes to it AND Joomla screwed up the installation of an update to our extensions (leaving it half-installed) they would NOT break the updates.

As a side note, people from the Joomla project blaming our code for this issue is disingenuous at best and outright malicious if we want to be perfectly honest. Had Joomla not failed to install the updates in the first place we wouldn’t end up with a mix of old and new code. This is a Joomla bug. I had proposed a better installer with full rollback support back in 2014 but I was ignored. Had Joomla not lost the extra_query from the #__update_sites table this issue wouldn’t have been triggered anyway. This is a Joomla bug. Had the Joomla extensions updater not used a query cache or had the decency of clearing it when you click on Clear Cache in the Updates page (what a misnomer, since it does NOT fully clear the cache!!!) the workaround to that problem would be much easier than having to do an extra release and give complicated instructions. This is again a Joomla bug. I don’t see how we are responsible for Joomla bugs, especially since these bugs are not even in the code that I contributed to Joomla eight years ago (on a broken promise by the project that I could rewrite it, but I digress). If the Joomla project thinks that third party developers are responsible for working around Joomla’s bugs that the Joomla project won’t fix I’d like to see that written on official letterhead — it would be quite the admission of incompetence.

How the second option (installing an update to FOF 4) works

We already touched that, but let’s flesh it out.

Remember that the #__update_sites needs the developer to populate the extra_query column of the extension being installed? Because Joomla 3 does not have a user-accessible GUI or a developer-accessible code interface to do so we are implementing this with custom code. The custom code is called ever time you visit the Control Panel of Akeeba Backup, Admin Tools or Akeeba Ticket System.

The custom code was common across our extensions so we have long moved it into our framework (FOF). This was back in 2014, in FOF 2. Of course this code made it to FOF 3 and FOF 4, the latter first released on March 2nd, 2021 and used in the current versions of our extensions.

In previous versions of the framework the custom code would only kick in if you had just entered a Download ID in the component options after having no Download ID there. This is why our 18-step instructions had steps 2 to 8. In the updated version of FOF 4 we changed this common code. It checks the extra_query column content of the #__update_sites record of our extension and compares it to what it should have been. If it differs, it updates it. This renders steps 3 to 8 obsolete; we take care of that for you automatically.

The other problem we had to deal with is that the #__updates records may still be using the old extra_query with an out of date or no download ID in it. Further to that, the hidden _system cache may have been kicking in, causing the updates to fail, despite having already updated #__update_sites. This is why our instructions had steps 11 to 17. In the new version of FOF we go through all the #__updates records linked to our extension’s update site and check the extra_query against what it should have been. If it differs, we update it. Finally, if we made any change to the #__update_sites or #__updates table we clear the _system cache and other associated Joomla caches, rendering steps 10 to 17 obsolete; we take care of that for you automatically.

This means that the 18 step process now becomes a much simpler, five step process:

Go to the backend of your site
Go to our software's main page e.g. click on Components and then on Akeeba Backup or Admin Tools or Akeeba Ticket System Professional depending on which of our software you have.
If you have more than one Professional version of our software installed repeat the previous step for each one of: Akeeba Backup Professional, Admin Tools Professional, Akeeba Ticket System Professional.
Go to Extensions, Manage, Update.
You should now be able to perform the updates to our software

As it’s very clear to anyone who actually read this article to the end, we wrote a workaround to Joomla’s issues regarding its built-in extensions updater, the one which we are forced to use since 2017 and which Joomla declines to fix sine 2014 when I first contributed a quick, imperfect, temporary solution for paid extensions updates. The fact that Joomla still uses this temporary solution is, frankly, mortifying but what can you do?

There are more bugs, of course

This is not the total amount of Joomla bugs with regards to extension updates. There are other bugs we work around or are impossible to work around. Let’s discuss them.

When a package type extension, that’s nearly every Joomla extension these days, stops including an extension (e.g. a plugin, module etc) Joomla does NOT uninstall this removed extension. It leaves it behind. In the case of modules and plugins this can cause major problems since this code may run anywhere on your site. If the rest of the extension is updated but the discontinued module or plugin is not you might end up with a mix of new and old code which fails with a PHP fatal error. This is also something that happened with our updates on March 2nd. We call this a Joomla omission rather than a bug because while it’d make sense for Joomla to remove extensions no longer included in a package it’s never been documented that it should do that.

We address this issue by manually uninstalling obsolete extensions when you install an update to our software. For good measure we forcibly delete their folders when you install an update so that they don’t get in the way. For plugins this is more complicated because they are loaded BEFORE anything else and long before the pre-update code executes. This means that at worst the first update attempt will fail but the second one will not. This was added since Akeeba Backup 8.0.2, Admin Tools 6.0.2 and Akeeba Ticket System 4.0.2.

Another problem with Joomla is that when it installs an update to an extension, e.g. a component, it might not always copy all of the files from the update package. We are already dealing with this annoying issue (which has existed since 2007 but is proving really hard to reliably reproduce so it can be positively fixed) by having post-installation code which goes over the extension files and copies them over if the installed file does not match the file included in the package.

We had forgotten to work around this Joomla bug in FEF 2.0.0 which caused some problems using our earlier March 2021 releases. We have since published an update to FEF 2 which includes our workaround to Joomla’s annoying update installation bug. This is a bug we have been privately reporting to the Joomla project for TEN YEARS, if you are wondering.

Joomla does not clear the op-code cache (PHP OPcache) after installing an update. This means that that update is installed but PHP is not loading it. This can cause various issues, up to and including inability to access your site.

We have included a workaround for this issue since Akeeba Backup 8.0.1, Admin Tools 6.0.1 and Akeeba Ticket System 4.0.1. We reported this issue to the Joomla project on March 6th, 2021 when we first determined that it does indeed happen and is perfectly reproducible. The bug report is the one which was met with open hostility and threats so don’t hold your breath about anything being done about it.

Finally, there is an issue in Joomla 4 when Joomla finds updates for a package extension and some of the extensions included in it. Our software is delivered as “package” type extensions which include several extensions: a component, some modules, some plugins and two “file” type extensions for our frontend and backend framework (these are not “library” extensions to work around yet another bug in the Joomla extensions installer — but that’s a long story for another time).

The framework extensions have their own update sites so you might indeed see updates to them as well. What happens when you see an update for a package extensions (e.g. Akeeba Backup), FOF and FEF at the same time? If you select all three and click on Update Joomla will iterate through all three and try to install them.

Installing the update package also installs FEF and FOF included in the package. As a result, these two extensions are updated. At this point Joomla itself removes the #__updates table records for these two extensions. So when Joomla’s extension updater tries to install the next update in its list, e.g. FOF, it does not find the #__updates table record. Instead of skipping it over or failing gracefully it simply errors out. This is very clearly a Joomla bug.

We do not think that the latter three bugs are reasonable, nor is blaming third party developers for them. These bugs occur in Joomla’s own extensions installer or updater code. They are trivial to fix. When we are being accused that we are doing something “strange” in our extensions to trigger them when they can be reproduced TRIVIALLY with blank packages that have ZERO code in them it’s pretty obvious that there's incompetence or malice involved...

Finally, on a personal note

I have spent countless hours dissecting the extensions installer and updater code and I know its pain points. I’ve spent a decade trying to have Joomla let me fix these darned issues. The Joomla project has consistently declined, on the pretext that it’d be a major change that has to go into a major release. All the while they have introduced many major changes, breaking backwards compatibility, making this point if not a joke than at least moot. If you don’t believe me try installing any complex extension last updated 2012, when Joomla was still in version 3.0, and tell me if it works. Of course it doesn’t. There’s no backwards compatibility to speak of. It’s an empty promise.

I have shared most of this technical analysis in private with several crops of Joomla leadership and core maintainers. Some of them are still active at the time of this writing. Nothing ever happened. All the while I had to suffer every time I publish an update to my software because I know that the next 2-3 days will be spent doing endless hours of support about Joomla bugs I’ve long reported and never seen fixed. I was doing that consistently for a decade even though this was causing loss of trust with my clients and loss of profit.

What I got in return for my troubles was open hostility. Any Joomla bug reported by yours truly, or at-mentioning yours truly or using our software as an example is quickly written off as us doing something “weird” with our code. Never mind that our code doesn’t even run at this point in time or that you can trivially reproduce the issue with blank, dummy packages that have zero third party code in them.

I am sick and tired of the toxic environment surrounding contributing to Joomla. I was in it for helping people these past 12 years. I am done. I have already stopped contributing since 2020 and I am never coming back. All GitHub emails from the Joomla CMS project are immediately deleted at the mail server level so I don’t even get notified when I am at-mentioned. I am now here only to make a living selling subscriptions to software, be it Joomla extensions or WordPress plugins. When Joomla dies I won’t shed a tear. We’ll reimplement our site in WordPress and it will be business as usual.