Today Joomla released a new version patching what they called a vulnerability. They said that it (also) affects FOF, our backend framework. However, what they addressed is not really a vulnerability. We had already explained the how and why to the Joomla Security Strike Team (JSST) since January 27th at 12:11 PM EET. Half an hour later we were told in no uncertain terms that it will still be considered a vulnerability. We strongly oppose incorrect information and would like to set the record straight.
The executive summary
The Joomla Security Strike Team (JSST) issued a vulnerability report about Joomla 3.2.0 to 3.9.24, saying it affects the random bytes generator in FOF 2.x (end of life since 2016).
However, the issue they have reported is not a security vulnerability, it's a fix to one. It's not something we invented, either. It's an implementation of RFC 4086 which was issued in 2005 and is still in effect. Moreover, it was Joomla itself which provided the original version of the code since Joomla 2.5.3.
There is no reason for alarm using our software. Your data has never been at risk. As a matter of fact, FOF 3.x had not been using this code anyway since late May 2020. As for older versions of FOF 3.x and even FOF 2.x — the latter included with Joomla since Joomla 3.2 — it could never run this code anyway, rendering the whole debate moot even without going into the fact that the code in question is a fix to a potential security issue instead of a security issue in itself.
We did explain these facts to them on January 27th at 12:11 PM EET, allowing them to review the facts and conclude that this is not a security issue. Less than half an hour later, an inadequate amount of time to review and discuss the facts, they came back telling us that they will consider this a security issue regardless of the evidence we provided to the contrary. We explained that in this case we would post publicly about our findings and they agreed to that. Hence the existence of this article.
Please note that this is a long, technical read. The takeaway is that you were never at risk and today we are releasing new versions of our software which do not include the code in the centre of this debate as we had planned since 2018.
The alleged vulnerability
A security researcher contacted the Joomla project, claiming that the code to provide random bytes in the end of life version of FOF included in Joomla and used to generate the Two Factor Authentication random keys looked like it was “a very obvious example of overengineering” and that it does “something that looks really wacky”. These are his exact words, quoted verbatim, them being the complete assessment of the code in question.
His report was based on the following assessment which we copy verbatim for your entertainment: “While practically attacking this is probably difficult and would require a very thorough analysis, this does not look confidence inspiring.”
He could not provide a theoretical attack mode or any other rational explanation of why or how this code is vulnerable in his opinion.
Creating randomness is a hard problem
When developing the Two Factor Authentication code for Joomla 3.2 back in 2012 we were in need of a cryptographically secure random number generator to ensure that we can generate a truly random secret key. The secret key is shared with the user and is used to generate the time-based six-digit codes. The requirement was that the source material that produced the key must be random and unpredictable. This combination is called “cryptographically secure” randomness.
Nine years ago PHP did not have a cryptographically secure random number generator built-in. Therefore we used three different approaches to get truly random numbers. The first two are using PHP’s OpenSSL or mcrypt extensions, whichever is available on the server. These are cryptography extensions which provide cryptographically secure random number generators.
If neither is available we needed to create truly random numbers by mixing using sources which are not guaranteed random, such as the system’s /dev/urandom, PHP’s uniqid() method and PHP’s mt_rand() pseudorandom number generator. The JSST didn't get the name of mt_rand() right in their report, either. There’s a great explanation of the mathematics of how that works in this blog post by Anthony Ferrara, the author of PHP-CryptLib and other security-oriented PHP software (and a person who actually understands the mathematics that underpins cryptography). This article also explains the intricacies of random versus unpredictable and how non-stochastic machines, like computers, can generate computational randomness.
The code used to do this randomness mixing was not even written by us. It was written by Joomla itself right before the Joomla 2.5.3 release. It was part of the JCrypt package. We saw that it indeed followed Anthony’s advice to make use of RFC 4086, a Best Current Practice document released in 2005 and not withdrawn since, i.e. it’s still valid. As a result we copied that code inside FOF 2.x since, at the time, FOF 2.x was also targeting older versions of Joomla which did not have the JCrypt library.
This code IS NOT INSECURE by any means. The JSST deciding to unilaterally call RFC 4086 vulnerable is hubris for this implies that they know better than the cryptographers who wrote, validated and voted on that RFC. How can a team without any cryptographers and mathematicians have an opinion on a scientific field they have no relation with? It's like a group of butchers disputing advice given by a team of highly trained surgeons.
That said, this code is something fairly bad: unused and obsolete. But we’re getting ahead of ourselves.
This code was never actually used
As we said before, this code is only a fallback. It’s only called when neither OpenSSL nor mcrypt extensions are present.
However, Two Factor Authentication in Joomla stores all per-user information encrypted. Wanna guess how it encrypts this information? Using either OpenSSL or mcrypt (newer versions of PHP only support OpenSSL; mcrypt support was dropped).
Therefore, you will never get to use the slow, obsolete fallback code for random byte generation because Two Factor Authentication will only work if you have the PHP OpenSSL extension installed. When you have the OpenSSL extension installed it’s what is used to generate random numbers as well.
Therefore, the code in question is inert (unused). It never executes.
As to why it was included. The first iteration of Two Step Verification didn't encrypt the data going into the database. Therefore it was conceivable that it would run on a server environment which doesn't have either of the mcrypt or OpenSSL PHP extensions. Hence the need for a fallback that could create computational randomness with guaranteed unpredictability out of randomness sources which were NOT guaranteed to be unpredictable. That's exactly what RFC 4086 is all about. Therefore its implementation was solving a potential security issue (predictable randomness). Before Joomla merged the Two Factor Authentication implementation they asked us to encrypt the data in the database — we wrote and contributed the Two Factor Authentication feature in Joomla 3.2. As a result the requirements changed and the code could no longer run on servers which had neither the mcrypt nor the OpenSSL PHP extension installed.
The same point about whether this code runs applies to all of our software which uses random byte generation, be it Akeeba LoginGuard (the much superior evolution of the Two Factor Authentication code) or Akeeba Backup (when encrypting your settings or your archives), or anything in between. We only ever use this code in conjunction with OpenSSL, therefore the only code that was being executed was OpenSSL’s random number generator which is cryptographically secure — at least in the PHP versions we support, but we’ll get back to that later.
As a result saying that code that can never run is vulnerable is a leap of logic. We are not sure why the JSST didn't spot that right away. Did they not read Joomla's code before making an assessment on its security? This is disconcerting.
PHP 7 can create randomness much more easily
Starting with PHP 7.0, PHP comes with a cryptographically safe random number generator (CSPRNG) in the form of two functions called random_int() and random_bytes(). The first generates a random integer between 0 and 2,147,483,647 on 32-bit versions of PHP and between 0 and 2^64 - 1 on 64-bit versions of PHP. The number is too long to type, let alone read. The latter creates a string of randomly generated bytes using the same CSPRNG. Neither is available in any PHP 5 version.
Joomla was notified in 2015 that they could finally use these two methods in PHP 5 using a polyfill, i.e. a third party library which provides PHP 5 implementations for random_int() and random_bytes(). The third party library used is paragonie/random_compat, an industry standard library for this purpose. Therefore Joomla replaced the code in its JCrypt library — the same code we had copied in FOF 2.x — with a call to random_bytes(), relying on paragonie/random_compat to provide a backwards compatible implementation on PHP 5.
That third party library uses a similar approach to the code in FOF 2.x that’s being replaced in Joomla 2.5.25. Namely, it will try to use OpenSSL, mcrypt or /dev/urandom. It does not have an RFC 4086 implementation.
The latter is important on older Linux kernels, before 4.8 (see Myths about /dev/urandom). If you are on a really old PHP version chances are you are also on a very old kernel. This could indeed cause /dev/urandom to reuse the entropy pool, possibly resulting in less random bytes being generated. Of course this is a theoretical issue inasmuch the raw random data is never shared publicly in a context that would compromise generated key material, i.e. the use case of Joomla itself and our own extensions. More on that later, when we discuss if this can be exploited.
This is also why we decided not to change our code until we could guarantee that our software would only ever run on PHP 7. We had seen too many live servers with older Linux kernels, making OpenSSL a better source of randomness than /dev/urandom, even considering the problems in PHP's implementation of the OpenSSL extension in older PHP versions.
We had already replaced that code anyway
We had already identified that the RFC 4086 implementation in FOF 2.x and 3.0 to 3.5 was not used when used with our software. However, FOF is used for software beyond our own so we have the reasonable expectation that someone, somewhere, somehow is using it to generate random strings outside the strict scope we use it, quite possibly sharing raw generated data in a way that could be used to tip off an attacker. Again, we don’t do that ourselves BUT we also have a better understanding of the security implications of the code we use and can steer clear of any problem areas.
For this reason and for the simple reason of better performance (the binary compiled C code PHP is written in is much faster than interpreted PHP code) we wanted to use random_int() and random_bytes() in FOF. This required a. Dropping support for PHP 5 and b. Dropping support for old versions of Joomla. The former meant that we could use random_int() and random_bytes() without ever falling back to the polyfill code in PHP 5 which we couldn’t guarantee would be random enough on Linux kernels before 4.8. The latter is a consequence of the former; we couldn’t drop PHP 5 support without dropping support for Joomla versions which could only run on PHP 5.
This decision was made in late 2018. We did a lot of planning on how to go about it. Starting in 2019 we began the process of dropping support for these obsolete versions of PHP and Joomla. On May 20th, 2020 we made changes in FOF 3. We replaced the calls to mt_rand(), the insecure pseudorandom number generator, with calls to random_int(), the CSPRNG used in Joomla in the RFC 4086 implementation. We still kept the former login of using OpenSSL first and only falling back to this RFC 4086 implementation if OpenSSL wasn’t available until we could verify with further tests that there would be no backwards incompatible changes. This latter work got delayed more than we expected due to the Joomla 4 release schedule suddenly accelerating, forcing us to reevaluate our priorities.
In any case, FOF 3.6.0 and later are NOT using the code that Joomla claims is vulnerable but is actually not. Any claims to the contrary are simply misguided.
And we are replacing it yet again
In November 2020, after having dealt with the Joomla 4 release schedule, we started the second phase of our removing support for old versions of Joomla and PHP. We increased the minimum PHP version to 7.2 and dropped support for all Joomla versions older than 3.9. This allowed us to do a second, more thorough, round of refactoring.
Having performed further tests and finding no issues, part of that refactoring was replacing the random bytes generator login with random_bytes(). This didn’t happen between December and January since we implemented a voluntary release freeze over the holiday period. People were already at the end of their tether having spent most of 2020 in lockdown due to the global pandemic. We were not going to spoil their holiday time by making releases. So we started the second round of refactoring in mid-January 2021.
Less than 10 days later the JSST contacted us about the alleged vulnerability. We explained it’s not a vulnerability and that their contact came in the middle of refactoring all instances of that code across our software. We had already replaced two instances of this code in the backup engine and our WordPress software (we start refactoring from the edges to the core, a technique that allows for stabler software when doing massive refactors). We were just about to do the same in FOF itself.
The Joomla project didn’t agree with our well documented assessment that it’s not a vulnerability and didn’t accept us being able to release our software while making the change and marking it for what it is, a simple modernisation of the code by removing an obsolete and slower implementation. That’s how we ended up doing a coordinated release with Joomla today.
Still, we did release Akeeba Backup 7.5.3 with the change in the backup engine. Nobody noticed the change, as we had expected and told the Joomla project would be the case. Nobody even noticed the disparity between the backup engine using random_bytes() but FOF itself doing something different.
Despite all that, can this code really be exploited?
Theoretically, yes, on some older PHP versions. Practically, no, it cannot.
If you are using PHP 5.3 (any version, assuming no back-ported security fixes by your Linux distribution), PHP 5.4.0 to 5.4.43 (inclusive), PHP 5.5.0 to 5.5.27 (inclusive) or PHP 5.6.0 to 5.6.11 (inclusive) you are affected by a PHP bug which meant that openssl_random_pseudo_bytes() was not necessarily using a cryptographically safe pseudorandom number generator to generate the random bytes.
This means that the old random bytes generator code that was prioritising OpenSSL over RFC 4086 could create random data that could theoretically be predicted. But that’s the theoretical aspect. The practical aspect is that no, it can’t be exploited in practice.
The first obvious reason is that this refers to old PHP versions which have long been replaced and fixed. In fact, none of these PHP versions are supported by our software for well over a year now.
Even if you are on an affected PHP version, an attacker would need to know the internal state of OpenSSL’s pseudorandom number generator. To the best of our knowledge, PHP creates an instance of OpenSSL every time it runs and OpenSSL initialises the PRNG instance with a system-specific entropy source therefore a compromise would require knowing the exact output of OpenSSL’s random bytes generator and the sequence they were generated.
In the context of where this random bytes generator is used this doesn’t pose a problem. There are three contexts where it can be used:
- Generating a TOTP secret key. This is communicated to the user and is meant to be secret. If the secret key is divulged to an attacker they don’t need to reverse engineer the state of OpenSSL’s PRNG; they have they key to generating Two Factor Authentication codes i.e. you have been pwned.
- Generating Initial Vector (IV) data for AES-128 encryption, as used for example when encrypting your Akeeba Backup settings or your JPS archives. The IV is conveyed in the clear, so it only needs to be random, not unpredictable. The key material is generated from your password using the PBKDF2 algorithm. The static salt used in the PBKDF2 algorithm also needs to be random, not unpredictable, and is communicated in the archive. Therefore even using a non-CS PRNG in these cases is not a security issue.
- Creating passwords. In our software this an operation taking place in a single page load and the result is not communicated. Therefore an attacker cannot infer the state of the PRNG as they don’t get the results of its execution.
Things are a bit more complicated if you are using an affected PHP version as an Apache module. In this case it’s not guaranteed that OpenSSL would be re-initialised in a new page load which creates a theoretical window of opportunity for an attacker able to execute arbitrary code on your server to determine the state of the OpenSSL PRNG and infer any generated passwords and TOTP secrets as long as they can guarantee that their inference code executes right before a secret is generated.
However, if an attacker has the ability to execute arbitrary code on your site it’s FAR EASIER and FAR MORE EFFICIENT for them to take over your site than use a precarious technique which may or may not give them half of what they need to log into your site.
Therefore, even in this theoretical case there is no practical exploit for the uses cases concerning Joomla and our software.
Moreover, the point is completely moot if you are using a newer version of our software and / or an up to date PHP version.
Why did the JSST declare it vulnerable?
That’s a great question.
We don’t think the JSST has ever been aware of the OpenSSL bug in PHP or at least they didn’t tell us anything to that effect. Even so, it wouldn’t be a practical attack in the context of where and how the code is used.
The only information shared with us by the JSST was the “vulnerability report” where the security researcher wrongly identified the RFC 4086 implementation as “wacky” and “overengineered” while admitting they had no way to see any kind of practical attack. This tells us that the researcher didn’t actually do his research very thoroughly and is not a cryptographer. The phrasing he used should have raised several red flags with the JSST which instead chose to take the report on blind faith.
Even more troubling is that when we explained what this code does (which is the opposite of being vulnerable!) JSST decided to still consider it a vulnerability. We even linked them to the mathematical explanation, therefore they have no excuse of feigning ignorance. We do not understand the rationale behind ignoring objective, indisputable facts.
We find it disconcerting that the JSST decided to issue a CVE for this non-issue, the first we understand they issued after Joomla became a CVE issuing authority. We are concerned that this may undermine Joomla's credibility as a CVE issuing authority and harm its reputation.
We were alarmed when they told us that the same code wouldn't be fixed in Joomla 4 because Joomla 4 is still in beta. This raises a number of questions, such as whether any security fixes applied to Joomla 3 during the 5 ½ years of Joomla 4's development have been back-ported to Joomla 4 or if the release of Joomla 4 will see a mountain of security regressions.
We are concerned that the JSST doesn't seem to have considered whether the code runs at all in any circumstances. This is the first thing we noticed when we started following the code. Within 30' we had concluded it's inert code in the context of the "vulnerability report" the JSST shared with us.
Finally, we can’t understand why in 2015 replacing the same code — which was definitely in active use — was just code modernisation that got a passing reference in the full changeling nobody reads but in 2021 doing the exact same thing is suddenly a security fix.
We are perturbed by these unanswered questions. We would like to hope that they are just the result of bad communication coming from the JSST contact person and that the JSST actually took all of that into consideration. We can see why someone would like to replace this code with the PHP native random_bytes() implementation — after all, we had come to the same conclusion. We disagree that it should be treated as a security issue or even a security improvement. There is no indication the RFC 4086 implementation was ever anything but secure. We would call it either modernisation or a security improvement, the latter qualified with the disclaimer that the previous code wasn't insecure but PHP's CSPRNG is definitely guaranteed to by crypto-safe and regularly audited by actual cryptographers making it guaranteed secure, not assumed secure in lack of evidence to the contrary.
We have a lot of releases coming this week
We did not decide to modernise our code base just to keep it to ourselves. The goal has always been making new releases to make use of it. Today and the next few days we are making a number of new releases.
We are releasing FOF 3.8.0. The only change between that and the previous FOF 3.7.x releases is the change of the random number generator. This is the final release cycle of FOF 3. FOF 3 is now considered a legacy product and will officially become End of Life on December 31st, 2021.
We are releasing a brand new major version of our FOF backend framework, FOF 4.0.0. It is a deep refactor of FOF 3 to make it faster, more logically arranged, make use of PHP 7.2+ scalar type hints (that’s why we dropped PHP 7.1 support) and include some much needed improvements regarding Joomla 4 support/
This is complemented by the release of FEF 2.0.0, the major new version of our frontend framework. Instead of just a CSS framework, FEF 2 now includes a JavaScript framework which lets us completely remove jQuery from all of our software and stop copying around the same JavaScript and CSS code between our extensions. The JavaScript part of FEF 2 allows us to get rid of event handlers in HTML attributes and makes our JavaScript possible to use with defer by default to optimise the speed of your sites. When used with Joomla 4 it even makes use of HTTP/2 Push by default to make your sites even faster for your visitors.
We are also publishing new versions of all of our Joomla extensions making use of FOF 4 and FEF 2. They have all received a major version bump as a result. There are many improvements in them, not just a version bump. We strongly advise you to upgrade to them as they become available.
What about older versions of Akeeba extensions?
There is nothing to worry about, as explained above.
All versions of our software released since the end of May 2020 do not include this code in FOF anyway.
All older versions of our software never had an execution path which would see that code run.
If you are still on a PHP version which was affected by PHP’s OpenSSL extension bug you are not practically at risk. However, given that you’re still using PHP 5.3 to 5.6 which have been end of life for 2 to 8 years already we strongly advise you to upgrade your PHP version immediately. There are more important security issues that have not been fixed in your End of Life version of PHP.
In short, there is no reason to be alarmed and your data is perfectly safe. The reported vulnerability is not a vulnerability at all.