Support

Admin Tools for WordPress

#32996 – Security exception email spam on update to Admin Tools 1.2.1

Posted in ‘Akeeba Admin Tools for WordPress’
This is a public ticket. Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.
Wednesday, 06 May 2020 10:56 CDT
jferro
Hi,

I am curently in the process of updating Admin Tools to the latest version 1.1.1 > 1.2.1. My process is to take backups of my live sites, restore them to a local test server (using the excellent Akeeba Backup!), run the updates, test, then roll out to live. So far, so good.

However with the 1.2.1 plugin update, as soon as it completes, I am getting spammed with security exception emails from the site. I initially thought it was some sort of DDOS, but wiping the site down, restoring from backup and completing the process again, more or less yields exactly the same number of emails every time the plugin is updated.

On the face of it, it looks like prior security exception notifications are being reset and it's rescanning the logs and treating them as new events.

I'm a little bit hesitant now to update one of my main sites, as there are over 100K exceptions logged!

If anything, I'm kind of glad this has happened as I hadnt realised we were getting so many exceptions per day! There is obviously some more security hardening I can do to mitigate it. However, with regard to the problem mentioned, I havent had this problem updating from previous versions of Admin Tools?

To clarify this is happening on two different websites, wordpress 5.3.3 & 5.41, ubuntu 18.04.4, PHP 7.2.24-0
 
Thursday, 07 May 2020 01:42 CDT
nicholas
On the face of it, it looks like prior security exception notifications are being reset and it's rescanning the logs and treating them as new events.


No, this is not the case. Emails are sent when the security exception is triggered, not after the fact. It wouldn't make sense to get an email notification for a security exception that happened in the past. That would be useless.

I'd also like to note that security exceptions typically happen when someone is attacking your site from outside your server. If someone is blindly running a bot against your site or has launched a DoS / DDoS attack deleting your site and reinstalling it won't help you at all; your actions have absolutely no effect on what a third party, outside your server and outside your control, has decided to do.

In any case, the information you've provided so far is unusable for me to help you. You have excluded all the relevant information.

Have you turned on email throttling and set up a reasonable throttling threshold in each Admin Tools email template? If not, you will be receiving an email for each and every security exception. If your site is being targeted for a DoS, DDoS or being scanned / attacked by an automated script ("bot") you will be inundated with email.

What kind of exceptions are you receiving? This is very important to understand the nature of your alleged issue. For example, if you are getting thousands of failed login security exceptions it means that you've enabled the Treat Failed Logins as Security Exceptions option (enabled by default) and someone is blindly using an automated script to try combinations of common usernames and passwords in hope they will be able to gain Administrator privileges on your site.

Have you enabled IP blocking of repeat offenders and IP blacklisting of persistent offenders? If not, the attacks are logged and stopped but the offending IP address is not prevented from retrying. If there's an ongoing automated attack you will see thousands upon thousands of security exceptions logged and emailed to you.

Without this basic information for starters I can only speculate. I don't know about you, I am a really bad oracle and an even worse mind reader :) Please give me this information so I can help you understand better what is going on.


Nicholas K. Dionysopoulos

Lead Developer and Director



🇬🇷Greek: native

🇬🇧English: excellent

🇫🇷French: basic



Please keep in mind my timezone and cultural differences when reading my replies. Thank you!



Thursday, 07 May 2020 05:57 CDT
jferro
Hello Nicholas,

Thank you for responding to my ticket.

To try and answer your points in order:

The reason for reaching my conclusion about 'security exception notifications being re-sent', is because in one scenario, the site is being restored into a secure network. You just cant hit those machines from outside the network. The security events in question are mainly login failures (99%) and DFI shield (1%). So if these security events were not being caused by logs being rescanned, I'm not sure how it was generating ~150 login failure warnings in 10 minutes on a closed network? As the restored site is technically bringing data across from another server that did experience the 'real login failures' as it was an publically accessible machine on the internet, it wouldnt seem too unreasonable to assume these events were being imported?

As I said though, it was purely speculation. If that isnt happening then I'll take your word for it. But I hope you can see my reasoning why I might think that? I really like your software and I rasied this issue out of a genuine desire to try and find a resolution. I'm not attacking it. But let me be clear. These event emails are literally being triggered (or re-triggered) by the update of the plugin. I can restore the site to a clean test environment, use it as normal. Update wordpress and plugins, no problems. The second Admin Tools is updated, this flurry of emails are sent. I have re-run this scenario 6 times and each time I can replicate the same behaviour. I also ran it on the live server to the same effect.

To further complicate the matter, the security exception log in Admin Tools isnt reflective of these events either. I have just now (07/05/2020) restored Tuesday's site backup to a test server (prior to updating Admin Tools to the latest version). As per my attachments below, I recieved ~164 emails today from 10.56 onwards telling me about repeated login failures in the space of 10 minutes. This also happens to coincide with the time the plugin was updated. I have also sent a screen shot of the exception log a good 10 minutes after recieving the 164th email and it does not show any of these events?

Have I turned on email throttling?
No these are at the default of 5 emails in 1 hour for each event type. This is not being respected anyway given the volume of 'login failure' messages I'm recieving in such a short space of time (see attachments). As to your other question, yes, login failures are listed as security exceptions.

As already mentioned the security exceptions are mostly 99% login failures with the odd DFI.

Have you enabled IP blocking?
Yes, we are running fail2ban (medium threshold) and having had a number of real DDOS attacks in the past and penetration tests by a 3rd party company within the last 6 months, I am faily conifident that it is working as intended. The IP blocking function is also enabled in admin tools, prioir to updating the plugin.

We've been using the the system for ages and are really happy with it. This newest update is causing some wierd behaviour. I'm trying to rule out Admin Tools as the culprit. It's not a deal breaker. I'm not going to stop using the software. I work in software development and raised the issue out of a genuine desire to flag a potential bug so it can be squashed. But of course, it could just be a quirk of our own workflow and processes. If you need more information, please dont hesitate to contact me.

 
Thursday, 07 May 2020 09:28 CDT
nicholas
I can tell you in absolute certainty that the emails are sent without looking at the log entries. Moreover, I'm an engineer (Mechanical Engineer by field of study, Software Engineer by profession). I don't make assumptions, I look at the code :)

Speaking of code, please open wp-content/plugins/admintoolswp/app/plugins/waf/util/exceptionshandler.php around line 167. As you can see the email is part of the security exceptions logging process. The email is sent based on the information sent into the logBreaches() method which as you can trace yourself is only called when a security exception is triggered for the current request.

The fact that you are on a private network does not necessarily mean that nothing else on your internal network is compromised and trying to brute force the login. Moreover, it's possible that you have some other automation going on which might be trying to perform a login. In fact, you can see in wp-content/plugins/admintoolswp/app/plugins/waf/admintools/main.php, method registerFeatures(), that we merely telling WordPress to register our onUserLoginFailure method to the wp_login_failed handler. If your site is built in a way that triggers WordPress' wp_login_failed continuously then yes, you will get thousands of emails.

For what it's worth, we didn't change anything regarding tracking failed logins as security exceptions in the past 11 months and that was simply removing the ability to log the failed login's password. The implementation of that feature dates back to 2017.

I don't see any other change which could be relevant to your issue either. Maybe if I had the Target URL of the exception I could help you better.

Regarding some other points I asked you about.

Setting the limits in each email template is necessary but not enough to set email limits. You have to go to Configure WAF, Logging and Reporting and set Enable security exception email throttling to Yes. Do note that if you have dozens of malicious requests hitting your site at the same time the limit might be exceeded.

Regarding IP blocking, I was not asking you about your server configuration. I was asking you about your Admin Tools configuration. This is in the Configure WAF page under Auto-ban. If it's enabled as you say and the offending IP is the same it should be blocked. But hold your thought on that because I found more clues.

The very fact that your security exceptions log doesn't list the security exceptions you are being emailed about shoots down your suspicion about the log being parsed to send out emails – even if you don't look at the code like I did.

Moreover, your log states that the IP address that causes the security exception (at least the one you screenshotted for me) is 17.58.101.202. This IP address belongs to Apple. Namely, it's part of AppleBot. It is normally used by things like Siri and Spotlight Suggestions. This could be a clue as to what is going on. Furthermore, if it's AppleBot hitting your server it would explain why it doesn't get blocked; it uses a lot of different IP addresses. Each request comes from a different IP address, therefore a single IP address wouldn't be blocked as it would individually not trigger enough exceptions.

I can't tell you WHY AppleBot is hitting your server, I cannot tell you why that comes through to your internal network and I cannot tell you why WordPress believes that there is a failed login when this happens. I can only tell you that based on the one email you shared with me it's AppleBot hitting your site that triggers the security exceptions.


Nicholas K. Dionysopoulos

Lead Developer and Director



🇬🇷Greek: native

🇬🇧English: excellent

🇫🇷French: basic



Please keep in mind my timezone and cultural differences when reading my replies. Thank you!



Saturday, 06 June 2020 17:17 CDT
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.
This ticket is closed, therefore read-only. You can no longer reply to it. If you need to provide more information, please open a new ticket and mention this ticket's number.