No, you shouldn't be worried. You are not "under attack" or a "target" of any malicious user. Let Admin Tools do its job. The fact that you see too many security exceptions means that Admin Tools is already handling the situation for you.
So what is going on? What you are experiencing is most likely a high number of automated attacks. Some common types of attacks are the following:
Probes. There are software tools which try to detect whether a site is vulnerable to a number of known attacks in older versions of software. These tools are normally used by security researchers and companies you hire to assess the security of your site. Sometimes they are also used for nefarious purposes. Just because your site is being scanned for vulnerabilities does not mean it has one, though! Think of it as a locksmith or a burglar testing the security of your door's lock.
Brute force. Another common attack tries to guess the password for a Super User. Typically, the attacker will try using very common username/password combinations such as admin/admin or admin/password and so on.
Blind attacks. Unskilled, wanna-be hackers (commonly called "script kiddies" or "skiddies") will try blindly using some very old attacks they found on the Internet against any site they see in front of them. While this sounds scary, this is rather dumb as the attacks they try are known for years and are, therefore caught very easily. Moreover, they don't even check that they have the right version of Joomla! or even the right CMS. We routinely see attacks on our site targeting the obsolete Joomla! 1.5 (retired in 2011) and WordPress (which we never used on our main site's domain).
In all of these cases Admin Tools does its job well. It will intercept the attack and ban the IP of the attacker if they are repeatedly causing security exceptions. The attacker will eventually give up on your site and simply move on to the next target in their list.
So don't worry too much about it, Admin Tools has your back!