How the Manage Backups page works with local and remote backup archive files and where quota management fits in

This is a longer page explaining the non–obvious connection between what you see in the Manage Backups page, locally and remotely stored files and the Quota settings in the Configuration page of each backup profile. This is based on numerous conversations we've had with clients, taking into account their challenges, questions and considerations. We hope it will shed some light on why certain things are implemented they way they are and exactly how they work.

The Manage Backups page lists backup attempts, not files

Akeeba Backup keeps a history of all of your backup attempts in your site's database. This is what is displayed to you in the Manage Backups page, not the backup archives on your server or remote server.

There are several reasons for this. Having the information of all in–progress, successful and failed backups allows to implement features beyond taking a backup from the backend of the site where we are guaranteed to have a working user session. For example, this allowed us to create the front–end backup URL and the Akeeba Backup JSON API which can be used to schedule backups using services external to your site, running without the need for a logged in Joomla user.

Furthermore, this implementation is much better for performance. Querying the database records for a thousand backups takes less than 0.2 seconds. Querying the local filesystem for this many backup records takes about 1 second. Querying a remote storage service takes several minutes. The only way to have a workable Manage Backups page is to query database data; anything else would either time out (making the page completely unusable) or would be so slow that it would be practically unusable.

Local vs Remote files

Each backup record saves, among other things, the directory on your server where the backup archive was created in, the base name of the backup archive and the remote path (remote storage service type, path and filename); the latter is stored only if you have set up a Post-processing Engine other than None in the Configuration page and only applies to Akeeba Backup Professional.

By default, Akeeba Backup will create your backup archive on the same server the backup is running. This is intentional. Creating a backup archive requires being able to not only append to the archive file but also go back and change things towards the start of the backup archive (the very beginning of the first part) where the total number of files, the total compressed and uncompressed size of these files and whether this is a multi–part backup archive information is stored. Both of these features are only supported when dealing with files on your server, i.e. local files.

The backup archive file or, in the case of a multi–part backup archive, the part files of the backup archive file can be optionally transferred to remote storage based on your configured Post-processing Engine options. The files stored outside of your server are called remote files.

It is possible for a backup record to have only local files (its status is shown as OK), only remote files (its status is shown as Remote), both (its status is also shown as OK) or neither (its status is shown as Obsolete).

Whether local files exist is something which is determined every time you access the Manage Backups page. As noted above, this is a very cheap operation which only takes fractions of a second.

Whether remote files exist is something which is ONLY updated when you manage those files through Akeeba Backup. That is to say, if you delete your remotely stored backup archive files outside of Akeeba Backup your backup record in Akeeba Backup's Manage Backups page will STILL appear as Remote, indicating there are remote files. This is not a bug. As explained, trying to do otherwise would take a very long time and make the Manage Backups page unusable.

Deleting files: local vs remote

The Delete Files button in the Manage Backups page only deletes the locally stored files. Likewise, the Delete button in the Manage Backups page will only delete locally stored files and the backup record itself; it won't touch the remotely stored files. This happens for several reasons.

The least important of all reasons is that deleting a remotely stored backup takes a lot of time. If you have a remotely stored backup with dozen of parts it would time out. That's why the Manage Remotely Stored Files page has this large spinner and reloads several times when you have a remotely stored backup archive consisting of many parts (therefore many files which need to be deleted one at a time).

However, this is not the most important reason. That could be worked around. What cannot be worked around is the need to manage locally and remotely stored files separately from each other and remotely stored files separately from backup records. It's easier to give you two common use cases.

You start taking a backup. The backup does complete successfully but the files are not transferred to remote storage because of a temporary network issue, the remote storage service is down or you had used the wrong credentials. At this point the backup archive is stored locally. You can rectify the error and use the Manage Remote Files to upload the backup archives to remote storage. Now you have both local files and remote files. One of the points of storing backups remotely is that you don't want them taking up space on your web server. If you cannot delete the locally stored files without deleting the remotely stored files you'd have to use FTP/SFTP to do that, therefore Akeeba Backup would not be the one and only administration interface you need for your backups which beats the reason for having it in the first place. Therefore, having local and remote files managed separately solves this problem.

It's also possible that you want to delete the backup record from your database but keep the backup archive stored remotely. Here's an example from our own use case. Before July 2019 we were issuing invoices ourselves instead of using a third party service. We are legally required to keep copies of these invoices for at least seven years, preferably ten (7 is the minimum set by law, 10 is the recommendation of the tax office in case they have a case that dragged on due to bureaucracy). We need to keep that July 2019 backup until at least 2029. However, we don't want it to be listed in our backend to prevent accidental restoration to a version of the site that won't even run correctly on our current server.

There are many use cases like that where people want to keep an old version of the site in remote storage but not let another less experienced administrator try to restore it, breaking everything. The way to handle it is deleting the backup record without deleting the remotely stored files. If we deleted the remotely stored files when deleting the backup record you'd have to use a database client such as phpMyAdmin to achieve that which would again be a problem as you cannot manage your backup records with just Akeeba Backup.

Having backup records and local files managed independently from remotely stored files solves these problems.

As to why locally stored backups are deleted together with the backup record, it once again comes down to the same requirement to have Akeeba Backup as the sole tool necessary to manage backup records and backup files. If you delete a backup record without deleting the locally stored file you won't know that there's something taking up space or how to delete it.

You might wonder, why not do that for remote stored files too? Apart from the use cases we gave you above, there's a big difference to the context of locally and remotely stored files. Locally stored files take up space on your web server where space comes at a premium. Remotely stored files are stored on far cheaper external storage. Locally stored files are only required if you want to restore your site (i.e. they are “hot” files). Remotely stored files are used for archiving backups in case you need them (they are “cold” files). It makes sense to treat hot and cold file storage pools differently since the context of storing a file in each one of them is different.

You can of course delete the remotely stored archive files or move files between local and remote storage using the Manage Remotely Stored Files in the Manage Backups page. That's why this feature exists. You can delete remote files, download remote files back to your server (remote files to local files) and uploading archive files stored on your server to the remote storage (local files to remote files).

Quota management

Akeeba Backup offers a feature called Quota Management. It can automatically delete old local and remote files you don't need at the end of the backup process.

Quota management works on the backup records in the database, not the backup archive files. In other words, Akeeba Backup will NOT check the contents of your backup output directory for local files or the folder of the configured remote storage for remote files when processing quotas. It will trust what is stored in its database. There are, again, many reasons for that.

Beyond the performance issues we already mentioned, it is possible that different backup profiles or even different sites share the same backup output directory or remote storage directory. If quotas were applied on files stored on disk / remote storage you would never be able to have different quota settings per backup profile or even different quota settings across files. According to our experience this would lead to backup archives you need to end up getting automatically deleted. A backup gone when you do not expect it is a disaster waiting to happen. That's another reason we erred on the side of caution.

Quotas are not a substitute for manual management. They are an optional feature on top of manual backup archive management. They allow you to implement a custom retention policy for backup archives and backup records.

In fact, backup files quota management and backup record quota management are separate and work independently. Yes, this means that you can shoot your feet since quota management requires the backup records of said backups to exist in the database to figure out what to remove and you can accidentally configure Akeeba Backup to remove these records, essentially making the quotas never fire. There is method to what on first glance appears to be madness. We implemented it this way because it also gives you far greater flexibility. In the end of the day we follow the time–honored UNIX dogma that the user is a responsible adult in ultimate control. Our software won't question your intentions, it will execute your instructions with precision and efficiency.

For example, you may choose to retain remotely stored backup archives for the last 30 days and the 1st of each month but also remove backup records older than the last 45. Assuming daily backups, this lets you see when backups were taken in the last month and a half, making sure backups were not skipped due to a scheduling problem. It also lets you keep the first of the month backup archives (typically, for regulatory reasons) without seeing them in the Manage Backups page (so you don't accidentally mess up and restore a backup that shouldn't be restored).