#34232 – Incremental backups and quota management

Posted in ‘Akeeba Solo (standalone)’
This is a public ticket. Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.
Wednesday, 16 December 2020 08:09 CST
lukaszpp

Hi,

I have problems setting up incremental backups with quota management. I have relatively large website (few GB) and limited backup space.

What I am trying to achieve is to allow Akeeba automatically manage space and be able to restore backup from incremental file.

Backup without quota management goes good - full website backup is uploaded, and then incremental files are set up. Although when I have enabled quota management, then main backup file was removed making it impossible to restore website from incremental files only. Example:

  1. October 1st - full backup - was removed
  2. October 2nd - incremental - was removed
  3. October 3rd - incremental - was kept
  4. October 4th - incremental - was kept
  5. etc

Instead, I would like it to behave like this:

  1. October 1st - full backup - kept
  2. October 2nd - incremental - removed
  3. October 3rd - incremental - removed
  4. October 4th - incremental - kept

Is this possible to set up in Akeeba? Or I have to do it somehow with multiple backup profiles? Anyway, I don't also see an option for incremental backup profile to "force full backup" again on given interval (ex. on 1st every month), which would actually allow to achieve something like this, and that would solve my issue:

  • October 1st - full backup - was removed
  • October 2nd - incremental - was removed
  • October 3rd - incremental - was kept
  • October 4th - incremental - was kept
  • etc, but then
  • November 1st - full backup - was kept
  • November 2nd - incremental - was kept
  • etc

In such case October incremental files would be useless, but November files would be all good

Custom Fields

PHP version (in x.y.z format) 7.3.25
Akeeba Solo version (x.y.x) 7.4.0.1
 
Wednesday, 16 December 2020 08:43 CST
nicholas

Incremental backups and quota management are incompatible concepts on the face of it. The whole idea of an incremental backup is that you have a full backup and one or more incremental backups which have to all be restored sequentially. The whole idea of quotas is that older backups are going to be removed. Of course, if you remove incremental backups you can no longer restore the site.

The only way to make it work is to take a full site backup every X days and incremental backups in between. This requires two different backup profiles: one for the full backups and one for the incremental backups. Your quotas in both profiles must be set up so that backups from the last X days are not removed.



Nicholas K. Dionysopoulos

Lead Developer and Director



🇬🇷Greek: native

🇬🇧English: excellent

🇫🇷French: basic



Please keep in mind my timezone and cultural differences when reading my replies. Thank you!



Wednesday, 16 December 2020 08:59 CST
lukaszpp

Hi Nicholas,

I don't agree that quota management and incremental backups are incompatible concepts - in fact it seems they make a lot of sense in long run, when ex. you are required to run incremental backup very often (1 hour), website is relatively big, and script is running for more than a year. Just in one year that would produce 8700 files, which is relatively big amount - recovery of such set is real pain, I have been actually through such set. Ability to reset incremental backup (ex. every 1st of the month, as I described in second part of ticket) adds a lot of reliability to such case, because each set of backups would be just around 730 files.

"The only way to make it work is to take a full site backup every X days and incremental backups in between."

Would those two profiles work together? Isn't it that every profile is separate backup set? Example:

  1. Profile "full backup":
    1. September 1st
    2. October 1st
    3. November 1st
  2. Profile "incremental backup"
    1. September 1st - is not run
    2. September 2nd - incremental file is created, that is based on "September 1st" full backup
    3. September 3rd  - regular incremental file is created
    4. etc.
    5. October 1st - is not run
    6. October 2nd - incremental file is created, that is based on "October 1st" full backup
    7. October 3rd  - regular incremental file is created
    8. etc., and same for November

In such case would recovery work? Example, if I wanted to run recovery for October 4th, I would take files:

  1. From "Full backup" October 1st file
  2. From "incremental backup" following incremental files:
    1. October 2nd
    2. October 3rd
    3. October 4th

In other words - would such mixing of files from different profiles work?

 
Wednesday, 16 December 2020 09:54 CST
nicholas

I explained why the concepts would be incompatible: if you delete any of the incremental backups taken since the last full backup you can't restore your site to a state that makes sense. That's basically what you were doing which doesn't make sense and can't work.

Regarding your comment about backup profiles: each backup profile is a different set of options and filters. They are all taken against the same site. You can of course restore a partial backup on top of a site. An incremental backup is, in fact, a partial backup: it has only some files of the site. By itself it can't restore a site but if it's restored on top of a full site it can do something useful. Ah! Therefore you can have one profile, let's call it X, taking a full site backup and another profile, let's call it Y, taking an incremental backup. Then you can restore them one after the other, in chronological order. That's the whole concept of incremental backups in Akeeba Backup.

Let's stop and think for a minute. How does that even work?

A full site backup includes all of your site's files and database table content.

An incremental backup includes one of the following:

  • Incremental backups: all of the database content plus only the files that have changed since the last incremental backup based on the files' last modified timestamps.
  • Incremental, files-only backups: only the files that have changed since the last incremental backup based on the files' last modified timestamps

I assume you have the former kind.

So, let's say that you have a full site backup from October 1st and two incremental backups from October 2nd and third.

You start by restoring the full site backup from October 1st. This restores the entire site: all of its files and all of its database contents.

Then you restore the incremental backup from October 2nd. All of the database content is overwritten with the database content from this backup profile. Any files added or modified between October 1st and 2nd are created or replaced respectively.

Then you restore the incremental backup from October 3rd.  All of the database content is overwritten with the database content from this backup profile. Any files added or modified between October 2nd and 3rd are created or replaced respectively.

So your site is now more or less in the state it was on October 3rd. The only caveat is that files that have been deleted between October 1st and 3rd are NOT deleted. So it's imperative that you take a new full site backup when you expect files to be critically deleted, i.e. whenever you update Joomla and its extensions.

Since you're short on space you also definitely need to keep a limited number of backups at any one time. Therefore you need to be able to remove, say, all backup archives older than 30 days. Given how restoring incremental backups work you need to ensure that you have a full site backup taken at most 30 days prior. That's why you need to schedule a full site backup at least once every 30 days. Of course full backups take a lot of space but you only really need to keep the last one of them. So you can have a count quota of one.

Let's combine all those concepts together.

Create one backup profile that is a full site backup. Set up count quotas and set the quota count to one. Schedule this backup profile to run on the 1st of every month. You also need to manually run this backup after updating Joomla and its extensions.

Create a second backup profile of the Incremental Backup type. Set up count quotas for it and set the quota count to 30 (each month has up to 30 days but we don't take a backup on the 1st of each month, therefore we need up to 30 backups). Schedule this backup profile to run every day except the 1st of each month. Advanced mod: you can exclude database tables which have temporary data or infrequently modified data you can afford to lose (that's very site specific so I can't really guide you there).

Restoring your site means that you need to restore, in this order:

  • The last full site backup.
  • All of the incremental backups which were taken after the last full site backup, in chronological order.


Nicholas K. Dionysopoulos

Lead Developer and Director



🇬🇷Greek: native

🇬🇧English: excellent

🇫🇷French: basic



Please keep in mind my timezone and cultural differences when reading my replies. Thank you!



Friday, 15 January 2021 20:17 CST
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.
This ticket is closed, therefore read-only. You can no longer reply to it. If you need to provide more information, please open a new ticket and mention this ticket's number.

Support Information

Working hours: Typically we work Monday to Friday, 9am to 7pm Cyprus timezone (EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets, but we cannot respond to them, outside of our working hours.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!