Support

You need two backups profiles. One that takes a full backup, and one that takes incremental file backups.

For example, you could have the full backup profile run every week, and the increment one run every day. When you want to restore your site you'd restore the latest full backup, then each and every of the incremental file backups.

I think you conflate “intuitive” with “advanced”. They are almost always polar opposites.

If you were thinking “intuitive”, sure, it's very easy to do. In fact, that's the direction I was originally going with that. The original idea was to give you two options: full backup every at least X amount of hours, and full backup every X backup runs. Sounds great, right? You only need to set up one CRON job.

Well, that idea works great if your use case is something like “I want to take a full backup every Sunday and an incremental backup Monday to Saturday” Yes, this is a valid use case… but it's really the only one supported.

A very simple use case where this fails is a simple one: I want to take a full backup once a month, and an incremental backup every other day. This now becomes more convoluted because you need another set of options which are mutually exclusive with the previous options. That was the point where I realised that “intuitive” was going to be problematic for the users. I can add options all day long and test them fairly easily with Unit Tests… but people would a. not understand why some options are mutually exclusive (because nobody tries to think what happens if they combine two options whose intersection is the null set), and b. there are always use cases which cannot be covered by existing options, no matter how advanced they are.

Here is another valid, real world use case a business client shared with me: I want to take a full backup on the first Thursday of a month and a partial backup every other Thursday. You cannot express that as every X time, or every X runs, or using month and day of the month constraints. Each month has a different number of days and weeks than each other, and starts at a different day of the week. This has a real world business policy reasoning behind it. Site backups were stored to on-site storage. The on-site storage was backed up every Friday at midnight to tape. Tapes were stored in an on-site safe, and moved to an off-site location the first Friday of every month. I don't know of any kind of general purpose options which could express that. I would have to write a special rule for this and any other oddball use case. This neither scales, nor makes for an intuitive user interface after all.

Yes, I get it, but I told you why it had to be set up this way. What you envision is what I originally had in mind. However, it would only cover the one use case you and me can see in our heads. Unfortunately, this is not how 90% of the users want to use the software.

Trying to reconcile all those different use cases ends up requiring a re-implementation of the CRON system. I know because I put a lot of work into dissecting use cases, figuring out how to implement this, until I realised that I am basically re-inventing CRON.

Yes, you definitely need a bit of documentation for the site. But… you should have that already. Do you really remember how everything is set up on every site? Like, do you really have a mindmap of all the category and user group choices you made on each site, just two things which require far more structure and thought than your backup plan? You can't. Anything beyond an absolutely trivial site with 2–4 categories and just the basic Joomla! user groups requires documentation. Most trivial sites don't need incremental backups. So, what exactly would be the problem I am solving by introducing a restrictive interface?

Besides, your backup strategy is rather simplistic. I am pretty sure that your site does not grow massively in multiple folders. It only does so within a few specific folders, right? Let's say that it only grows by adding files into the images folder. It also grows by adding database data to certain tables, let's say #__content (which also means #__assets, as these two go hand-in-hand) Here's a good backup strategy:

• Profile 1: Full site backup.
• Profile 2: Full site backup with incremental files. Exclude all top-level files and folders except images. Exclude all tables except #__content and #__assets.

Now your incremental backup is even more precise, and smaller.

As for storage limits, each backup profile has Quota settings. Enable remote quotas and set up your limits. For example, you could have the full site backup have a maximum of 2 backups kept, and the incremental backup have a maximum of 12 backups kept. Assuming a once-a-week full backup and a daily incremental backup (except for the full backup days) that would give you two weeks of rollback.

The problem arose when I made backup number 6, 7, etc. The system continued to make copies of only what changed, and the full backup from the remote server was deleted. If I didn't have the information I gained from you, I would have to create a new profile or figure out another solution.

I am sorry to tell you, but a. you failed to use it correctly and b. this is another reason why your idea would fail in the real world.

You should have marked the full site backup as a Frozen backup. Frozen records are never deleted automatically. Since you didn't freeze it, it got reaped by the quota settings you set up. You did not think it through.

And this is why your idea would fail in the real world. Let's say a user wants to take a full backup every Sunday and incremental backups every day. Let's say we magically let them do that through the interface. They would also be able to set up quotas. What if they set a count quota of 3? What if they set a size quota which is met on a Thursday? In both cases they delete their full backup and now they don't actually have a backup of their site. Are they going to blame themselves for their own choices? No. As you demonstrated, they will blame the software for doing exactly what the user told it to do.

FWIW, a few years ago we thought about creating a "wizard" for incremental backups. The wizard would create the two backup profiles and tell you how to set up their CRON jobs (or even create Joomla! Scheduled Tasks for you). However, we decided against it because it ultimately requires a modicum of responsibility from the user which appears to be incompatible with the target audience of such a feature.

The best we can do is document how incremental backups are supposed to work. This is already on my to-do list.

I know that you didn't blame me, sorry if I gave you the wrong impression. What I was trying to say is that even you, who are one of the most focused and understanding users, could not figure out that your configuration was wrong. This tells me that the less focused and understanding people would also miss that and then come back to blame the software. I hope it's a bit more clear now.

You are right in that not a lot of users use incremental backups. However, those who do tend to have very different use cases. Only half of the people who have reached out to me about incremental backups have a straightforward use case. The other half have use cases which are quite convoluted. I often have to sit down with pen and paper (okay, iPad and Apple Pencil) to come up with a plan which will work for them. The use case I told you yesterday where the first Thursday of a month is a full backup is convoluted. The incremental backups require two CRON jobs, one running Friday to Wednesday, and another one running on the second, third, and fourth Thursday of the month because of a limitation on what and how can be expressed with CRON expressions.

After replying to you this morning I did write the first draft of the documentation. I will go through it again before publishing it in a couple of weeks :)

Have a wonderful day!