Support

Akeeba Solo

#36369 Failed to restore a jpa file

Posted in ‘Akeeba Solo (standalone)’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

PHP version
n/a
Akeeba Solo version
n/a

gvitry

Hi,

I use your products on Joomla sites for several years without any problems. I think I know them pretty well.

I've a Moodle site on which I've decided to install solo version to create backups of this pretty big site (around 5Gb data and 1.3 Gb database).

There are 2 cli jobs. One for a full data backup excluding DB and the other one for the DB only (sql file). The jpa setting was set to 50Mb because of a ftp post-processing to another server. The result is around one hundred files per backup.

Following a corrupted site after upgrading Moodle, I had to restore the data file from the backup. Starting from the local backup, I tried to restore using kickstart. It failed quickly. Following your documentation, I've tried several settings to tune but it was still failing.

Since it failed quickly, I decided to download the first 10 files on my personal desktop and decompress them using the windows utility. It failed quickly. Step by step, I finally found the third file was broken. I decided to open it in notepad++ to have a look. And surprisingly it was a plain text readable file instead of a compressed binary one.

I finally discovered that it was a bit of a 260 Mb log file from a Moodle cron.

It seems that if the granularity of the jpa archive is less than a given plain text file (named cron.log), this could rise a non-compressed badly formatted file instead of a splited normal archive.

I ‘ve changed the setting to split the jpa file every 256 Mb and then everything went well. I did a full test and could be able to restore everything without problems.

Did I misunderstood something ?

Thank you.

 

Laurent Collongues p/o Grégoire Vitry (collongues.laurent@apchea.com)

nicholas
Akeeba Staff
Manager

Yes, you did misunderstand something.

The part size and the file size are unrelated. It is expected and perfectly normal for the part size to be smaller than the biggest file you are backing up.

In fact, part files ARE NOT standalone archives, as I have explained a million times over the past 15 years and clearly documented. Split archives simply take one big archive file and chop it down to chunks. You can read the JPA specification if you don't believe me: https://www.akeeba.com/documentation/akeeba-solo/jpa-archive-format.html

The problem is that your log file changes size while it's being archived. As a result, we started backing up a file that's let's say 259.7Mb and by the time we were done archiving it's 260Mb. Same problem if it shrinks in size. This inconsistency between reading the file properties and finish putting it to the archive means that the File Data Block size does not match the Compressed Size of the Entity Description Block in the archive.

Kickstart (and Extract Wizard, which is not being developed since 2017) calculate the starting offset of the next file by taking the offset of the previously read File Data Block and add the Compressed Size to it. They expect that what lies at that offset is a File Entity Block. Because of the inconsistency we land in the middle of a File Data Block or another Entity Description Block. This is reported as an error: the archive is broken.

Note that the offset is calculated from the start of the first part (.j01). The offset is mapped to a specific part file and offset to the start of that part file as we know how long each part file is. Let's say we have the following situation:

.j01 10Mb

.j02 10Mb

.jpa (last part) 6.9Mb.

The offset 11Mb is 1Mb into .j02. The offset 23Mb is 3Mb into the last part (.jpa).

That's why the part size and the file size DO NOT have any correlation to each other.

The thing that you missed is that you need to exclude log files and generally any files which are expected to change size during the backup operation. In practical terms, yes, it's just log files. That's why Akeeba Backup for Joomla, for example, skips over the configured logs directory of the site. It also does not make sense backing up log files anyway.

The limitation has ultimately to do with PHP itself. PHP has a memory limit. We cannot read supermassive files into memory (thereby not caring how big they are and if they change size) because we will run out of PHP memory. The only thing we can realistically do is read them in chunks. Reading a file in chunks cannot place a lock on the file — even more so since we will be doing the chunked read over several page loads for web–based backup origins and over several seconds or minutes in any case. Even forgetting that it's impossible to put a file lock with PHP over several page loads, even if we somehow magically managed to do that it would cause your server to screech to a halt while the backup of that log file is in progress since the web application (Moodle in your case) would be unable to write into it. You'd be in an IO lock which would worsen with each request, possibly taking your server down.

TL;DR: Your problem is that you are backing up a log file which changes size while it is being backed up. You need to exclude the log file.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

gvitry

Thanks for your clear and detailed explanation.

Actually the file we are speaking about was a frozen log file. Not concerned by the file size modified during backup thing.

However, thinking about it, I found another log file that I did not exclude. This one could be the source of the problem as it is gowing by several 10k every minutes.

I'll double check the setting to be sure to go through that kind of files in the exclusion setup.

But it raises two questions :

1/ is it "normal" that one of the jxx chunck was a plain text file (part of the 260 Mb cron.log file) rather than a binary compressed one as the other jxx chunks ?

2/ even if doing backup at night during a time frame were the human activity is the lowest, it could happen that someone uploading a pretty big file (video...) while we are bakcuping. Also Moodle as an internal cron triggered every minutes by the system cron. This cron is doing a lot of tasks including some which can involve create, delete or modify files in size or content during the backup time frame. Would such situation generate a broken jpa backup ? if so, how could we be sure that a given backup is reliable ? ie restorable ?

 

On several years restoring joomla akeeba archive, I never had a problem at the uncompress step....

 

nicholas
Akeeba Staff
Manager

Yes, if there's a log file growing in size it will definitely make all offset calculations go awry. The next file you try to read will have the wrong offset which lands you in the middle of a random file. That's probably what happened. Luckily it was another log file so you filed a ticket and I could tell you where the problem might lie.

> 1/ is it "normal" that one of the jxx chunck was a plain text file (part of the 260 Mb cron.log file) rather than a binary compressed one as the other jxx chunks ?

You are looking at this wrong. What you are seeing is part of the File Data Block for that file. Since that file is immense (260 Mb) it's stored uncompressed. So what you see is not plain text, it's the exact binary representation of that file's uncompressed data. Why am I saying they are different? “Plain text” implies that the application modifies line ends to be OS–specific (CRLF on Windows, CR on macOS 9 and earlier, LF everywhere else). We don't do that. We keep the file content binary identical to the source.

So, the correct question is “is it normal that a big file is stored uncompressed“ to which I will reply yes of course and it's documented https://www.akeeba.com/documentation/akeeba-solo/archiver-engines.html#archiver-jpa. It's the Big File Threshold which has the following documentation block:

Files over this size will be stored in the archive file uncompressed. Do note that in order for a file to be compressed, the application has to load it in its entirety to memory, compress it and then write it to disk. As a rule of thumb, you need to have free memory equal to 1.8 times the size of the file to compress, e.g. 18Mb for a 10Mb file. Furthermore, compression is resource intensive and will increase the time to produce a backup. If this value is too high, you might run into timeout errors. On most servers a value between 1 and 5Mb works best.

So, you now get the reason why it's stored uncompressed :)

> 2/ even if doing backup at night during a time frame were the human activity is the lowest, it could happen that someone uploading a pretty big file (video...) while we are bakcuping. Also Moodle as an internal cron triggered every minutes by the system cron. This cron is doing a lot of tasks including some which can involve create, delete or modify files in size or content during the backup time frame. Would such situation generate a broken jpa backup ? if so, how could we be sure that a given backup is reliable ? ie restorable ?

Files being created or deleted after the directory listing is produced are not a problem. They will just not be backed up.

Files becoming larger are typically not a problem because it's very unlikely that the file will grow while it's being read to be backed up. The only problem is when you have a very big file which takes several seconds or minutes to read. We DO try not to read past the file size the filesystem reported when we started backing up the file but this may still not work properly for various esoteric reasons that have to do with filesystems and operating systems. The biggest problem is large files being truncated or deleted while they are being backed up — we have no data to read, the archive WILL be broken. However, this is detected and you are given a warning.

In practice, only REALLY large files changing size are a problem and these are almost always log files. In fact, the only three use cases I know of in the PHP world where a very big file can change size over time are 1. log files; 2. backup archives; 3. chunked uploads (Moodle does not use that). Therefore in practice the only thing you need to take care of is exclude log and temporary files. This will minimise the chance that the backup will be broken.

Ideally you would need to disable these housekeeping CRON jobs during the backup. They can resume afterwards.

Uploading videos etc is not a big deal, as I said, because the file appears instantly: PHP uploads to an off-site temporary folder, Moodle then does a file move to the final location which performs an atomic filesystem operation. One moment the file isn't there, the next moment it's there in whole.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

nicholas
Akeeba Staff
Manager

I should also probably mention that we have dozens of people using Moodle, either standalone or together with a Joomla or WordPress site. They don't have a problem with their backups as long as they exclude logs. One of them has a backup running for 10 hours every weekend — it's a 90GB backup when the dust settles. He's told me that he has restored it successfully at least once so there's that anecdotal piece of information to consider.

In general, a backup's worst enemy is big files changing size. Even standard UNIX tools like tar and rsync would have a problem with that. They have the benefit of being able to go back and amend the file metadata (tar) or decide whether to drop the file or keep on transferring (rsync). We don't have that luxury unless we remove the option to immediately upload each part file to remote storage which means our backup software could not work on space constrained servers, exactly like tar.

It's one of those things where you take a good, hard look at all the information and decide which one's the lesser evil. Occasional breakage on backing up ever changing log files OR not being able to back up unless you have ~50% of your hosting limit free? In this case the lesser evil is the former. The vast majority of our clients are on Joomla or WordPress where we already exclude most of the log files automatically. If something breaks it will be caught in the test backup restoration because everyone tests their backups — if they don't, they don't have backups, as the age-old systems administrators saying goes. If this happens they will file a ticket and we'll tell them to exclude logs, even which logs if we have a backup log file. On the other hand, if our software flat out does't work if there's not a lot of free space would be a no–go situation for a very large number of clients. So, lesser evil it is, then.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

gvitry

I thought I have read all your clear and abundant documentation... Not enough obviously !

Thanks again for you complete and clear explanations which make everything clear now. I feel confident to keep going on using Akeeba solo for our Moodle's backup and will again and again test restoring backup for time to time.

Congratulations for your software and support quality and best wishes for this new year !

Laurent

nicholas
Akeeba Staff
Manager

You're welcome, Laurent! I am happy I could help :)

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

Support Information

Working hours: Typically we work Monday to Friday, 9am to 7pm Cyprus timezone (EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets, but we cannot respond to them, outside of our working hours.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!