Support

Akeeba Backup for Joomla!

#37166 Remotely storing to S3 Object storage server fails when backup size is larger than 200MB

Posted in ‘Akeeba Backup for Joomla!’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

Joomla! version
3 (not yet tested on 4)
PHP version
8
Akeeba Backup version
8.1.4

stavrosz

I have created a profile named "Online Backup" configured so that its backups be stored into a remote S3 Object storage server (Contabo Object Storage - https://contabo.com/en/object-storage).

I have setup the same setup in a couple dozen different websites and setup a cron job to each for an automatic weekly backup.

Everythng works fine as long as the backup files are less than 200MB. If a backup file happens to surpass the 200MB at a certain week Akeeba Backup fails to upload the file to the remote Object Storage server with the warning message as per below: (I have placed certain names within "< >" for clarity purposes)

Warnings
Failed to process file /home/<accountID>/public_html/administrator/components/com_akeeba/backup/<fileName>.jpa Error received from the post-processing engine: Unexpected HTTP status 413 Post-processing interrupted -- no more files will be transferred
  The Object Storage server has a maximum size limitation is 5TB per object. I tried to manually upload, through the Object Storage server's control panel, one of those files that Akeeba Backup failed to upload, and the uploading went through just fine.    Is there something that can be done at the Akeeba Backup program side, or should I check with the Object Storage server's support, and what should I ask them when the uploading problem is faced only with the Akeeba Backup program?

nicholas
Akeeba Staff
Manager

HTTP 413 comes from outside our code and means Payload Too Large: https://developer.mozilla.org/en-US/docs/web/http/status/413

Now, here is the thing. As you said, Amazon's limits are far bigger than that. On top of that, Amazon does not return an HTTP 413, ever. It has its own error code reporting through the XML response it sends back.

I am pretty sure that your host has a proxy sitting between your server and the Internet. Most likely it's SQUID set up as a transparent proxy and there's a bit of iptables magic to make it truly transparent (as in: any request to an Internet resource over TCP ports 80 and 443 automatically goes through SQUID, very simple to do once someone has pointed out how it's done). Indeed, SQUID will return an HTTP 403 when the request body exceeds a preset threshold, see https://community.sophos.com/utm-firewall/f/management-networking-logging-and-reporting/31375/http-1-0-413-request-entity-too-large. I have actually tried that configuration on an Ubuntu box and yes, I do get the same result you do (I set the limit to 5Mb as I didn't want to die of old age waiting for a 200Mb backup to happen). I also happen to know that this SQUID setup is definitely and for a fact how many commercial hosts configure their servers. That's what makes me pretty sure that this is what is going on in your case.

There are two ways to deal with your host's restriction.

Easy and recommended method: uncheck the “Disable multipart uploads” option in your profiles' Configuration, Post-processing Engine, Configure pane. This will upload the barckup archive in small chunks. Each chunk is up to 10Mb big so you never get to hit the limit on your server. This also allows spreading the upload between multiple page loads, making it far less likely that it will time out.

Virtually impossible method: Try to convince your host that they need to remove the HTTP request body limits in SQUID. Unless you are on a dedicated or managed server this would be a no-go.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

stavrosz

I have the "Disable multipart uploads" checked because initially I was getting an error message and only with this option checked I managed to make the remote storing work.

Nevertheless, I unchecked this option and tried again and got the following error as expected:

Warnings

  • Failed to process file /home/<userID>/public_html/administrator/components/com_akeeba/backup/<fileName>.jpa
  • Error received from the post-processing engine:
  • Upload cannot proceed. Amazon S3 returned an error.
  • Akeeba\Engine\Postproc\Connector\S3v4\Connector::uploadMultipart(): [403] Unexpected HTTP status 403 Debug info: SimpleXMLElement Object ( [Code] => SignatureDoesNotMatch [RequestId] => tx00000970b863f8f09b354-00628baeda-78a2b-default [HostId] => 78a2b-default-default )
  • Post-processing interrupted -- no more files will be transferred

The same error appears for backups of any size even with Akeeba Backup version 9.2.2 on Joomla 4. It seems that in my case, remote storing works only with this option checked and with file sizes smaller than 200MB.

Does this error message say to you something new?

nicholas
Akeeba Staff
Manager

Yes, it tells me that the host is messing with the request headers. Most likely they are caching the request we send to Amazon S3.

The way the upload works is this. We take a chunk of the backup archive. We calculate its size and its MD5 sum. We combine this information with the current date and time, your Amazon access, the region, the bucket name and the path of the file in the S3 bucket (in a certain order, prescribed by Amazon) using the HMAC-SHA-256 algorithm using the Amazon secret key as the password. The result is the signature which is sent in the HTTP authorisation header to Amazon S3. Amazon S3 does the same process and compares the signature it calculates to the one we send. If there is a difference it throws the error you are receiving.

Since a whole file is uploaded just fine it means that the inividual components are calculated correctly, including the current date and time (Amazon allows a maximum of ±15 minutes to the actual time the request was sent). The fact that a smaller request fails immediately tells me that your host has cached the request and does not send to Amazon the HTTP headers we calculated and asked to be sent. Therefore the request fails.

Unfortunately, you wil have to contact your host and tell them to fix their broken caching.

Nicholas K. Dionysopoulos

Lead Developer and Director

🇬🇷Greek: native 🇬🇧English: excellent 🇫🇷French: basic • 🕐 My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

stavrosz

You are absolutely right.

Their proxy is CloudFlare. They cooperated with CloudFlare and managed to have those restrictions being lifted.

Now everything works excellently, though only with "Disable multipart uploads" checked (but the result is satisfactory).

Thank you!

Support Information

Working hours: Typically we work Monday to Friday, 9am to 7pm Cyprus timezone (EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets, but we cannot respond to them, outside of our working hours.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!

Summer vacations: Our support will be closed for replies and new tickets from August 6th to August 21st, 2022 due to summer vacations.