Support

The video is not useful for determining the root cause. I need the log file of the failed backup attempt. Please ZIP and attach it to your next post, as requested when filing the ticket. Thank you!

The log file reads:

You must enter your Download ID in the application configuration before using the “Upload to Google Drive” feature.

Have you entered a valid Download ID in Akeeba Backup on that site?

Starting with Akeeba Backup 7 some things changes for every storage engine which uses OAuth2: Box, DropBox, Google Drive, Google Storage, OneDrive (legacy and the unified OneDrive and OneDrive for Business implementation) and pCloud.

You cannot connect to these services without an access token. Their APIs do not allow you to create that token directly. They require a public, known in advance authentication endpoint which will exchange a secret key known to the client and the initial response of the storage provider's authentication server with an access token. Furthermore, each and every time you take a backup you need to use the secret key to refresh the access token (exchange the expired one with a new one). The two problems with OAuth2 and mass distributed software is that a. the endpoint needs to be known in advance and b. the secret cannot be disseminated with the software because it'd jeopardise everyone's security.

There are two ways to go about a solution.

One way is to ask you, our clients, to host that endpoint on your site. You'd need to sign up for a developer account on the storage engine provider you want to use. Some of them require payment to do so. You'd need to host a special file that runs outside Joomla on your site and make it web accessible. You'd need to navigate the very unfriendly and ever changing interface of the storage provider you'd like to use, create an OAuth2 login screen and configure the endpoints correctly. You'd need to write and publish Privacy Policy and Terms of Service pages for the OAuth2 integration on your site, even if you're the only one using it. You'd need to go through a verification process. Then and only then would you be able to start transferring backups to the remote storage. This is impractical, to say the least. 

The other way to go about it is us going through all that process and host the authentication mediation endpoint on our own server. The downside is that it costs us real money to develop and maintain this endpoint. Do remember that each and every of the backups you take use it which costs us money.

Between Akeeba Backup 3 and 6 inclusive we made these endpoints available free of charge. This led to a significant number of clients who let their subscription lapse but were still using our services to facilitate backups on their sites. We reached a point where this was no longer sustainable so we enforced a Download ID check starting with Akeeba Backup 7 -- and put the code to convey the Download ID in the last Akeeba Backup 6 release as well. That's why you need an active Download ID to upload a backup to these storage providers, as documented.

So, that explains the Google Drive problem.

Amazon S3 does not need a Download ID because it does not go through our server. Same thing applies for every other storage provider that does not need to go through our server. If you don't need to consume our resources to take a backup we don't need to ask for rent for our resources, to put it very bluntly.

Looking at your logs, your problem with Amazon S3 is different. It says that the archive part file disappeared while trying to upload it. This means that something outside the backup process removed it. If you are starting another backup or visiting Akeeba Backup's control panel page while the upload to any remote storage is in progress you will cause the archive part files not already uploaded to disappear. Same goes if an external service tries to access Akeeba Backup's API at that point. Or maybe you have a CRON job, yours or your host's, which deletes these files automatically. Or a plugin which does the same. The problem is not coming from the backup process, it's external to it.

I hope this helps.

I was not talking about a web proxy or CDN in the issue you've linked to. I am talking about the server proxying its outbound requests through something. Typically this is Squid or Varnish. This is typically done in a belief that by proxying requests FROM the site TO the outside world (the exact opposite of what CloudFlare does) any requests to third party services would be cached and your site would be faster. However, this falls flat when we are trying to send data to the remote service or read up-to-date content to make decisions, i.e. what Akeeba Backup does.

Also note that this might be outside your VPS. A VPS nowadays is just a virtual machine. Proxying would very likely take place at the VM host level which is outside your control. Therefore you need to talk to your web host about it.

One last thing (sorry) but getting back to the original problem now- that the backups were being marked as complete when in truth they did not actually transfer all of the files.  Is there anything that can be done about that? 

Not in any practical way.

When we send data to the remote server we expect to receive a reply back telling us if the upload succeeded or failed. When we receive a successful reply we have to assume that it's legitimate. The problem here is that the server undermines the communication, spitting out lies back to us. This situation is essentially a man-in-the-middle attack.

In theory we could re-download all the files we uploaded and check them for integrity. If you only check them for existence there are many cases where you cannot differentiate between a partially uploaded and a fully uploaded backup archive file.

However, this comes with two problems. First and foremost, the files are not guaranteed to show up immediately upon uploading. For example, if you uploaded a 5GB backup archive in 5MB chunks to S3 it might take several seconds for the archive file to appear as available: S3 needs to first stitch together all the parts (which are possibly on different physical machines). The way to implement that is with an exponential back-off strategy which is unworkable in the context of a mass-distributed application running on diverse server environments many of which have very tight time limits.

Even if that wasn't an issue, we are still downloading the entire backup back to the server. This is a slow process which might trigger a backup script failure because of CPU time limits in CRON jobs. It will also eat up your server's bandwidth because each backup is transferred twice. Furthermore, not all remote storage engines support chunked downloads which would most definitely cause a timeout during the transfer back to the server (that's why there's no Fetch Back to Server button for all remote storage engines). Last, but not least, we'd have to assume that the local server has enough disk space to transfer back the files which is not a given – that was the reason behind the immediate archive part transfer option implemented in all remote storage engines.

So while in theory this is possible, in practice it would only work if you are using specific storage engines from the CLI backup without any CPU usage limits on your account on a server with practically unlimited bandwidth and disk space. That's not a realistic set of requirements in the context of who is using on our software, on which servers and what for.