Support

Akeeba Solo

#23555 phpbb3 Cache exclusion

Posted in ‘Akeeba Solo (standalone)’
This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Environment Information

PHP version
n/a
Akeeba Solo version
n/a

Latest post by on Saturday, 09 January 2016 17:20 CST

pigga
Hi Akeeba-Team.
1st of all thanks for this brilliant piece of software and the great documentation.
However...I ran into some trouble(s) when I tried to migrate a PHPBB3 forum: The forum always tried to connect to the old database and at 1st glance I had no reason why, because the config.php had been modified by "ANGIE" as expected.
After some googling I found out that the Cache (that had been restored as well) was the root cause.
My question:
Is there a way to backup the .htaccess and the index.html file within the "cache" folder and leave all the other files behind? I noticed that Akeeba understands "Reg Ex", but to be honest I have no idea how to figure this out.
Thanks a lot ex ante!
Regards,
Thomas

nicholas
Akeeba Staff
Manager
You will have to exclude included files of this folder using the Files and Folders Exclusion feature. This should NOT exclude the .htaccess file from that directory. Please note that you do not need the index.html file since you already have a .htaccess file in there.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pigga
Hi Nicholas.
Ok, so .htaccess files are not affected by the filter settings and will be backed up anyhow? That's good to know! In this case I was thinking way to complicated. You are totally right, It will work without the index.html file. Ok, it's not a big deal to delete the cache folder after migration/restore manually but I want to minimize potential trouble for the future (the remaining cache really caused me some headache...). However, I will try this out.
Thanks a lot for your help and have a nice week!
Thomas

nicholas
Akeeba Staff
Manager
Yes, the .htaccess files should be included. Please test that yourself. If they are not included let me know, I'll have to check.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pigga
καλημΞ­ρα :-)
Hi Nicholas (and the entire Akeeba-Team)!
I would like to give some feedback. I did not want to over-stress my shared web-hosting so I waited until front-end backup via webcron.org did it's job.
I downloaded the file, and restored it on my local XAMPP test-environment. Everything went fine!
Until now all akeeba-components clicks like a clockwork.
Of course you were right: Due to the fact that the .htaccess contains
<Files *>
	Order Allow,Deny
	Deny from All
</Files>

there is no need to add an additional index.html. So I set it up the way you recommended and now the backup/restore works "out of the box". And YES: The .htaccess was backed up altough I excluded the files of the "cache"-Folder. It works as intended.
Maybe this may help anyone in the future: If phpbbb3 connects to wrong database after restore, you will have to delete the "cache" folder although the *.php files inside of it may look a bit "too important" to delete them manually. This will do the trick.

Finally there were four minor issues I noticed:
  • Security Issue: As stated in your documentation, the *.jpa file will probably "tell" everyone (who gehts this file) about database pw, ftp credentials and so on.
    I found it a bit confusing that there are two steps of setting a password: 1st of all the "ANGIE password". But this only sets a pw to the installation routine. I did not realize that this will not affect the data archive at 1st glance. Later on I noticed that I had to change from "use JPA" to "use JPS". However, once enabled, this works like a charm.
    After running the "Configuration wizard" the archive will be reset to "JPA". I ran into this trap one time. (Meanwhile I found the hint in the documentation but I was a bit astonished that the wizard changes my settings. I was assuming that -once set- the wizard tries to apply my settings ans will tell whether they work or not).
  • from time to time, backend is hanging slightly. Error log says
    [Thu Oct 29 17:54:03.472243 2015] [authz_core:error] [pid 22219:tid 140401883690752] [client xx.yyy.134.115:33324] AH01630: client denied by server configuration: /opt/users/www/mypath/html/subfolder/backup/Solo/assets/private/config.php, referer: http://www.mydomain.de/backup/index.php?view=main

    Does not crash but "lags" sometimes and most likely at that point of the error log.
    I did not check if it went away after the update yesterday, but maybe this info is helpful for you?
  • The phpbb3 I am backing up with your software is running for over 10 years right now. Akeeba-Solo-Backup succeeded, but I noticed that the front end of the forum went nearby down during update, it definitely brought the server down to it's knees. So I re-read the documentation and noticed that the "smart file scanner" does not suit for all situations. I had a closer look and currently there are nearby 7500 Files of about 1Gb stored in the "files"-Folder of the phpbb installation plus about the same amount in the "Thumbnails"-folder. Referring to your manual I should change the scanner for "big sites"? The backup succeeded, but I do not want to run into trouble with the hosting company due to the fact it is shared webhosting...
  • Last but not least: For some reason the backend says "last backup (front end) FAILED". But the referring *.JPS file is in my dropbox and i was able to restore it today. Unfortunately I (most likely) logged into the Akeeba console just right before the front-end-backup had been finished. Maybe that's the reason?
However.
Long short story: Akeeba Backup on phpbb3 works like a charm. 1.3Gb in total are backed up in 21 minutes to the dropbox. When I remember how long this took via manual backup...Akeeba core it's worth its money. Definitely.
Kind regards,
Thomas

nicholas
Akeeba Staff
Manager
Regarding the "security issue" it's not anything like that. What you described is failing to read the documentation. Sure, if you do not read the documentation you will do things that may compromise your security. That's why we tell you to read the documentation.

The Configuration Wizard is meant to be ran ONCE per profile, when you create a brand new profile. Additionally you can run it if you've screwed up everything in the backup profile and want to make a fresh start. The job of the Configuration Wizard is to benchmark your site and reset your settings to what it believes will work optimally to take a backup. It should be self-evident that resetting the settings actually means resetting the settings. I mean, come on, if it couldn't override non-default settings what should it do if you run it on an existing profile? Tell you "Sorry, Dave, I don't think I can do that"? ;)

from time to time, backend is hanging slightly. Error log says


This is intentional. Akeeba Solo is checking your security. It needs to make sure that the file Solo/assets/private/config.php which contains its configuration –therefore its database and probably FTP/SFTP connection information– can NOT be read from the web. If it ever detects that it can read that file it will show you a Big, Scary, Red Warning. We are actively concerned about your security.

Referring to your manual I should change the scanner for "big sites"? The backup succeeded, but I do not want to run into trouble with the hosting company due to the fact it is shared webhosting...


Yes, but it's not the end of the story. By definition, a full site backup means that the backup software has to go through every single record on each and every database table, through every single file in every single directory and put them all into an archive file. This DOES require CPU and memory to accomplish. You could make the backup slower but you risk losing archive integrity AND stressing the server for a prolonged period of time which is actually worse.

Perhaps you should consider a different backup strategy. Backup everything except images and thumbnails daily, then have an Incremental Files Only backup for the images and thumbnails running once a week, when your forum is least used. This keeps a great balance between performance and data integrity. We do the same for the files of our downloads area.

Last but not least: For some reason the backend says "last backup (front end) FAILED". But the referring *.JPS file is in my dropbox and i was able to restore it today. Unfortunately I (most likely) logged into the Akeeba console just right before the front-end-backup had been finished. Maybe that's the reason?


Yes. If a backup step takes more than 3 minutes to execute and you log in to the control panel page or visit the Manage Backups page the backup is marked as failed. Since you're uploading a massive file to Dropbox it will take a while. Please try not using our software's Control Panel during that time. Also, it would make more sense if you moved your backups earlier in the night when there are fewer people on your server :)

1.3Gb in total are backed up in 21 minutes to the dropbox.


That's a very fast server :) Speaking of which, here's a trick to lower the CPU usage on your server during backup – assuming that you're OK with prolonging the backup time. In the Configuration page set:
Minimum execution time: 15 seconds
Maximum execution time: 10 seconds
Execution time bias: 75%
Also, in the Dropbox configuration section of that page do enable chunked uploads.
The backup should take around 35 minutes to complete with almost half the CPU usage.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pigga
Hi Nicholas.
Yes, you are right, the whole stuff requires some more RTFM from my side. Thanks for your helpful input.
In the Configuration page set:

Minimum execution time: 15 seconds

Maximum execution time: 10 seconds

Execution time bias: 75%

Also, in the Dropbox configuration section of that page do enable chunked uploads.

The backup should take around 35 minutes to complete with almost half the CPU usage.

Just to be sure: Would you recommend the "smart scanner" with this settings (due to the fact it is apparently working that way) or choose the "large file" scanner instead?
I tried it with the "large file scanner" and your recommended settings. Max Ex. Time 10s and Bias 75% were already set by default, so I only increased the "Minimum execution time" from 5 to 15 seconds. As expected, backup process took a bit longer (just below half a hour AFAIR) but with no noticeable lagging of the forum's front end.
So I'd like to know which scanner type you recommend (just to be sure).
Unfortunately I ran into trouble with the JPS file (see attachment). It would be nice to have another layer of security by protecting the archive file, but makes the encryption process of (such a large) archive the system significantly more error-prone? In this case I would better leave pw protection behind for the sake of a 100% extractable archive file...
I mean, come on, if it couldn't override non-default settings what should it do if you run it on an existing profile? Tell you "Sorry, Dave, I don't think I can do that"? ;)

Uhm...in combination with a red-flashing camera eye this would take my user experience to a totally new level I think :-)
Perhaps you should consider a different backup strategy. Backup everything except images and thumbnails daily, then have an Incremental Files Only backup for the images and thumbnails running once a week, when your forum is least used. This keeps a great balance between performance and data integrity.

Yes, I think this would be the best compromise.
I did some reading in the forum and in the manual.
Just for my better understanding: I create a new profile and set it to "files only, incremental" and exluce evertything except the files/upload/whatever folder.
When I run this profile the 1st time it will create a huge backup with all 7500files in it.
From now on it will "remember" the date when last backup was taken and only backup newer files, right? To restore all data, I would need the main backup (database+files except the large images-folder) plus all incremental files from the large folder. I think this will accumulate many incremental backup files after some time.
Is there an easy way to clean this up from time to time and to set a new starting-point to prevent it from getting too confusing?
Thanks for your patience...
Regards,
Thomas



nicholas
Akeeba Staff
Manager
Just to be sure: Would you recommend the "smart scanner" with this settings (due to the fact it is apparently working that way) or choose the "large file" scanner instead?


Always use the Large Site scanner when you have big sites. The reason we have not enabled it by default is that it's up to 10% slower while most of our clients' sites are not that big and properly processed by our trusted Smart Scanner.

I tried it with the "large file scanner" and your recommended settings. Max Ex. Time 10s and Bias 75% were already set by default, so I only increased the "Minimum execution time" from 5 to 15 seconds. As expected, backup process took a bit longer (just below half a hour AFAIR) but with no noticeable lagging of the forum's front end.


Awesome :) If you're wondering, this trick creates about 30% idle CPU time in each backup step. This is enough to let the server "breathe", i.e. finish processing other threads without the backup thread stealing all the resources.

Unfortunately I ran into trouble with the JPS file (see attachment). It would be nice to have another layer of security by protecting the archive file, but makes the encryption process of (such a large) archive the system significantly more error-prone?


It depends. The JPS archive encrypts everything with 128-bit Rijndael encryption. This does require extra processing power. Extra processing power means longer running backups with increased CPU usage. Can that cause problems on some servers? Yes, it can.

Also, your problem MAY be caused by eXtract Wizard. Have you tried using Kickstart on a local host instead? I see you're using Ubuntu. You can install a local LAMP stack if you haven't already done so (I bet you have – you didn't strike me as a beginner user). Kickstart is the preferred way to extract archives, always.

Uhm...in combination with a red-flashing camera eye this would take my user experience to a totally new level I think :-)


Something to consider for April 1st this year :D

From now on it will "remember" the date when last backup was taken and only backup newer files, right? To restore all data, I would need the main backup (database+files except the large images-folder) plus all incremental files from the large folder. I think this will accumulate many incremental backup files after some time.


Yes, correct. The way we usualyl set it up is having two profiles. One Files Only running once per month, one Incremental Files Only running daily or weekly – depending on how frequently your files change. This means that in case of a disaster you'd have to restore the latest Files Only backup and all incremental backups from that date onwards.

Is there an easy way to clean this up from time to time and to set a new starting-point to prevent it from getting too confusing?


Yes, there is, with the TWO profiles option only. For the Files Only profile enable and set count quotas to 2. For the Incremental Files Only profile enable and set the count quotas to keep the last 30 backups – assuming that the big Files Only profile runs twice a month. The idea is to set the count quota of the incremental files to 2x as many incremental backups you'll need in the worst case scenario. This may sound a waste of space but it's your insurance policy in case a massive pile of poo hits the proverbial fan. When it comes to backups and security there's no such thing as "too paranoid" ;)

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pigga
Hi.
Thank you very much for your Feedback and pointing me into the right direction!
Yes, there is, with the TWO profiles option only. For the Files Only profile enable and set count quotas to 2. For the Incremental Files Only profile enable and set the count quotas to keep the last 30 backups – assuming that the big Files Only profile runs twice a month. The idea is to set the count quota of the incremental files to 2x as many incremental backups you'll need in the worst case scenario

Bingo! Oh dear...sometimes the solution is easier than expected. Never had thought about such a strategy but you are totally right. There is not too much activity on the site so I assume for the worst case a backup/restore interval of 1 week is sufficient (this is the worst case i.e. it also can be less days).
So I ended up in such a strategy: I set up three profiles
1) backup the phpbb filesystem (except Files/Media Folder) plus complete Database including ANGIE Installer on a weekly basis
2) Do an incremental backup of the file/media folder after that (i.e. on a weekly basis as well)
3) Make a complete backup of the whole files/media folder at the beginning of every month.
Okay, as stated somewhere else in this forum: Restoring an incremental backup will not work 100% automatically, but the data is there and can be restored. The phpbb backup (including database) is quite lean (<30Mb), the incremental backup most likely too. The complete files/media backup is the biggest part, but will only take place once a month.
By the way:
If you're wondering, this trick creates about 30% idle CPU time in each backup step. This is enough to let the server "breathe", i.e. finish processing other threads without the backup thread stealing all the resources.

I can confirm this. During the backend-backup process I noticed that there always was a little pause between the steps. But that's totally okay. You can not have both: Shared hosting with several users on a server and consume all resources for your backup at the same time. This will not work. Furthermore, a backup does not take much longer than before. It's totally ok that way.
As suggested, I will increase the quota a bit, so in worst case there will even be the chance to roll back to the major backup from the month before (including additional incremental backups) and so on... Great stuff. Setting it up took some time, but after all it will (hopefully!) do the work on its own in the future.
The idea is to set the count quota of the incremental files to 2x as many incremental backups you'll need in the worst case scenario. This may sound a waste of space but it's your insurance policy in case a massive pile of poo hits the proverbial fan. When it comes to backups and security there's no such thing as "too paranoid" ;)

Yes, +1 on this. I learned it the hard way this summer. I am keeping an eye on the Jommla install of a friend of mine for a small company, just to keep it up to date, add some article from time to time and so on. Suddenly Images from the site disappeared. From a retrospective, a data burp of server's HDD (or whatever) must have corrupted either parts of the file system or database or both.
Site still worked but the URL-router linked images wrong (not to images/stories but to category_name/images/stories which does not exist of course). Additionally, Akeeba Backup had stopped working and throw an AJAX Error. I.e. strange things happened, but there was no indication of any attack to the site. So I was totally lost.
I ended up in restoring a *.JPA backup to the server. Lucky me I had taken a backup some months before akeeba had broken down (and stored on my HDD). With this backup, site was back to normal! Murphy's law is always on your side :-)
But this was the point where I decided to use the full version and automatize things. This hopefully helps me to sleep better at night in the future.
Also, your problem MAY be caused by eXtract Wizard. Have you tried using Kickstart on a local host instead? I see you're using Ubuntu. You can install a local LAMP stack if you haven't already done so (I bet you have – you didn't strike me as a beginner user). Kickstart is the preferred way to extract archives, always.

You are totally right, I double checked it with a local server install as well :) Yes, extract Wizard failed under Win7 and Ubuntu as well. But Kickstart returned an error as well. So I assumed the archive was broken. I switched back to *.JPA backup format and now it works flawlessly.
After that, I checked it with backups from the Joomla install mentioned above: In this case the extract-Wizard failed as well, but kickstart worked like a charm and was able to extract all three *.JPS backups I have taken so far (all about 150Mb size). So it's hard to say what is causing this. At least it shows me that kickstart should be the way to go, as you recommended. I found the information that the extract wizard probably will be deprecated soon, right?
Once again thank you very much for your support!
Kind regards,
Thomas

nicholas
Akeeba Staff
Manager
eXtract Wizard is already deprecated. The last time I wrote code for it was 2011. It's a massive surprise to me that it still runs – even though it was originally compiled for Windows XP, Mac OS X Snow Leopard and Ubuntu Linux 10.10 (yeah, that old). The biggest problem with eXtract Wizard is handling JPS archives. There were surprisingly few AES-128 libraries available for FreePascal / Delphi and none I could find with a FOSS license compatible with GPL v3. Sigh...

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pigga
Hi Akeeba-Team.
I checked it, within the last few weeks, and -thanks to your instructions- automatic PHPBB3 backup works like a charm. Even the incremental backup works as it is expected. At least 100% better than the out-of-the box backup solution that phpbb3 provides.
There is only one thing about which I am scratching my head: For some reason, the "status" Indicator remains blank.
I noticed that in the documentation it's the same:

From Joomla I can remember that the start page always shows a status of the last taken backup.
So, Do any partial and/or remote backups not set any backup status?
See screnshot below.
Thanks a lot for clarification!
Regards,
Thomas

nicholas
Akeeba Staff
Manager
Thank you for the heads up! I see what's wrong. The label doesn't show up for successful backups. The fix will be included in the next version of our software.

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

pigga
Hi
(I just recycle my old ticket, hopefully it's ok?)
There is one minor oddity I noticed. I'm not sure if this is the way it's expected to work:
The latest version offers "one-click-backup"-Buttons. One for every profile I have created.
I would expect that every button triggers the backup profile it is associated with.
In my installation, all of the "one-click-backup" Buttons have the same effect as if I klick on the general "backup now"-button. That means you will have to choose the desired "Active backup profile" in the dropdown-list at top of the page first and then it will start backing up with the right profile.
Funny thing: When I hover with the mouse cursor over the "one-click-backup" buttons, they will show the right link at the bottom of the browser window, something like http://www.mydomain.com/akeeba/index.php?view=backup&autostart=1&profileid=3 for example.
Regards,
Thomas

nicholas
Akeeba Staff
Manager
That's a bug. Thank you for the heads up!

Nicholas K. Dionysopoulos

Lead Developer and Director

πŸ‡¬πŸ‡·Greek: native πŸ‡¬πŸ‡§English: excellent πŸ‡«πŸ‡·French: basic β€’ πŸ• My time zone is Europe / Athens
Please keep in mind my timezone and cultural differences when reading my replies. Thank you!

System Task
system
This ticket has been automatically closed. All tickets which have been inactive for a long time are automatically closed. If you believe that this ticket was closed in error, please contact us.

Support Information

Working hours: We are open Monday to Friday, 9am to 7pm Cyprus timezone (EET / EEST). Support is provided by the same developers writing the software, all of which live in Europe. You can still file tickets outside of our working hours, but we cannot respond to them until we're back at the office.

Support policy: We would like to kindly inform you that when using our support you have already agreed to the Support Policy which is part of our Terms of Service. Thank you for your understanding and for helping us help you!