Support

Documentation

Advanced Site Transfers

Nicholas K. Dionysopoulos

Akeeba Ltd

Introduction

Transferring a development site to a live site will overwrite the existing live site and replace it with the content backed up from the development site. Many times this is undesirable, for example when users are registering continuously on the live site, forum posts are made, sales are performed and –generally– business goes on as usual. What is required is a way to “merge” the development and live site, without losing the valuable user-generated content. This walkthrough describes the steps to this process using Akeeba Backup. It is written for an intermediate to advanced audience and assumes that the reader is already familiar with the basic concepts of Akeeba Backup and fluent in backing up and restoring sites.

The problem we're trying to solve

Here's the typical scenario. You have a live site and a development (usually called “dev”) site. They usually begin as identical twins. Then you start working on your dev site, restructuring it, changing its template, adding and removing extensions and what not. At the same time, user activity never ceases on your live site. There are people registering, buying merchandise, posting in the forum... the works! By the time you are done with working on your dev site, the live and dev sites have drifted apart so much that they seem like they are two different sites. However, you need a way to transfer the dev site to the live site without losing the valuable live site content. A big question emerges: how can you “merge” these two sites into one?

Coming up with a plan

Akeeba Backup and its tools do not destroy by default any existing files or database contents before restoration. Files are added or replaced on the site. Files already on the site but not present in the backup are left alone. Backed up database tables replace existing ones. Database tables not present in the backup are left alone.

We can take advantage of the way restoration works to “merge” an existing live site with a partial backup of our development site. All we need to do is to figure out what we need to leave out of the dev site's backup to prevent overwriting any useful information on the live site. We need a plan for that!

Table exclusions

The first thing we don't want to replace are users. Joomla! stores user information in several tables. For Joomla! 1.6, 1.7, 2.5, 3.x and 4.0 these tables are (not all tables exist in all versions of Joomla):

  • #__action_logs_users Notification preferences for user action logs since Joomla! 3.9

  • #__user_keys Used for cookie authentication

  • #__user_notes Stores user notes

  • #__user_profiles Additional user profile settings, used by privacy controls and third party extensions

  • #__users The actual user information.

  • #__user_usergroup_map Maps users to user groups. If you changed the user group assignment of any user you will need to manually redo that on the merged site.

Do note that #__ is your site's table name prefix, e.g. f0o_. We use #__ because that's how it's displayed in Akeeba Backup's interface and it's how Joomla internally represents this prefix in a generic way. While we are at this, just go to Akeeba Backup, click on Database Table Exclusion and find those three tables. Click on the leftmost icon next to each one so that it turns yellow. Now your live site's users will not be replaced when we restore the backup.

The next thing we have to determine is where each extension stores modified content. This applies only to extensions which have content that your users have modified on your live site, not every extension on your site. There is no easy way of figuring this out. If unsure, ask the developer of each extension.

What about the case where you have, for example, defined new products on your e-shop? In this case you'd need to overwrite the live site's products with the dev site's products but you'd want to leave the orders intact. How do you do that? We can't really tell you. You have to take a look at the database tables yourself and figure it out. Most likely the developer is using sensible table names, e.g. name the orders table something like jos_fooshop_orders instead of jos_xyz_abcd1234. The idea is that you will have to exclude all of the extension's tables except the tables holding the data you want to transfer from the dev to the live site, e.g. the products table. If unsure about which tables holds which data, please consult the developer of that particular extension.

Beware of #__assets

There is one very special table in Joomla!, #__assets. This is a tree structure containing the access control assets. An access control asset can be an extension, a menu item, a module instance, an article, a category etc. As a result, this table links everything on the site together.

You cannot restore a site where you have modified content, menus, modules, categories and what not without restoring the #__assets table. Because of that, you CAN NOT skip backing up any core Joomla content. If your users have created articles or other content that ends up being referenced in the #__assets table you'll have to use a third party extension (e.g. SP Transfer) to transfer it to your dev site BEFORE taking a backup of your dev site.

Files exclusions

Something not very obvious is the need to exclude any files. Usually, there are only a handful of cases where you'd need to exclude files. Most likely you want to exclude forum attachments, product image descriptions or similar media files. The exact location of such files is highly dependent on the extension and how its developer designed it. If you're unsure, please consult the developer of your extension. When you have identified the files which need to be excluded, please use the “Files and Directories Exclusion” page in Akeeba Backup to exclude them.

Merging the two sites

Now that you know what you don’t want to transfer from the dev to the live site and have applied the relevant exclusions, you can take a backup of your dev site. If everything went to plan, the backup itself contains only the information we need to transfer. It’s what we call a selective backup. We will use it to merge the two sites.

Testing the merge on a local site

It is advisable to test the merge on a local testing site before attempting that on the live site. This way you will be able to identify and fix problems without accidentally ruining your live site. Begin by taking a backup of your live site and restoring it on a local server or any other hosting location other than your live site. Using a subdirectory or subdomain of your live site is not advisable. We will call the site you restored from the live site’s backup the test site.

Next thing, we will restore the dev site backup on top of the test site. It’s just like any other site restoration. Put your dev site’s backup archive in your test site’s root along with Kickstart and run Kickstart. Follow the whole restoration process just like you would do with any other site. What happens is this:

  • Existing files are replaced with their updated versions

  • Database tables in the backup replace the existing tables

Now it’s the time to make sure that everything went fine.

First, check in your back-end to make sure that there are no missing users. Try logging in as a user which was created after you started working on your dev site (so this user doesn’t exist in the dev site). Try creating new users, posts, test sales and generally run a scrutinized review of your site’s features. If something is not working, now is the time to review your backup exclusions and adjust them to cope with this problem. Do note that almost never will you get this process perfectly the first time you attempt it. Most likely you will have to go through 2-3 iterations to get it right.

Once everything is in working order, it’s time to wrap up and go live.

Going live

Take a backup of the dev site. It's advisable to use the ANGIE Password feature to prevent random visitors from seeing the restoration interface.

Take a backup of the live site and download it to your local PC using FTP in binary transfer mode. You really need to do that. If something goes awry, you need a way to get your site back. Having a fresh backup is the only way to do that.

If you have removed extensions in the dev site, be advised that the site merge will remove their database records but not their files. This means that extensions present in the test site but not the dev site will be partially removed. You won’t see them installed in Joomla!, but their files will still be on your site. So, before you restore the backup, remove those extensions from the live site.

Now run the restoration of the dev site’s backup to the live site using Kickstart. Go through all of the restoration steps. When this process is finished, your dev and live sites will be merged.

Check that everything works as expected. If not, restore the live site from its backup. You will have a working site before you can figure out what went wrong.

Is this practical?

It depends.

Does your site have user-generated content? Is losing any of that content a plausibly business-ending event? Are you unable to afford a downtime long enough to redesign / refresh / update the site? If the answer to all these questions is yes then yes, it is practical and pretty much your only choice.

If you have a site which only receives updates by a small group of administrators / authors you can probably afford to do a complete redesign and use something like SP Transfer to transfer the new content to the dev site. Then you can simply back up the dev site, delete the live site and restore the dev backup to the live server. It's far less complicated.

As for us, we've used this method many times to deliver substantial updates and complete redesigns of our site. Every single iteration of our site since we started our forum back in 2008 has been delivered using this method. Usually they were small changes, slightly altering the menu structure and adding a new template.

In 2017 we completely changed the user groups, view levels, template, menu structure, active plugins, plugin settings, component settings and modules on our site (basically creating a completely different site!) with 5 seconds of downtime – we performed the restoration unattended, using Akeeba UNiTE, to reduce the downtime to the absolute minimum. While the live action took 5 seconds, there was a full week of testing, tweaking, rinse and repeat until we could guarantee that the process would be smooth. For us, it was worth it. The redesign took over 4 months to complete and there was no way we could survive shutting down our site that long or losing client information from that period.