Updating Joomla! websites with JoomlaPack Kickstart: a step by step guide

This article is long owed to the community. I apologise for taking forever to write it down, but development of the 1.3 version was much more higher in my priority list. In the light of the new Joomla! 1.5.7 release - and having two sites demanding urgent update - I have decided to share the secret with you: It's upgrade time and Kickstart can be used to apply the Joomla! update patches to your site easily. It's really simple, too. Let's go!

First of all, you'll need the SVN version of Kickstart which supports plain file or FTP file access modes, integrates AJAX and JavaScript redirects modes and includes a few more improvements. Since this is not released yet, you can download a developer's preview - but very stable already! - version from here. Extract the PHP file in the archive you've just downloaded.

Next up, you'll need the ZIP version of the Joomla! patch, which you can download from the Joomla! project's file repository site. Depending on which Joomla! version you've already installed, download the appropriate ZIP file. The tar.gz file will not do! For example, if you've got Joomla! 1.5.6 and want to update to 1.5.7 you must download the Joomla_1.5.6_to_1.5.7-Stable-Patch_Package.zip.

Upload the Joomla! ZIP file and kickstart.php to your site's root. If you used FTP, make sure both files' permissions are 0777. Make sure you have a working backup of your site before you proceed! In the unlikely event you run into a problem we want to be sure you can roll back. There are many ways to take a backup of your site, but we obviously suggest you to use the open source JoomlaPack backup component.

Now visit the kickstart.php URL, which is something like http://www.example.com/kickstart.php, substituting www.example.com with the full URL to your Joomla! site. In the page presented to you, make sure the update ZIP is selected in the topmost combo box. For the operation method we suggest using the "JavaScript Redirects" method, unless you know what you're doing. Depending on your site configuration you can either use "Write directly to files" or "Use FTP", supplying your FTP connection information as well. Most users will need the second (FTP) option.

After you're done setting up Kickstart, hit the big green "Start" button. You will see your browser reloading the page many times; this is normal. On the final page, click the second link titled "here". As the page reads, you're (almost) done! If you had an .htaccess file on your site, it is now renamed to htaccess.bak. Using an FTP client, rename it back to .htaccess and you're really done, in - virtually - no more than a few seconds!

Try it! I just upgraded two sites in 3 minutes flat (including upload times!).

Editor's Pick and site look

JoomlaPack is the new Joomla! Editor's pick for extensions. We are greatly honored to be chosen. An official announcement of this will be made later on the site.

JoomlaPack has a new site template. The template was designed by JoomlaPraise and it is their Joomla! 1.5 template contest entry. We chose this template because of its clean design and its emphasis on content. There still are some minor problems with the forum version of this template so we're going to give it one last shot before we release it as is. As with anything that changes, things may go wrong so please do tell us on the forums any problems you see with the design.

The upcoming JoomlaPack 1.3 brings new features

We might be just a few days before the release of 1.2.1 stable, but the JoomlaPack development team doesn't call it a day. We are actively developing the upcoming minor number version of the component. As with all past minor number versions, it brings along many new features. We feel it's only fair to share with you.

• Joomla! 1.5.x native mode only. If you follow Joomla! announcements you should have known - since about 8 months - that Joomla! 1.0.x development has stopped. No further updates are planned for that version of our favourite CMS. In other words, Joomla! 1.0.x is dead. Period. Our single most major step for JoomlaPack 1.3 was removing J! 1.0.x support and sticking with J! 1.5.x. Not constrained by the limitations of the old generation Joomla! and taking advantage of the empowering Joomla! Framework we have set to develop a modern code base, based on the principles of the MVC pattern, which will be much more easily maintenable in the future.
• Backup profiles. Until now, JoomlaPack was constrained to a single configuration. Setting configuration parameters and filter options was of an all-or-nothing approach. If you wanted to tweak some filters to get a different type of backup (let's say, filtering out some ill-behaving component) and then revert to your regular settings it was impossible. JoomlaPack 1.3 incorporates backup profiles, an unlimited number of profiles storing configuration option and filter settings. What's more, you can select which profile to use at backup time, even with front-end backup!
• Backup statistics. You asked for and you get it. JoomlaPack 1.3 will keep a lot of information about each backup taken. This has some nice implications you're gonna love:
  • No more need to reset after failed backup attempts. Based on information from Backup Statistics, JoomlaPack's engine will detect failed backups upon visiting the Control Panel page or starting a new backup, cleaning after them.
  • Describe and comment on your backups. No more need to remember what changes were made between individual backups (applies to backups started from the backend)
  • Ability to keep a limited amount of backup archives on your output folder automatically.
  • More informative backup administration page.
• Optional non-AJAX mode for the administration. Up until now, if your configuration was incompatible with AJAX you wouldn't be able to use JoomlaPack. After all, it relied on AJAX to operate the filters pages. This is no longer true. All basic operations on the filters pages are now possible to be executed in non-AJAX mode. Just select "JavaScript Redirects" for the "Backup Mode" in the Advanced tab of the configuration. The non-AJAX mode does not apply to some more advanced operations, like checking for database connectivity while editing a Multiple Databases setting, or fetching the default output directory in the configuration page.
• No more temporary directory setting. Joomla! 1.5.x provides with a standard temporary files location and JoomlaPack uses it. Of course, it needs to be writable!
• Embedded documentation. The basic set of documentation (help for each JoomlaPack page) will be included with the component, available by pressing the "help" button on the toolbar.
• Automatic problem detection. There are a number of server configurations which might cause JoomlaPack to fail. We know about them and know how to detect them. The next big step is warning you in advance, right from the Control Panel! It's just like the "You're using the default output directory" warning of JoomlaPack 1.2.1, only it covers a whole lot more situations.
  
• Revamped Kickstart. The new Kickstart will have a few changes over its predecessor. For starters, there will be no more two different files for AJAX and non-AJAX mode. The mode selection will be a radio button on Kickstart's first page. We are also planning to implement a FTP mode, to get past servers using Safe Mode and/or have "wrong" permissions.
  

Roadmap to JoomlaPack 1.2.1

Just over a week ago the stable 1.2 version was released to the public. As always, the stable release signifies the beggining of a new development cycle. This time, instead of jumping straight to developing version 1.3, we decided to improve the latest release in order to produce a safer, more performing, mature component.

This is the list of features we are currently working on, for version 1.2.1:

• Administrator directory restructuring and change of class names. CUBE is turning to a framework of its own right, built atop the Joomla! Framework. A big step towards full Joomla! 1.5 framework adoption, much of the guts of our component is being recoded to use only J! 1.5 API calls, while providing a backport layer for J! 1.0.x compatibility.
• Reduce and concatenate the steps (domains) needed to produce a backup. It occured to me that only two domains are actually required and should be present: database backup and file backup. In order to achieve this, a number of changes have to take place:
• Avoid extracting the installer files to temporary directory. Use the packaged installer files to "seed" the backup file instead. This also has a security enhancement bonus, as no "live" PHP files are written to the temporary directory, eradicating a direct access attack exploiting such files.
• Use name mangling on temporary files to minimize possibility of direct file access attacks
• Store the temporary file names and use this table when cleaning up. Moreover, clean up each temporary file right after it's been put in the archive. This is much more reliable than the current method which almost always leaves leftovers.
• The file list creation and packing steps are merged. This also means that much less data (in number of SQL queries and in total data size alike) is required to be written to database, solving many errors related to db server overload.
• Drop the "fast" algorithm. It is meaningless on most servers (crashes with timeouts) and the "smart" algorithm is already fast enough.
• Introduce Magic Numbers for smart algorithm. This will let you fine-tune the smart algorithm's performance to accomodate for slow, overloaded servers.
• Re-implement translations. It serves a double purpose: get closer to J! 1.5 compatibility and allow the use of semi-automated translation tools on the translation INI files. Some highlights:
• Introduce upercase INI keys by concatenating header and key, inserting an underscore between them
• Separate front-end and back-end language files
• Make a static method CLangManager::_($key) which translates the string like JText::_($key)
• Error and warning propagation and handling. Right now, if an error occurs in some part of the backup engine, it is not propagated and the process crashes in an unrelated point or completes to a partial backup (or, worse, no backup at all!). The idea is that if an error occurs anywhere in the code, the execution stops immediately, the error message is displayed to the user and also gets logged. In the case of warnings, they will be displayed just below the backup process messages.
• Distributed AJAX handling. The AJAX proxy will be taken apart and AJAX handlers will be grouped by page, hopefully creating less overhead during the actual backup process.
• JoomlaPack Installer 3. We are planning on creating the third iteration of our smart restoration script. This time it will be J!1.5-specific, based on the original J! 1.5 installer application. This is much harder than the traditional approach we took with JPI and JPI2, but it will be more forgiving in the case of errors - as you well know JPI2 is notorious for crashing silent when an error occurs during a J! 1.5 site's restoration. It will also cater for changing the cache and temp directories to their defaults if the pre-configured directories do not exist or are not writable, fulfilling a frequently made feature suggestion.

These are quite a mouthfull for a sub-minor number release! Right now the bulk of heavy refactoring is complete, namely the first two points. The rest of them are quite easy to implement, with the exception of the last two points which are bound to consume lots of our time.

The planned release date for 1.2.1 stable is mid-October 2008. Stay tuned!

Migrating a site to Joomla! 1.5: a success story

One of the tasks I had to perform less than a month ago was to migrate a site from Joomla! 1.0.15 to Joomla! 1.5.x. The site's operation couldn't be suspended for more than a few minutes. This was an exceptional challenge for testing JoomlaPack's tools in a real world task. Let's sum up the mission objectives and get to work:

• The original Joomla! 1.0.15 site must be upgraded to the latest Joomla! 1.5.
• The same or equivalent components will be used without loss of data, especially in regards to translation and mailing list contacts.
• The site must have virtually no downtime; a maximum of 10 minutes is acceptable.
• The resulting site must be fully operational. No broken links, missing images or any other mishap will be tolerated.

Sounds like Mission: Impossible, right? Impossible is nothing, as long as you have the right tools.

Step 1: Back the old site up

One of the things experience teaches up - albeit in a harsh manner - is that things can and will get awry, unless you're prepared for the unthinkable. In this case, the unthinkable is to bring the whole site down by accident, without a way to get it back online to its previous state.

That's why I used JoomlaPack 1.2.b1 (right then it was in a pre-release state) to get a snapshot of the site. Since this was a J! 1.0.x site, I used the -j10 package, which is Joomla! 1.0.x native. In 5 minutes flat the backup copy was sitting happily archived on my hard drive and USB key. I know, but redundancy makes me feel safer.

Step 2: Define the migration strategy

One of the common misconceptions is that when it comes to migration you can only act on a live site. Right? Well, since I did have a site snapshot at hand and a XAMPP for Linux powered machine on my desk, the only safe decision was to try migrating on a locally operated "clone" of the site. The result could be then uploaded and deployed within minutes on the live server.

Furthermore, I decided to have a "safe haven" should the restoration go wrong. For this reason I created a subdomain, with the intent to put the old site in it. It was actually easy; after uploading the site backup archive and kickstart.php I merely clicked my way through the restoration procedure and - presto! - the clone was right there, in the new subdomain.

During this process I also safely checked Kickstart's compatibility with my live server. It turns out that even when I deactivated PHP's Safe Mode, I was still unable to get Kickstart to work. Some head-scratching later, it turns out that the site's root was not writable to the web server user. I just connected to the server with my FTP client and changed this folder's permissions to 0777. That did the trick. This was a useful thing, since I used my newly found knowledge to "fix" the permissions on my main domain's site root folder to avoid any troubles during the restoration of the migrated site (more on that later!).

Next, I repeated the procedure on my local server. Now I had a perfectly working "clone" of the live site on my local server. It's time to have some fun!

Step 3: The actual migration process

It turns out migration isn't as straightforward as you might have thought, or read about. After a couple of frustrating attempts, I came up with the right (as in "worked for me") procedure:

• Install the migrator component on the original Joomla! 1.0.x site
• If you had translated the site using Joom!Fish, you'll have to install the respective plugins. Too bad they're only available in their SVN and you have to guess that they even exist! I don't want to be too harsh; after all at the time of this writing Joom!Fish 2 is still in beta. Installing them is simple. Just download the plugins and tables folders and place in the com_migrator's corresponding folders.
• Run the migrator component, which results in a SQL file. Keep a copy of it, you'll need it.
• Make a new folder for the 1.5.x site and extract the latest Joomla! 1.5 distribution there. Do not run the installer yet!
• Copy the SQL file in the new site's installation/sql/migration folder.
• Make sure the installation folder, subfolders and all contained files have read/write privileges for everyone.
• Run the Joomla! installation, making sure you select the migration content instead of sample content towards the end of the installation. Tip: Use the same prefix as the old site. Hopefully, nothing went wrong in the process. If not, you'll have to copy the SQL file again and then retry the process. During migration the SQL file gets altered. Why does it have to? Beats me.
• You have a semi-usable site now. Copy over any media (like, images) from the old site to the new.
• Install third party components. Remember that you need to activate the Legacy Plugin for components which are not Joomla! 1.5 Native (that's about 80% of the components I guess).
• If the components offer no backup/restore for their data, copy their tables from the old site's database to the new site's database.
• Make sure everything is in working order. Otherwise, keep on tweaking.

Step 4: Getting ready for transfer

After all this process was over, I had a fully working site on my local server, all aspects tested out and everything to the client's liking. The next step is to get it online. JoomlaPack to the rescue!

I installed the 1.2.b1 release, using the Joomla! 1.5 Native -j15 package, so I hadn't have to activate the Legacy plugin. In less than 4 minutes a snapshot of the shiny new site was ready on my hard drive. Hey, that was easy!

Step 5: Showtime!

The big moment is here. Everything has to be done in military precision. The site is not to be left off-line for more than 10 minutes. Hey, don't panic. It is really easy, trust me!

First off, I made sure the site's root folder permissions were 0777 and that PHP Safe Mode was disabled. This will keep headaches away later in the process. Then, I uploaded kickstart.php and the new site's backup archive.

I now needed a way to remove the old site and restore the new one. I chose to use SSH to remove the old folders. I could as well have used FTP, but removing the thousands of Joomla! files with FTP takes forever, especially if your host doesn't allow multiple concurrent FTP connections from the same IP address. A few seconds away everything was gone, except kickstart.php and the archive.

Up next, I used Kickstart to get the archive unpacked, ran the installer to get my site restored and than clicked on the relevant links on Kickstart's page to get rid of kickstart.php and the archive.

Phew! Everything was ready now. A quick test and I was done.

Conclusions

JoomlaPack and its accompanying scripts can play an important role during site migration to Joomla! 1.5.x. The steps you can use its time and frustration reducing services are:

• Backing up the original site (JoomlaPack, J! 1.0.x package)
• Making a local clone of the original site (Kickstart, JoomlaPack Installer 2)
• Creating a snapshot of the new site (JoomlaPack, J! 1.5 Native package)
• Deploying the new site (Kickstart, JoomlaPack Installer 2)

JoomlaPack's multipurpose functions as a site backup and cloning tool can save you time during the migration process, letting you focus on the important part: transferring data to the new Joomla! version. It can provide you with confidence that the end result can be deployed in a matter of minutes, without risking bringing down the live site during the migration process, or to iron out migration-related issues discovered along the way.

But if you don't believe me...

... I can prove my success story. Yes, I know what you're thinking. I am JoomlaPack's author, so it is reasonable - if not selfish - to praise my software. You might even think I made this all up.

The example is very real and the only pitfall in the process was the initial migration (the migration only worked when I copied the darn SQL file instead of HTTP uploading it).

Need more proof? The site I was talking about is that of the Hellenic Association for Adult Education. The new site, facelifted is here, whereas the original site's clone is here. As you can see, the content is there, just the forum format and software changed and the past newsletter issues have been temporarily removed, on client's request.

If you do believe me, after all, ...

... you can use JoomlaPack on your site today. If you have a great success story about how JoomlaPack saved the day for you, send it to us.

Getting closer to 1.2.b1

After having done a lot of work the past few days, we are getting closer to releasing 1.2.b1. The most notable additions are:

• Support for accessing the backend with HTTPS (still needs some intensive testing)
• Single file exclusion filter
• Backing up without using AJAX

The codebase had the HTTP protocol hard-coded in the URL generation part, which made using the component impossible for people accessing their backend through HTTPS. It only got worse if you were actually enforcing the use of HTTPS with an appropriate .htaccess file. The fix I did was rather rudimentary: JoomlaPack chooses its protocol prefix based on whether the server has informed PHP that we were called through the SSL protocol. This feature still needs extensive testingto make sure there are no loopholes in the code and that it supports non-standard HTTPS ports (which means, anything but port 443).

As far as the single file exclusion is concerned, this was an asked for feature for quite a while. Many people want to exclude just that pesky unreadable file their host puts on the website's root. Or, maybe, those two big video files from the downloads directory. You get the picture... The interface is slightly weird, mainly because it just doesn't look like Joomla!. There are two panes, the left one displaying directories and the right one displaying files. You click on the directories to visit them, click on the files' checkboxes to toggle the exclusion status. Pretty easy, but I'd certainly prefer to code a proper tree view on the left side, a la Konqueror, or Windows Explorer :/

The "backup without AJAX" was a last minute addition, after some user had terrible problems with using AJAX during the backup, for no apparent reason. This can happen due to a million reasons, one of them being that the server is overpopulated. Instead of giving up and let frustration get the better of you, you can just switch to the brand new "JavaScript Redirects" backup mode. This is very simple, indeed. After each piece of work performed (a "step" in JoomlaPack jargon), a JavaScript redirect is issued to make the process proceed. It's not the same as the front end backup because it neither uses HTTP 301 headers, nor does it output a blank page during the backup. On the contrary, it outputs the familiar "backup status" messages, albeit in a funny looking way, yet.

All these exciting features were added in SVN 113, just a few minutes ago :)

Introducing Kickstart and JPA

One of the most notable additions in JoomlaPack 1.2 - which is still in Alpha - is that we now offer a renowned archive format, the JoomlaPack Archive (JPA) format. Its major benefit is that it can be more reliably created in the context of PHP script, such as JoomlaPack. On the downside, it is a custom format, without support from external tools. In here I will tell you not only how you can extract the archive, but how you can easily restore a site's backup too! But first, let's see how and why we got there...

Most people upload files to their sites using FTP. Despite of being the most widely used file transfer protocol for uploading files, it comes with some drawbacks. The most prominent is slow uploads when you have lots of small files. A Joomla! installation is a typical example, with over 3000 files occupying a mere 15Mb. No matter how much bandwidth you have, uploading takes at best half an hour.

Wait a minute! Half an hour for 15Mb worth of files?! It reminds me the dark days when I used PSTN to connect to the internet... But, what else can you do? If you upload the backup archive itself, it certainly gets uploaded fast (the more bandwidth you have, the faster it's uploaded), but you can't "run" the ZIP/JPA file. Or can you?

Using this thought, I decided to create a script which can "bootstrap" the installation process. I won't take credit for the original idea; I took it from the way software such as Joomla.Start works. I just enhanced the original idea in order to:

• use AJAX-powered, multiple step unpacking, so that it can operate on huge archives
• support both ZIP and JPA formats
• handle .htaccess renaming automatically
• delete itself, the installation folder and the archive after the installation is complete

It has one fundamental limitation, which is actually a limitation of PHP itself: the Safe Mode has to be turned off, or the folder on which you'll extract the archive has to be owned by the same user as the one the web server runs under. Just setting the permissions won't help it.

So far this software is alpha as well and you can get it through our SVN repository (directory kickstart). In order to use it just upload the backup archive and the kickstart.php to your server, then visit http://www.mysite.com/kickstart.php. Just follow the instructions from that point. In the end you'll have a fully functional site set up in -literally- minutes!