Our blog has moved

Well, for the last few months our official blog has been moved inside our site, at http://joomlapack.net/resources/blog.html.

We do recognize that some people have occasional problems with accessing our forum, e.g. their registration confirmation email is caught in the spam box and they can't retrieve it. As a result we'll keep the comments of this blog post open so that you can contact us in such a case. However, this is NOT the preferred way to do so. You may want to try these methods first:

• Send a Twitter message @nikosdion or @joomlapack
• Drop me a line in the contact form of my personal site.
• Send me an email; my email address is present in each and every JoomlaPack PHP file. It is also accessible from your site's back-end, as the author's email address for the JoomlaPack component.

Thank you!

Nicholas K. Dionysopoulos

JoomlaPack Lead Developer and project leader

Split ZIP Support in upcoming JoomlaPack 2.3

In my previous blog posts I described some of the new features arriving with JoomlaPack 2.3 and presented the brand new JPI4 installer. This time, I'd like to talk about a different feature, less visible, but very important nonetheless: Split archive support.

Some hosts impose a hard limit on the maximum size a file can have on their server. It's different than the account quota we're all used to. Whereas account quota limits the total size of files (and database contents) your account can hold in total, the maximum size limit imposes a restriction on the size a single file can have on the server. When your JoomlaPack backup archive tries to grow beyond this size (usually, that's 10Mb), your host abruptly ends the backup process throwing an HTTP 500 Internal Server Error.

The ZIP archive format supports a feature called spanning or spliting. A ZIP archive utilizing this feature can span multiple files of a predetermined maximum file size. For instance, you can have 20Mb of data spanning four 5Mb files. The unarchiver application will treat them as a single archive when extracting the ZIP file. JoomlaPack 2.3 is now able to take advantage of this feature with a new configuration option in the Advanced section. You simply set the maximum size of each part and let it work out the spanning details.

This new feature also caused an avalanche of required changes in pretty much all of the JoomlaPack Suite software components. For example, the component must support downloading the multiple parts. Kickstart has to expect and handle multi-part ZIP files. The same goes with eXtract, as it presently doesn't know about split ZIPs. Last, but not least, the JoomlaPack Remote application will also have to handle downloading the multi-part ZIP files.

Rome wasn't built overnight, neither does JoomlaPack. So far we have managed to implement the generation of split ZIPs and their handling in the Administer Backup Files page. Before the beta is released, Kickstart will also support split ZIPs. By the time 2.3 goes stable, eXtract and Remote will also support split ZIPs.

Our next goal is to give the JPA engine the same support. It's an equally engaging process, so it's not possible to predict if this will make it to the stable. It highly depends on the available time I've got to develop such a thing.

Meanwhile, you can try - at your own risk! - the split ZIP support by downloading the latest developer's release from the Bleeding Edge.

Introducing JoomlaPack Installer 4

One of the features we were working on with utmost secrecy – or not so much of it if you are following us on Ohloh – is that of a new installer script, duped JoomlaPack Installer 4, or JPI4 for short. JPI4 is supposed to be a site installer script which, unlike its predecessors, doesn’t depend on the Joomla! Framework and has less chance of failing when some ACL components are installed. It’s also supposed to be lightweight and easy to use. Here is a sample video of it in action:

A few quick conclusions from this video:

• I write better English than I speak
• I can’t speak and use the computer at the same time
• jQuery UI effects are quite impressive

You are welcome to share your thoughts with us. Your feedback is most welcome!

JoomlaPack 2.2 imminent and 2.3 feature plan

With JoomlaPack 2.2 Stable being only one week away from official release (it's scheduled for May 24th), work has began on JoomlaPack 2.3. You may think it's a little too soon, but it's never early enough to design new exciting features, don't you think?

Before I start ranting about new features, I want to share with you some exciting last-minute features added in JoomlaPack 2.2:

• Easy to navigate, translatable back-end menu. It was there in 1.2.x, it was removed in 2.0, so it's back again. With proper icons. Cool!
• Back-end Control Panel quick backup icon. Integrates JoomlaPack right into your Joomla! Control Panel. The icon displays a warning icon if your last backup has failed or if backing up your site is overdue. Useful and cool.
• Display optimization. Since some JoomlaPack text was unreadable on certain browsers (Chrome, anyone?), I made the font-size uniform throughout.
• One-click profile change. having to select the active profile and click on the Change Profile button was an overkill. Now you just have to select the new profile. Convenient.
• Plus some important bug fixes you are about to find out. Just wait for May 24th :D
  

Since JoomlaPack is in a very mature state, the releases in the 2.x series really focus on detail, automation, and plain cool stuff. About this latter category, even though less than 1% of you will ever use them - or get to know about them - I know that those who will use them will really appreciate them! What's on the menu, then?

• Easier site migration with DirectFTP file packer engine. Annoyed by having to backup, download the backup, upload the backup to the new server and unpack the backup file there to create a clone or transfer your site? Me too. The optimal solution would be backing up directly to the target server, sparing you from the need to download / upload the backup file. This is what DirectFTP does. Instead of backing up to an archive file, it backs up to a remote FTP server. Life is neither fair, nor perfect, so getting it to run requires some trial and error adjusting the Magic Numbers. This is a feature for geeks and professional web developers, not the average layperson. Preliminary tests show that it has some good potential.
• CRON made easy. Setting up your CRON script is a two step process. First, you have to set up the PHP file which runs the backup. Then, you have to set up the CRON job on your server. While JoomlaPack can't help you with latter step (it exists in the realm of server setup which JoomlaPack doesn't have access for security reasons) we can help you with the former part. Let's face it. Few people understand enough PHP and cryptic parameters to successfully set up the script which runs the backup. To this end, we'll include a "CRON script manager" feature. It will create those scripts for you. What's more, it can be set up to email you the backup file. This is nowhere near being rock-solid, as the maximum size of an attachment is usually too small to hold a complete site backup. It can, however, efficiently hold a medium-sized database dump. So, there you go, you can have something like daily database backup delivered to your doorstep... umm... email address, I mean.
• Split ZIP files. On popular request! Some servers put a cap on the maximum size of files on their server, for example 10Mb. If you try to create a larger file, they just fail. The ZIP format has this handy multi-volume feature, but so far JoomlaPack can't make use of it. This is about to change. This is a very advanced feature, useful for a handful of occasions, so don't expect it to integrate perfectly with JoomlaPack's "Administer Backup Files" feature or JoomlaPack Remote. The only way to download the split backup file will be using FTP. Of course, something is much better than nothing and the people to whom this feature is addressed can attest to this fact.

Every time I write a post on new features I get the impression that the only thing missing from JoomlaPack is the kitchen sink. Of course, I am always wrong; you prove me wrong. That's not something bad, on the contrary. Every time you prove me wrong, you get to have even more features implemented for everyone to benefit. Don't feel shy, please come by our Feature Requests forum and ask for something crazy and useful. Who knows? Maybe the next useful feature is the result of your request!

And since we are at this, don't forget to vote for JoomlaPack in JoomlaShack's competition for the Ultimate Joomla! Extension. Thank you!

What you don’t see matters the most

Whereas it’s true that most JoomlaPack features are exposed by means of its graphical user interface, its the most important aspect is the backup core engine. What matters the most is having reliable backups every time, under any circumstances. This is what the backup core engine (CUBE) does and that’s why I had to improve it for the upcoming JoomlaPack 2.2.

Up until now, certain bottlenecks in the file packing code made it virtually impossible to use JoomlaPack on big sites running on slow servers. The reason was really simple. JoomlaPack had to scan all contained files and folders of a given folder in one go. If this operation took long enough (on slow servers it did), it would crash due to time outs. Same goes with packing big files. If many smaller files had been archived before hitting the big file, a timeout may had occurred.

Up until now we had to work around these limitations with certain fuzzy procedures, such as fiddling with Magic Numbers and changing to the Slow algorithm. The effect was that fewer operations would happen in a single step, reducing (but not always eliminating) the possibility of a time out occurring. Of course, this was a half-baked attempt to circumvent the problem, rather than solving it.

JoomlaPack 2.2.b1 will include my fixes to these problems. JoomlaPack is now smart enough to detect if a long file operation might cause a timeout, stop it, roll back and execute it in a new step, having the maximum amount of execution time available. There is also an improved filesystem scanner engine which detects large directories and forces their scanning to occur inside a single new step, to avoid timeouts.

The benefit of this approach is that JoomlaPack will now be able to work on sites where it wouldn’t work out-of-the-box and it will be substantially faster on sites where workarounds had to be implemented. For example, my own site required the Balanced settings in order to be possible to back it up. The backup took 15 minutes to complete. Now it works with the Optimistic settings and backs up in under 3 minutes! Wow!

Moreover, some servers couldn’t handle the amount of information required to be stored for large sites in the database between subsequent steps and crashed with a database error. I have included an option to force storage of temporary data in files instead of in database. This is required if you have directories deep down the fileystem hierarchy, containing more than 100-200 files.

Finally, there is a new feature you’re gonna love: Live Update. That’s right, JoomlaPack can detect if there is a new version released, notify you and install it with a single click. Please note that this feature requires that your server can draw data from other Internet sites. If RSS syndication with other sites works for you (i.e. you can display other sites’ RSS feeds in Joomla!) then Live Update will probably work as well. Also note that on servers where JoomlaPack’s “alternate installation” is required the single click update will fail. This is a limitation of the server.

The Earth Hour 2009

With only half an hour before 20:30 local time in Greece, I am turning off the lights on my house. Don't know what I'm talking about? It's this year's Earth Hour. We just turn off all the lights for an hour, 20:30 to 21:30 local time, wherever we are on the planet. It's the least we can do for the planet we live in.

Being a person who cares, I've decided to participate. For this hour I am turning off all lights, my PC (which consumes more electricity than a light bulb), light up some candles and enjoy the atmosphere. If I get a sudden Internet connectivity craving, there's always my EeePC. With less than 20W of power consumption, it's the most eco-friendly device I own :)

So, did you turn off your lights today?

Still on the free side

In my last blog post I shared my thoughts with you about having two versions of JoomlaPack, one paid and one for free. Well, 2.1 Stable is released and – as you noticed – it remains free of charge.

There was a number of factors which led me to this decision. The most important of which was the sentimental factor. I began creating JoomlaPack as a small personal project in order to help me accomplish a task I had at hand. I released it under the GPL in hope that other people would find it useful. The rest is history…

I simply found out that the spirit of JoomlaPack commanded that it should remain Free, both as in “free speech” and as in “free beer”. It is my personal attempt to prove that Free software doesn’t have to be an ugly, cryptic, undocumented and unsupported piece of code, meant only for the privileged elite of “hackers” and “geeks”. JoomlaPack was designed with the user in mind, right from the start.

But, let’s face it, the developer, yours truly, does have the need to make a living. Let alone the need to finance this expensive hobby: developing, writing documentation, supporting, improve the web infrastructure of this project does cost money and time. That’s why I decided to keep accepting donations and added advertisements on JoomlaPack.net. I sincerely hope that you’ll keep on donating . This is what keeps this project both Free and “for free”.

As a side note, I’d like to point out that both the “Free Edition” and the “Special Edition” are fully functional and distributed without any charge. You should think of their names as in “free from unnecessary features” and “made for sites and people with special interests in mind”. I know, this isn’t probably the most thought out or correct marketing approach, but you know me… I am just a code monkey, not an advertising agency :)

Free of charge, unless you want to pay for it

I have been considering a way to make a little money out of JoomlaPack. The donations are not very disappointing, but let’s face it: donations can only go that far. On the other hand, I detest proprietary (closed-source) software. Moreover, I believe that users should be able to pay in order to use a software only if they choose to do so. So, how do you combine Free (as in speech and as in beer) with Commercial?

To this end, beginning with the JoomlaPack 2.1 Stable release scheduled for mid-March, there will be two versions of the JoomlaPack component and JoomlaPack Remote: the Free version and the Special Edition version. Both versions will be 100% Open Source software (licensed under the GPL, so you can download their source code for free, anytime) and you can use them on an infinite number of sites for an unlimited period of time. Their main difference is that the advanced stuff (like multiple database backup, multiple backup profiles and extension filters) will only be available to the Special Edition version. You’ll gain 1 year access to Special Edition downloads by donating 20 Euros to the project.

So, your options for using JoomlaPack will be:

1. Don’t pay a dime; use the Free version. About 90% of users won’t need anything more than that in order to backup, restore or move their site between hosts.
2. Evaluate Special Edition for free. Why pay for something you can’t get to use beforehand? Our “Free Ride” plan gives you time-limited (24 hours) access to the full Special Edition downloads. You can keep the software after this period expires and use it legally on an unlimited number of sites. Yes, it’s like giving away free beers!
3. Download the source code and build it yourself. Free and Open Source Software (FOSS) is about freedom of choice. If you don’t like paying, you can always download the source code for free and build it yourself. It’s not for the faint at heart, though… BTW, the Bleeding Edge will only serve SVN builds of the Free version.
4. Donate 20 Euros and get 1 year of access to the Special Edition downloads. This is – obviously! – what I am hoping you’ll do if you are a serious webmaster or a web professional. Our site will have an integrated “donate & register” sign-up feature to support this option. Everybody who has donated the equivalent of 17 Euros or more since January 2008 will be entitled to a free account.

Please note that no matter if you use the Free or Special Edition version, how you got it and if you have a current JoomlaPack Club membership, you can use our software on an unlimited number of sites – commercial or not-for-profit – and for an unlimited amount of time. There will be no restrictions whatsoever. This is possible thanks to the provisions of the GNU General Public License under which JoomlaPack is licensed.

Some things were, are and will eternally remain free of charge. All of the other tools, like Kickstart and eXtract, will remain free of charge. Support will remain free of charge, for all. We take great pride in our support and we are willing to keep it this way! Documentation will be provided on-line and free of charge (albeit with a few unobtrusive advertisements) through our JoomlaPack.net site.

All of these changes will be applied somewhere in the first half of March, so don’t ask me for details yet! I hope you don’t mind this small change of policy. It’s just that the developer needs money to buy something to eat and rent a place to stay (and write code – computers don’t like rain).

Getting closer to 2.1

We are getting closer to the release of 2.1.b2, a landmark release of JoomlaPack. With this version, we are getting closer to what we considered feature-complete all along those two years. The new features – compared to 2.0 – are:

• Easy Mode. We were all newbies and inexperienced users. Some of you have installed Joomla! only a short while ago and find the plethora of JoomlaPack’s features, options and settings intimidating. Feel intimidated no more! The Easy Mode basically provides a cut-down, easy to grasp, oversimplified interface without compromising on the effectiveness. The configuration page is much easier to use, as you only have the choice between three predefined sets of configuration options. Newcomers to the world of Joomla!, welcome the easiest way to backup your site.
• Off-site directory inclusion filter. Backup folders outside your web server’s root!
• Extension filters. Single click exclusion of components, modules, plug-ins, templates or even languages from your backup! In other words: build a site with everything installed; if you need a customized version, just select which extensions not to back up. Cool feature for professional web developers.
• Front-end “light mode” backup. You can now back up from any Internet-capable device: EeePC’s, Pocket PC’s, cellphones, Wi-Fi enabled iPods… you name it!
• Support for tar and tar.gz. Note that tar.gz requires an external gzip executable. A niche feature for all you *NIX fans.
• Automatic troubleshooter. JoomlaPack detects if it failed last time and tries to apply two sets of “safe but slow” defaults. This will let it run smoothly on even more hosts without asking us for help and without you having to tweak Magic Numbers. Nice and clean!
• Better notification of potential issues. Potential problems will show up before backing up; this is the best approach to help you to keep out of harm’s way.

JoomlaPack also received its fair share of bug fixes and minor adjustments. From the easiest to the fairly complex, these fixes and adjustments help our software work in even more “difficult” environments.

As always, we are commited to providing the community with a professional grade, reliable, open source and Free (as in speech and as in beer) backup solution, complete with documentation, support and a set of incredible utilities.

Coming up next: JoomlaPack 2.1

It's almost a week since 2.0.1 was released, but we haven't been lying dormant. On the contrary, this has been a very productive week with major work having been carried out towards the next beta release of our component, JoomlaPack 2.1 Beta 1. The major highlights of this upcoming release are:

• Off-site directory inclusion. You can now backup directories above your site's root, provided that your server's open_basedir restrictions allow it. Many webmasters use directories above the site's root to store download repositories and such, therefore it's a handy feature to them :) The off-site directories are stored inside a new folder in the archive, one which does not already physically exist on your site.
• Extension filters. I've been planning this ever since 1.0, but the code base wasn't mature enough. With this filter you can completely exclude any installed component, module, plug-in, template or even language! This is the absolute step in creating customized sites by performing nothing more than a backup! Note: components' database tables are not automatically excluded because there is no technically sound way to determine which table is owned by which component.
• Front-end "Light Mode". With an increasing number of ultra-portable Internet-connected devices, ranging from Netbooks to PDA's to cellphones, it's desirable to be able to perform a site backup right from your palm, while on the move. In order to ensure this possibility, the "Light Mode" was invented. Using very plain pages (so users priced on data volume won't suffer) and front-end access, authenticating by means of the "secret word", the light mode enables you to backup your site anywhere, anytime, even while commuting to work or attending that boring meeting!
• Tar and tar.gz support. The popular uncompressed tar format is based on a modified version of PEAR's Archive_Tar library. The optional gzip compression is realized using your operating system's gzip binary; doing this inside PHP has proved to be an awfully bad idea in the past (remember memory outages on JoomlaPack 1.0.x?)
• Integrated news reader. We know we release a new version every now and then and that makes it hard for you to keep up. Since the "old" system of updates querying had the potential to cause insane amounts of traffic to our website (therefore it was eradicated in 2.0), we decided to implement a different idea. The new version fetches, caches and displays JoomlaPack's latest headlines through RSS, using the trusted Joomla! newsfeed syndication technology. So, no more remembering to visit JoomlaPack.net, the component's Control Panel does it for you ;)
• Tons of bug fixes. It's bug hunting season and we came back from our "bug safari" with quite a few trophies! Fallen pray to our hunting skills: a dozen or so sources of PHP notices, JPI3 displaying odd behaviour with multiple database restoration pages, a broken email feature, undesired behaviour of quotas, erratic handling of cache and temporary directories and some few other minor bugs.

Now it's minor tweaking and QA time, so expect a formal beta 1 release somewhere around New Year. It'll be our present to you for the holiday season ;)

Of course, we're still working on other stuff as well, including our Native Tools. But that's the topic of another post, in due time!

It's the final countdown!

After 6 months in development, countless hours of testing by JoomlaPack Developers, JATS and regular users, fixing a pile of bugs, one alpha and one beta release, we are finally ready to release JoomlaPack 2.0 Stable. So, spread the word around...

JoomlaPack 2.0 Stable
Global Launch
Sunday, December 7th, 2008 - Noon, UTC

Our site will be down for 2-3 days for maintenance and a necessary face-lift. It will be back on-line at the official launch date and time. Meanwhile, let me give you the highlights of this release...

Coded around the Joomla! 1.5 API, the JoomlaPack 2.0 component delivers a high-profile backup solution for your site, while remaining Free, both as in "freedom" and as in "zero price". More powerfull than ever before and compatible with a wide assortment of server setups, it's designed to protect your valuable site. With exciting new features, like live full site or database only restoration and multiple database-only backup, it makes backing up your site a child's play. Our thorough documentation will guide you through and - should the need arise - our stellar support team is always willing to lend a hand, right now and for free!

We wanted to push the PHP site backup technology to its limits ever since the first release, more than two years ago. So, as you readers of this blog already know, we decided to take site backup... to your desktop! AFAIK, nobody attempted this before, but we did. Full fledged multiple site backups running in parallel, right from your desktop, thanks to JoomlaPack and a hefty XML-RPC plugin. The whole thing is called JoomlaPack Remote and will be shipped with this release.

As far as the backup archive extraction is concerned, we've got the traditional command-line suite of utilities called the JPA Utilities (JPAUtils) or, a new addition to the webmaster's arsenal, the vastly improved JoomlaPack eXtract graphical wizard for Windows and all x86 systems running WINE. It's click, click, backup archive is now extracted, thank you very much ;)

Our versatile web-based ZIP and JPA archive extraction tool, Kickstart, is even more powerful and smart. It can write files to your site using the traditional PHP direct file access mode, or the new FTP mode for maximum compatibility with common shared hosts setups. Using the FTP mode, it can automatically work around permission and ownership problems ("ownership hell") which would otherwise give you, the webmaster, a tough challenge trying to restore or upgrade your site.

The default embedded installer is now the JoomlaPack Installer 3 (JPI3), based on the Joomla! 1.5 installer, improved under the hood. It even supports the "auto mode" which - in conjunction with Kickstart - allows for fast two-click site restoration/migration for advanced web professionals.

Welcome to the future of Joomla! site backup.

Tools of the trade

A topic I was willing to cover for quite a while is the toolchain used to build JoomlaPack. Some programmers have even asked me what are the best tools to build Joomla! extensions. While probably not perfect, the combination I'll present here has worked wonders for me.

• Integrated Development Environment. After having tried a number of different PHP IDE's, I settled for Eclipse PHP Development Tools. An Open-Source project by Zend, the makers of the PHP core engine, it uses the famous Eclipse IDE to provide a thorough PHP IDE. Another alternative is PHPeclipse, but lacks some features which facilitate development and management of large and/or complex problems.
  
• Version Control. Tracking your changes and being able to roll back when pooh happens is a must-have for anything more complex than "hello world" software. I regularly use Subversion (affectionately called SVN be everyone). In order to integrate it with Eclipse, I use Subclipse. By the way, this makes up for the best Subversion client I've used so far!
• Testing server. Every developer needs a local testing server. The easiest solution is to use a prepackaged, easy to install solution, like XAMPP or WAMPserver.
• Reference book. Joomla! 1.5 offers a very rich and powerful framework. Before setting out to develop an extension for it, it is very useful to read a book on Joomla! extension development. The two books I have read so far are Mastering Joomla! 1.5 Extension and Framework Development (wonderful reference book) and Learning Joomla! 1.5 Extension Development (more like a tutorial, recommended for beginners). Of course you can shop for something else and it probably takes a few tries to find the one which is best for you.
• Debugging tools. I regularly use J! Dump to gain insight on my code. To be perfectly honest, once I found out about it and began using it, I started to wonder how could I code without it! It replaces your needs for ugly calls to var_dump(), die(), etc.

That's about it, guys and gals. Developing for Joomla! is neither hard, nor expensive. All the tools, save the books, are Free and Open Source software.

Have fun writing some high quality code!

Taking site backup... to the desktop!

Lately I have sporadically "leaked out" some of my thoughts on JoomlaPack's future and features I am working on lately. One particular feature, is providing XMLRPC services. Why is it so important, you might wonder? Well, let's see what XMLRPC is all about.

Most people have published on a blog using a desktop application, be it ScribeFire, Windows Live Writer, w.bloggar, etc. The common feature of all these tools is that you are writing your article off-line and when you're ready you can publish it on your blog. The magic behind the scenes is XMLRPC. This buzzword really means that the client (your blogging software) sends a chunk of XML data to the server (your blog). This is specially formatted so that the server understands it has to perform an action (in this case, post an article). When it's done, it sends a response in XML format back to the client. The client processes it and tells you that your article is posted.

It wouldn't be so much of a deal if it weren't for the XML format itself. It is generic enough to cater for every use imagineable, but it's also very robust that it can be very well described. It is analogous to paper in written communication. While it does not provide the language, it provides the medium on which the language can be carried between the two parties involved. The choice of language (and the interpretation of the messages) is left to the two parties. But it's really cool to know that the message will always be written on paper, neither on stone, nor on papyrous.

Joomla! 1.5.x has two built-in XMLRPC plugins for posting articles. The first is the Joomla! XMLRPC services plugin and the other one is the Blogger API XMLRPC services. This way you can use an off-line blogging tool with Joomla!. In fact, it's the best way to have a newbie post articles in a Joomla! site, because it carries virtually no risk of messing up settings.

This is all very cool, but you are probably wondering where does JoomlaPack fit in? What does this article posting thing got to do with site backups? Well, remember when I said XMLRPC is very generic? It is so generic it can be used any time you want to tell the server to do something and inform you on the outcome. For example, we could tell the server to load JoomlaPack's engine (CUBE), starting a backup and returning us backup status information. In fact, this allows us to control the backup process from any networked device as long as it can connect to our site's server.

The missing piece of the solution is the client part. Ideally, a client would run on any operating system (cross-platform) and be able to backup any site on which JoomlaPack runs. It should even grab a local copy of the backup archive for safekeeping. This piece of software would truly take site backup to the desktop!

Today, in SVN revision 313 (notably, this revision number is the same as Donald Duck's car registration number - I'm a lifetime fan of Donald :D - and has a number of notable properties ), I have commited the first part of the solution, the XMLRPC services plugin. I will now start writing the desktop application in Lazarus, a cross-platform programming environment similar to Delphi. My expectation is that this will be stable somewhere in mid-Q1 2009.

Stay tuned. JoomlaPack is about to redefine site backup.

A new tool in the 'hood!

You talk, we hear. We hear you and we think hard. Then, we come up with a solution, code it, test it and release it. That sums up the development process here at JoomlaPack Developers.

This time, we heard a lot of you complaining that sometimes backup archives are hard to extract. It seems that PHP4 users have a lot of grief with ZIP files, because on their platform the larger files appear to be corrupt with most traditional archivers. A lot of people who don't have an option but to use JPA files complain - rightfully - that this format seems arbitrary and lacks support from visual tools. You talked. We heard you.

Please, welcome JoomlaPack eXtract! eXtract is the latest addition to our utilities collection, with a twist. It is visual extraction wizard running on the Windows(tm) platform, similar in look and feel to the integrated Windows' ZIP extraction wizard. eXtract can extract ZIP and JPA backup archives generated by JoomlaPack. It comes with an installer to make your life easier during installation. Despite being version 0.1, it's usable.

Download eXtract now from our JoomlaCode.org FRS or the SourceForge.net mirror.

Updating Joomla! websites with JoomlaPack Kickstart: a step by step guide

This article is long owed to the community. I apologise for taking forever to write it down, but development of the 1.3 version was much more higher in my priority list. In the light of the new Joomla! 1.5.7 release - and having two sites demanding urgent update - I have decided to share the secret with you: It's upgrade time and Kickstart can be used to apply the Joomla! update patches to your site easily. It's really simple, too. Let's go!

First of all, you'll need the SVN version of Kickstart which supports plain file or FTP file access modes, integrates AJAX and JavaScript redirects modes and includes a few more improvements. Since this is not released yet, you can download a developer's preview - but very stable already! - version from here. Extract the PHP file in the archive you've just downloaded.

Next up, you'll need the ZIP version of the Joomla! patch, which you can download from the Joomla! project's file repository site. Depending on which Joomla! version you've already installed, download the appropriate ZIP file. The tar.gz file will not do! For example, if you've got Joomla! 1.5.6 and want to update to 1.5.7 you must download the Joomla_1.5.6_to_1.5.7-Stable-Patch_Package.zip.

Upload the Joomla! ZIP file and kickstart.php to your site's root. If you used FTP, make sure both files' permissions are 0777. Make sure you have a working backup of your site before you proceed! In the unlikely event you run into a problem we want to be sure you can roll back. There are many ways to take a backup of your site, but we obviously suggest you to use the open source JoomlaPack backup component.

Now visit the kickstart.php URL, which is something like http://www.example.com/kickstart.php, substituting www.example.com with the full URL to your Joomla! site. In the page presented to you, make sure the update ZIP is selected in the topmost combo box. For the operation method we suggest using the "JavaScript Redirects" method, unless you know what you're doing. Depending on your site configuration you can either use "Write directly to files" or "Use FTP", supplying your FTP connection information as well. Most users will need the second (FTP) option.

After you're done setting up Kickstart, hit the big green "Start" button. You will see your browser reloading the page many times; this is normal. On the final page, click the second link titled "here". As the page reads, you're (almost) done! If you had an .htaccess file on your site, it is now renamed to htaccess.bak. Using an FTP client, rename it back to .htaccess and you're really done, in - virtually - no more than a few seconds!

Try it! I just upgraded two sites in 3 minutes flat (including upload times!).

Editor's Pick and site look

JoomlaPack is the new Joomla! Editor's pick for extensions. We are greatly honored to be chosen. An official announcement of this will be made later on the site.

JoomlaPack has a new site template. The template was designed by JoomlaPraise and it is their Joomla! 1.5 template contest entry. We chose this template because of its clean design and its emphasis on content. There still are some minor problems with the forum version of this template so we're going to give it one last shot before we release it as is. As with anything that changes, things may go wrong so please do tell us on the forums any problems you see with the design.

The upcoming JoomlaPack 1.3 brings new features

We might be just a few days before the release of 1.2.1 stable, but the JoomlaPack development team doesn't call it a day. We are actively developing the upcoming minor number version of the component. As with all past minor number versions, it brings along many new features. We feel it's only fair to share with you.

• Joomla! 1.5.x native mode only. If you follow Joomla! announcements you should have known - since about 8 months - that Joomla! 1.0.x development has stopped. No further updates are planned for that version of our favourite CMS. In other words, Joomla! 1.0.x is dead. Period. Our single most major step for JoomlaPack 1.3 was removing J! 1.0.x support and sticking with J! 1.5.x. Not constrained by the limitations of the old generation Joomla! and taking advantage of the empowering Joomla! Framework we have set to develop a modern code base, based on the principles of the MVC pattern, which will be much more easily maintenable in the future.
• Backup profiles. Until now, JoomlaPack was constrained to a single configuration. Setting configuration parameters and filter options was of an all-or-nothing approach. If you wanted to tweak some filters to get a different type of backup (let's say, filtering out some ill-behaving component) and then revert to your regular settings it was impossible. JoomlaPack 1.3 incorporates backup profiles, an unlimited number of profiles storing configuration option and filter settings. What's more, you can select which profile to use at backup time, even with front-end backup!
• Backup statistics. You asked for and you get it. JoomlaPack 1.3 will keep a lot of information about each backup taken. This has some nice implications you're gonna love:
  • No more need to reset after failed backup attempts. Based on information from Backup Statistics, JoomlaPack's engine will detect failed backups upon visiting the Control Panel page or starting a new backup, cleaning after them.
  • Describe and comment on your backups. No more need to remember what changes were made between individual backups (applies to backups started from the backend)
  • Ability to keep a limited amount of backup archives on your output folder automatically.
  • More informative backup administration page.
• Optional non-AJAX mode for the administration. Up until now, if your configuration was incompatible with AJAX you wouldn't be able to use JoomlaPack. After all, it relied on AJAX to operate the filters pages. This is no longer true. All basic operations on the filters pages are now possible to be executed in non-AJAX mode. Just select "JavaScript Redirects" for the "Backup Mode" in the Advanced tab of the configuration. The non-AJAX mode does not apply to some more advanced operations, like checking for database connectivity while editing a Multiple Databases setting, or fetching the default output directory in the configuration page.
• No more temporary directory setting. Joomla! 1.5.x provides with a standard temporary files location and JoomlaPack uses it. Of course, it needs to be writable!
• Embedded documentation. The basic set of documentation (help for each JoomlaPack page) will be included with the component, available by pressing the "help" button on the toolbar.
• Automatic problem detection. There are a number of server configurations which might cause JoomlaPack to fail. We know about them and know how to detect them. The next big step is warning you in advance, right from the Control Panel! It's just like the "You're using the default output directory" warning of JoomlaPack 1.2.1, only it covers a whole lot more situations.
  
• Revamped Kickstart. The new Kickstart will have a few changes over its predecessor. For starters, there will be no more two different files for AJAX and non-AJAX mode. The mode selection will be a radio button on Kickstart's first page. We are also planning to implement a FTP mode, to get past servers using Safe Mode and/or have "wrong" permissions.
  

Roadmap to JoomlaPack 1.2.1

Just over a week ago the stable 1.2 version was released to the public. As always, the stable release signifies the beggining of a new development cycle. This time, instead of jumping straight to developing version 1.3, we decided to improve the latest release in order to produce a safer, more performing, mature component.

This is the list of features we are currently working on, for version 1.2.1:

• Administrator directory restructuring and change of class names. CUBE is turning to a framework of its own right, built atop the Joomla! Framework. A big step towards full Joomla! 1.5 framework adoption, much of the guts of our component is being recoded to use only J! 1.5 API calls, while providing a backport layer for J! 1.0.x compatibility.
• Reduce and concatenate the steps (domains) needed to produce a backup. It occured to me that only two domains are actually required and should be present: database backup and file backup. In order to achieve this, a number of changes have to take place:
• Avoid extracting the installer files to temporary directory. Use the packaged installer files to "seed" the backup file instead. This also has a security enhancement bonus, as no "live" PHP files are written to the temporary directory, eradicating a direct access attack exploiting such files.
• Use name mangling on temporary files to minimize possibility of direct file access attacks
• Store the temporary file names and use this table when cleaning up. Moreover, clean up each temporary file right after it's been put in the archive. This is much more reliable than the current method which almost always leaves leftovers.
• The file list creation and packing steps are merged. This also means that much less data (in number of SQL queries and in total data size alike) is required to be written to database, solving many errors related to db server overload.
• Drop the "fast" algorithm. It is meaningless on most servers (crashes with timeouts) and the "smart" algorithm is already fast enough.
• Introduce Magic Numbers for smart algorithm. This will let you fine-tune the smart algorithm's performance to accomodate for slow, overloaded servers.
• Re-implement translations. It serves a double purpose: get closer to J! 1.5 compatibility and allow the use of semi-automated translation tools on the translation INI files. Some highlights:
• Introduce upercase INI keys by concatenating header and key, inserting an underscore between them
• Separate front-end and back-end language files
• Make a static method CLangManager::_($key) which translates the string like JText::_($key)
• Error and warning propagation and handling. Right now, if an error occurs in some part of the backup engine, it is not propagated and the process crashes in an unrelated point or completes to a partial backup (or, worse, no backup at all!). The idea is that if an error occurs anywhere in the code, the execution stops immediately, the error message is displayed to the user and also gets logged. In the case of warnings, they will be displayed just below the backup process messages.
• Distributed AJAX handling. The AJAX proxy will be taken apart and AJAX handlers will be grouped by page, hopefully creating less overhead during the actual backup process.
• JoomlaPack Installer 3. We are planning on creating the third iteration of our smart restoration script. This time it will be J!1.5-specific, based on the original J! 1.5 installer application. This is much harder than the traditional approach we took with JPI and JPI2, but it will be more forgiving in the case of errors - as you well know JPI2 is notorious for crashing silent when an error occurs during a J! 1.5 site's restoration. It will also cater for changing the cache and temp directories to their defaults if the pre-configured directories do not exist or are not writable, fulfilling a frequently made feature suggestion.

These are quite a mouthfull for a sub-minor number release! Right now the bulk of heavy refactoring is complete, namely the first two points. The rest of them are quite easy to implement, with the exception of the last two points which are bound to consume lots of our time.

The planned release date for 1.2.1 stable is mid-October 2008. Stay tuned!

Migrating a site to Joomla! 1.5: a success story

One of the tasks I had to perform less than a month ago was to migrate a site from Joomla! 1.0.15 to Joomla! 1.5.x. The site's operation couldn't be suspended for more than a few minutes. This was an exceptional challenge for testing JoomlaPack's tools in a real world task. Let's sum up the mission objectives and get to work:

• The original Joomla! 1.0.15 site must be upgraded to the latest Joomla! 1.5.
• The same or equivalent components will be used without loss of data, especially in regards to translation and mailing list contacts.
• The site must have virtually no downtime; a maximum of 10 minutes is acceptable.
• The resulting site must be fully operational. No broken links, missing images or any other mishap will be tolerated.

Sounds like Mission: Impossible, right? Impossible is nothing, as long as you have the right tools.

Step 1: Back the old site up

One of the things experience teaches up - albeit in a harsh manner - is that things can and will get awry, unless you're prepared for the unthinkable. In this case, the unthinkable is to bring the whole site down by accident, without a way to get it back online to its previous state.

That's why I used JoomlaPack 1.2.b1 (right then it was in a pre-release state) to get a snapshot of the site. Since this was a J! 1.0.x site, I used the -j10 package, which is Joomla! 1.0.x native. In 5 minutes flat the backup copy was sitting happily archived on my hard drive and USB key. I know, but redundancy makes me feel safer.

Step 2: Define the migration strategy

One of the common misconceptions is that when it comes to migration you can only act on a live site. Right? Well, since I did have a site snapshot at hand and a XAMPP for Linux powered machine on my desk, the only safe decision was to try migrating on a locally operated "clone" of the site. The result could be then uploaded and deployed within minutes on the live server.

Furthermore, I decided to have a "safe haven" should the restoration go wrong. For this reason I created a subdomain, with the intent to put the old site in it. It was actually easy; after uploading the site backup archive and kickstart.php I merely clicked my way through the restoration procedure and - presto! - the clone was right there, in the new subdomain.

During this process I also safely checked Kickstart's compatibility with my live server. It turns out that even when I deactivated PHP's Safe Mode, I was still unable to get Kickstart to work. Some head-scratching later, it turns out that the site's root was not writable to the web server user. I just connected to the server with my FTP client and changed this folder's permissions to 0777. That did the trick. This was a useful thing, since I used my newly found knowledge to "fix" the permissions on my main domain's site root folder to avoid any troubles during the restoration of the migrated site (more on that later!).

Next, I repeated the procedure on my local server. Now I had a perfectly working "clone" of the live site on my local server. It's time to have some fun!

Step 3: The actual migration process

It turns out migration isn't as straightforward as you might have thought, or read about. After a couple of frustrating attempts, I came up with the right (as in "worked for me") procedure:

• Install the migrator component on the original Joomla! 1.0.x site
• If you had translated the site using Joom!Fish, you'll have to install the respective plugins. Too bad they're only available in their SVN and you have to guess that they even exist! I don't want to be too harsh; after all at the time of this writing Joom!Fish 2 is still in beta. Installing them is simple. Just download the plugins and tables folders and place in the com_migrator's corresponding folders.
• Run the migrator component, which results in a SQL file. Keep a copy of it, you'll need it.
• Make a new folder for the 1.5.x site and extract the latest Joomla! 1.5 distribution there. Do not run the installer yet!
• Copy the SQL file in the new site's installation/sql/migration folder.
• Make sure the installation folder, subfolders and all contained files have read/write privileges for everyone.
• Run the Joomla! installation, making sure you select the migration content instead of sample content towards the end of the installation. Tip: Use the same prefix as the old site. Hopefully, nothing went wrong in the process. If not, you'll have to copy the SQL file again and then retry the process. During migration the SQL file gets altered. Why does it have to? Beats me.
• You have a semi-usable site now. Copy over any media (like, images) from the old site to the new.
• Install third party components. Remember that you need to activate the Legacy Plugin for components which are not Joomla! 1.5 Native (that's about 80% of the components I guess).
• If the components offer no backup/restore for their data, copy their tables from the old site's database to the new site's database.
• Make sure everything is in working order. Otherwise, keep on tweaking.

Step 4: Getting ready for transfer

After all this process was over, I had a fully working site on my local server, all aspects tested out and everything to the client's liking. The next step is to get it online. JoomlaPack to the rescue!

I installed the 1.2.b1 release, using the Joomla! 1.5 Native -j15 package, so I hadn't have to activate the Legacy plugin. In less than 4 minutes a snapshot of the shiny new site was ready on my hard drive. Hey, that was easy!

Step 5: Showtime!

The big moment is here. Everything has to be done in military precision. The site is not to be left off-line for more than 10 minutes. Hey, don't panic. It is really easy, trust me!

First off, I made sure the site's root folder permissions were 0777 and that PHP Safe Mode was disabled. This will keep headaches away later in the process. Then, I uploaded kickstart.php and the new site's backup archive.

I now needed a way to remove the old site and restore the new one. I chose to use SSH to remove the old folders. I could as well have used FTP, but removing the thousands of Joomla! files with FTP takes forever, especially if your host doesn't allow multiple concurrent FTP connections from the same IP address. A few seconds away everything was gone, except kickstart.php and the archive.

Up next, I used Kickstart to get the archive unpacked, ran the installer to get my site restored and than clicked on the relevant links on Kickstart's page to get rid of kickstart.php and the archive.

Phew! Everything was ready now. A quick test and I was done.

Conclusions

JoomlaPack and its accompanying scripts can play an important role during site migration to Joomla! 1.5.x. The steps you can use its time and frustration reducing services are:

• Backing up the original site (JoomlaPack, J! 1.0.x package)
• Making a local clone of the original site (Kickstart, JoomlaPack Installer 2)
• Creating a snapshot of the new site (JoomlaPack, J! 1.5 Native package)
• Deploying the new site (Kickstart, JoomlaPack Installer 2)

JoomlaPack's multipurpose functions as a site backup and cloning tool can save you time during the migration process, letting you focus on the important part: transferring data to the new Joomla! version. It can provide you with confidence that the end result can be deployed in a matter of minutes, without risking bringing down the live site during the migration process, or to iron out migration-related issues discovered along the way.

But if you don't believe me...

... I can prove my success story. Yes, I know what you're thinking. I am JoomlaPack's author, so it is reasonable - if not selfish - to praise my software. You might even think I made this all up.

The example is very real and the only pitfall in the process was the initial migration (the migration only worked when I copied the darn SQL file instead of HTTP uploading it).

Need more proof? The site I was talking about is that of the Hellenic Association for Adult Education. The new site, facelifted is here, whereas the original site's clone is here. As you can see, the content is there, just the forum format and software changed and the past newsletter issues have been temporarily removed, on client's request.

If you do believe me, after all, ...

... you can use JoomlaPack on your site today. If you have a great success story about how JoomlaPack saved the day for you, send it to us.

Getting closer to 1.2.b1

After having done a lot of work the past few days, we are getting closer to releasing 1.2.b1. The most notable additions are:

• Support for accessing the backend with HTTPS (still needs some intensive testing)
• Single file exclusion filter
• Backing up without using AJAX

The codebase had the HTTP protocol hard-coded in the URL generation part, which made using the component impossible for people accessing their backend through HTTPS. It only got worse if you were actually enforcing the use of HTTPS with an appropriate .htaccess file. The fix I did was rather rudimentary: JoomlaPack chooses its protocol prefix based on whether the server has informed PHP that we were called through the SSL protocol. This feature still needs extensive testingto make sure there are no loopholes in the code and that it supports non-standard HTTPS ports (which means, anything but port 443).

As far as the single file exclusion is concerned, this was an asked for feature for quite a while. Many people want to exclude just that pesky unreadable file their host puts on the website's root. Or, maybe, those two big video files from the downloads directory. You get the picture... The interface is slightly weird, mainly because it just doesn't look like Joomla!. There are two panes, the left one displaying directories and the right one displaying files. You click on the directories to visit them, click on the files' checkboxes to toggle the exclusion status. Pretty easy, but I'd certainly prefer to code a proper tree view on the left side, a la Konqueror, or Windows Explorer :/

The "backup without AJAX" was a last minute addition, after some user had terrible problems with using AJAX during the backup, for no apparent reason. This can happen due to a million reasons, one of them being that the server is overpopulated. Instead of giving up and let frustration get the better of you, you can just switch to the brand new "JavaScript Redirects" backup mode. This is very simple, indeed. After each piece of work performed (a "step" in JoomlaPack jargon), a JavaScript redirect is issued to make the process proceed. It's not the same as the front end backup because it neither uses HTTP 301 headers, nor does it output a blank page during the backup. On the contrary, it outputs the familiar "backup status" messages, albeit in a funny looking way, yet.

All these exciting features were added in SVN 113, just a few minutes ago :)