The Point

In the last installment, I covered how to create your own custom components. This time I'm going to tackle something that should have been much less involved.

For every page request that uses that file uploader, I wind up downloading approximately a bazillion .js files.  Reasonable caching should cut that down so it only happens once (-ish) per session.  Still, bandwidth is money.

More importantly, my hosting environment puts rather extreme limitations on the number of files I can use.  It's a long story.

The ideal solution would be to use a CDN.  As I discussed in Part 2 (FIXME: make that a link), that doesn't work because of Flash's cross-site scripting security.  Besides, custom components just don't work with the CDN (at least, they choke with an error to that effect every time I've tried).

So, it's time to make my files smaller and work with fewer of them.

Option A

There's an Adobe AIR Toolbox for Dojo.  It includes an API reference and a wrapper over the compression tool.  The basic idea is that you specify a profile file (more about that below), the location of your Dojo source tree, and an output directory.   Whatever I'm doing is wrong.  It always freezes about 2/3 of the way through.

If it works for you, great!  If you're like me, it's time to take it to the command line.  (Don't be such a wuss.  This is a blog for programmers).

Merging Files

There are all sorts of options available for this.  But the one that makes the most sense is Dojo's.  It already understands the various files involved and can handle tracking down all (well, most of) the dependencies.

Of course, it can't work magic.  You have to specify which files you're using.  This is where that profile file (mentioned above) comes into play.

Start out by downloading the latest source distribution (you want either the .zip or .tar.gz file, depending on your druthers, with "src" in its name) and extracting it somewhere.

In there, under the utils\buildscripts\profiles folder, you need a profile file (mentioned above) that describes your build.

Profile File

You need this even if you're using the Dojo Toolbox.

I was tempted to take an example taken straight from the Dojo Book.  But you can look that up just as easily as I can.  So let's go with something slightly more interesting.

Each "layer" in that file will be built into a source file, along with all its dependencies.

The "prefixes" array tells the "compiler" where to find files associated with various namespaces.  In this case, dijit and dojox are in the directory next to the one where the dojo code lies.  Things in the "internal" prefix are in a directory parallel to the dojo root.  These dependencies are the ones you specify directly (e.g. in dojo.require() calls).  The "magic" part of this is that you don't have to work out any of their dependencies.

wherever
- dojo
|-dijit
|-dojo
|-dojox
-internal

The builder will create a core dojo.js file by default.  This profile specifies three more files to be generated: dijit.js (which will include the basic dijits), YourNameBase.js which includes some default widgets that are used all over the place, and YourNameUploader.js, which references the file uploader we built last time.

N.B. The list of dependencies you include in your layers is vitally important.  If you reference dependencies you don't actually use, you can wind up with ballooning files that totally defeat the purpose of this.  On the other hand, if you forget to specify a dependency, it should be pretty obvious when you test a page that requires it.  You could keep an intern very busy tweaking this thing.

Building

Once you have that set up to your satisfaction, open up a command line in the src\util\buildscripts folder (you do have the Command Line Here powertoy installed, don't you?) and run build.bat with the appropriate options.

What are those options?  Like so much else in life, it depends.  In general, you want to specify which profile and how much compression to use. The link to the page in the Dojo Book that I included at the top has lots of discussion about what the various options mean in the comments.  For now, let's just stick with the basics.  Assuming your profile file was named MyProfile.profile.js, you might run

Running it without specifying an action will give you a list of the various options available. There are various issues involved with many of these options that you should be aware of.  Particularly the xd options for building pieces designed to work cross-domain (see all the pain I've had trying to use the CDN). The comments about cssOptimize, optimize, layerOptimize, and mini are the ones you probably need to pay most attention to.  (The nutshell version is that specifying commentscomments.keepLines leaves newlines to try to keep things readable/debuggable.  packer tells the builder to run those files through Dean Edwards' packer. The mini option tells the builder to delete several files that you don't want on your production site, such as unit tests).

The builder should run for a bit, spitting out lots of log text, then save the results in src\release\initial.  In this example, the interesting files are src\release\initial\dojo\dojo.js, src\release\initial\dojo\dojo.uncompressed.js, src\release\initial\interal\YourNameBase.js, src\release\initial\internal\YourNameBase.uncompressed.js, src\release\initial\internal\composites\YourNameUploader.js, src\release\initial\internal\composites\YourNameUploader.uncompressed.js, src\release\initial\dijit\dijit.js, src\release\initial\dijit\dijit.uncompressed.js, src\release\initial\dijit\dijit-all.js, and whichever theme you might be using (e.g. src\release\initial\dijit\themes\tundra\tundra.css).

As you might guess, the files with "uncompressed" in their names are the versions that were mashed together, but not actually compressed.  Use them for debugging. The dijit-all.js file contains a reference to all the dijits.  (There are options to mash the entire dijit part of the tree into this file instead of just using references).

The CSS files...some you'll need, others you won't. You can pull the basics from CDN, but several controls have their individual CSS files that you'll need.

Using Them

Copy the files you'll be using into the appropriate folder on your web server.  You might have others (with their dependencies) that you didn't add to any layer, because you use them so rarely. The directory structure should mimic the output from Dojo's builder.  The various documents and blog posts (with their comments) that I've found claim that you need to include the entire generated source tree.  That isn't strictly true (and would be a deal-killer in my situation).

Change your script references to include the most-specific script you'll need, along with all its dependencies:

1. Because I'm not including the dojo._base parts of the tree, the isDebug option to djConfig must now be false (you only really need to deploy dojo._base._firebug to get around this).
2. You'll get annoying 404 errors trying to retrieve translation files in the NLS directories. You'll have to copy these files into the appropriate directories as you need them (and you will need them).
3. The error isn't showing up client-side, but the server is complaining about a missing dojoRoot/dojo/resources/blank.gif.  It's a 43 byte file.  Might as well add it.
4. The profile file was missing a line that specified that YourNameUploader.js depends on dojox.embed.flash.  It probably shouldn't need that, since it's going up the dependency tree.  Still, it's dojox.
5. Yet another cryptic error message: '[Exception... "'Error: Bundle not found: validate in dijit.form , locale=en-us' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001c (NS_ERROR_XPC_JS_THREW_JS_OBJECT)" location: "<unknown>" data: no' (Hey, it's a lot better than you get from, say, SharePoint or SSIS). 
   

This originally looked like a bug the dojo developers seem to have blamed on FireFox 3: Dojo Bug #7280 (there probably is some sort of distant relation).  But I got basically the same result in IE 7. This is actually a side-effect of that 404 error mentioned above. I had to copy in the English versions of the .js files for YourNameBase.js and YourNameUploader.js. When the site goes international, the rest will probably need to be added as well.

Fixing those errors left me with a 404 on the uploader.swf.  Client-side, the error message was "'this.flashMovie' is null or not an object" on IE and a pair of errors in FireFox: "this.flashMovie is null" and "this.flashMovie.setFileMask is not a function."

Total Failure (sort of)

I added the SWF and dojoRoot/dojox/embed/IE/flash.js and ran headlong into another example of the dangers of dojox.  IE (running flash 10) now switched to an "Unknown Error" when I clicked the file browse button. It works as expected in FireFox (running flash 9). Following a reboot, I ran the upgrade to flash 10.  The install said the change would not take effect until I rebooted again.  But, after that, clicking "Browse..." in firefox caused this error: '[Exception... "'Error calling method on NPObject! [plugin exception: Error in Actionscript. Use a try/catch block to find error.].' when calling method: [nsIDOMEventListener::handleEvent]" nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)" location: "<unknown>" data: no].'

This is a known issue with Flash 10.  svn trunk has pretty much a complete re-write of the file uploader (since 1.2.3 was released on 12/4) that deals with this, along with a change to let you use CDN, and quite a few other bugs that have been plaguing this thing.  The joys of open source and an active community!

I'm not about to go into that, though.  Messing with dojox in the first place is silly enough (and not someone anyone should ever really consider except on projects where they just want to be on bleeding edges).  Writing about changes that aren't part of any official release would just be a waste of my time and yours.

Besides, the point to this article was reducing your application's throughput.

Final Note

This should probably become a regular part of the build process on any sizable project using dojo.  Along with a script to generate the profile file and copy the built files, based upon a dependency list, as part of your continuous integration.  Don't forget to run the output through something like jsLint.

On the other hand, like a lot of "best practices," that might be overkill at earlier stages in the project. In the beginning, you might be better off specifying a single compressed .js file that contains everything you need.  Later, you can tweak it into multiple files with layered dependencies like in the example I gave. Until you have numbers to balance requests vs bandwidth, it's just premature optimization.

Quick Hint and Update:

Redoing the build every time you make a change is a complete change.  Especially if you're working with other scripting languages, and you've gotten spoiled to not having to wait on the compiler.

The trick here is to take the file(s) you're working on out of the build profile, and copy them manually into the destination directory.  The devilish detail here is remembering to move the changes back out into your source tree after you're finished.  Caveat emptor.