JSON-P Makes Progress Cross Domains With Apache, Passenger, and jQuery

As yet another part of the be-all, end-all media upload processing solution, my client wanted to provide a nice progress bar for tracking the status of file uploads. Let me note, that providing user feedback as to the state of an extended upload turns out to be a very important UI feature.

Luckily, the problem of a nice way to track file upload progress had already been solved several times. My own platform of choice at the moment is Apache running with Passenger, and cool dude Peter Sarnacki aka drogomir had kindly provided a handy version of the same JSON based file upload progress tracking that had been done so well on nginx, except this time for Apache in the form of apache-upload-progress-module. He even had created a jquery-upload-progress plugin too, which is exactly what I needed.

Only one small problem... those pesky cross-domain javascript calls. As in, browsers do not as a rule, allow them. Then again, browsers do turn out to know how to be promiscuous. Which is to say, execute arbitrary JavaScript code handed to them by strange servers. Say wha? Yes, that is correct.

Enter JSON-P, which is the JavaScript Object Notation you know and love, with an extra layer of "Padding". What does the P in JSON do? Sorry, I could not resist that. Anyhow, the padding consists of a special temporary javascript function created just to handle the data callback from a server that is not in the same domain as the calling server.

Here is an example from our need for upload progress tracking. A normal JSON request to the upload component, like this:

   http://www.yourdomain.com/progress?X-Progress-ID=1234

Would return JSON data like this:

   new Object({ 'state' : 'uploading', 'received' : 35587, 'size' : 716595, 'speed' : 35587 })

But if we are using JSON-P, the request would include the name of the 'magic' callback function, so that the data returned from the cross-domain request is available to the original calling page, even though it is in a different domain, or even just subdomain.

For example, this request:

   http://uploads.yourdomain.com/progress?callback=jsonp123&X-Progress-ID=1234

Would return the JSON-P function:

   jsonp123(new Object({ 'state' : 'uploading', 'received' : 35587, 'size' : 716595, 'speed' : 35587 }));

The calling page executes the 'jsonp123' function after the data is returned from the cross-domain request, and then the returned data is available to scripts on the original page. This is how Twitter and other Javascript API's are able to function with these cross-domain requests.

So I whipped up patches for each the apache-progress-module, and the jquery progress plugin, and now that drogomir has accepted them, we can all enjoy the benefits of cross-domain uploads, along with JSON-P powered shiny progress indicators. Well, as long as it is a jquery based progress indicator you seek. I will try to get around to looking at the Prototype progress plugin, but since I am mostly working with jquery now, perhaps someone else will jump into the fray.

Have fun, and make some progress.

Getting Async With Sinatra And Passenger Using spork

Everyone wants it all. In the case of upload processing, this means we want both the convenience of Sinatra coding, plus the performance that a nice process/threading model gives us. Go off and do some work I just assigned you, and don't make me wait around to see you do it.

If you are using Passenger, it already is buffering your uploads, and not calling your Sinatra application until the upload is complete. That saves system resources for more important things, like even more uploads. Pretty nice.

But a lot of work often has to take place after the upload is completed. A common example these days would be a video transcoding or image processing server. Once your user has uploaded their file, it would be considerate to let them go off and do something else. Sites like YouTube and Flickr have been doing exactly this for some time.

For certain highly transactional system, a solid message queue would be essential. But more humble needs, like an upload/transcoding/image thumbnail server, does not really need all of those extra moving parts.

Thanks to the power of the underlying Ruby language itself, it is very simple to fork off blocks of code into a separate process. Combining this with Passenger's already pretty robust process handling, and you get a pretty well performing solution with not much coding effort.

But wait... it does not work?! Sure enough, there are issues in the underlying interaction between Passenger and Rack, that you must address, or your long-running code will be just that, long-running, making the user sit there and wait for it to complete before Passenger allows Sinatra to return a response.

Enter spork... a blatant ripoff of both the spoon and the fork... I mean of the Spawn plugin for Rails, except with no actual Rails stuff in it, of course. spork handles the monkeypatching required to get things working the way they should be, and also lets you run under thin with no code changes should you so wish, like when running in development mode locally.

Anyhow, it may not be a glamorous implement, but it is a useful one. Have a look at spork if you are using Sinatra on Passenger, and you want to get all asynchronous on your users.

Sinatra and Passenger Back On The Rack

There is a new release of the Sinatra gem, version 0.9.1. This happens to work out well, since the newest Passenger 2.0.6 has a strong dependency on Rack 0.9, and the previous Sinatra 0.3 simply did not work at all due to dependency on Rack 0.4. The last few days have had some of us apoplectic over Ruby gem version hell.

And lots of interesting and even important things require all three. For example, I have been messing around with Integrity, a very cool little continuous integration server which is built using Sinatra. Foca and the rest of Team Integrity are still struggling with upgrading a morass of DataMapper gems, but this should help.

The point here, is getting everything up on Rack 0.9 seem to have made these problems go away, at least for my own Sinatra-based projects. Thanks to everyone who worked on these new releases, and special thanks to the IRC dwellers that provide support for the community.