Crossing The PhoneGap For Multiplatform Mobile Applications

I had first heard of the PhoneGap open source framework for multiplatform mobile development last year at the FutureRuby conference in Toronto. Honestly, I did not really concentrate on all they were saying at the time, and in the flurry of info including back-to-back mobile sessions with Rhomobile, I did not fully retain a clear picture of what they had to offer. My bad.

It was not until late last year, while working on plans for a very cool mobile application, that I was reminded about PhoneGap by one of my colleagues. After a brief evaluation of their benefits vs. the other multiplatform options, we decided to use PhoneGap on this particular project. We have made some amazing progress with using PhoneGap for mobile development since then, and I thought I would share a few of the lessons learned.

First of all, a quick explanation of how PhoneGap works. Jesse MacFadyen, one of the programmers at Nitobi, the primary developers of PhoneGap, has a good blog post where he breaks down how the PhoneGap framework works on iPhone. Here is my much condensed take on what he is saying.

All of the mobile platforms supported by PhoneGap have some kind of web browser control. A PhoneGap application is a packaged up application which is a webpage or mini-website, that executes inside whatever web browser control is available on that platform. Add in a standard JavaScript API to a wrapper that accesses the device-specific functionality like GPS or the accelerometer, and you can hook up the JavaScript on your "page" to the hardware.

But don't make the mistake of thinking you should just slap together some web pages formatted for mobile. Rather, you can and must think of it as an application that is written in the form of a single web page with a bunch of JavaScript. To access server data, you will need to write some AJAX code to access the remote resources, then update your UI accordingly. The good news is you can use familiar JavaScript libraries such as jQuery to do so. We chose jqTouch to get a very iPhone-like UI.

The PhoneGap framework is under very active development. New code is being committed to their github repo frequently. As such, you really do need to have git installed, and some working knowledge of how to use it, to get things setup.

In fact, the device specific functionality for each platform is contained within the main PhoneGap git repository as a series of git submodules such as phonegap-iphone, phonegap-android, etc. This makes it all but impossible to install the latest and greatest without git. Being a git user myself, this does not pose any problem, and if you are not gitmotized yet, this is an ideal time to become so.

Here are the series of steps I followed to create my own PhoneGap project...

Prerequisites

• Install iPhone SDK (yes, you must join the iPhone developer program)


• Install Android SDK (http://www.talkandroid.com/android-sdk-install-guide/ has some good instructions. I am not using Eclipse, so I skipped all that)


• Install Apache Ant (sudo port install apache-ant)

Install PhoneGap

• Get latest PhoneGap (git clone git://github.com/phonegap/phonegap.git)


• Get submodules for iPhone & Android (cd phonegap && git submodule update)


• Build iPhone Lib and install


• Build Android libs

Setting Up A New Multiplatform PhoneGap Project

• Create a new iPhone project using the phonegap-iphone template. The www directory in the new project will be becoming the shared part of your project, with all your UI and application logic


• Create the Android project using the phonegap-android build script. Use the www directory from the iphone project as the www-dir param. You will be replacing this with a reference to the git submodule


• Create a new git repo in the www directory in the iPhone project, and commit all the files in the www directory


• Change directories to the parent of where you want to put the directory for the shared git repo, then clone the repo located at /path/to/iphoneproj/www to /path/to/sharedrepo. You may also want to create a remote branch at this point for the new shared repo


• Remove the www directory from the iPhone project


• Create a new git repo in the iphone project directory, then commit all files. You may also want to create a remote branch at this point


• Put the shared code into the iPhone project by using "git submodule add" to put a reference to the shared repo into /path/to/iphoneproj/www


• Create a new git repo in the Android project directory, then commit all files. You may also want to create a remote branch at this point, as you probably did for the iPhone project files


• Remove the /path/to/androidproj/assets/www directory from the Android project, and use "git submodule add" to put a reference to the shared repo at /path/to/androidproj/assets/www

You now should have 3 git repos, with the shared code, the iPhone specific project code, and the Android specific project code, each in their respective places. Much easier and nicer to work on.

Anyhow, I hope this info is useful to any PhoneGap developers that want to get things setup cleanly for multi-platform mobile programming. Please let me know any feedback or improvements that people out there come up with, for this neat open source framework for doing JavaScript-based mobile development.

Sparkline Some Interest With Ruby on Rails

Recently, I added some sparkline graphs to a Ruby on Rails application. A sparkline is a very small graphic that displays a large amount of information, typically shown over time, and usually embedded in some other text. Invented by the father of modern infographics Edward Tufte, the sparkline has become a fixture of many online applications that want to visually display some stats in a simple, integrated way.

When it come to adding sparklines into a Ruby on Rails application, there are a couple of different options. You can chart the data on the server, output being an image file. You can also chart the data on the client, with a couple of different JavaScript libraries as available options.

I started my plan, intending to implement my charts on the client-side. The project that appears to have the most flexibility, speed, and options, is an amazing jQuery plugin called jQuery Sparkline. Unfortunately, the project to which I needed to add the sparklines is a bit older, and does not use jQuery. As a result, I was not able to use jQuery Sparkline for this current project.

Another interesting JavaScript sparkline library is lethain's Sparkline.js, but it has not been updated in some time, and is not compatible with current Internet Explorer versions. There is also an interesting looking newer lib topfunky-sparkline-js but I have not tried it out yet.

So that brings us to server-side sparkline generation. If you are using ImageMagick/RMagick then @topfunky once again provides, with the Ruby sparklines gem. This gem provides lots of options for doing all sorts of fancy sparklines.

In my case, I am not using ImageMagick for anything else, so I did not want to install it just for this. What I really wanted was something much lighter-weight, and I was willing to accept a lot fewer options to get it.

It turns out that madrobby has written a library for generating very simple sparklines in pure Ruby code, called spark_pr. The project uses _why's pure Ruby implementation of a PNG generator to do the low-level work.

spark_pr has also spawned an interesting application of it from @technoweenie called Sparkplug, which is a Rack module that generates spaklines from CSV data on the fly, using Rack handlers and Rack caching.

Once I had seen Sparkplug's minimal elegance, it seemed spark_pr was the option for me. I decided to incorporate spark_pr into my application. Given that the app was written with a dedicated approach to keeping a clean RESTful interface, and that the database already contained the time-series data, it was quite easy to incorporate. Here is what I had to do:

Step 1: Put spark_pr.rb file into lib directory. I just grabbed the code from the repo, and dropped it into my project. You may decide to have a more sophisticated way to do it, such as using git submodules.

Step 2:

require "spark_pr"

in your controller

Step 3:

include Spark

in your controller

Step 4: Add PNG format to controller action that was already returning my time-series data, and return the sparkline PNG data:

Note that this controller action is already returning XML or JSON. PNG is just another format to be added, if you have a well-designed RESTful controller action.

Step 5: Put image_tag that requests the sparkline PNG image file into the view where you want it to appear, like this one:


Voila! Refresh that view and you should now be looking at your ultra-hipster-chic sparkline graph. It is surprising how much more information comprehension a person has, when they are seeing a visual representation of their data. Plus it looks cool.

Thanks For A Great #code2009

It has been an amazing year for both personal and professional code development.

Starting with the inspiration to begin Project Flying Robot, to the prestige of presenting at LARubyConf, FutureRuby, TWTRCON, IgniteLA, 140 The Twitter Conference, RubyConf, and Conferencia Rails, and lastly the year-end fun of starting out the ongoing #code2009 Twitter meme, so popular that it spawned a couple of mashups and got picked up by Hacker News and uber-language blog Lambda the Ultimate.

In between were numerous meetups, hackfests, code jams, code dojos, pull requests, and casual codeslinging with friends. And the Maker Faire!

To everyone who welcomed me, listened to me, helped me, or taught me something, I am indeed grateful. Thank you. Let's do this 2010 thing right!

Flying Robot: World Tour 2009 Continues

As usual, no blog posts = a lot of other activity here at Flying Robot HQ. Among other personal stuff, my brother Damen Evans and I have been getting ready for the last public demos of @flyingrobot for 2009. And we are going out with style!

Later this week, we roll up to San Francisco to present at the prestigious RubyConf! Then, next week @flyingrobot and I will fly off to Madrid, Spain, to do our first European appearance at the awesome-looking Conferencia Rails.

Anyhow, if you have been waiting eagerly for more Flying Robot news and gadgets, be patient. We will be unveiling our mysterious cool new stuff in just a few days...

PostgreSQL on Ubuntu on EC2: Backing It All Up

This post continues what I started with "PostgreSQL on Ubuntu on EC2: The Installation Guide". Once you have your PostgreSQL database server instance running, you will need to backup two different things: your database data, and the instance itself. The database data will be backed up using Elastic Block Storage (EBS) snapshots. Once we have the instance running the backups correctly, we will then create an Amazon Machine Image (AMI) that will allow you to launch a new instance to replace the database server in case it goes down.

Backing Up The Database
First, we need to connect to our database server instance via SSH using the ubuntu user.

We will need to install some dependencies to get our backup script to run:

sudo apt-get install build-essential
sudo apt-get install ruby1.8-dev
sudo apt-get install rubygems
sudo gem update --system


You will need to tweak RubyGems so that the update works correctly, as described here.

Now you can install Gemcutter, which is the new ultra cool repository for gems:

sudo gem install gemcutter
sudo gem tumble


Finally we are ready to install the Amazon EC2 rubygem:

sudo gem install amazon-ec2


Now we can create our backup script. Save this code into the ~/ directory under the name backup_database.rb. You will need to substitute the Amazon ACCESS_KEY_ID and SECRET_ACCESS_KEY, as well and entering the correct EBS volume for the DATABASE_VOLUME constant:

#!/usr/bin/env ruby

require 'rubygems'
require 'AWS'

ACCESS_KEY_ID = 'YOUR_ACCESS_KEY'
SECRET_ACCESS_KEY = 'YOUR_SECRET_ACCESS_KEY'
DATABASE_VOLUME = 'vol-XXXXXXXX'

puts "Starting database snapshot..."
ec2 = AWS::EC2::Base.new(:access_key_id => ACCESS_KEY_ID, :secret_access_key => SECRET_ACCESS_KEY)
ec2.create_snapshot(:volume_id => DATABASE_VOLUME)
puts "Database snapshot completed."


Due to the finicky way that Ruby runs as part of a cron job, we are best off creating a shell script that then runs the Ruby backup script. Save this code into the ~/ directory under the name backup_db.sh:

#!/bin/sh
cd /home/ubuntu
ruby /home/ubuntu/backup_database.rb


Don't forget to make the backup shell script executable:

chmod +x /home/ubuntu/backup_db.sh


Now we just need to configure this script to run as part of a cron job, so that the backups take place automatically. The crontab command brings up the list of configured cron tasks for the current user:

crontab -e


This example crontab entry runs the backup daily at midnight, but you may want it to run more frequently:

0 0 * * * /home/ubuntu/backup_db.sh


At this point, you should have a fully functional automated backup system. Verify after midnight that the script has run as you expect, by looking to see if a new snapshot has been created, using Elastifox or however you administrate your EC2 instances.

Creating The AMI
Creating the AMI to backup the entire database instance is pretty easy. First, you need to upload the PEM files. Remember you are authenticating as the "ubuntu" user:

scp -i id_rsa-gsg-keypair pk-YOUR.pem cert-YOUR.pem ubuntu@domU-12-34-31-00-00-05.usma1.compute.amazonaws.com:


Use your SSH connection into the database server instance to copy the PEM files to the /mnt directory:

sudo cp /home/ubuntu/*.pem /mnt


Now create the bundle. Make sure you use your Amazon account number (without dashes) as the value for the -u parameter. This can take quite a while, so do not get impatient:

sudo ec2-bundle-vol -d /mnt -k /mnt/pk-YOUR.pem -c /mnt/cert-YOUR.pem -r i386 -u YOURUSERACCOUNTNUMBER


You can now upload the bundle to your Amazon S3 account, in preparation for making available as an AMI. Use something versioned for the -b parameter which is the name of the bundle:

sudo ec2-upload-bundle -b my-database-server-1.0-ami -m /mnt/image.manifest.xml -a YOUR_ACCESS_KEY -s YOUR_SECRET_ACCESS_KEY


Final step is going back to your local machine, and making the newly created bundle available to be used to start a new instance:

ec2-register my-database-server-1.0-ami/image.manifest.xml


You can now now launch a brand new database server instance based on this AMI, and it will be a clone of your existing database server. This is the procedure you would follow if you need to restore your database server instance from backups.

Restoring Your Database Server From The Backups
In the case that something goes terribly terribly wrong, you can get back to normal as follows:
- startup a new EBS volume from your most recent snapshot backup,
- start up a new server from your database AMI
- configure your new server instance to use the new volume started from the backup data
- switch your elastic IP to point to the new server, or update the references in your application to point to the new server

This concludes part 2 of the great PostgreSQL config post for the EC2 cloud. Hopefully it will help you with a nice simple way to take the basic PostgreSQL instance that you got up and running on Ubuntu/EC2 using the directions in the part 1 post, and add the confidence that backed up data and a completely reproducible configuration provides.

Happy 200 Posts: My 10 Personal Favorites

I was shocked to discover this morning that this is to be my 200th blog post. Wow! It has been a good run so far since I restarted the Dead Programmer Society in 2006, and I really appreciate the awesome feedback and support that I have received from the community.

To commemorate this personal event, here is a list of my top 10 favorite posts, in no particular order:

1. "I'd Rather Be A Jazz Programmer"
2. "Fear And Loathing At RailsConf 2009"
3. "Programming Zombies Will Crush You"
4. "The Twitter 1-2-3 Rule"
5. "Goldilocks and the Three Icons"
6. "Money In The Ghetto"
7. "I Speak For The Code"
8. "The Folly Of Accountabalism"
9. "The Planning Game Vs. The Crying Game"
10. "Architect Is Not An Honorary Title"

Once again, I thank everyone for your support, and I look forward to telling more tales from the Dead Programmer Society.

PostgreSQL on Ubuntu on EC2: The Installation Guide

For some time, I have had clients hosting a couple different applications on Amazon EC2 using Ubuntu. One of these apps uses PostgreSQL, and has been running without event for quite a while. Yesterday, I got to catch up for lost time, by spending the entire day wrestling with data recovery issues related to a failed apt-get upgrade on an important database server. Luckily, the awesome Eric Hammond was around on IRC, came to my rescue, and coached my thru my self-inflicted pain.

If you are not interested in PostgreSQL, you probably just stop here. Nothing to see folks, move along. However, if you are looking for the well-lit path to getting PostgreSQL installed on Amazon EC2 will all the trimmings, read on.

I went to find various web pages to use as source material, expecting since the last time I went through this, someone would have written a nice definitive guide to installing PostgreSQL on Ubuntu, running it on a dedicated instance on Amazon EC2, and using Elastic Block Storage (EBS). Naturally, you want to be using the XFS file system too. However, no such luck: just a big collection of pages of instructions on the various parts, without any nice simple path to getting things working together.

Hence, this post tries to provide a set of instructions for getting things working, and avoiding a couple of problems that I have run into while running Postgres in production for the last couple years.

Step 0 - You are signed up for Amazon EC2, no? If not, there are plenty of pages with instructions on how to do so.

Step 1 - Choose your AMI
There are several AMI's available to you. I currently run Hardy 8.04 LTS x86 architecture in the USA, so I am using ami-5d59be34, but you may have other requirements. The Ubuntu EC2 starter guide has good info on your options.

Step 2 - Launch your instance
I like to use Elasticfox, cause I am super lazy. The command line works well, also.

What size instance? This AMI supports small and medium. PostgreSQL is pretty efficient these days, and especially using a dedicated instance and not running anything but the database server improves raw database performance considerably. You would probably be pretty surprised how well a small instance can perform, but choose medium if you think you will have more significant needs.

One key pattern I use for my EC2 hosted apps, is creating security groups in EC2 to separate my database servers from the public internet. I never use the default security group, but instead create a group for each tier of my application like "database", "web", "transcoder" and then allow specific groups to communicate with each other.

Step 3 - Create the EBS Volume
You can do this via Elasticfox, or via command line. Either way, make sure you do two key things: make sure you create the EBS volume in the same availabilty zone as your database server instance, and also make sure you create a volume with enough space. Here is how you would use the command line tools to create a 10GB volume in the 'us-east-1a' zone:

ec2-create-volume -z us-east-1a -s 10

One the volume is ready, attach it to the database instance. For example, this attaches an EBS volume named 'vol-VVVV1111' to the instance 'i-IIII1111' on device /dev/sdh:

ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111


Step 4 - Connect to the database instance
You need to SSH in to configure you new instance. Remember, you cannot connect as 'root' user in Ubuntu, you need to connect using the 'ubuntu' user. This page has good details about using sudo and SSH on the official Ubuntu EC2 AMIs.

OK, so now you are connected via SSH to your server. Of course, start with the usual update/upgrade:

sudo apt-get update && sudo apt-get upgrade -y


Step 5 - Install XFS
We will need to install the XFS file system. Actually, your could use some other file system, but XFS is quite mature and has good performance. Plus if you are crazy, you can scale up to a massive virtual RAID drive that will cost $4000 per month.


sudo apt-get install -y xfsprogs
Step 6 - Format the EBS volume using XFS
We need to install a file system on the EBS volume before we can do anything with it. Here is an example:

sudo modprobe xfs
sudo mkfs.xfs /dev/sdh

echo "/dev/sdh /data xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir /data
sudo mount /data

Now we have a /data directory that maps to our EBS volume. Anything we write to /data will be persisted, even if the database server instance itself terminates.

Step 7 - Install PostgreSQL
Now we need to get PostgreSQL installed. This page has a very nice simple set of instructions on how to do that correctly for Ubuntu, but here is a synopsis especially for a headless server. First install Postgres:

sudo apt-get install postgresql postgresql-client postgresql-contrib

Now reset the password for the postgres account in the PostgreSQL server:

sudo su postgres -c psql template1
template1=# ALTER USER postgres WITH PASSWORD 'password';
template1=# \q

And then change the password on the user account to match:

sudo passwd -d postgres
sudo su postgres -c passwd


Now we need to modify the postgres configuration file postgresql.conf. First, to allow other machines to connect to our instance, and also to have PostgreSQL use our nice shiny /data directory.


sudo nano /etc/postgresql/8.3/main/postgresql.conf

Change the line containing

data_directory = '/var/lib/postgresql/8.3/main'

to

data_directory = '/data/main'

Now change

#listen_addresses = 'localhost'

to

listen_addresses = '*'

and also change

#password_encryption = on

to

password_encryption = on

Save the file, then open the pg_hba.conf file so we can control who can access the server:

# DO NOT DISABLE!
# If you change this first entry you will need to make sure that the
# database
# super user can access the database using some other method.
# Noninteractive
# access to all databases is required during automatic maintenance
# (autovacuum, daily cronjob, replication, and similar tasks).
#
# Database administrative login by UNIX sockets
local all postgres ident sameuser
# TYPE DATABASE USER CIDR-ADDRESS METHOD

# "local" is for Unix domain socket connections only
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5

# Connections for all PCs on the subnet
#
# TYPE DATABASE USER IP-ADDRESS IP-MASK METHOD
host all all 0.0.0.0/0 md5 # wide-open, you may want to make this more specific to your database


Step 8 - Move the database files
We need to stop the PostgreSQL server, move the database files to our EBS volume, then restart the server.

sudo /etc/init.d/postgresql-8.3 stop
sudo mv /var/lib/postgresql/8.3/main /data
sudo /etc/init.d/postgresql-8.3 start

You are now running PostgreSQL on Amazon EC2 using EBS for your database, with the XFS file system. Congratulations!

I will write a followup post on how to setup your database server for self-backing up using EBS snapshots, but that is all I have time for right now. Hopefully this pared-down set of instructions has been useful to you. Thanks again to Eric Hammond, and everyone else who's blogs were culled together into this post.