Configuring Hybrid Ruby on Rails Applications On EY-Cloud

This last weekend, we completed migration of a slightly more complex Ruby on Rails application than the usual to Engine Yard's EY-Cloud. It took a few tries, and a tiny bit of research, before we completely understood what to do to get it up and running with all of the needed capabilities. Naturally this was due to gaps in my own knowledge, and I wanted to post about what I had learned, and how awesome EY-Cloud really is!

If you do not know about it, EY-Cloud is Engine Yard's cloud platform that is based on Amazon's EC2. Readers of this blog know I am not afraid to get hardcore when it comes to EC2 server configurations, and with EY-Cloud I feel like I can now get the best of everything. But I am getting ahead of myself.

Let me briefly describe the architecture of the application I was migrating. The main application is written in Ruby on Rails and uses MySQL as its data store. The app provides UI and an API for interactions with the application, mostly called by a custom iPhone app written using PhoneGap. The main app is secured using an SSL certificate. In addition, a separate API for a specific high-performance capability is written using Sinatra and DataMapper. This "special API" is also secured via SSL.

OK, you got that? Perhaps not, so here is a diagram to illustrate what I am talking about:



We call this kind of application a "hybrid" application, meaning that it is built using Ruby on Rails + "something else". For this app, the "something else" is Sinatra + DataMapper.

The previous installation used the same basic setup, but was created manually using hand-configured EC2 instances. This required a lot more management to keep the application going, and especially to add or remove instances to the cluster. Backing up the entire configuration by creating custom AMI's was also very labor intensive.

This was the scenario that led us to wish to migrate the app from raw EC2 to EY-Cloud. So if you are planning to set something like this up yourself, here are my notes to share my experience migrating a hybrid Ruby on Rails application to EY-Cloud.

Prerequisites
You need to have an account setup with Engine Yard for EY-Cloud. In addition, you should have installed the Elasticfox plugin for Firefox to easily administer the security groups. You can also do this via the command line, but it is a lot more effort.

Configure Main Environment & Application
Configuring your main application for EY-Cloud is a pretty simple process. It basically requires setting up private key access to the git repository and the application settings using the web based interface that Engine Yard provides. There is a complete article that describes this in detail here. The main steps are:

• Create main environment


• Create main application and configure access to its git repo


• Config ssh key you will use to connect to instances


• Add ssl certificate, and config main app to use it


• Config any needed gems and unix packages to be used by main app


• Launch app server and separate database server in main environment

Configure Secondary Environment & Application
Now you are ready to configure the environment and application for the other public accessible SSL protected service that is part of the application. This needs to be a secondary application because EY-Cloud currently only allows a single SSL cert per environment, and only allows you to associate it with one application. Hence needing a second environment configured. The main steps are:

• Create specialapi environment


• Create specialapi application and configure access to its git repo


• Config ssh key you will use to connect to specialapi instances


• Add ssl certificate, and config specialapi app to use it


• Config any needed gems and unix packages to be used by specialapi app


• Launch specialapi app in specialapi environment. No database is necessary.


• Copy contents of database.yml from /data/mainapp/shared/config/database.yml on "main app" server to /data/specialapi/shared/config/database.yml on "specialapi" server. Yes, you will need to use SSH to login to both servers.
  
• Use keep file on the database.yml location on the specialapi environment to point to the master database in the main app environment. This page explains about keep files.

EC2 Security Configuration
In order to allow the secondary environment to get access to the database located in the main environment, you need to make a small but critical change to the configuration in the EC2 Security Groups. Otherwise the internal firewall at Amazon will not allow the application to access the database. It is very easy to configure this using Elasticfox. Here is how:

• Setup your EC2 security credentials for your EY-cloud account. You can download them as part of the ey-cloud.yml file


• Once you are connected to that account using elasticfox, click on the "Security Groups" tab. You should see 3 security groups "default", one for the main app named something similar to "ey-myapp_production-12341234" that refers to the main environment, and another one named something like "ey-secondaryapp_production-56785678" that refers to the secondary environment


• Click on the "ey-myapp_production-12341234" group in the left pane labeled "Your Groups" then click on the green checkmark icon at the top of the "Group Permissions" pane. This will bring up the "Add New Permission For Security Group" dialog. Click on the "Group" tab, then choose the "ey-secondaryapp_production-56785678" group from the Groups dropdown, and click on the "Add" button. Your secondary environment should now be able to access the main environment's database

Conclusion
It is pretty great to be able to take advantage of both the cool management interface and scalable clustered database and application stack, as well as being able to get down and dirty via SSH, something you really do need in order to configure more complex hybrid Ruby on Rails applications. EY-Cloud provides this kind of capability, and is certainly a lot easier than managing it all yourself. Hopefully, these notes will make it easier for you to migrate your own complex application. Please let me know how it works for you.

PostgreSQL on Ubuntu on EC2: Backing It All Up

This post continues what I started with "PostgreSQL on Ubuntu on EC2: The Installation Guide". Once you have your PostgreSQL database server instance running, you will need to backup two different things: your database data, and the instance itself. The database data will be backed up using Elastic Block Storage (EBS) snapshots. Once we have the instance running the backups correctly, we will then create an Amazon Machine Image (AMI) that will allow you to launch a new instance to replace the database server in case it goes down.

Backing Up The Database
First, we need to connect to our database server instance via SSH using the ubuntu user.

We will need to install some dependencies to get our backup script to run:

sudo apt-get install build-essential
sudo apt-get install ruby1.8-dev
sudo apt-get install rubygems
sudo gem update --system


You will need to tweak RubyGems so that the update works correctly, as described here.

Now you can install Gemcutter, which is the new ultra cool repository for gems:

sudo gem install gemcutter
sudo gem tumble


Finally we are ready to install the Amazon EC2 rubygem:

sudo gem install amazon-ec2


Now we can create our backup script. Save this code into the ~/ directory under the name backup_database.rb. You will need to substitute the Amazon ACCESS_KEY_ID and SECRET_ACCESS_KEY, as well and entering the correct EBS volume for the DATABASE_VOLUME constant:

#!/usr/bin/env ruby

require 'rubygems'
require 'AWS'

ACCESS_KEY_ID = 'YOUR_ACCESS_KEY'
SECRET_ACCESS_KEY = 'YOUR_SECRET_ACCESS_KEY'
DATABASE_VOLUME = 'vol-XXXXXXXX'

puts "Starting database snapshot..."
ec2 = AWS::EC2::Base.new(:access_key_id => ACCESS_KEY_ID, :secret_access_key => SECRET_ACCESS_KEY)
ec2.create_snapshot(:volume_id => DATABASE_VOLUME)
puts "Database snapshot completed."


Due to the finicky way that Ruby runs as part of a cron job, we are best off creating a shell script that then runs the Ruby backup script. Save this code into the ~/ directory under the name backup_db.sh:

#!/bin/sh
cd /home/ubuntu
ruby /home/ubuntu/backup_database.rb


Don't forget to make the backup shell script executable:

chmod +x /home/ubuntu/backup_db.sh


Now we just need to configure this script to run as part of a cron job, so that the backups take place automatically. The crontab command brings up the list of configured cron tasks for the current user:

crontab -e


This example crontab entry runs the backup daily at midnight, but you may want it to run more frequently:

0 0 * * * /home/ubuntu/backup_db.sh


At this point, you should have a fully functional automated backup system. Verify after midnight that the script has run as you expect, by looking to see if a new snapshot has been created, using Elastifox or however you administrate your EC2 instances.

Creating The AMI
Creating the AMI to backup the entire database instance is pretty easy. First, you need to upload the PEM files. Remember you are authenticating as the "ubuntu" user:

scp -i id_rsa-gsg-keypair pk-YOUR.pem cert-YOUR.pem ubuntu@domU-12-34-31-00-00-05.usma1.compute.amazonaws.com:


Use your SSH connection into the database server instance to copy the PEM files to the /mnt directory:

sudo cp /home/ubuntu/*.pem /mnt


Now create the bundle. Make sure you use your Amazon account number (without dashes) as the value for the -u parameter. This can take quite a while, so do not get impatient:

sudo ec2-bundle-vol -d /mnt -k /mnt/pk-YOUR.pem -c /mnt/cert-YOUR.pem -r i386 -u YOURUSERACCOUNTNUMBER


You can now upload the bundle to your Amazon S3 account, in preparation for making available as an AMI. Use something versioned for the -b parameter which is the name of the bundle:

sudo ec2-upload-bundle -b my-database-server-1.0-ami -m /mnt/image.manifest.xml -a YOUR_ACCESS_KEY -s YOUR_SECRET_ACCESS_KEY


Final step is going back to your local machine, and making the newly created bundle available to be used to start a new instance:

ec2-register my-database-server-1.0-ami/image.manifest.xml


You can now now launch a brand new database server instance based on this AMI, and it will be a clone of your existing database server. This is the procedure you would follow if you need to restore your database server instance from backups.

Restoring Your Database Server From The Backups
In the case that something goes terribly terribly wrong, you can get back to normal as follows:
- startup a new EBS volume from your most recent snapshot backup,
- start up a new server from your database AMI
- configure your new server instance to use the new volume started from the backup data
- switch your elastic IP to point to the new server, or update the references in your application to point to the new server

This concludes part 2 of the great PostgreSQL config post for the EC2 cloud. Hopefully it will help you with a nice simple way to take the basic PostgreSQL instance that you got up and running on Ubuntu/EC2 using the directions in the part 1 post, and add the confidence that backed up data and a completely reproducible configuration provides.

PostgreSQL on Ubuntu on EC2: The Installation Guide

For some time, I have had clients hosting a couple different applications on Amazon EC2 using Ubuntu. One of these apps uses PostgreSQL, and has been running without event for quite a while. Yesterday, I got to catch up for lost time, by spending the entire day wrestling with data recovery issues related to a failed apt-get upgrade on an important database server. Luckily, the awesome Eric Hammond was around on IRC, came to my rescue, and coached my thru my self-inflicted pain.

If you are not interested in PostgreSQL, you probably just stop here. Nothing to see folks, move along. However, if you are looking for the well-lit path to getting PostgreSQL installed on Amazon EC2 will all the trimmings, read on.

I went to find various web pages to use as source material, expecting since the last time I went through this, someone would have written a nice definitive guide to installing PostgreSQL on Ubuntu, running it on a dedicated instance on Amazon EC2, and using Elastic Block Storage (EBS). Naturally, you want to be using the XFS file system too. However, no such luck: just a big collection of pages of instructions on the various parts, without any nice simple path to getting things working together.

Hence, this post tries to provide a set of instructions for getting things working, and avoiding a couple of problems that I have run into while running Postgres in production for the last couple years.

Step 0 - You are signed up for Amazon EC2, no? If not, there are plenty of pages with instructions on how to do so.

Step 1 - Choose your AMI
There are several AMI's available to you. I currently run Hardy 8.04 LTS x86 architecture in the USA, so I am using ami-5d59be34, but you may have other requirements. The Ubuntu EC2 starter guide has good info on your options.

Step 2 - Launch your instance
I like to use Elasticfox, cause I am super lazy. The command line works well, also.

What size instance? This AMI supports small and medium. PostgreSQL is pretty efficient these days, and especially using a dedicated instance and not running anything but the database server improves raw database performance considerably. You would probably be pretty surprised how well a small instance can perform, but choose medium if you think you will have more significant needs.

One key pattern I use for my EC2 hosted apps, is creating security groups in EC2 to separate my database servers from the public internet. I never use the default security group, but instead create a group for each tier of my application like "database", "web", "transcoder" and then allow specific groups to communicate with each other.

Step 3 - Create the EBS Volume
You can do this via Elasticfox, or via command line. Either way, make sure you do two key things: make sure you create the EBS volume in the same availabilty zone as your database server instance, and also make sure you create a volume with enough space. Here is how you would use the command line tools to create a 10GB volume in the 'us-east-1a' zone:

ec2-create-volume -z us-east-1a -s 10

One the volume is ready, attach it to the database instance. For example, this attaches an EBS volume named 'vol-VVVV1111' to the instance 'i-IIII1111' on device /dev/sdh:

ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111


Step 4 - Connect to the database instance
You need to SSH in to configure you new instance. Remember, you cannot connect as 'root' user in Ubuntu, you need to connect using the 'ubuntu' user. This page has good details about using sudo and SSH on the official Ubuntu EC2 AMIs.

OK, so now you are connected via SSH to your server. Of course, start with the usual update/upgrade:

sudo apt-get update && sudo apt-get upgrade -y


Step 5 - Install XFS
We will need to install the XFS file system. Actually, your could use some other file system, but XFS is quite mature and has good performance. Plus if you are crazy, you can scale up to a massive virtual RAID drive that will cost $4000 per month.


sudo apt-get install -y xfsprogs
Step 6 - Format the EBS volume using XFS
We need to install a file system on the EBS volume before we can do anything with it. Here is an example:

sudo modprobe xfs
sudo mkfs.xfs /dev/sdh

echo "/dev/sdh /data xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir /data
sudo mount /data

Now we have a /data directory that maps to our EBS volume. Anything we write to /data will be persisted, even if the database server instance itself terminates.

Step 7 - Install PostgreSQL
Now we need to get PostgreSQL installed. This page has a very nice simple set of instructions on how to do that correctly for Ubuntu, but here is a synopsis especially for a headless server. First install Postgres:

sudo apt-get install postgresql postgresql-client postgresql-contrib

Now reset the password for the postgres account in the PostgreSQL server:

sudo su postgres -c psql template1
template1=# ALTER USER postgres WITH PASSWORD 'password';
template1=# \q

And then change the password on the user account to match:

sudo passwd -d postgres
sudo su postgres -c passwd


Now we need to modify the postgres configuration file postgresql.conf. First, to allow other machines to connect to our instance, and also to have PostgreSQL use our nice shiny /data directory.


sudo nano /etc/postgresql/8.3/main/postgresql.conf

Change the line containing

data_directory = '/var/lib/postgresql/8.3/main'

to

data_directory = '/data/main'

Now change

#listen_addresses = 'localhost'

to

listen_addresses = '*'

and also change

#password_encryption = on

to

password_encryption = on

Save the file, then open the pg_hba.conf file so we can control who can access the server:

# DO NOT DISABLE!
# If you change this first entry you will need to make sure that the
# database
# super user can access the database using some other method.
# Noninteractive
# access to all databases is required during automatic maintenance
# (autovacuum, daily cronjob, replication, and similar tasks).
#
# Database administrative login by UNIX sockets
local all postgres ident sameuser
# TYPE DATABASE USER CIDR-ADDRESS METHOD

# "local" is for Unix domain socket connections only
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5

# Connections for all PCs on the subnet
#
# TYPE DATABASE USER IP-ADDRESS IP-MASK METHOD
host all all 0.0.0.0/0 md5 # wide-open, you may want to make this more specific to your database


Step 8 - Move the database files
We need to stop the PostgreSQL server, move the database files to our EBS volume, then restart the server.

sudo /etc/init.d/postgresql-8.3 stop
sudo mv /var/lib/postgresql/8.3/main /data
sudo /etc/init.d/postgresql-8.3 start

You are now running PostgreSQL on Amazon EC2 using EBS for your database, with the XFS file system. Congratulations!

I will write a followup post on how to setup your database server for self-backing up using EBS snapshots, but that is all I have time for right now. Hopefully this pared-down set of instructions has been useful to you. Thanks again to Eric Hammond, and everyone else who's blogs were culled together into this post.

April Is For AOL

Last night was the always fun Los Angeles web application developer Meetup, this time at AOL in Beverly Hills. After navigating a veritable maze of cubes, we found ourselves in a fairly large screening room chock full of AV equipment, accompanied by a large pile of pizza boxes. A few moments later, the throng of LA-based developers shuffled in, ready for the latest and greatest. The prerequisites fulfilled, we got down to some serious techno-babble.

First up was Omer Singer from DigiTrust, talking about Web Application Firewalls. Although it strayed a little closely to a sales pitch, Omer was able to tie it together with his definition of "defense in depth", which is simply enough having multiple layers of overlapping protection. One good point that he raised which is often overlooked is the cost of "brand-damage" in the marketplace from a web application being hacked. There were several good security related question asked afterwards, so yes the crown was paying close attention.

Next, was Mitch Hashimoto recent hire at CitrusByte, with a brief overview of the newly beta released Google App Engine. Mitch took us on a brief tour of the main capabilities of the Python-based system, like a Datastore API (based on BigTable), and the Users API (based on the Goog's own account authentication database). If you like Python, and don't mind doing everything the Google Way and no other, then the App Engine might be for you.

After Mitch, I did a presentation about Frankie, my newly released Ruby gem for easy development of Facebook applications. If you don't yet know, Frankie uses the Sinatra web server and the Facebooker gem to allow you to create a basic Facebook application with only 10 lines of Ruby. I also did a brief demo of a new, super-secret web thing that I have been working on for a little while... more will be revealed soon!

Last up was programming chum and Ruby expert Ari Lerner, talking about Amazon's EC2 service, and showing a little of another upcoming pet project of ours called "Pool Party", which makes it easy to deploy an auto-scaling application to the EC2 cloud.

It was a really great meetup, and the turnout was really good. Thanks to the persistent efforts of organizer Will Jessup, the community in LA is really thriving, and we were even able to keep the annoyance of non-technical recruiter sales pitches away. Great job, everyone!

Thanks For Paying Attention

This last week has brought some very nice and favorable attention to the Dead Programmer Society.

The ever-popular "I'd Rather be A Jazz Programmer" post popped up again, this time in the stodgy old school publication Computerworld. They ran an article on the "Rock Star Programmer" meme, and decided that my little post was part of the "Kill Rock Stars" contrarian perspective they were looking for. Thanks for making me look so good, guys!

Actually it really is pretty amazing to make an article that also mentions Linus Torvalds, Joel Spolsky, Paul Graham, Obie Fernandez, Zed Shaw, and "why the lucky" (sic). Hey, I told you Computerworld was stodgy. Sorry they mangled your handle, why!

On to more recent work. Ari Lerner and I have been getting some good attention to our introduction of the EC2 ProcessorPool gem. We managed to get covered in both Rails Envy and Ruby Inside this last week. Thank you to the editors for helping us let people know about this quite useful tool.

Anyhow, that is about enough self-congratulation for one post. Next time, back to our irregularly scheduled programming.

Everybody Into The Pool

A few weeks back, I indulged my current obsession with grid computing, and Amazon's EC2 and S3 services in particular, with a lengthy post. I mentioned how friend Ari Lerner and I had built a really neat solution, and were going to put all of our processing farm goodness into a nice neat gemified package. Well, mostly thanks to Ari, we now have a new Ruby gem called ProcessorPool that uses EC2 and S3 to do all the hard work for you.

Based on Sinatra, with a bunch of S3 goodness mixed in, the ProcessorPool gem makes it staggeringly easy to exploit the power of the EC2 grid. More info about it is available at the Blog @ CitrusByte, and you can also check it out on RubyForge here.

All kinds of fun applications await you in the pool... image processing, video/audio transcoding, heavy computational stuff, long crawls of content... and it is fun to delegate to a pool of worker machines in the EC2 cloud.

Come on in... the pool is fine.

On The Grid With EC2

Lately, I have had a chance to get seriously into Amazon's Elastic Compute Cloud (EC2) service. I doubt any reader of this blog is unfamiliar with the existence of EC2, but many of you have probably not even gotten to the tire kicking stage, like I had been until recently.

Even after attending several sessions about Amazon's EC2 at RailsConf 2007, I still had only a very general idea about the service. However, it is true that there is no substitute for experience, and now that I have finally been getting hands-on, I am pretty excited about what you can do with it.

A client with a pretty complex Ruby on Rails application needed to create a processing farm for doing video conversion, as well as serving up the resulting video files. Sounds like the perfect application for Amazon's utility computing services. I dug in to the myriad of online sources of information available...

I decided to base my Amazon Machine Image (AMI) on Ubuntu 7.1. Oh, Ubuntu, such a modern operating system you are! That turned out to be a very good decision, and apt-get was quite co-operative getting the myriad of prerequisites my open source video processing farm would require.

I also decided to base my upload processor on Sinatra. My application's entrypoint is a single stateless call, but there are lots of these calls, and each requires a bunch of intensive back-end processing. Something nice and multithreaded, like some kind of mongrel handler, was called for. In this case, even Merb seemed like it was too heavyweight!

Luckily, I had colleague Ari Lerner available to help me strap Sinatra into my processing farm, who proved to be indispensable. We ran into some issues with Rack, which I am sure Ari will be blogging about. Thanks, Ari, you are the man.


Some lessons learned about working with EC2 and S3:

• When you can launch a new instance whenever you want, you can really work on your machine configurations in an iterative way. Treat your AMI like source code, and don't be afraid to bundle and upload iterations of the AMI as you go along. If you mess up something, you will be glad you did. And there is no bandwidth charge between EC2 and S3 so go ahead, pile it into S3! At $0.15 per GB-Month of storage used, you cannot afford not to.



• Amazon provides very complete instructions on how to create an image. When you are "developing" an AMI, by iteratively modifying a running instance, and re-bundling and re-uploading, you can skip the step of uploading your keys. However, you will need to delete /mnt/image before you can re-run the "ec2-bundle-vol" utility, or else you get an "the specified image file /mnt/image already exists" error.



• You can get DRY with your AMIs. You can use a single AMI, and pass in launch parameters to set various configuration settings. For example, when I launch a processor instance, I use my main AMI and just pass in the required launch parameters to tell it if is staging or production, and which pool of processors to join itself to. And how do you retrieve this individual instance data at runtime? You use a REST call to get it. Neat!



• At some point, I did something wrong with the code that generated keys for each file I was saving to S3, which in S3 corresponds to the path and file name. For whatever reason, it messed up, and I could not even delete the S3 objects using the S3 Firefox Organizer! Fortunately, the somewhat ugly, but much more functional Bucket Explorer was able to get rid of the offending files.



• To really take advantage of the distributed nature of the solution, I decided to use a client-side load balancing scheme like that described by Lei Zhu in this article.
  
  The solution described does load balancing without a separate load balancer. It accomplishes this as follows: when a new processing node comes online, it registers itself by saving its name into a special bucket. Any server listed in this special bucket is available to serve requests. Add in regular peer to peer checking to verify that a node has not gone offline, and the peer removes it from the bucket. If that node, or a replacement, comes online, it will register itself. And so on... Zhu's solution is both effective and minimalist.
  
  As I said, the solution proposed at the end of the article was really cool, but no actual code was provided, since it is part of what appears to be the codebase for the author's startup company. So, I decided to implement it myself, which proved to be really fun and easy. A fully Ruby-based client side load balancer for EC2 is a pretty useful thing to have around. If anyone else is interested, I will publish as a gem or something.

We are not quite finished with our video processing farm, but it is almost there, and the results are a surprisingly small amount of code. I am been extremely impressed by the relative ease with which we have put together a pretty sophisticated solution.

Grid computing... it's not just the future, it's the present.