PostgreSQL on Ubuntu on EC2: Backing It All Up

This post continues what I started with "PostgreSQL on Ubuntu on EC2: The Installation Guide". Once you have your PostgreSQL database server instance running, you will need to backup two different things: your database data, and the instance itself. The database data will be backed up using Elastic Block Storage (EBS) snapshots. Once we have the instance running the backups correctly, we will then create an Amazon Machine Image (AMI) that will allow you to launch a new instance to replace the database server in case it goes down.

Backing Up The Database
First, we need to connect to our database server instance via SSH using the ubuntu user.

We will need to install some dependencies to get our backup script to run:

sudo apt-get install build-essential
sudo apt-get install ruby1.8-dev
sudo apt-get install rubygems
sudo gem update --system


You will need to tweak RubyGems so that the update works correctly, as described here.

Now you can install Gemcutter, which is the new ultra cool repository for gems:

sudo gem install gemcutter
sudo gem tumble


Finally we are ready to install the Amazon EC2 rubygem:

sudo gem install amazon-ec2


Now we can create our backup script. Save this code into the ~/ directory under the name backup_database.rb. You will need to substitute the Amazon ACCESS_KEY_ID and SECRET_ACCESS_KEY, as well and entering the correct EBS volume for the DATABASE_VOLUME constant:

#!/usr/bin/env ruby

require 'rubygems'
require 'AWS'

ACCESS_KEY_ID = 'YOUR_ACCESS_KEY'
SECRET_ACCESS_KEY = 'YOUR_SECRET_ACCESS_KEY'
DATABASE_VOLUME = 'vol-XXXXXXXX'

puts "Starting database snapshot..."
ec2 = AWS::EC2::Base.new(:access_key_id => ACCESS_KEY_ID, :secret_access_key => SECRET_ACCESS_KEY)
ec2.create_snapshot(:volume_id => DATABASE_VOLUME)
puts "Database snapshot completed."


Due to the finicky way that Ruby runs as part of a cron job, we are best off creating a shell script that then runs the Ruby backup script. Save this code into the ~/ directory under the name backup_db.sh:

#!/bin/sh
cd /home/ubuntu
ruby /home/ubuntu/backup_database.rb


Don't forget to make the backup shell script executable:

chmod +x /home/ubuntu/backup_db.sh


Now we just need to configure this script to run as part of a cron job, so that the backups take place automatically. The crontab command brings up the list of configured cron tasks for the current user:

crontab -e


This example crontab entry runs the backup daily at midnight, but you may want it to run more frequently:

0 0 * * * /home/ubuntu/backup_db.sh


At this point, you should have a fully functional automated backup system. Verify after midnight that the script has run as you expect, by looking to see if a new snapshot has been created, using Elastifox or however you administrate your EC2 instances.

Creating The AMI
Creating the AMI to backup the entire database instance is pretty easy. First, you need to upload the PEM files. Remember you are authenticating as the "ubuntu" user:

scp -i id_rsa-gsg-keypair pk-YOUR.pem cert-YOUR.pem ubuntu@domU-12-34-31-00-00-05.usma1.compute.amazonaws.com:


Use your SSH connection into the database server instance to copy the PEM files to the /mnt directory:

sudo cp /home/ubuntu/*.pem /mnt


Now create the bundle. Make sure you use your Amazon account number (without dashes) as the value for the -u parameter. This can take quite a while, so do not get impatient:

sudo ec2-bundle-vol -d /mnt -k /mnt/pk-YOUR.pem -c /mnt/cert-YOUR.pem -r i386 -u YOURUSERACCOUNTNUMBER


You can now upload the bundle to your Amazon S3 account, in preparation for making available as an AMI. Use something versioned for the -b parameter which is the name of the bundle:

sudo ec2-upload-bundle -b my-database-server-1.0-ami -m /mnt/image.manifest.xml -a YOUR_ACCESS_KEY -s YOUR_SECRET_ACCESS_KEY


Final step is going back to your local machine, and making the newly created bundle available to be used to start a new instance:

ec2-register my-database-server-1.0-ami/image.manifest.xml


You can now now launch a brand new database server instance based on this AMI, and it will be a clone of your existing database server. This is the procedure you would follow if you need to restore your database server instance from backups.

Restoring Your Database Server From The Backups
In the case that something goes terribly terribly wrong, you can get back to normal as follows:
- startup a new EBS volume from your most recent snapshot backup,
- start up a new server from your database AMI
- configure your new server instance to use the new volume started from the backup data
- switch your elastic IP to point to the new server, or update the references in your application to point to the new server

This concludes part 2 of the great PostgreSQL config post for the EC2 cloud. Hopefully it will help you with a nice simple way to take the basic PostgreSQL instance that you got up and running on Ubuntu/EC2 using the directions in the part 1 post, and add the confidence that backed up data and a completely reproducible configuration provides.

On The Grid With EC2

Lately, I have had a chance to get seriously into Amazon's Elastic Compute Cloud (EC2) service. I doubt any reader of this blog is unfamiliar with the existence of EC2, but many of you have probably not even gotten to the tire kicking stage, like I had been until recently.

Even after attending several sessions about Amazon's EC2 at RailsConf 2007, I still had only a very general idea about the service. However, it is true that there is no substitute for experience, and now that I have finally been getting hands-on, I am pretty excited about what you can do with it.

A client with a pretty complex Ruby on Rails application needed to create a processing farm for doing video conversion, as well as serving up the resulting video files. Sounds like the perfect application for Amazon's utility computing services. I dug in to the myriad of online sources of information available...

I decided to base my Amazon Machine Image (AMI) on Ubuntu 7.1. Oh, Ubuntu, such a modern operating system you are! That turned out to be a very good decision, and apt-get was quite co-operative getting the myriad of prerequisites my open source video processing farm would require.

I also decided to base my upload processor on Sinatra. My application's entrypoint is a single stateless call, but there are lots of these calls, and each requires a bunch of intensive back-end processing. Something nice and multithreaded, like some kind of mongrel handler, was called for. In this case, even Merb seemed like it was too heavyweight!

Luckily, I had colleague Ari Lerner available to help me strap Sinatra into my processing farm, who proved to be indispensable. We ran into some issues with Rack, which I am sure Ari will be blogging about. Thanks, Ari, you are the man.


Some lessons learned about working with EC2 and S3:

• When you can launch a new instance whenever you want, you can really work on your machine configurations in an iterative way. Treat your AMI like source code, and don't be afraid to bundle and upload iterations of the AMI as you go along. If you mess up something, you will be glad you did. And there is no bandwidth charge between EC2 and S3 so go ahead, pile it into S3! At $0.15 per GB-Month of storage used, you cannot afford not to.



• Amazon provides very complete instructions on how to create an image. When you are "developing" an AMI, by iteratively modifying a running instance, and re-bundling and re-uploading, you can skip the step of uploading your keys. However, you will need to delete /mnt/image before you can re-run the "ec2-bundle-vol" utility, or else you get an "the specified image file /mnt/image already exists" error.



• You can get DRY with your AMIs. You can use a single AMI, and pass in launch parameters to set various configuration settings. For example, when I launch a processor instance, I use my main AMI and just pass in the required launch parameters to tell it if is staging or production, and which pool of processors to join itself to. And how do you retrieve this individual instance data at runtime? You use a REST call to get it. Neat!



• At some point, I did something wrong with the code that generated keys for each file I was saving to S3, which in S3 corresponds to the path and file name. For whatever reason, it messed up, and I could not even delete the S3 objects using the S3 Firefox Organizer! Fortunately, the somewhat ugly, but much more functional Bucket Explorer was able to get rid of the offending files.



• To really take advantage of the distributed nature of the solution, I decided to use a client-side load balancing scheme like that described by Lei Zhu in this article.
  
  The solution described does load balancing without a separate load balancer. It accomplishes this as follows: when a new processing node comes online, it registers itself by saving its name into a special bucket. Any server listed in this special bucket is available to serve requests. Add in regular peer to peer checking to verify that a node has not gone offline, and the peer removes it from the bucket. If that node, or a replacement, comes online, it will register itself. And so on... Zhu's solution is both effective and minimalist.
  
  As I said, the solution proposed at the end of the article was really cool, but no actual code was provided, since it is part of what appears to be the codebase for the author's startup company. So, I decided to implement it myself, which proved to be really fun and easy. A fully Ruby-based client side load balancer for EC2 is a pretty useful thing to have around. If anyone else is interested, I will publish as a gem or something.

We are not quite finished with our video processing farm, but it is almost there, and the results are a surprisingly small amount of code. I am been extremely impressed by the relative ease with which we have put together a pretty sophisticated solution.

Grid computing... it's not just the future, it's the present.