Dead Programmer Society

Gabba: A Ruby Gem For Doing Server Side Google Analytics Tracking

2010-11-30T23:05:00.000-08:00

Yo, Gabba! Gabba, hey? Gabba is a Ruby gem to do easy server-side Google Analytics tracking of page view and custom events. Gather around and I'll show you how it works.

Google Analytics is the gold standard of website traffic reporting. In addition to tracking page views, one of the interesting capabilities it has is to track custom events. These could be shopping cart checkouts or video plays from clicking on a button in a Flash player.

The way that Google Analytics is implemented is using the typical tracking image technique. An image is fetched from the server, and based on the infomation pased with this request, the information about the visit is tracked. One twist on this is that Google Analytics generates the tag for the "traxel" using a piece of asynchronous JavaScript. This means that clients without javascript are not able to use Google Analytics to track visitor metrics... or does it?

There is another lesser-known way to integrate that can generate the needed info entirely on the server. This is normally used to handle older mobile browsers that do not have JavaScript implementations. Since all you are doing is fetching a 1x1 pixel GIF image, all you really need is the magic URL.

This is in fact exactly what you would need to do if you want to track custom events, the only real difference being the specific parameters passed to Google Analytics.

There are a number of cases where you might want to track custom events for non-browser devices or other unique situations. Tracking when a purchase is completed, or when a video is played by Flash player or a phone call is initiated, are just a few things you might want to track entirely on the server, with no browser interaction at all.

I did a bit of searching, but did not find any Ruby library that did what I needed. You know what that means! Fire up the editor, I'm going in! And that is how the gabba gem was born.

It is very easy to use:


# track page views
Gabba::Gabba.new("UT-1234", "mydomain.com").page_view("something", "track/me")

# or track custom events
Gabba::Gabba.new("UT-1234", "mydomain.com").event("Videos", "Play", "ID", "123")

That is all there is to using it. Simple way to get server side Google Analytics tracking of page view and custom events. Fork gabba on github, or just "gem install gabba" and start tracking all your fun stuff.

10 Cool Things From RubyConf X

2010-11-17T07:20:00.000-08:00

This last week was the fantastic RubyConf X. It has been ten years of RubyConf and in celebration of this notable occasion, the organizers located it in New Orleans and opened up the registration to a lot more people then most years past. This had the clearly foreseeable consequence of turning the conference into a really fun time!

So here, in no particular order, are 10 cool things I saw or heard at the conference.

1. Rite

During Matz's keynote he introduced "Rite" which is a embedded version of Ruby designed to run on devices such as digital TVs, and other neat small gadgets. It will have a different implementation that the CRuby implementation, optimized for the smaller lower powered profile of the next generation gadgets. The project is sponsored by the Japanese government and is already being used in development by a major game company that could not be named but is will known. My palms started sweating the minute I heard about it, and I am already getting excited about the prospect of flying_robot 2.0 based on Rite.

2. Evergreen w/CoffeeScript

CJ Kihlbom and Jonas Nicklas gave an excellent overview of client-side UI testing. They also demoed their own highly useful Evergreen testing framework which works together with Ruby and Jasmine looks like a very succinct and useful approach toward browser testing. Writing specs in CoffeeScript looks very tasty.

3. Gentlemanly Git Tricks

Scott Chacon's talks on git always leave me feeling knowing both more than I did, and less then I thought I did, at the same time. This session also had a hilarious and yet useful section on 'how to be a gentleman' including tips on whiskey, how to dress, how to tip, and how to treat a lady. Keeping it classy, Scott!

4. "F-Bombs, Zeds, _whys and a Missing Brain Area"

The keynote speeches from Dave Thomas, David Heinemeier Hansson and Matz could not have been more different on the surface, and yet had a similar theme: defy the limitations of the status quo, and seek out new ideas that are actually better. Dave's admonishment of the community to get over our gender bias, David's F-bombs, weed photos, and cry of "Freedom!", and Matz's love of diversity from _why to Zed, and adopting of David's amusing yet appropriate term "freedom-patching" in lieu of the less pleasant-sounding "monkeypatching" were all different ways of showing our desire as a community to seek out these truths. As long as we keep talking to each other, and also listening with an open mind.

5. Ruby 1.9.2 is Ready

Several people have been saying it, I am been using it from some testing, and Jim Wierich said it in the 'hallway track': Ruby 1.9.2 is ready for prime-time. If you are not transitioning your apps to 1.9.2 you should be. Performance, and cool tools like minitest are just two reasons to do it today.

6. Beyond The Code

Paul Campbell, Joe O'Brian, Keavy McMinn, Jon Dahl, Tom Preston-Werner and several other interesting developers spoke about more than just programming. Hacking business, art, music, and code, are all just aspects of hacking. RubyConf is a great place to find out not just about Ruby code, but the motivations and inspiration behind people we respect, and look to to improve ourselves.

7. Lightning Talks

The number of people that are eager to signup for a lightning talk was huge at RubyConf, as always greatly exceeding the available time. Even trimming down each to 4 minutes, there was only time to get thru relatively few, and I was lucky to get to speak about the Ticketmaster gem.

8. Kid Programmer Mini-Conf

Sarah Allen, Sarah Mei, Jim Meyer (who sadly could not make the conference at the last minute), Liah Hansen put together a parallel kids programming track, for the first time ever at RubyConf. The idea was brilliant, and I hope to bring my own son next year to participate.

UPDATE: Sarah Mei informs me that Maxwell Salzberg is also an organizer of the kids track. My bad!

9. Hallway Track

As always, the conversations at RubyConf are amazing. The people who attend are doing amazing things, and the change to hang out for a few days talking is a fantastic chance to learn and find out incredibly useful and interesting things. You have to actually attend RubyConf to experience this, you just cannot get it by watching the videos.

10. Music

From the live music in New Orleans, to the DJs, it is really great to have been in a city where music is something living, vibrant, and cherished. Getting to do my music/brain hacking talk on "How To Jam In Code" was just the icing on the cake for me in the city of Jazz. Getting to create a bio-computer made of living people is a special privilege, thank you very much to everyone who participated.

But most of all, thank you very much to Kelly Fowler, Chad Fowler, Rich Kilmer, and David Black for doing so very much for all of us. It is a lot of hard work, and we all appreciate it!

RubyConf X was a tremendous affirmation of the strength and depth of sharing within the Ruby community. I look forward to seeing what the next ten years have in store for us.

Happy Ada Lovelace Day 2010

2010-03-24T09:44:00.000-07:00

Once again, it is Lady Ada Lovelace day, and time for me to celebrate some women software developers that I know personally. This year I want to call out four women that I have especially noticed their knowledge and dedication. In no particular order...

First, a big shout out to Sarah Mei (@sarahmei). Sarah is a founder of RailsBridge, a non-profit organization dedicated to helping people learn about and make use of Ruby on Rails. She is also a conference speaker who has been going around hyping up the community about one of the subjects most near and dear to my heart: teaching kids programming.

Next, a special thanks to Sonia Ramirez, my colleague at our Los Angeles based software development consultancy The Hybrid Group. Sonia is a tireless and detail oriented software developer. She is always eager to learn, and brings a great sense of humor to all of her work with us.

Third, a special thanks to Andrea O.K. Wright. Andrea has been a very active member of the Ruby community, and has given a number of presentations, must recently an incredibly ambitious overview of NoSQL databases at RubyConf 2009.

And rounding out this small sampling of the XX chromosome-based programming talent, is a new acquaintance Sarah Allen (@ultrasaurus). Sarah is also involved with the RailsBridge project, and is a mobile software developer who has published several projects based on the Rhodes framework. She is also a conference speaker, and published the RubyConf mobile application that we all made use of at LARubyConf earlier this year.

These persons of the female gender are merely a small sampling of the awesomeness that exists within the Ruby community. They just happen to be ladies that I have either worked with, or benefited from their contributions directly.

Thank you all, and have a great Ada Lovelace Day. And get back to some coding!

Configuring Hybrid Ruby on Rails Applications On EY-Cloud

2010-03-04T10:30:00.000-08:00

This last weekend, we completed migration of a slightly more complex Ruby on Rails application than the usual to Engine Yard's EY-Cloud. It took a few tries, and a tiny bit of research, before we completely understood what to do to get it up and running with all of the needed capabilities. Naturally this was due to gaps in my own knowledge, and I wanted to post about what I had learned, and how awesome EY-Cloud really is!

If you do not know about it, EY-Cloud is Engine Yard's cloud platform that is based on Amazon's EC2. Readers of this blog know I am not afraid to get hardcore when it comes to EC2 server configurations, and with EY-Cloud I feel like I can now get the best of everything. But I am getting ahead of myself.

Let me briefly describe the architecture of the application I was migrating. The main application is written in Ruby on Rails and uses MySQL as its data store. The app provides UI and an API for interactions with the application, mostly called by a custom iPhone app written using PhoneGap. The main app is secured using an SSL certificate. In addition, a separate API for a specific high-performance capability is written using Sinatra and DataMapper. This "special API" is also secured via SSL.

OK, you got that? Perhaps not, so here is a diagram to illustrate what I am talking about:

We call this kind of application a "hybrid" application, meaning that it is built using Ruby on Rails + "something else". For this app, the "something else" is Sinatra + DataMapper.

The previous installation used the same basic setup, but was created manually using hand-configured EC2 instances. This required a lot more management to keep the application going, and especially to add or remove instances to the cluster. Backing up the entire configuration by creating custom AMI's was also very labor intensive.

This was the scenario that led us to wish to migrate the app from raw EC2 to EY-Cloud. So if you are planning to set something like this up yourself, here are my notes to share my experience migrating a hybrid Ruby on Rails application to EY-Cloud.

Prerequisites
You need to have an account setup with Engine Yard for EY-Cloud. In addition, you should have installed the Elasticfox plugin for Firefox to easily administer the security groups. You can also do this via the command line, but it is a lot more effort.

Configure Main Environment & Application
Configuring your main application for EY-Cloud is a pretty simple process. It basically requires setting up private key access to the git repository and the application settings using the web based interface that Engine Yard provides. There is a complete article that describes this in detail here. The main steps are:

Create main environment

Create main application and configure access to its git repo

Config ssh key you will use to connect to instances

Add ssl certificate, and config main app to use it

Config any needed gems and unix packages to be used by main app

Launch app server and separate database server in main environment

Configure Secondary Environment & Application
Now you are ready to configure the environment and application for the other public accessible SSL protected service that is part of the application. This needs to be a secondary application because EY-Cloud currently only allows a single SSL cert per environment, and only allows you to associate it with one application. Hence needing a second environment configured. The main steps are:

Create specialapi environment

Create specialapi application and configure access to its git repo

Config ssh key you will use to connect to specialapi instances

Add ssl certificate, and config specialapi app to use it

Config any needed gems and unix packages to be used by specialapi app

Launch specialapi app in specialapi environment. No database is necessary.

Copy contents of database.yml from /data/mainapp/shared/config/database.yml on "main app" server to /data/specialapi/shared/config/database.yml on "specialapi" server. Yes, you will need to use SSH to login to both servers.
Use keep file on the database.yml location on the specialapi environment to point to the master database in the main app environment. This page explains about keep files.

EC2 Security Configuration
In order to allow the secondary environment to get access to the database located in the main environment, you need to make a small but critical change to the configuration in the EC2 Security Groups. Otherwise the internal firewall at Amazon will not allow the application to access the database. It is very easy to configure this using Elasticfox. Here is how:

Setup your EC2 security credentials for your EY-cloud account. You can download them as part of the ey-cloud.yml file

Once you are connected to that account using elasticfox, click on the "Security Groups" tab. You should see 3 security groups "default", one for the main app named something similar to "ey-myapp_production-12341234" that refers to the main environment, and another one named something like "ey-secondaryapp_production-56785678" that refers to the secondary environment

Click on the "ey-myapp_production-12341234" group in the left pane labeled "Your Groups" then click on the green checkmark icon at the top of the "Group Permissions" pane. This will bring up the "Add New Permission For Security Group" dialog. Click on the "Group" tab, then choose the "ey-secondaryapp_production-56785678" group from the Groups dropdown, and click on the "Add" button. Your secondary environment should now be able to access the main environment's database

Conclusion
It is pretty great to be able to take advantage of both the cool management interface and scalable clustered database and application stack, as well as being able to get down and dirty via SSH, something you really do need in order to configure more complex hybrid Ruby on Rails applications. EY-Cloud provides this kind of capability, and is certainly a lot easier than managing it all yourself. Hopefully, these notes will make it easier for you to migrate your own complex application. Please let me know how it works for you.

I Was A Guest Blogger At LARubyConf 2010

2010-02-22T11:40:00.000-08:00

Longtime readers of this blog know that I can really get into writing about Ruby conferences, and I had a great time this last weekend at LARubyConf 2010. This time, I got to try some thing new: I was guest blogging the conference for TheBitSource, which has been providing some great coverage of other tech events like PyCon and SCALE.

Please check out my complete recap from the 2nd annual Los Angeles Ruby Conference LARubyConf 2010 here. And thanks!

Crossing The PhoneGap For Multiplatform Mobile Applications

2010-01-18T08:30:00.000-08:00

I had first heard of the PhoneGap open source framework for multiplatform mobile development last year at the FutureRuby conference in Toronto. Honestly, I did not really concentrate on all they were saying at the time, and in the flurry of info including back-to-back mobile sessions with Rhomobile, I did not fully retain a clear picture of what they had to offer. My bad.

It was not until late last year, while working on plans for a very cool mobile application, that I was reminded about PhoneGap by one of my colleagues. After a brief evaluation of their benefits vs. the other multiplatform options, we decided to use PhoneGap on this particular project. We have made some amazing progress with using PhoneGap for mobile development since then, and I thought I would share a few of the lessons learned.

First of all, a quick explanation of how PhoneGap works. Jesse MacFadyen, one of the programmers at Nitobi, the primary developers of PhoneGap, has a good blog post where he breaks down how the PhoneGap framework works on iPhone. Here is my much condensed take on what he is saying.

All of the mobile platforms supported by PhoneGap have some kind of web browser control. A PhoneGap application is a packaged up application which is a webpage or mini-website, that executes inside whatever web browser control is available on that platform. Add in a standard JavaScript API to a wrapper that accesses the device-specific functionality like GPS or the accelerometer, and you can hook up the JavaScript on your "page" to the hardware.

But don't make the mistake of thinking you should just slap together some web pages formatted for mobile. Rather, you can and must think of it as an application that is written in the form of a single web page with a bunch of JavaScript. To access server data, you will need to write some AJAX code to access the remote resources, then update your UI accordingly. The good news is you can use familiar JavaScript libraries such as jQuery to do so. We chose jqTouch to get a very iPhone-like UI.

The PhoneGap framework is under very active development. New code is being committed to their github repo frequently. As such, you really do need to have git installed, and some working knowledge of how to use it, to get things setup.

In fact, the device specific functionality for each platform is contained within the main PhoneGap git repository as a series of git submodules such as phonegap-iphone, phonegap-android, etc. This makes it all but impossible to install the latest and greatest without git. Being a git user myself, this does not pose any problem, and if you are not gitmotized yet, this is an ideal time to become so.

Here are the series of steps I followed to create my own PhoneGap project...

Prerequisites

Install iPhone SDK (yes, you must join the iPhone developer program)

Install Android SDK (http://www.talkandroid.com/android-sdk-install-guide/ has some good instructions. I am not using Eclipse, so I skipped all that)

Install Apache Ant (sudo port install apache-ant)

Install PhoneGap

Get latest PhoneGap (git clone git://github.com/phonegap/phonegap.git)

Get submodules for iPhone & Android (cd phonegap && git submodule update)

Build iPhone Lib and install

Build Android libs

Setting Up A New Multiplatform PhoneGap Project

Create a new iPhone project using the phonegap-iphone template. The www directory in the new project will be becoming the shared part of your project, with all your UI and application logic

Create the Android project using the phonegap-android build script. Use the www directory from the iphone project as the www-dir param. You will be replacing this with a reference to the git submodule

Create a new git repo in the www directory in the iPhone project, and commit all the files in the www directory

Change directories to the parent of where you want to put the directory for the shared git repo, then clone the repo located at /path/to/iphoneproj/www to /path/to/sharedrepo. You may also want to create a remote branch at this point for the new shared repo

Remove the www directory from the iPhone project

Create a new git repo in the iphone project directory, then commit all files. You may also want to create a remote branch at this point

Put the shared code into the iPhone project by using "git submodule add" to put a reference to the shared repo into /path/to/iphoneproj/www

Create a new git repo in the Android project directory, then commit all files. You may also want to create a remote branch at this point, as you probably did for the iPhone project files

Remove the /path/to/androidproj/assets/www directory from the Android project, and use "git submodule add" to put a reference to the shared repo at /path/to/androidproj/assets/www

You now should have 3 git repos, with the shared code, the iPhone specific project code, and the Android specific project code, each in their respective places. Much easier and nicer to work on.

Anyhow, I hope this info is useful to any PhoneGap developers that want to get things setup cleanly for multi-platform mobile programming. Please let me know any feedback or improvements that people out there come up with, for this neat open source framework for doing JavaScript-based mobile development.

Sparkline Some Interest With Ruby on Rails

2010-01-12T12:12:00.000-08:00

Recently, I added some sparkline graphs to a Ruby on Rails application. A sparkline is a very small graphic that displays a large amount of information, typically shown over time, and usually embedded in some other text. Invented by the father of modern infographics Edward Tufte, the sparkline has become a fixture of many online applications that want to visually display some stats in a simple, integrated way.

When it come to adding sparklines into a Ruby on Rails application, there are a couple of different options. You can chart the data on the server, output being an image file. You can also chart the data on the client, with a couple of different JavaScript libraries as available options.

I started my plan, intending to implement my charts on the client-side. The project that appears to have the most flexibility, speed, and options, is an amazing jQuery plugin called jQuery Sparkline. Unfortunately, the project to which I needed to add the sparklines is a bit older, and does not use jQuery. As a result, I was not able to use jQuery Sparkline for this current project.

Another interesting JavaScript sparkline library is lethain's Sparkline.js, but it has not been updated in some time, and is not compatible with current Internet Explorer versions. There is also an interesting looking newer lib topfunky-sparkline-js but I have not tried it out yet.

So that brings us to server-side sparkline generation. If you are using ImageMagick/RMagick then @topfunky once again provides, with the Ruby sparklines gem. This gem provides lots of options for doing all sorts of fancy sparklines.

In my case, I am not using ImageMagick for anything else, so I did not want to install it just for this. What I really wanted was something much lighter-weight, and I was willing to accept a lot fewer options to get it.

It turns out that madrobby has written a library for generating very simple sparklines in pure Ruby code, called spark_pr. The project uses _why's pure Ruby implementation of a PNG generator to do the low-level work.

spark_pr has also spawned an interesting application of it from @technoweenie called Sparkplug, which is a Rack module that generates spaklines from CSV data on the fly, using Rack handlers and Rack caching.

Once I had seen Sparkplug's minimal elegance, it seemed spark_pr was the option for me. I decided to incorporate spark_pr into my application. Given that the app was written with a dedicated approach to keeping a clean RESTful interface, and that the database already contained the time-series data, it was quite easy to incorporate. Here is what I had to do:

Step 1: Put spark_pr.rb file into lib directory. I just grabbed the code from the repo, and dropped it into my project. You may decide to have a more sophisticated way to do it, such as using git submodules.

Step 2:

require "spark_pr"

in your controller

Step 3:

include Spark

in your controller

Step 4: Add PNG format to controller action that was already returning my time-series data, and return the sparkline PNG data:

Note that this controller action is already returning XML or JSON. PNG is just another format to be added, if you have a well-designed RESTful controller action.

Step 5: Put image_tag that requests the sparkline PNG image file into the view where you want it to appear, like this one:

Voila! Refresh that view and you should now be looking at your ultra-hipster-chic sparkline graph. It is surprising how much more information comprehension a person has, when they are seeing a visual representation of their data. Plus it looks cool.

Thanks For A Great #code2009

2009-12-31T09:07:00.000-08:00

It has been an amazing year for both personal and professional code development.

Starting with the inspiration to begin Project Flying Robot, to the prestige of presenting at LARubyConf, FutureRuby, TWTRCON, IgniteLA, 140 The Twitter Conference, RubyConf, and Conferencia Rails, and lastly the year-end fun of starting out the ongoing #code2009 Twitter meme, so popular that it spawned a couple of mashups and got picked up by Hacker News and uber-language blog Lambda the Ultimate.

In between were numerous meetups, hackfests, code jams, code dojos, pull requests, and casual codeslinging with friends. And the Maker Faire!

To everyone who welcomed me, listened to me, helped me, or taught me something, I am indeed grateful. Thank you. Let's do this 2010 thing right!

Flying Robot: World Tour 2009 Continues

2009-11-16T08:15:00.000-08:00

As usual, no blog posts = a lot of other activity here at Flying Robot HQ. Among other personal stuff, my brother Damen Evans and I have been getting ready for the last public demos of @flyingrobot for 2009. And we are going out with style!

Later this week, we roll up to San Francisco to present at the prestigious RubyConf! Then, next week @flyingrobot and I will fly off to Madrid, Spain, to do our first European appearance at the awesome-looking Conferencia Rails.

Anyhow, if you have been waiting eagerly for more Flying Robot news and gadgets, be patient. We will be unveiling our mysterious cool new stuff in just a few days...

PostgreSQL on Ubuntu on EC2: Backing It All Up

2009-10-10T11:19:00.000-07:00

This post continues what I started with "PostgreSQL on Ubuntu on EC2: The Installation Guide". Once you have your PostgreSQL database server instance running, you will need to backup two different things: your database data, and the instance itself. The database data will be backed up using Elastic Block Storage (EBS) snapshots. Once we have the instance running the backups correctly, we will then create an Amazon Machine Image (AMI) that will allow you to launch a new instance to replace the database server in case it goes down.

Backing Up The Database
First, we need to connect to our database server instance via SSH using the ubuntu user.

We will need to install some dependencies to get our backup script to run:


sudo apt-get install build-essential
sudo apt-get install ruby1.8-dev
sudo apt-get install rubygems
sudo gem update --system

You will need to tweak RubyGems so that the update works correctly, as described here.

Now you can install Gemcutter, which is the new ultra cool repository for gems:


sudo gem install gemcutter
sudo gem tumble

Finally we are ready to install the Amazon EC2 rubygem:


sudo gem install amazon-ec2

Now we can create our backup script. Save this code into the ~/ directory under the name backup_database.rb. You will need to substitute the Amazon ACCESS_KEY_ID and SECRET_ACCESS_KEY, as well and entering the correct EBS volume for the DATABASE_VOLUME constant:


#!/usr/bin/env ruby

  require 'rubygems'
  require 'AWS'

  ACCESS_KEY_ID = 'YOUR_ACCESS_KEY'
  SECRET_ACCESS_KEY = 'YOUR_SECRET_ACCESS_KEY'
  DATABASE_VOLUME = 'vol-XXXXXXXX'

  puts "Starting database snapshot..."
  ec2 = AWS::EC2::Base.new(:access_key_id => ACCESS_KEY_ID, :secret_access_key => SECRET_ACCESS_KEY)
  ec2.create_snapshot(:volume_id => DATABASE_VOLUME)
  puts "Database snapshot completed."

Due to the finicky way that Ruby runs as part of a cron job, we are best off creating a shell script that then runs the Ruby backup script. Save this code into the ~/ directory under the name backup_db.sh:


#!/bin/sh  
cd /home/ubuntu
ruby /home/ubuntu/backup_database.rb

Don't forget to make the backup shell script executable:


chmod +x /home/ubuntu/backup_db.sh

Now we just need to configure this script to run as part of a cron job, so that the backups take place automatically. The crontab command brings up the list of configured cron tasks for the current user:


crontab -e

This example crontab entry runs the backup daily at midnight, but you may want it to run more frequently:


0 0 * * * /home/ubuntu/backup_db.sh

At this point, you should have a fully functional automated backup system. Verify after midnight that the script has run as you expect, by looking to see if a new snapshot has been created, using Elastifox or however you administrate your EC2 instances.

Creating The AMI
Creating the AMI to backup the entire database instance is pretty easy. First, you need to upload the PEM files. Remember you are authenticating as the "ubuntu" user:


scp -i id_rsa-gsg-keypair pk-YOUR.pem cert-YOUR.pem ubuntu@domU-12-34-31-00-00-05.usma1.compute.amazonaws.com:

Use your SSH connection into the database server instance to copy the PEM files to the /mnt directory:


sudo cp /home/ubuntu/*.pem /mnt

Now create the bundle. Make sure you use your Amazon account number (without dashes) as the value for the -u parameter. This can take quite a while, so do not get impatient:


sudo ec2-bundle-vol -d /mnt -k /mnt/pk-YOUR.pem -c /mnt/cert-YOUR.pem  -r i386 -u YOURUSERACCOUNTNUMBER

You can now upload the bundle to your Amazon S3 account, in preparation for making available as an AMI. Use something versioned for the -b parameter which is the name of the bundle:


sudo ec2-upload-bundle -b my-database-server-1.0-ami -m /mnt/image.manifest.xml -a YOUR_ACCESS_KEY -s YOUR_SECRET_ACCESS_KEY

Final step is going back to your local machine, and making the newly created bundle available to be used to start a new instance:


ec2-register my-database-server-1.0-ami/image.manifest.xml

You can now now launch a brand new database server instance based on this AMI, and it will be a clone of your existing database server. This is the procedure you would follow if you need to restore your database server instance from backups.

Restoring Your Database Server From The Backups
In the case that something goes terribly terribly wrong, you can get back to normal as follows:
- startup a new EBS volume from your most recent snapshot backup,
- start up a new server from your database AMI
- configure your new server instance to use the new volume started from the backup data
- switch your elastic IP to point to the new server, or update the references in your application to point to the new server

This concludes part 2 of the great PostgreSQL config post for the EC2 cloud. Hopefully it will help you with a nice simple way to take the basic PostgreSQL instance that you got up and running on Ubuntu/EC2 using the directions in the part 1 post, and add the confidence that backed up data and a completely reproducible configuration provides.

Happy 200 Posts: My 10 Personal Favorites

2009-09-13T15:42:00.000-07:00

I was shocked to discover this morning that this is to be my 200th blog post. Wow! It has been a good run so far since I restarted the Dead Programmer Society in 2006, and I really appreciate the awesome feedback and support that I have received from the community.

To commemorate this personal event, here is a list of my top 10 favorite posts, in no particular order:

1. "I'd Rather Be A Jazz Programmer"
2. "Fear And Loathing At RailsConf 2009"
3. "Programming Zombies Will Crush You"
4. "The Twitter 1-2-3 Rule"
5. "Goldilocks and the Three Icons"
6. "Money In The Ghetto"
7. "I Speak For The Code"
8. "The Folly Of Accountabalism"
9. "The Planning Game Vs. The Crying Game"
10. "Architect Is Not An Honorary Title"

Once again, I thank everyone for your support, and I look forward to telling more tales from the Dead Programmer Society.

PostgreSQL on Ubuntu on EC2: The Installation Guide

2009-08-15T01:27:00.000-07:00

For some time, I have had clients hosting a couple different applications on Amazon EC2 using Ubuntu. One of these apps uses PostgreSQL, and has been running without event for quite a while. Yesterday, I got to catch up for lost time, by spending the entire day wrestling with data recovery issues related to a failed apt-get upgrade on an important database server. Luckily, the awesome Eric Hammond was around on IRC, came to my rescue, and coached my thru my self-inflicted pain.

If you are not interested in PostgreSQL, you probably just stop here. Nothing to see folks, move along. However, if you are looking for the well-lit path to getting PostgreSQL installed on Amazon EC2 will all the trimmings, read on.

I went to find various web pages to use as source material, expecting since the last time I went through this, someone would have written a nice definitive guide to installing PostgreSQL on Ubuntu, running it on a dedicated instance on Amazon EC2, and using Elastic Block Storage (EBS). Naturally, you want to be using the XFS file system too. However, no such luck: just a big collection of pages of instructions on the various parts, without any nice simple path to getting things working together.

Hence, this post tries to provide a set of instructions for getting things working, and avoiding a couple of problems that I have run into while running Postgres in production for the last couple years.

Step 0 - You are signed up for Amazon EC2, no? If not, there are plenty of pages with instructions on how to do so.

Step 1 - Choose your AMI
There are several AMI's available to you. I currently run Hardy 8.04 LTS x86 architecture in the USA, so I am using ami-5d59be34, but you may have other requirements. The Ubuntu EC2 starter guide has good info on your options.

Step 2 - Launch your instance
I like to use Elasticfox, cause I am super lazy. The command line works well, also.

What size instance? This AMI supports small and medium. PostgreSQL is pretty efficient these days, and especially using a dedicated instance and not running anything but the database server improves raw database performance considerably. You would probably be pretty surprised how well a small instance can perform, but choose medium if you think you will have more significant needs.

One key pattern I use for my EC2 hosted apps, is creating security groups in EC2 to separate my database servers from the public internet. I never use the default security group, but instead create a group for each tier of my application like "database", "web", "transcoder" and then allow specific groups to communicate with each other.

Step 3 - Create the EBS Volume
You can do this via Elasticfox, or via command line. Either way, make sure you do two key things: make sure you create the EBS volume in the same availabilty zone as your database server instance, and also make sure you create a volume with enough space. Here is how you would use the command line tools to create a 10GB volume in the 'us-east-1a' zone:


ec2-create-volume -z us-east-1a -s 10

One the volume is ready, attach it to the database instance. For example, this attaches an EBS volume named 'vol-VVVV1111' to the instance 'i-IIII1111' on device /dev/sdh:


ec2-attach-volume -d /dev/sdh -i i-IIII1111 vol-VVVV1111

Step 4 - Connect to the database instance
You need to SSH in to configure you new instance. Remember, you cannot connect as 'root' user in Ubuntu, you need to connect using the 'ubuntu' user. This page has good details about using sudo and SSH on the official Ubuntu EC2 AMIs.

OK, so now you are connected via SSH to your server. Of course, start with the usual update/upgrade:


sudo apt-get update && sudo apt-get upgrade -y

Step 5 - Install XFS
We will need to install the XFS file system. Actually, your could use some other file system, but XFS is quite mature and has good performance. Plus if you are crazy, you can scale up to a massive virtual RAID drive that will cost $4000 per month.


sudo apt-get install -y xfsprogs

Step 6 - Format the EBS volume using XFS
We need to install a file system on the EBS volume before we can do anything with it. Here is an example:


sudo modprobe xfs
sudo mkfs.xfs /dev/sdh

echo "/dev/sdh /data xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir /data
sudo mount /data

Now we have a /data directory that maps to our EBS volume. Anything we write to /data will be persisted, even if the database server instance itself terminates.

Step 7 - Install PostgreSQL
Now we need to get PostgreSQL installed. This page has a very nice simple set of instructions on how to do that correctly for Ubuntu, but here is a synopsis especially for a headless server. First install Postgres:


sudo apt-get install postgresql postgresql-client postgresql-contrib

Now reset the password for the postgres account in the PostgreSQL server:


sudo su postgres -c psql template1
template1=# ALTER USER postgres WITH PASSWORD 'password';
template1=# \q

And then change the password on the user account to match:


sudo passwd -d postgres
sudo su postgres -c passwd

Now we need to modify the postgres configuration file postgresql.conf. First, to allow other machines to connect to our instance, and also to have PostgreSQL use our nice shiny /data directory.


sudo nano /etc/postgresql/8.3/main/postgresql.conf

Change the line containing


data_directory = '/var/lib/postgresql/8.3/main'


data_directory = '/data/main'

Now change


#listen_addresses = 'localhost'


listen_addresses = '*'

and also change


#password_encryption = on


password_encryption = on

Save the file, then open the pg_hba.conf file so we can control who can access the server:


# DO NOT DISABLE!
# If you change this first entry you will need to make sure that the
# database
# super user can access the database using some other method.
# Noninteractive
# access to all databases is required during automatic maintenance
# (autovacuum, daily cronjob, replication, and similar tasks).
#
# Database administrative login by UNIX sockets
local   all         postgres                          ident sameuser
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               md5
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5

# Connections for all PCs on the subnet
#
# TYPE DATABASE USER IP-ADDRESS IP-MASK METHOD
host    all         all         0.0.0.0/0          md5 # wide-open, you may want to make this more specific to your database

Step 8 - Move the database files
We need to stop the PostgreSQL server, move the database files to our EBS volume, then restart the server.


sudo /etc/init.d/postgresql-8.3 stop
sudo mv /var/lib/postgresql/8.3/main /data
sudo /etc/init.d/postgresql-8.3 start

You are now running PostgreSQL on Amazon EC2 using EBS for your database, with the XFS file system. Congratulations!

I will write a followup post on how to setup your database server for self-backing up using EBS snapshots, but that is all I have time for right now. Hopefully this pared-down set of instructions has been useful to you. Thanks again to Eric Hammond, and everyone else who's blogs were culled together into this post.

The FutureRuby Revolution Will Not Be On AOL - Part 2

2009-07-25T09:37:00.000-07:00

FutureRuby Day 2 began in a seemingly calm and reflective way. Coffees were sipped, and hangovers nursed. As the self-inflicted wounds from the Pravda-Vodka-Kalashnikov faded, Pete Forde, our leader and spiritual adviser, began a short sermon.

His message was simple: Vegas is a horrible place to hold RailsConf. And we should live in a manner that follows the "Four Agreements". Seriously, yes, he said both of these things.

Pete told us of the source of his sudden enlightenment: Portland's Jupiter Hotel. Instead of a Gideon bible, they have copy of Four Agreements in each room. Pete, being a curious guy, started to read the book. To save us time, he summarized it in nice Twitter-sized chunks.

1. Use your words for good... do not gossip
2. Do not take anything personally
3. Do not make assumptions
4. Always try your best

With our spiritual bootstrapping complete, we proceeded to have a consciousness-expanding session from Collin Miller presentation called "Transc/Ending Encoding". This was NOT about video encoding.

Collin Miller - "Transc/Ending Encoding"

If the 60's revolt gave the counterculture heroes like Leary and Hoffman, it gave us tech heroes like Englebart and Kay.

When writing software, we edit text files. We use textual encoding is a way to flatten down information to a simpler structure. But what does editing text lack? There are other options to make programs without text.

There is this high priesthood of text, however, programming does not need to be difficult to be useful. The future is the ONLY frontier... so where are we as programmer-monks going?

Martin Fowler has his concept of "illustrative programming". As another example, a spreadsheet is non-textual programming.

Charles Simonyi's Intentional Programming in a different approach. It allows users to change names easily, or even program in two different natural languages. It does this by maintaining a constant set of references to everything in program. By doing this, different users can edit the same source database, without using the same editing style.

Another example is Subtext (http://www.subtextual.org/). In Subtext, everything is just a reference. It is like "googling the code". Subtext uses decision tables, and a syntax tree editor.

It was a very interesting talk, and it seems like many people were inspired to think differently about code. I was having Smalltalk flashbacks, and my brother Damen Evans was reminiscing about how cool HyperCard used to be.

Dr. Nic - "Living with 1000 Open Source Projects"

Next up was "Dr. Nic" aka Dr. Nic Williams who is actually a PhD in CS, so not just granting himself an honorarium. His talk was called "Living with 1000 Open Source Projects". I have heard Dr. Nic speak before, and he is a very intelligent and funny speaker.

There are two types of open source project founders:
Type A. Nurture and converse "Do you care?"
Type B. People who were previously type A

"Who ever looked at their old code and thought 'that's better than what I write now'?"

If you look after your old projects, you will end up with 500/hr. week of projects

"Open source projects don't scale, but neither does raising pets and children"

The question is which OSS projects to maintain? The pet projects you NEED every day

Goal: ZERO maintenance

How to reduce bad karma from "abandoning" your project:
- publish project status
- facilitate group therapy
- forward emails to mailing list

Put a badge on project home page that says last time someone contributed to the project

Aim for community to be self-sufficient

Github makes things easier with centralized patches. The github gem is great for laziness.

"Easy to give away commit rights, if you think 'this is not MY project, I just look after it'"

Aim: ZERO process cost

Aim for Zero
- don't use it? do not maintain it
- manage expectations
- community self-sufficient
- zero process cost
- zero defects

How to use your spare time
- find a hobby
- talk to your spouse
- create more projects

"you can do less"

Dr. Nic's talk really resonated with many of us. I, for one, immediately on getting back from the conference gave commit rights on two of my projects to two worthy individuals. Wow, what a relief!

Matt Knox - "Crimes Against Humanity, Writ Small"

After Dr. Nic, was a great talk called "Crimes Against Humanity, Writ Small" from Matt Knox. I have been hanging out for the last couple of years with Matt at various Ruby conferences, but I had no idea how awesome he really is, till he got to show his stuff at FutureRuby.

The message behind his talk was really about taking responsibility for one's own actions. This was a very important recurring theme, starting right from Nathanial Talbott's talk at the very beginning of FutureRuby "you write the software for the nukes, you own responsibility if they are used". In Matt's case, the "nukes" in question were adware. The kind that attaches itself to your machine like a vampire squid, and will not let go.

Like all roads to hell, Matt's started with the best of intentions. His wonderful idea was killing adware on Windows with Scheme. As in LISP. That sounds like a really fun job... and it was, at first.

From an auspicious start "kill this worm", the job progressed to "kill lots of worms/malicious ad clients". Then the job became "somewhat edgy" aka "kill competitors and keep us from being killed... by anything"

As a result of all this, there were major negative repercussions that took down the company. In the aftermath, Matt was able to do some amazing self-exploration. "What just happened? Is this just who I am?"

That brings us around to the famous Milgram experiments. The incredible part was that 70% went the distance, and did what they thought was "torturing" another human being. Most human evil lives here.

What does this mean?
- The human brain has a remote root exploit in 70% of the installed base
- Knowing is 1/2 the battle

Good
- don't be evil

Better
- recognize that people who do evil may not be evil
- this makes it easier to not hate them

Best
- set up structures to insure this does not happen

The world forgives. But to provoke forgiveness, one needs to own your actions, and their results.

There is a remote root exploit in human brain, but the world forgives.

Matt, thank you very much for your bravery and honesty, in sharing what was clearly a very painful learning experience.

Paul Dowman - "Between the Battleship and the FAILWhale"

After the raw psychology of Matt's talk, it was not easy for me to switch gears to Paul Dowman's talk "Between the Battleship and the FAILWhale". However, it was full of solid info with why's and how's about scaling. Here are a few highlights:

Scalability != performance
Performance is faster load time
Scalability is handling greater load on same hardware

2 kinds of scaling
Vertical scaling - increase power of a single unit of your architecture
Horizontal scaling - adding units to your architecture

Why is scaling so hard? It cannot be an afterthought.
Should I forget about my scaling problem till my app is a hit? It's a biz decision

Developers and shareholders should talk about the tradeoffs, because scaling has costs: it requires more capital, and makes system more complex.

You can do some simple things to prepare to scale, without a major engineering effort. The goal is to be able scale just by adding more servers

HTTP Caching like squid, varnish or Rack::Cache

Use a queue for anything not needed to render the page right then. You will get a faster response, have a more consistent system load, and have less contention for locks.

Amazon SQS is pretty cool. SQS is slow but scalable, simple and requires no maintenance or deployment.

Memcached is inherently distributed, and you can just add more instance to scale. But it is not a database, so do not treat it like one. Data can/will disappear, since it is not persisted.

For scaling your database, you have various options:
- Use an RDBMS like MySQL or PostgreSQL
- SimpleDB
- Tokyo Cabinet
- CouchDB
- something else

Traditional RDBMS cannot scale horizontally forever. However, a lot of data does fit the table paradigm and SQL is powerful. Do not confuse data storage with data management.

Joe Wilk - "Cucumbered"
Next up was Joseph Wilk, all the way from London, to talk about Cucumber. Joe is a very unassuming but smart and witty fellow. The way he structured his talk was really clever. He used BDD itself to describe BDD... brilliant!

So why use something like Cucumber? So that the customer can use something less syntactically stripped to describe their needs, THEN translate that to Ruby. It is a token of the conversation, and defines the acceptance criteria for the "customer". It is useful as a design tool, and provides executable documentation for the project.

Cucumber has a gateway for different human languages, so that the developer and customer can interact in the customer's own human language. Like Swedish, Spanish, or LOLCATS. In fact there are currently over 30 languages already supported.

Really, that is part of getting the most value out of Cucumber, is getting customers using Cucumber THEMSELVES... you can even just send around the "plaintext" using email, Google Docs, whatever allows you to share the plaintext data.

The Art of "plaintext"
- don't force structure
- avoid noise
- avoid inconsistency
- balance abstraction
- use Ruby language building blocks to keep things DRY

There are a couple of cool features that have gotten into Cucumber while I was not paying attention. One is Tagging, which allows you to tag a feature, and run only those features.


@any plaintext word
  
cucumber --tags ~@in-progress

Another is Continuous Integration (Work In Progress) (--wip) which looks very useful since running each and every feature story can be time-consuming and slow down a CI build. Running all of the features as part of a nightly build is a workable compromise, and this look pretty useful to me for Integrity integration etc.

It was a very cool talk from Joe, and if you are not using Cucumber you really should be. It is an amazing source of insight into the needs of the user, and a great way to explain WHY you are doing things, not just WHAT you are doing.

Avi Bryant - "Failure: An Illustrated Guide"
Next up, Avi Bryant gave a fun talk called "Failure: An Illustrated Guide". He basically took us thru 30+ iterations (I lost count) of UI variations trying to create an important part of the functionality for his new site Trendly.

It was interesting to see all of the different attempts that they made, in finally reaching what is a pretty cool and different UI metaphor for visualizing time-series data from website stats. Once the presentation video is online, it is for sure worth watching, not so much because of what they did, but more from how it will make you revisit your own UI.

Jon Dahl - "Programming and Minimalism"
Jon's talk about "Programming and Minimalism" delved into the comparisons and contrasts of music and programming. He played a number of musical examples that showed stylistic development of musical genres from simple forms, to complex ones, and then evolved to simpler ones as part of a new "branch" of development.

It was interesting to consider these parallels, especially since I happened to be sitting next to friend Greg Borenstein, who's classical musical vocabulary is much greater than mine, and had interesting side-channel comments. Like Avi's presentation, the video/audio is probably needed to in order to get more than a superficial explanation of his points.

Brian Marick - "Artisanal Retro-Futurism and Team-Scale Anarcho-Syndicalism"
I was really looking forward to the next talk. Brian Marick is one of the original authors of the "Agile Manifesto" and a very interesting thinker. I had heard him bandy about this phrase "Artisanal Retro-Futurism and Team-Scale Anarcho-Syndicalism" and I was eager to hear what it meant. FutureRuby was about to get radical. The video of this talk is now online, so I really suggest you check it out for yourself.

"When I say agile, I mean Ruby... the way that Ruby projects are run"

"Switching to scrum, at least my job doesn't suck as much as it used to"

"I don't want to see on my gravestone, 'he made agile projects suck a little less'"

"Even the wage-slave can have joy-in work"

"the cubical is the single worst design of people and space to do software development"

What is "anarcho-syndicalism"? It is a political/economic trade-union movement peaked in 1923, crushed in 1924 by the U.S. government.

Here were a few of their tenants:
- getting rid of the government, and getting rid of private corporations
- worker self management
- direct action
- worker solidarity

Brian suggests adopting some of the ideas of the anarcho-syndicalists but at Team-scale" meaning within your own team.

Teams should band together more than they do, and we need more power in the hands of team to counterbalance the power in the corporation.

So on to this "Artisanal" thing. Brian used the example of artisanal cheese. The people who make this cheese are very into cheese. They care about the cheese! They do not just do it for profit, profit is the result of their caring.

Lastly back to the "retro-futurism". Recently, the New Yorker magazine did an issue about innovation. However, to capture the idea of innovation, they used images from the past, like the jet pack.

The idea of "retro-futurism" is trying to recapture the spirit of hopefulness from the past. Books like Freeman Dyson's "Infinite In All Directions" capture this endless sense of possibility.

Do not let the context drive you, you control the context. Brian calls for a revolution in how software development projects are run, and challenges us to be scrappy, care, and keep our spirit of naive optimism.

Start doing something about this: go to arxta.net, and talk to your teammates. And yes, I do have a sticker on my MacBook Pro.

Jesse Hirsh - "Fighting the Imperial Californian Ideology"
The final presentation, from Jesse Hirsh, was even more radical than Brian Marick. Jesse challenged all of us by taking some of those same principles that we had all just agreed with coming from Brian about software, but extended them further. Much further.

There have already been a couple of good posts that summarize or comment on Jesse's talk. Go check them out if you want more detail.

A couple of books that have influenced Jesse are "Snow Crash" and "Imperial San Francisco". The reason this is important is that ideologies are viral. In the mid 1800's the US sent surveyors into California, and once the mineral wealth there are been established, declared war on Mexico to get mines.

This was only the first of many "gold rushes" to take place in CA, although subsequent ones would develop other resources than mines. San Francisco technology invented the mining shaft to extract greater amounts of resources from the same mine. Taking those same technological achievements, a mining shaft turned upside down was a skyscraper - mining human labor instead of minerals.

The Hearst mining family was the most successful of these robber barons of mining, and the most responsible for many of the negative outcomes that resulted. As an example, Hearst Mining is responsible for 8 of 10 Superfund hazardous waste cleanup sites.

But it did not end there. As is well documented, the Spanish-American War, which resulted in the brutal colonial occupation of the Phillipines, was triggered by the first manufactured war, created by the first media mogul William Randolph Hearst.

San Francisco built all arms, and still responsible for all advanced military technology today. One big example is U.S. nuclear weapons, which are designed at the Lawrence Livermore National Labs.

This all established California as a place where a few small elites could conquer the world. The end of the cold war, was replaced by new imperial project - the California Ideology. The acolytes of this new ideology were Kevin Kelley, Stewart Brand, and the Global Business Network.

Many saw the emergence of magazines like Wired and Mondo2000 (shout-out to RUSirius!) as the frontier of new techno-utopia. However, not everywhere has the silicon valley infrastructure. This new world was still under the dominance of SF.

The corrupting influence and domination of SF was exemplified by BALCO - the Bay Area Labratory Co-Operative known for the designer steroids that have altered professional sports irrevocably.

When Chris Anderson wrote "The Long Tail", Jesse says we all recognized it as brilliant. However, it reinforced the hierarchy to allow the few to get all of the best parts, while relegating everyone else to the skinny end of the long tail.

Jesse goes on to attack Chris's latest manifesto "Free". He says there is something fundamentally wrong with his argument, however NOT the free part. Jesse says the fatal flaw is the ethic of waste. Chris says that now that bandwidth is in such abundance, we must waste it, because only then can we reach innovation.

Jesse rails on waste as an ethic in CA (cars, weapons, mining etc). He prefers a revolutionary wholeism. Wholeism is a flip on relativism. Making everything the same, is NOT the answer, according to Jesse. We are in similar time with social tools like as when AOL took over Internet and turned it into total crap.

When you have neighborhoods in the net, you can clean them up. We need to take the best tools available, merge into a coherent vision. Take a page from Barack Obama's playbook and become community activists.

Who can you trust? Not the corporation, only your comrades, which is whoever you have social capital with.

The struggle for human rights never ends, the question is which side are you on? The era of the nation-state is done, it is time for the new rise of the city-state. Get involved.

Jesse had given an intense and fascinating talk. We could not complete the FutureRuby agenda, without some serious rabble-rousing. We surely have to individually take responsibility for what we choose to do with our power as technologists. Agree with Jesse on any individual point, or not, there was a lot of food for further thought.

Aftermath
The conference sessions were now over, but the FutureRuby festivities had not yet ended. After so many ideas compressed into so little time, we needed to hang out and process things together, while allowing it to be unstructured. Meghann had come up with the innovative thought of putting the after-party into 3 different walkable nearby locations: a cool coffeehouse with retro video games, a classic little dive bar with live music, and HackerspaceTO. Not to mention a hilarious street performance that could only happen somewhere open-minded like Toronto.

Jamming on harmonica with a cyborg who played the water organ was just part of my personal awesome experience. Where is that video? So was getting to hang out at HackerspaceTO where they have frickin' laser beams. We timed it poorly, and missed the band dressed in Farscape garb, but there was so much to see and do, right up to the end.

FutureRuby was not just a fun conference. And it was not just a chance to learn about a bunch of new things. It opened me up to new possibilities, and helped re-affirm my personal commitment. I thank all of the staff, volunteers, speakers, and attendees for making it an inspirational experience.

I Have Seen The FutureRuby, And It Is Amazing - Part 1

2009-07-16T17:05:00.000-07:00

It was with tremendous excitement that my brother Damen and I had arrived in Toronto for FutureRuby. Not only were we getting to attend the reprise of what had been by all accounts the "Best. Conference. Ever.", but we were going to be speaking about Project Flying Robot.

There had been many interesting interactions with various security personnel on the journey, thanks to the many small homemade electronic devices that make up our tiny squadron. All of them were extremely friendly and professional as they carefully unpacked, swabbed, scanned, then repacked our cases full of joysticks, Arduinos, electric motors, batteries, and many wires. MANY wires.

By the hour that we arrived, we were too late to attend failCAMP (failed to make it?), but there would be many opportunities to interact with our fellow comrades in Ruby. @peteforde and @meghatron of Unspace had designed the conference with the kind of architectural integrity only a geek could conceive. It was not until the final sessions that the master plan became clear, but I will get to that.

As a result, the next morning we had no post-fail hangovers to slow us down with our last minute assembly and attempts at troubleshooting, combined with walking all over Toronto. Once the evening came, we were eager to connect with our fellows, and happy to climb the stairs to Unspace's cool digs. Pinball machine FTW! And Greg Borenstein's robotic drummer pounding the skins on Pete Forde's drum kit, controlled by Archaeopteryx. It was an excellent party, and they had to kick us out at midnight with the reminder of the talks in the early AM, not to mention the festivities yet to come.

Opening up the first day of the actual conference with the first talk was Nathaniel Talbott with a rabble-rousing speech on "How Capitalism Saves Ruby From Corporatism, or, Owning The Means of Production". This was an immediate shot across the bow of the status quo, and gave us all a clue that the 'collectivist' theme was not just a cool design style for the schwag, but also a serious theme for the conference content.

Next on was Ilya Grigorik with "Lean and Mean Tokyo Cabinet Recipes". If you do not know about it, Tokyo Cabinet is an open source key-value database, that also has server and full-text capabilities. Ilya gave a very hardcore presentation that went all the way into many of the cool things that can be done with TC right now. This was a departure against the traditional SQL way of doing things, and tied in with the revolutionary theme. You HAVE been getting up to speed on one or more non-SQL databases already, haven't you?

The next session was one I was particularly eager to hear. Austin Che spoke about "Programming Life". As in, "Hello World in a petri dish" kind of programming. I had missed the actual workshop, where some lucky people were successful as growing their own glowing bacteria. However, the excellent talk from Austin took us on a wild ride through the current state-of-the-art in biohacking. Let me put it another way: we already have the rough biotech equivalents of both Github, with the Open Bioinformatics Foundation, and Sparkfun with Auston's own Gingko Bioworks. Other sites like biobricks.org and openwetware.org are also there for anyone who wants to get started with this fascinating technology at home.

Following this was Anita Kuno with "Version Control: Blood Brain & Bones" reminding us that the human mechanism needs to be correctly maintained, and developed for correct performance. She had a bunch of specific eating techniques and foods to share, and almost immediately, it seemed that we were more conscious of what nutritional input we were routing into our individual biocomputers.

Next up was one of the best presentations of the entire conference. Foy Savas gave a talk named "polyglots Unite!" which spoke about multi-language programming, and using a takeoff on Rack named Crack to providing a kind of Rack-adapter for other web-backends other than Ruby. It is a neat concept, and I look forward to seeing where it goes. The presentation itself was absolutely fantastic. The timing, the clarity... in a word, he "killed". One of the best speakers of the conference.

Only something pretty different and amazing could follow up that, and Misha Glouberman's "Terrible Noises for Beautiful People" satisfied. It was a laptops closed participatory session that had our entire group singing, clapping, and sushing together. Not only that, but we actually played Conway's game of life using musical interaction with the entire group as the cellular automata. You can't do THAT at home! Absolutely brilliant.

Next were my brother Damen Evans and I with our "Flying Robot" presentation. Despite a few small technical glitches (hardware!) we pulled it off, and the crowd was enthusiastic. We had a great time, and congrats to the winners of the 2 Blimpduino kits @_krispy_ and @maplealmond. There is some cool video here, and lots of great photos like here, and here. Thank you to everyone who participated, we had a great time doing it!

Once we had demonstrated Ruby air superiority over the skies within the Metropolitan Hotel, it was all mobile all the time for the remains of the day. First, a 3-way talk from the guys at Phonegap, followed with a demo by Adam Blum from Rhomobile. I had seen Adam's basic pitch before at LARubyConf, one nice change was that they no longer seem to be trying to charge a per-user license. Per-user license, what's that?? I haven't seem one of those since last century, I think.

Finally, the sessions for the first day were complete. We all put on our finery, and took over Pravda, a Russian-mobster-styled vodka bar that pulled out all the stops, with many people staggering out of the vodka-freezer with smiles on their faces. My personal favorite moment was when we gave a spontaneous group loud "ahhh-clapping-shushing" in response to the wonderful announcement that Shopify was going to pay to keep the bar open longer. There was large quantities of amazing food as well. That @meghatron really knows how to throw a party!

After a pleasant stroll through the streets of Toronto, powered by Russian jet-fuel, we collapsed, to get a few comfortable, if short, hours of rest before FutureRuby Day 2.

Getting Ready For Takeoff At FutureRuby

2009-07-05T09:55:00.000-07:00

I just realized it has been an entire month since my last post. Sorry! In case you were wondering, the always overambitious plans for Project Flying Robot have taken up more time than expected. And parts. Especially parts.

Lucky for us, the benefit of a hard deadline approaches: FutureRuby is coming up next week. My brother Damen Evans and I are going to be showing off our latest works in Unmanned Aerial Vehicles (UAV) based on Ruby Arduino Development (RAD). I don't want to let on too much, so as to eliminate the surprise element, but this should be our biggest spectacle yet.

So if you're going to be in Canada next week, we look forward to seeing you. If not, I'm sure there will be plenty of video to watch in either amazement or amusement, depending on how well we can pull this off...

Project Flying Robot: Supporting The Blimpduino

2009-06-03T21:12:00.000-07:00

As Maker Faire approached, my brother Damen and I were very busy working on something cool: support for the now-shipping Blimpduino kit!

Thanks to the tireless efforts of Chris Anderson and Jordi Muñoz, the long awaited Blimpduino kit is now shipping at Makershed. As readers of this blog know, we have drawn a lot of inspiration from the Blimpduino. Now, we actually have 2 of them, and you can get your own. For less than $100, plus a few other items, and flying_robot software of course, you have everything you need for a complete experimenters kit for Unmanned Aerial Vehicles at home.

There are a few mods you need to make to your Blimpduino, if you want to be cool like us, and control/reprogram it using a linked pair of XBee modems. We will post complete directions soon on how to mod your blimpduino into a flying_robot using Ruby Arduino Development.

In the meantime, you can look at the almost completed flying_robot for Blimpduino code here.

Fear And Loathing At RailsConf 2009

2009-05-17T15:17:00.000-07:00

We were around Barstow on the edge of the desert, when the drugs began to take hold... wait, that was someone else's story. OK, restart.

We were around Barstow on the edge of the desert when the excitement began to take hold... we were on our way to RailsConf 2009! No screaming bats, just loud pumping techno music to power the PT Cruiser. My designer, who was not old enough to be pouring beer on his chest, nor interested in facilitating the tanning process, said "What the hell are you yelling about?". I aimed the Cruiser toward the horizon without slowing down, "I need an In-n-Out milkshake."

Las Vegas... what a place. Putting RailsConf there is the sort of idea that makes sense on paper, but could turn a previously mild-mannered group of Ruby programmers into a mob of raging lunatics. Come to think of it, a group of Ruby programmers IS a mob of raging lunatics. Case in point? Video slot machines... the worst odds in vegas, but the best graphics. How will a group of perpetually partially attentive people be able to resist the siren call of millions of sensory distractions each designed to exert psychological pressure to LOOK AT ME? Seems like an interesting Milgram-like experiment.

The back of the PT Cruiser was full of musical gear for the RailsConf music jam. With a small but effective PA and a few spare guitars, this session should be the best one yet. Could we play Vegas? Without offending the locals, or running afoul of some Musician's Union enforcers, that is.

I had meant to keep meticulous notes, and post a flurry of blog entries as I have done as RailsConf's past. But the dull fog of Vegas combined with the mad dog sentiments already awakened in the Rails community at GoGaRuCo, left me with the sure knowledge that no matter how hard I might try to offend the insiders, no one would even notice with the continuous drunken flame wars that RailsConf Vegas would become quickly known for throughout the Twitterverse.

The madness had taken hold long before we hit Vegas, and adding alcohol and neon fueled hyperstimulation only had the effect of pushing us into a raging frenzy. "Tim Ferris? How DARE he tell ME to exercise. Bob Martin? How dare he accuse me of not testing? Everyone else? How dare they dare to dare, or else how dare they not dare to dare! Forgeddaboutit!"

Just then my already tenuous sanity began teetering, and I started yelling that the White Rabbit and I had been pair programming together for years. The wild-eyed activist within me leaped into action, and I practically elbowed people out of my way to get to the mic, to ask Uncle Bob the burning question on my mind: "What happened to the social revolution you started with Kent Beck and Ward Cunningham?"

From the look in his eyes, I know the question haunted him, just as it still haunts me. If this is the utopia, why are we all fighting so much? "I saw the best minds of my generation destroyed by madness, starving hysterical naked..." and still Twittering away trying to validate justify, explain, strengthen, while simultaneously eroding it, tearing it away.

I had to escape, find a place to hide and collect my shattered illusions. Fortunately, the safety zone of CabooseConf greeted me. The comfort and sanity of watching my programming buddies hacking together an LLVM implementation for AVR was like slipping under a warm, soft blanket, after the frenzy that had started while I was sitting in the Reptile Room, watching some giant lizards get ready to feast on fresh ideas.

Days had passed, but in the strange netherworld between Vegas's clockless existence, and the constant Twitter flow of new input, I had lost all sense of temporality. It was a surprise that we had already come to the final keynote aka Q&A session. It was an odd demonstration of our shared exhaustion and sensory overload, that pretty much no one wanted to ask any questions.

"Time to get out of here!" I said to my designer. We piled the PT Cruiser full of our gear, plying the staff with dollar bills like we were mythical high-rollers. I drove like the wind, but it was not quickly enough. Leaving behind a cloud of gritty, baked dust, we fled from a man-made 24-hour spectacle that even Dante could have never imagined, even if he had taken all the drugs available to an Italian in the 14th century at the same time.

With apologies to, and in memory of HST, we need his free spirit now more than ever

Heroku Has Launched

2009-04-29T09:28:00.000-07:00

Well, just a very short time after I started using Heroku, they went commercial. Yes, after their very successful beta period, where apparently 24,999 web sites other than mine were already hosting, Heroku is now offering a paid version of their service. I had a sneak peak at the pricing a few days ahead of time, but I was not able to talk. And despite my intentions of blogging this right away, the other demands on my time have kept me occupied till just now.

Heroku is mimicking the successful "freemium" pricing plan of other services, but brings it into the Ruby web application hosting space, within some pretty generous limitations. Yes, exactly. Heroku still allows you to get started with their service at no charge at all. Wow. I do not know of any free web hosting service that does not at minimum plaster your site with hideous ads. Let alone quality Ruby hosting. Let alone Ruby powered cloud computing.

As your traffic needs increase, or database storage needs, they have a variety of pricing tiers. Thanks to a slick AJAXified pricing tool, the complexity of so many pricing options is somewhat mitigated. Plus it's fun to play with.

The evolution of most startups seems to track really well with Heroku's overall business strategy: as a customer becomes larger and more successful, their increased traffic and database needs will cause them to start paying Heroku. If your venture does not really go anywhere, it is not really taking up much in the way of resources anyhow. This aligns Heroku nicely with the needs of their customers, instead of pitting them against them trying to extract revenues too early in the growth curve.

Despite a few growing pains, they have had pretty decent uptime on my app so far. Even with my new app Thumbfight getting a few sudden traffic bursts, as well as having a major reliance on Twitter for back-end processing (more about Thumbfight in an upcoming post).

Heroku is a work in progress, but so is most everything else on the entire Internet. Heroku provides an amazingly easy and insanely cheap way to jumpstart your Ruby-based web application hosting and deployment, while still getting some real expertise. As long as you can work within their current technical limitations, for a Ruby-powered startup, I think Heroku is a great way to go.

Project Flying Robot: Getting RAD With The ATMega328

2009-04-18T15:08:00.000-07:00

I have been wanting to upgrade the hardware used in our Dorkboards for flying_robot, from the ATMega168, to the newer better faster ATMega328. More memory, and a faster UART for serial communications with the XBee modems in the same pinout = easy win. Thanks to a quick shipping turnaround from @adafruit I got them in before the weekend, so I could play a little bit today.

The first step was to upgrade my hard-working Arduino Diecimila to a 328. I now have it working great with Ruby Arduino Development (RAD), but since RAD was really setup for Arduino 12, I had to make a couple changes. Here is what I did:

1. D/l and install Arduino 15 (brave, I know, since that is the latest release, and many people run one version down from the latest)
2. Change my hardware.yml entry

mcu: atmega328p

3. Change my software.yml entry

arduino_root: /Applications/arduino-0015

4. Lastly, since the ATMega328 bootloader runs at a faster rate, I had to tweak the RAD code itself to support it. The file "/vendors/rad/generators/makefile/makefile.erb" is the template used to create the makefile that compiles and uploads the code to the Arduino. Line 77 in that file controls the baud rate, which needs to be set like this for the '328:

UPLOAD_RATE = 57600

Once I had done this, I was easily and quickly able to recompile/re-upload the latest flying_robot code to my test board. Yeah! Hopefully tomorrow I can upgrade Rogue 1 and try a flight at the new, higher communication speed.

Heroku, Why Haven't I Been Using You Till Now?

2009-04-14T11:23:00.000-07:00

Last night, I finally got around to deploying something on Heroku, an interesting service founded by my formerly LA-based Ruby programming chums Adam Wiggins, James Lindenbaum, and Orion Henry. I had played with their previous incarnation of the service, now known as "Heroku Garden" but only recently have I gotten to know a little bit more about the incredible offering they have evolved into.

Basically, the Heroku crew have addressed the question "how can I deploy my Ruby on Rails, Sinatra, or other Rack-based web application into a dynamic cloud of servers with ridiculous ease?" They have done this with an ingenious architecture that takes advantage of Amazon's EC2 to provide their internal infrastructure. This allows Heroku to concentrate on their most important core value proposition, of a simple way to take your Ruby code and just push it into the cloud.

Notice I said "push". Heroku requires that you use git for source control of your application. You are using git for everything now, right? If not, git with it! Sorry, could not resist that. Anyhow, by simply adding a remote master to your existing git repo that points to Heroku, along with a few Ruby gems that they provide, you can deploy your app just by pushing your current branch to the Heroku master.

Doing this, causes your app to get packaged up into "slug". Once you have an active slug, it will be be deployed to a "dyno" within the Heruku grid, which is what a virtual node within their architecture is called. As your app requires more resources, the slug can be deployed to more dynos within "less than 2 seconds for most apps". That is way faster than starting up a new Amazon EC2 instance yourself, and having this extra layer has a number of other interesting benefits as well.

Heroku has a quick start guide, which pretty much runs down what you need to do. I had found a slightly more simplified quickstart here. I already had an existing Sinatra-based app that I wanted to test on Heroku, so here were my steps:

1. Install heroku gem

sudo gem install heroku

2. Setup Heroku account info, and upload public key

heroku keys:add

This will prompt you for your Heroku account info. If you have not created one yet, better jump over to http://heroku.com/signup and create one

3. Create Heroku app from my existing app
I just changed so my current directory was the app I wanted to add to Heroku, then entered:

heroku create myappname

This creates the new app on Heroku, and creates a remote branch so you can deploy just by pushing the code.

4. Deploy my code

git push heroku master

That's it! If you have a really simple app, with no database access, you are done. What, you are deploying a Ruby on Rails app and need a database setup? OK, then...

5. Run database migrations

heroku rake db:migrate

NOW, you are fully deployed and running on Heroku. Unless you are not. I still had a minor problem with my app. I was writing my log file into "logs/production.log" but Heroku does not normally allow write access to disk. The two exceptions to this are the "tmp" directory and "log" directory (notice singular). They do provide an easy way to view your most recent log entries, by typing

heroku logs

which is how I figured out my problem with the log directory.

So, here is my total time required to deploy this app on Heroku:
- Reading quickstart = 3 minutes
- Installing gem and entering account info = 2 minutes
- Making my app a Heroku app = 1 minute
- Deploying my app for the first time to Heroku = 2 minutes
- Figuring out what I had done wrong from the Heroku documentation = 10 minutes
TOTAL = 18 minutes

Here were my bonus steps:
- Reading Heroku docs on using a custom domain with Heroku = 1 minute
- Realize I need to rename my app using Heroku command line = 1 minute
- Rename my app using Heroku command line = 1 minute
- Setting my DNS settings to point to Heroku = 5 minutes
- Telling Heroku about my custom domain = 1 minute
TOTAL = 9 minutes

So there you have it... a fully deployed app, living in the Heroku grid and consequently the Amazon EC2 cloud, in less than 30 minutes, having never used their tools before, including troubleshooting a minor configuration problem. That may seem unfair... and it is. That is exactly the kind of unfair I like on my side!

Much credit should go to the Heroku team for creating something extremely cool and functional. Important details are still not available, like pricing etc., but at least for now Heroku, is a great way to easily get your app up into the cloud within literally minutes.

What is the future for Heroku? Funded by Y Combinator, they have been quietly working away, and now with Sinatra team leads Blake Mizerany and Ryan Tomayko onboard as well, I think we will be hearing a lot from this exciting little company.

LARubyConf 2009 - Jim Weirich - "The Grand Unified Thoery of Software Development

2009-04-08T10:12:00.000-07:00

As the 2009 Los Angeles Ruby Conference (LARubyConf) drew to a close, our keynote speaker Jim Weirich took the podium. I have seen Jim speak several times, and he is both intelligent, as well as down to earth, which is a rare combination indeed.

The subject of his keynote would be anything but down to earth. In fact, it would have to be one of the most ambitious talks I have ever seen at a Ruby conference. Only Jim could have pulled it off as he did, with both humor and insight.

One thing Jim does is have to conduct Tech Interviews. One question he always asks is "What do you look for in a good design?" Most people's answer: "UMMMMMM..."

Then Jim seemingly shifted themes abruptly, to physics. Specifically, subatomic particles.

It is known that particles that are charged the same repulse each other. Furthermore, every time you change electric field, there is a changing magnetic field at 90 degree angle.

James Clerk Maxwell
Maxwell discovered 4 equations that describe relation between electrical and magnetic fields. By describing these two entirely separate forces, combines into single force. Maxwell's work probably greatest contribution to science.

Then, along came Ernest Rutherford's famous electron experiment. The one where electrons were supposed to evenly deflect onto a screen, but instead occasionally would deflect wildly. As a result, we now know that matter was mostly open space.

There are four known forces
- electromagnetic
- gravity
- strong nuclear
- weak nuclear

The search for the "Unified Field Theory" in physics is a search for a single explanation that accommodates all four forces.

This is very much like the search for a single explanation for software design.

Some Commonly Accepted Software Design Principles
- SOLID
- Law of Demeter
- DRY
- Small Methods
- Design by Contract

Lots of ideas about how to write software, but no grand unified theory.

"The Grand Unified Theory of Software Development"

Composite/Structured Design
- Glenford J Meyers - 1978

Coupling & Cohesion - from best to worst
- no coupling
- data coupling - local data, simple
- stamp coupling - local data, structured
- control coupling
- external coupling - global data, simple
- common coupling - global data, structured
- content coupling - when you reach inside of modules and mess with them from outside

control coupling
- method has flag parameter
- flag control which algorithm to use

Symptoms
- word OR in description

Example:

Array.instance_methods(true)

and

Array.instance_methods(false)

Which one lists only private methods?

Another example, Rails does this:

Customer.find(:all)

vs.

Customer.find(:first)

Myers' classification were OK, however failed to extend well to objects and dynamic languages

Meilir Page-Jones's book "What Every Programmer Should Know About Object-Oriented Design" has 3 sections, two of which are not too useful, but the third is very interesting. It talks about the idea of Connascence in software design.

Connascence - when two things are born and grow up together
Two pieces of code share Connascence when a change in one module requires a corresponding change in the other.

CoN - Connascence of Name
- when code linked by name
- can also apply to databases
- class name is NOT, but parameters are

Locality Matters
- if you have things grouped together, there is stronger connascence.
- as dist increase, you reduce connect between them

Connescence of Position
- when the order of the params matters

Low/high degree of CoP

When you encounter CoP it is better to transform it to CoN

Degree Matters

CoP in test data example


  User.find(:first)

CoM - Connescence of Meaning
- when two bits of code have to agree on the meaning of data

When you encounter CoM it is better to transform it to CoN

Contranesence is when things have to change opposite to each other
- for example, collision of class names in two different modules
- solution is to use namespaces

Another Example?
- do not monkeypatch unless you have to, and if so use namespaces

Connascence of Algorithm
- Two methods that do different things, but that are bound together by algorithm. For example, two different bits of code in two different languages that have to talk to each other. JavaScript client, Ruby server is good example.

CoA -> CoN
- also known as DRY

CoT - Connascence of Timing
- for example, a race condition

Summary
- Connascence is the 'quark' of software design
- Not really any tools to analyze code
- Seems like there is a relation between connascence and design patterns

Wow! Jim had taken us all the way from subatomic particles, to a start towards a unified theory of software design, and tied it all together nicely. And made it fun! It was a tremendous cap on an excellent conference, and we all had really appreciated Jim's contribution.

LARubyConf 2009 - Blake Mizerany - "Sinatra: the Ultimate Rack Citizen"

2009-04-07T12:19:00.000-07:00

I was very happy when the next presenter at the Los Angeles Ruby Conference 2009 (LARubyConf) was Blake Mizerany, creator of the very cool Ruby micro-framework Sinatra. As long-time readers of this blog know, I am very into Sinatra.

There has been an incredible amount of work going into Sinatra lately, so I was very interested to catch up on what the team has been up to.

What is Sinatra? A Ruby Domain Specific Language (DSL) Mapping REST to simple actions

Why?
- small
- fast
- great rack and ruby citizen
- strong focus on HTTP
- HTTP caching helpers built in before it was cool
- content negotiation
- no boilerplate
- dead simple config when the default are not enough
- smart configuration
- DOCS- sinatrarb.com
- extending is easy
- rack is the only dependency
- very low WTF to LOC ratio (jeremy mcnally's rubyfringe talk)

when?
- a few controllers models and views
- starting any web application
- you need reusable apps and/or middleware and/or resources
- you need speed

who?
- heroku
- github
- taps
- integrity

sinatra in your gems
- a mini-github for offline repo browsing
- a local plugin and play wiki
- memcached utilization graphs
- config reusable github hook

Example: NotTwitter

As classic Sinatra


set :username,
  Proc.new { fail "yo"}
  
get '/' do
  ...
end

Change

  require 'sinatra'

  require 'sinatra/base'

But I want to deploy to Passenger or Heroku! No problem.

./bin/install-not-twitter
Copy example config.ru to cwd
Copy .gems file

.ru is a standard Rack config file.

.gems is a Heruku configuaration file that will handle any needed Ruby gems installations


  git init && git add .
  git commit
  heroku craete
  git push heku  aster
  heroku

3 Awesome Features in Sinatra
pass - I cannot handle this request, try the next route
forward - sinatra as middle ware... done my job, let the next app take over... pop in front of rails metal
use - Sinatra loves rack so much, we made sure not to hide it

Resources
http://sinatrarb.com
http://github.com/rack/rack
http://github.com/rack/rack-contrib
http://github.com/rtomayko/rack-cache

If you are doing everything with Rails, you are probably using too much for the job. Sinatra is simple, fast, and extensible. I am using it in two production applications right now, along with Rails. Sinatra handles parts of the application better than how Rails does, so that is how I roll.

Especially with the ever increasing momentum behind Rack, Sinatra is a good bet for getting things done. Combined with Rails Metal, and you really have it all.

LARubyConf 2009 - Danny Blitz - "Herding Tigers: Software Development and the Art of War"

2009-04-05T20:51:00.000-07:00

I had no idea what I was about to experience at Los Angeles Ruby Conference 2009 (LARubyConf) when Danny Blitz took the podium as the next presenter. I had seen him hanging out with his distinctive pompadour, tattoos, and leather jacket. He is a big guy, and hard to miss. But he had been pretty quiet till then, which was about to change radically.

Herding Cats is a term commonly used when describing the management of software teams. But when Danny Blitz says cats, he means big cats aka tigers. So who is this guy? He has done TONS of stuff, from DOD to Dell, to the DARPA autonomous vehicle challenge. Very cool stuff.

Get Agility?

"A good plan violently executed now is better than a perfect plan executed next week" - Patton

Q. Why is software so difficult?
A. We don't want to face the truth

Q. Why don't we want to face the truth?
A. We're afraid

Q. What are we afraid of?
A. We're afraid we do not know anything about end result

Q. How can we deal with this?
A. Admit the truth

A tiger team is a small self improving team.

What can you expect from a tiger team?
Their first project was scheduled to take 5 weeks - took 5 days

TIGERS ARE A TEAM!

QA is part of tiger team

Tigers show leadership

Tigers self-improve

Almost no meetings on a tiger team

QA and test automation are the tip of the spear

QA should be there from very beginning

Shared psychology and intelligence

Winning, boldness, excellence

Not afraid of the dark

Who is on team?
- all staff needed to deliver product
- leader, 4 devs, 1 automation engineer, 1 Q, product staff m,ember
- in addition, architecture, system admins, any other support staff

why warfare?
- business is battle

There are two kinds of warfare: attrition and maneuver

Attrition warefare
- traditional, tactical
- clashing head-on

Maneuver warfare
- internet space
- rapid modern violent
- unexpected movements

Speed
- it is a competitive weapon
- undeniable advantage in business

Speed mitigates risk
- not a guarantee
- damage is contained by quickly compensating

Speed improves the team

Speed adds to job satisfaction

Speed allows agile to function properly
- max iteration length (usually 30 days)

Speed builds credibility
- shows a lot of work in short order

Cows and tigers
- cow is bigger, but who wins?

Disease: using agile terms to describe non-agile project

Corporate animal kingdom
- Tiger
cautious, calculating, looks to win

- Cow
herding
not known to be original
afraid of risk

- Bear
big, usually mellow
awesome battle skills
live and let live attitude

- Leopard
truly wild
will attack at any time

- Elephant
huge and tough
invincible
best to avoid battle

- Hyena
scavenger
steals food
evil

"Tiger teams are like Hell's Programmers" - Danny Blitz

Leadership
this is what makes or breaks
faith
love
hope
success belongs to the team
failure belongs to the leader
buck stops here
fearlessness
listener and learner
protector
outside influences
internally too
team members themselves

US marine management techniques
- manage by end state and intent
- reward failure
- demand to be questoned
- glorify the lower levels of organization

Politeness and professionalism
- that or poison

Agile is not a methodology, it is a mindset, it is inevitable

Danny says he is working on the book called "Herding Tigers". He has also started a blog at herdingtigers.com. All I can say is, he is a very dynamic and exciting speaker. Everyone was captivated, myself included. I'm still not sure how I took these notes.

Rock on, Danny!

LARubyConf 2009 - Jeremy Evans - "Sequel"

2009-04-05T20:22:00.000-07:00

The next session at the 2009 Los Angeles Ruby Conference (LARubyConf) was Jeremy Evans presenting Sequel, which is a very powerful database toolkit for Ruby.

Ruby originally had adapters for each database. The problem was that each was very database specific. This was a problem due to both SQL differences, as well as API differences.

no behavior

1997 - ruby-postgres was first created.

2000 - DBI

2004 - Active Record
Although AR made things easier, it had strong opinions. These opinions did not always map perfectly to any particular database.

2007 - Sequel

Completely DB independent API.

Example is concatanating strings, which requires a completely different syntax in each flavor of SQL database.

Concise

Optional Behavior

Opinions?

Ruby should be like clay in a child's hands - Matz

Sequel advantages:
- Simple - as possible but no simpler
- Flexible - opinions, not dogma
- Powerful
- Lightweight - 1/2 memory usage as active record
- Well maintained
- Easy to Contribute
- Easy to Understand
- More Fun

Show me the *** code!


require 'seq'
DB = Sequal.sqlite('larun')
DB[:attendees].count
# => 1

DB[:attendees].first
# => {:name ='', :address =>}

Transactions


DB.trans do
  DB[:entry],insert(:account_id => 1)

  ...

end

SQL Loggers


DB.loggers << Logger.new($stdout)

Each DB has its own connection pool


DB[:table].all # or .each

.update(:column => 'value)

.delete

DB[:table] # this is a dataset... like query cursor

Sequel is a functional API, where object methods return copies of themselves.

.select - while columns
.filter
.order

This means you can easily chain function calls, like jQuery. Awesome!

Sequel represents its internal objects using SQL's own internal representations

Sequel has 'core' and 'model'


class Attendee < Sequel::Model
  many_to_one
  one_to_many
  
  
end

Hooks & Validations - got 'em

Sequel it built entirely out of plugins. That sounds interesting, but experience with DataMapper has shown me that too many plugins may not be a good thing. However, I do not have any direct experience with Sequel yet, so this may be a non-issue.

13 - # of database adapters that Sequel supports today

Database graphing

Everything was going so well up to this point. However, Jeremy then tried to do his demo on Windows, just to show that if you are one of the poor souls using Windows, it can still work for you. He failed to account for the quirkiness of conference display adapters, and what that can do to a machine. Oh well, no demo.

That said, I was really impressed by what I heard about Sequel. I think I will have to try it out on something, just to see how it does.

LARubyConf 2009 - Aaron Patterson - "Journey Thru A Pointy Forest, or You Suck At XML"

2009-04-05T08:49:00.000-07:00

The second presentation of the day at Los Angeles Ruby Conference 2009 (LARubyConf) was Aaron Patterson, XML maniac. I had a run at hardcore XML/XSL a few years back, and it has been a while since I walked the razor's edge of the angle bracket. But Aaron is not just an aficionado, he is genuinely obsessed. Given that he is the maintainer of the Nokogiri and Mechanize gems, this makes me happy.

He covered four areas related to XML:
- XML Processing
- HTML Processing
- Data Extraction
- HTML Correction

The XML Processing section was as thorough a synopsis of XML as one could possible fit into just a few minutes. He went over the four main XML processing styles:
- SAX
- Push
- Pull
- DOM

SAX parsers are fast, and low on memory use. However, searching is hard, doc handlers are verbose, and programmer expense high.

Push Parsing works the same as SAX, the difference is the programmer controls document IO. Has low memory use, fast, fine control over IO.

Pull Parsers are handed XML, and yield a node object. They work like cursors, moving thru the document, so you only get one chance to process data without starting over from the beginning again.

DOM Interface is what most programmers are familiar with. Given XML, they build an in-memory tree. They can then be easily searched using XPath. DOM parsers provide easy data extraction, are programmer friendly, but have high memory use, and you pay a serious speed penalty.

HTML Processing is just like XML parsing, but it limited to the HTML DOM.

Data Extraction in XML can be done two different ways:
- CSS selectors
- XPath queries

XPath basics:


  //foo <!-- start at absolute root -->
  .//foo <!-- start a relative root -->
  //foo[@bar] <!-- has bar attribute -->
  //foo[@bar = 'baz'] <!-- has bar attribute with a value of 'baz' -->

Here is an example of a problem for parsers that support either CSS and XPath selector as parameter, like Hpricot:


p[b]

The problem is that it is both valid XPath AND CSS. Nokugiri has separate methods for searching CSS or Xpath, so as to avoid this problem.

XML Namespaces use URL's to remain globally unique, to avoid collisions between XML formats taht are different, but use the same node name. Here is an important point: XML Namespaces are as important as tag names.

HTML Correction is taking some invalid HTML, and "fixing" it. For example, making sure that all tags are properly nested.

Aaron wrote a tool called tree_diff, that compares XML trees, cause "they are interesting". With this tool he was able to process many HTML files with multiple HTML correctors/parsers, then compare the results to see if they matched. In many cases, they did not!

Fonts are hard
Attributes are harder

29% of the time it works all of the time

Seems like after all that, I will never use anything but Nokogiri for XML parsing again!

LARubyConf 2009 - Dan Yoder - "Resource Oriented Architectures and Why It Matters"

2009-04-05T08:35:00.000-07:00

Lead-off man Dan Yoder started off the day's proceedings at the Los Angeles Ruby Conference (LARubyConf) 2009, with a presentation on Ruby Waves called "Resource Oriented Architectures and Why It Matters". Despite not getting the same attention that some Ruby frameworks have, the Waves team has been tirelessly working on it. According to Dan, the foundation of Waves has gotten pretty solid. Waves adds a lot of support for things over and above just handling http requests. So what is Waves really? It is a layer on top of Rack for defining application frameworks.

The next thing beyond MVC is to help developers write more rest compliant apps.

But do the constraints in REST really buy anything? Actually, yes.

One nice thing about internet-based development, is that the existing infrastructure is already there like proxies, load balancers etc.

But why does it work? At the heart are the constraints.

The web is NOT MVC
- so why do we use it so often for web apps
- piggybacking off of the web browser

Example of busting out of the browser - RSS feeds from blogs and podcasts

OAuth SMART Proxies

Video Search
- edge caching

Resource Oriented Architecture (ROA) is just distributed objects, loosely based on Roy Fielding's definition

ROA solves an old problem, that there have been many attempts at solving previously with CORBA, COM, etc. But this time will be different.

Learning From Past Mistakes
- be platform neutral
- be wire neutral (any protocol)
- define meta-object protocols
- good performance, use edge and client caching
- allow layered architectures

Waves and ROA

Rich DSL for HTTP Requests


on (:get, ['location'],
  :query => {:lat => /\d{4}/, :long => /\d{4}/},
  :accept => [:json, :xml])

But it is still Ruby!

The One File Waves App
- influenced by Sinatra
- not quite as clean as Sinatra

Roadmap

A Resource DSL Example


class Blog
  include Waves::Resource::Server
  
  resource :list, :expires => 3.days, ['blogs'] do
    get { model.find_all}
  end
  
  ...
  
  schema :element, ['schema', 'blog', '2009-03'] do
    attributes :title, String, :descriptions => String
    link :entries, :list => Story
  end
  
  ...
end

What's with that "schema" block? This example also defines the RDF schema for the resource to provide machine-discoverability. That is one very cool aspect of ROA that I personally have not seen addressed much within the Ruby community.

Waves is really coming along, and I am planning to explore it a bit in the coming weeks. For more info, go check out http://rubywaves.com

LARubyConf 2009 Was A Big Success

2009-04-05T07:48:00.001-07:00

Yesterday was the first ever Los Angeles Ruby Conference. Despite being located a bit out of the way, down in Tustin, the venue was actually completely amazing. Called the Marconi Automotive Museum, the place was like "Robin Master's garage" quipped one attendee, being that it was FULL of Ferrari's and other highly valuable and hot sports cars.

Thanks to the tireless efforts of Coby Randquist and JR Fent, along with a host of others, this first conference was a tremendous success. There was an incredible lineup of speakers, with something for everyone. All of us who attended were really impressed by the quality of the event. There was even one guy who came all the way from Boston (this is great time of year to visit California from the east coast, people). After a short introduction, the program was off and running.

The sessions were:

Dan Yoder - "Resource Oriented Architectures and Why It Matters"
Aaron Patterson - "Journey Thru A Pointy Forest, or You Suck At XML"
Jeremy Evans - "Sequel"
Danny Blitz - "Herding Tigers - Software Development and the Art of War"
Wolfram Arnold - "Scaling 'most popular' lists: a plugin solution"
Blake Mizerany - "Sinatra: The Ultimate Rack Citizen"
Michael Fairchild & Ari Lerner - "Poolparty.rb"
Bill Lapcevic - "Managing Ruby on Rails for High Performance"
Ron Evans & Damen Evans - "Flying Robot: Unmanned Aerial Vehicles using Ruby and Arduino"
John Barnette & Aaron Patterson - "Johnson"
Pradeep Elankumaran - "Fast and Scalable Front/Back-end Services using Ruby, Rails, and XMPP"
Brendan Lim - "Mobilize Your Rails Application"
Adam Blum - "Rhodes Framework for Mobile Client Development"
Jim Weirich - Keynote speech - "The Grand Unified Theory of Software Development"

I took copious notes, and so I will be posting separately for each session, to avoid a massive blob post. The sessions were quite excellent, our demo of flying_robot was really well received, and Jim Weirich's keynote was by turns funny and thought provoking.

Thank you to everyone who participated by organizing, presenting, and/or attending LARubyConf. The 110 or so people there really appreciated it! I had a great time, learned a lot, met some great new people, and got to see some old friends. I really look forward to the next one!

The Twitter 1-2-3 Rule

2009-03-31T12:28:00.000-07:00

I just had a small realization: there is a perfect ratio for Twitter use. Wait, before you laugh and hit the back button, hear me out on the following idea.

There is an emergent trend in optimal Twitter use:


If you have X followers,
You should have 2X people that YOU follow
And have made 3X updates

Get it? Twitter 1-2-3. Nice simple rule...

Here is what is means if these ratios are out of whack:

Less then 2X people that you follow = who do you think you are, Shaq?

More than 2X people that you follow = perhaps you should do something besides Twitter?

Less then 3X updates = why are so many people following you?

More than 3x updates = you really like to listen to yourself, don't you?

Let's Celebrate The Female Side Of Tech

2009-03-24T09:59:00.000-07:00

It is Ada Lovelace Day today, and to support the women in technology, bloggers all over the world are celebrating the female side of tech. What do I mean? You may not know it from the throngs of hairy guys at tech events, but not only are there women in tech, they have done a lot to make YOUR life easier. Let's review a couple current ladies of technology that have been rocking my world.

First of all, no mention of modern geek tech can even begin without another Ladyada, aka Limor Fried. As the proprietor of Adafruit, she has done a lot to make physical computing with Arduino and other cool hardware a lot more accessible for novices, as well as helping out pros when they are stuck. As as engineer and MIT alumni, she is a great leader in the electronics community. We could not have made the progress we have on the flying_robot project, without her great hardware and helpful blogging.

Another female developer I know is Andrea O.K. Wright. Andrea is a very serious Rubyist, and did a really great talk on Ruby in gaming that I attended a while back. She was even kind enough to remember to send me a couple links to some of the tech she had mentioned in the presentation.

We smelly guys have to be be a bit more welcoming to the women of tech. Why? Not even for any of the standard reasons related to social progressivism. Simply enough, we all need all the help we can get, and not encouraging women to explore their interests in technology, is like losing 50% of the potential advancement we could be collectively making.

'Nuff Said!

Project Flying Robot: I Believe I Can Fly

2009-03-21T13:47:00.000-07:00

At long last, after many obstacles to be surmounted, yesterday we achieved liftoff. The blimp prototype "Rogue 1" of our flying_robot project to create a platform for Ruby Arduino Development powered Unmanned Aerial Vehicles has achieved an important milestone.

After our third complete rewiring of the main circuit board, two burned out Pololu micro serial motor controllers, some time reviewing the schematics of the Blimpduino project, and too many revisions to the basic software to list here, we had finally gotten everything ready for what would hopefully be a successful flight test.

There were some major limitations to this test, which was mostly intended to verify proper functioning and integration of the hardware. Happening inside my brother's small apartment, we would not be able to really do much flying. Also, we were going to measure our maximum payload capacity, but not really planning on balancing the blimp to achieve a proper neutral buoyancy.

The final assembly took place, we did a quick flight check, and you can see for yourself the results in this poorly shot and edited, but somehow still amusing video.

Now, this thing is really starting to take off!

Come Fly With Us At Los Angeles RubyConf 2009

2009-03-17T15:52:00.000-07:00

Yes, the rumors are true, our talk for LARubyConf was accepted. If you want to see the first ever public live demonstration of flying_robot, the real thing, actually flying around and such, come check it out live and direct at the first Los Angeles Ruby Conference on April 4.

I do yet not know exactly what our demo will include, but from what we have been doing so far in the lab, you won't want to miss this one. Unless what you get to witness is massive fail. Either way, it will be spectacular.

Not to mention, there are some really excellent presentations happening. Actually useful stuff you might put into daily practice. We're talking shining beacons of the Ruby community like Jim Weirich, Jeremy Evans, Aaron Patterson, Daniel Philpott, Dan Yoder, my boys Ari Lerner & Michael Fairchild, and many, many more. So if you have not registered yet, get with it!

Click here RIGHT NOW and register for LARubyConf. Do it!

Project Flying Robot: Better To Burn Out Than Fade Away?

2009-03-15T17:57:00.000-07:00

We had made incredible strides forward in our construction of "Rogue 1" our autonomous blimp that we are building as the first implementation of flying_robot, our Arduino-framework for Unmanned Aerial Vehicles based on top of Ruby Arduino Development (RAD). We are also heavily influenced by the cool Blimpduino project. We were finally ready for a live test... or so we thought. We inflated our envelope, and my brother attached the gondola that he has been carefully constructing.

We had already done some testing of the software/hardware, so we were heady with our previous successes. "Let 'er rip," said Damen. Next thing you know, the blimp was careening about, as the motors pulsed wildly out of control, then stopped abruptly. "That is not a good sign from an electric motor controller, " said my brother glumly.

We peeled the gondola off the envelope, and disassembled the components that had been connected so carefully, and yet not carefully enough. Multimeter in hand, Damen carefully traced each connection looking for power as I entered the commands into the serial interface.

Finally, after a few moments we looked at each other. There was one more test... we rewired everything properly, but using the spare Pololu Micro Dual Serial Controller and a breadboard. I entered the commands and the motors powered up. Now there was only one conclusion: we had fried our costly little Pololu.

Ouch! After spending a while on post-mortem analysis of our dead little bird, Damen muttered something about power. "Huh?" I looked up from debugging the Arduino code for the differential thrust calculations.

"Power. We need less power."

"Less power? I thought the specs said that controller was rated for more than our LiPo battery and more?"

"Specs lie."

I had to acknowledge the simple truth of that statement, especially given the obviously dead chip. With a lot more knowledge about our bimps power requirements, learned the hard way, Damen began planning the third iteration of our main board wiring, with a new extra power regulator. And an external kill switch.

Project Flying Robot: By Your Command

2009-03-10T13:52:00.000-07:00

Our flying_robot project for Unmanned Aerial Vehicles has been underway for a few weeks now. Much soldering, and playing around with cool Arduino tricks has been taking place. You can read more here, here, here, and here if you need to catch up.

One aspect that I have mentioned several times, but not gotten into much detail about, is our protocol for communicating with the UAV from the ground station. It is amazing to me, but there is still no dominant digital protocol for controlling UAV's. Many people are this using analog radio controllers, with some kind of sideband channel for autopilot commands.

There are a few characteristics that seem essential to me for a proper ground control system design:

A nice way to send control stick commands from a "virtual r/c" that can be used with any UAV

Consistent way to send autopilot commands

A way to be able to read the instruments remotely, and display them as part of your "control surface"

Another observation I have, is that there is every current Arduino UAV project has an entirely different code base. There also does not seem to be any practical way for these projects to share much, even though they are all targeting the same microcontroller.

Certain characteristics seem essential for developing on-board UAV system programming, but are lacking in any of the projects I was able to find:

A nice, modular way to put together the pieces you need in software that match the hardware configuration for that particular vehicle

A high-level domain specific language (DSL) to specify the UAV's behavior, and not just programming in C++ or assembly language

A consistent programming interface that corresponds to the protocol used by various ground controllers

With all of these design criteria in mind, I have been implementing the flying_robot command parser and interface. Here is an example of the minimum code required to create a UAV using the current implementation:

As you can tell, there is not all that much to it. The base for a UAV in 34 lines of code.

The most current code for the actual "Rogue 1" LTA vehicle is being committed to http://github.com/deadprogrammer/flying_robot_rogue_one/tree/master if you want to follow along with our progress.

Next post, we test the main thrusters and vectoring controllers, and nearly start a fire...

Playing With Hoops, or Every Step You Take

2009-03-06T11:35:00.000-08:00

I do not often pay full attention to Giles Bowkett. That is the flip side of being the provocateur that he is, some people will occasionally tune out. However, when anyone plays the Steve Krug card, I am compelled to take a look.

Yesterday, Giles posted a dead-on point about user registration, called "I Don't Care About Your Hoops". When you compare the success of simplicity, as in the Google and Apple examples, to what we all tend to fall into, I hope you are as embarrassed as I was. Time for some self-examination.

I was ready to hack down the user signup process to size. Here are my new rules:

4 or more registration steps = FAIL

3 registration steps = A long shot, but you might make it

2 registration steps = At the top of the key, worth taking a shot, but no sure thing

1 registration step = A free throw, you should easily sink it

0 registration steps = Slamdunk ala Michael Jordan. Ain't no stopping us now.

Project Flying Robot: The Flying Dorkboard

2009-02-28T08:53:00.000-08:00

As we proceeded along with our flying_robot project, one thing became abundantly clear: weight matters. The Arduino Diecimilla that we had been using up to this point, although a fine package for beginners, was just not going to make it when it came time to weigh in. Yes, there is the Arduino Nano, but it is not a cheap device. Also, it has a few options that were not important to us, like the integrated USB, since we planned on only using XBee based communication with the blimp.

I had been doing some research, and have learned a lot more about this whole Arduino ecosystem. There is incredible work being done by people like ladyada, and many others too numerous to list here. There are many variations on the Arduino, like the Boarduino, Sanguino, and so many others.

However, one option in particular caught my attention, called the Dorkboard. This thing is small, I mean tiny. Especially considering that it does not use the surface mount version of the ATMega microcontroller. And it has almost nothing on the board itself. You need to provide both power, and an interface to communicate with it, in our case, using the XBee modems.

This highly stripped down, minimalist approach was exactly what we had been looking for! Cheap, simple, light, and having everything that one needs from any Arduino/Freeduino platform. I ordered a few of the kits, and they were shipped from Wulfden just a few days later.

Once they came in, my brother was eager to get right into one. I dropped off the package, and he had soldered up one, plus the power supply, in less then an hour. We had the USB-BUB interface, so we just plugged it in. Once we had all the cable pinout correct, and we had downloaded the "hello, LED" code from RAD, we were watching it flash under our command.

The brain of our flying robot was ready! Now we needed a way to communicate with it, and update its code, all wirelessly. The next step, would be building an XBee breakout board, and doing some XBee hardware/firmware hacking...

RailsConf 2009 Program Is Looking Juicy

2009-02-27T15:07:00.000-08:00

I just noticed that there are already a whole bunch of the approved presentations for RailsConf 2009 on up the web site here.

It is looking really good already, go check it out. If you are into Ruby on Rails, you really should be going... I am!

Project Flying Robot: A Compass, Two Motors, And A Servo

2009-02-26T11:01:00.000-08:00

Last weekend was a fantastic hardware/software jam session with my brother Damen, working on our autonomous Ruby Arduino Development (RAD) powered blimp. Lots of little pieces had been coming in the mail, my brother had been soldering, and I had been programming away. Both of us were eager to start putting it all together.

The first item of business was getting our bearings. We had acquired from Sparkfun a very cool HMC6352 digital compass module. Based on the Honeywell chip, this tiny but powerful breakout board is a very easy device to use with Arduino, thanks to supporting the i2c protocol, also known as "two-wire interface" or TWI. No one had created a RAD plugin yet for this device, so I had whipped one together a couple days earlier using some Arduino code I found online, combining with the techniques used in some of the examples in RAD. Now, finally I would get a chance to see if it would work.

We plugged in the unit to our handy breadboarding setup, downloaded the code to the Arduino via USB cable, and viola! Amazingly, it worked perfectly the first time... we were able to obtain the heading, and turning around the unit resulted in the compass data updating correctly. Naturally, we started to get a little excited.

Of course, things would get a little bit trickier with our next item for integration: the Pololu Micro Dual Serial Controller. The pricey but useful Pololu chip allows for extremely simple serial command of 2 DC motors. Why would we want to do this, when we already have the Arduino itself? The ease of commanding a separate device that handles monitoring and setting the motor speeds, allows us to focus on the important parts of our application: integrating it all together. Plus, there is only so much processing power available in the ATMega microcontroller, so offloading that part of the work to the separate Pololu microcontroller frees up our Arduino to process more sensor inputs, and also do other interesting things we have planned.

Since the Pololu is a serial device, we used the Arduino's AFSoftSerial library to communicate with it. The soft serial library allows you to use any 2 of the Arduino's digital pins as a serial interface. Support is already in RAD for this library. With some more Google searching, and some assistance from the Pololu forums, I had written another RAD plugin. A few moments for Damen to wire things up, and we were ready for our test. We had only wired up one motor, and we had used the Arduino's own power to power both the Pololu's logic chip, as well as the motor. That would turn out to not be a very good idea.

The Pololu works by sending it simple serial commands. You tell it which of the two motors, which direction, and what speed to set it to. We fired things up set our motor to "forward speed 10" and it started to move. Glorious! So, overwhelmed with enthusiasm, we decided to try "20". Much to our surprise, setting the speed to 20 caused the motor to stop entirely. Say what? We looked at each other blankly.

You simply cannot power both the Arduino, and the motors, from the same power, unless you do some additional circuitry to provide both the higher voltage that the motors want, along with the 5V that the Arduino desires. Once we had thought about it for a few moments, it seemed obvious. We switched the motor supply to a separate battery, and the motor performed at 20. "Let's try full power", I said, and with a nod from my brother, I boosted the power to maximum. "Hey!" we both exclaimed at once a second later, as Damen grabbed for the now flying motor, which was trying it's best to lift the breadboard up by its power leads, and I frantically typed the commands that would shut down the motor. I wish we had THAT on video!

We then decided to keep going, and integrate our servo. The servo is used to change the vector of the thrust motors, a design that is also used by the Blimpduino. Unlike the main thrusters, we are just using the Arduino itself to control the servo motor, using one of the digital pins. It is really easy to use the servo library to move a servo to a particular position. In this case, no new plugin for RAD was required, cause others have already figured it all our for us. Which is nice.

We made the same mistake of connecting our servo to the same battery as the Arduino itself, due to the fact that the servo wanted the same voltage. By this point in the day, the poor thing just didn't have the force left in it. This caused the very odd side effect of resetting the Arduino itself when the servo would be ordered to move. We only realized that was what it was, once we tried getting power from the USB port. We had been testing using our wireless XBee modems. Once the power problem was solved, the servo did exactly what we told it.

It was a tremendous session, and we were now very satisfied that this project was, ahem, going to fly.

Prepare For The RailsConf 2009 Jam Session

2009-02-25T17:28:00.001-08:00

I will be running the music BOF at RailsConf 2009 in Las Vegas. What does that mean? We're gonna get down. All styles will be represented, so if you have an instrument or a voice, bring it.

We have a Google group started here.

So Long, And Thanks For All The SSH

2009-02-25T09:38:00.000-08:00

I read Jamis Buck's posting late last night, about how he was retiring from involvement with Capistrano, Net:SSH and a few other closely related projects. First of all, the community owes him a lot for doing so much work on these crucial parts of the Ruby on Rails ecosystem.

Seems like he has fallen victim to his own project's popularity. So many people using your code, and they start to expect things, and then demand them. I have seen friends in the Ruby community who are committers on a very popular tool, pretty much accosted at a conference, by fervent users of their project, who proceed to drag them off to the side to show them some obscure edge case. And like any good public servant, they go off to see whatever it is, regardless of whatever they were planning on doing the moment before.

Jamis was for sure getting burned out. I know that from personal experience, when programming pal Ari Lerner and I worked some patches for Net:SSH that solved a longstanding problem for public key authentication. Our first discovery on cloning the repo was that the original commit did not actually pass all of the unit tests. Then once we had corrected both that, and the original issue, and we contacted Jamis, he was pretty ambivalent. I am not knocking him here. Eventually, he accepted the patches. I think it was more that he had just had it with being a slave to his own creation.

Now we are on own, people. However, there does need to be some organization to the community. Github allows anyone to fork a project, sure. But having 25 different versions of the same project wandering around is too confusing for most people. New users would not know which version to install, the beauty of the standardization would be lost.

Jamis, you need to pick a successor to inherit the main responsibility for these projects, if for no other reason than this. Anyhow, we thank you again for all your efforts, and we look forward to seeing what you get into next.

Frankie Gets Some Fixes

2009-02-22T15:32:00.001-08:00

It has been very overdue to get Frankie updated to work with the latest versions of Sinatra and Facebooker.

But where I have been lagging, mjfreshyfresh has stepped up, and submitted a patch that gives compatibility with Sinatra 0.9.x and Facebooker 1.0.2. Thanks, my main man!

The Frankie gem has been updated to version 0.3, and it is available at your convenience. Note that you really do appear to need the 1.0.2 version of Facebooker for this release. Make sure you install facebooker using -v=1.0.2 to get only that version.

Let's all go huge on Facebook!

Project Flying Robot: Getting All XBee With It

2009-02-22T10:38:00.000-08:00

We have been making some amazing progress on our flying_robot project. This post is actually from almost a week ago, but I have not had time to catch up blogging with what we have been getting accomplished.

As I had mentioned in my first posting, one of the reasons that we did not simply copy the Blimpduino, is that we do not want to use a standard R/C controller for manual control. No, that would be too easy. Actually, it is because we have other kinds of ground control in mind... but I will get into that in future posts, once we have a working vehicle.

One focus of our project is to create a fully digital protocol to communicate wirelessly with UAV's. Something like MIDI is to music, the flying_robot command set is to flying. In other words, flying_robot is intended to provide a device-independent way to communicate with various kinds of unmanned aerial vehicles. So we started looking at digital wireless solutions. WiFi is common, but does not have the range, plus it uses too much power. Bluetooth just lacks range, period. Sounds like a job for XBee to me.

If you have not heard of it, XBee is a wireless data standard that has several priorities: it must have good range, use very low power, and be very cheap. The internet of things will likely be connected via XBee, since it it ideal for low-cost devices that wish to communicate with each other.

XBee modems come in several different flavors, depending on the needed range and desired type of network topology. The simplest version supports the 802.15.4 protocol, which is probably best for most point to point type applications, like a ground control to a base station. This is what they sometimes refer to as the "Series 1" modems.

The ZigBee modems, also sometimes called Series 2, add a protocol for mesh networking that looks to be very interesting. It could be used for a swarm of robots that communicate with each other, not just with a base station. This appealed to my inner geek, and so I ordered two of these modems, with the intention of experimenting with mesh networking in the near future.

On that note, there is an amazing deal right now for experimenters directly from Digi. You can get a development kit for the super-powerful next-generation XBee 868, that includes a pair of the modems, for only $99. I could not resist, and just ordered a pair of these to play around with, as well as the XBee-Pro 2.5's I already had acquired from SparkFun.

One thing I discovered quickly, was that the XBee modems I had purchased are really small. The next, was that I had to do some additional programming on them, before they would talk to each other. I found a useful Ruby script for configuring XBee modems from madrona, but it turned out to be a little more complicated, due to my choice of ZigBee over the simpler 802.15.4 protocol.

Fortunately, others have already been blazing this wireless trail, so I found these recent instructions on the Liquidware blog. It turns out that when dealing with ZigBee, one of the modems needs to be designated the "coordinator" for all the others in that mesh network, to hand out the all-important MY addresses. The coordinator needs different firmware than the individual nodes, which requires a utility provided by Digi, which is Windows only. Without this firmware on one of the modems in your mesh, you can send as many AT command as you like, the modems will never talk to each other. I dusted off a VM, and installed the Digi utility. Once I had updated the firmware, and updated the configurations of each modem using madrona's script, I was ready to go.

With one XBee modem plugged into my Mac via the USB port thanks to the XBee Explorer adapter, and the other plugged into the Arduino via the XBee Shield from Sparkfun, serial communication "just worked". Now I was ready to start coding on the Arduino itself... but that is another story.

Should We Call It Mails, or Rurb? Yahuda Katz Clears Things Up At The LA Ruby Meetup

2009-02-20T00:08:00.000-08:00

We just had a fun time at the latest LA Ruby Meetup, once again at UCLA. This time, we had presentations by James Foster from Gemstone, and Yehuda Katz from Engine Yard.

James Foster was showing MagLev, which is a Ruby VM based on top of the Gemstone Smalltalk VM. Not much to see here, just some Ruby code benchmarks running at about 4-5X faster than Ruby 1.8.6. That is, of course, not a very fair comparison, but no question that this is another Ruby VM to watch. His talk ended with the ever popular demo showing how Seaside does continuations, plus running the Smalltalk class browser right in the web browser and editing some code. Yes, this does kick serious butt, but it would be nice to see something new from the SmallTalk universe. I've seen all this from them two years ago, thanks to Avi Bryant.

The second presentation was where things got a lot more interesting. Yehuda Katz is well known to hard-core developers, thanks to his work on jQuery and Merb. He has also been very vocal about his differences of philosophy with the Rails way of doing things. Given the fairly recent announcement of the Merb project merging with the Ruby on Rails project, we all wanted to know, like, what's up?

As explanation, Yehuda gave a really complete history of Ruby on Rails. He showed slides of DHH's first Mac, he had dates and times of the first SVN commits for Rails. He explained the back story of what was going on in DHH's life while he was creating the first version of Rails. This was all ostensibly to justify that Ruby on Rails and Merb started with the same sort of motivations.

OK, great. That was fun. But what about the real dirt? He did not say anything directly, but my take on it was: when you really look at it, Merb was already turning into Rails++ so why not just merge all that effort into Rails itself? Putting too much emphasis into a separate framework also did not help sell services to the million Rails users out there, which is what Engine Yard's investors need them to be doing.

Yehuda was obviously comfortable with it, even thou he quipped about how many times he said "No!" when first presented with the concept of the great merge. But Merb seems to have the implementation, and Rails the sugar, plus the momentum, so your peanut butter is now in my chocolate. And the big winner? Has to be Rack, or the "Unix of web servers" as Yehuda called it.

Personally, I think it sounds great. Adding some API rigor to Rails will make it a lot more stable of platform for long-term development. And let us not forget that ezmobius has been contributing performance improvements back into Rails for a long time, so nothing shocking there. Just more good stuff for the Ruby web development community.

Project Flying Robot: The Adventure Begins

2009-02-18T08:22:00.000-08:00

I love robots. OK, there, I've gone and gotten that out into the open. Like many programmers, I have long dreamed of dominating the world with my robot legions. Or failing that, at least getting my RoboSapien to walk around in a little circle by itself.

Recently, my brother Damen Evans and I decided that we needed to realize our ambitions, and began work on a flying robot. Yes, that is what I said. Flying robot. After many years of respective obsessions about Lighter Than Air (LTA) vehicles and autonomous control, we are combining our forces and actually building something.

The "Rogue 1", as we are calling the airship, is heavily influenced (pun only, hopefully) by the very cool Blimpduino project. It will be controlled by an Arduino based system, like the Blimpduino. However, our focus is a bit different, which is why we are building our own hardware design, although also based on the open source Arduino hardware platform.

One major concept that we are working on, is creating a digital protocol for controlling the airship from a ground station. Where the Blimpduino uses a standard radio control setup, we are creating a fully digital protocol for Unmanned Aerial Vehicles (UAV) similar to what MIDI does for music. For example, an LTA vehicle would have a different way to perform a command like "rudder left 10 degrees" than an airplane or helicopter would. Especially since neither an LTA nor a helicopter even have rudders!

We have decided to use the Ruby Arduino Development (RAD) project for creating the software needed for flying_robot. It is still a bit rough in some ways, but it allows us to take advantage of a slick Ruby DSL-like approach for implementing the flying_robot command set for a specific UAV, as well as for integrating a bunch of the hardware we will be needing, like motors and sensors.

As the many small parts come in, both Damen and I will be blogging about the fun, so watch the skies... they're already here!

JSON-P Makes Progress Cross Domains With Apache, Passenger, and jQuery

2009-02-10T09:37:00.001-08:00

As yet another part of the be-all, end-all media upload processing solution, my client wanted to provide a nice progress bar for tracking the status of file uploads. Let me note, that providing user feedback as to the state of an extended upload turns out to be a very important UI feature.

Luckily, the problem of a nice way to track file upload progress had already been solved several times. My own platform of choice at the moment is Apache running with Passenger, and cool dude Peter Sarnacki aka drogomir had kindly provided a handy version of the same JSON based file upload progress tracking that had been done so well on nginx, except this time for Apache in the form of apache-upload-progress-module. He even had created a jquery-upload-progress plugin too, which is exactly what I needed.

Only one small problem... those pesky cross-domain javascript calls. As in, browsers do not as a rule, allow them. Then again, browsers do turn out to know how to be promiscuous. Which is to say, execute arbitrary JavaScript code handed to them by strange servers. Say wha? Yes, that is correct.

Enter JSON-P, which is the JavaScript Object Notation you know and love, with an extra layer of "Padding". What does the P in JSON do? Sorry, I could not resist that. Anyhow, the padding consists of a special temporary javascript function created just to handle the data callback from a server that is not in the same domain as the calling server.

Here is an example from our need for upload progress tracking. A normal JSON request to the upload component, like this:

   http://www.yourdomain.com/progress?X-Progress-ID=1234

Would return JSON data like this:

   new Object({ 'state' : 'uploading', 'received' : 35587, 'size' : 716595, 'speed' : 35587 })

But if we are using JSON-P, the request would include the name of the 'magic' callback function, so that the data returned from the cross-domain request is available to the original calling page, even though it is in a different domain, or even just subdomain.

For example, this request:

   http://uploads.yourdomain.com/progress?callback=jsonp123&X-Progress-ID=1234

Would return the JSON-P function:

   jsonp123(new Object({ 'state' : 'uploading', 'received' : 35587, 'size' : 716595, 'speed' : 35587 }));

The calling page executes the 'jsonp123' function after the data is returned from the cross-domain request, and then the returned data is available to scripts on the original page. This is how Twitter and other Javascript API's are able to function with these cross-domain requests.

So I whipped up patches for each the apache-progress-module, and the jquery progress plugin, and now that drogomir has accepted them, we can all enjoy the benefits of cross-domain uploads, along with JSON-P powered shiny progress indicators. Well, as long as it is a jquery based progress indicator you seek. I will try to get around to looking at the Prototype progress plugin, but since I am mostly working with jquery now, perhaps someone else will jump into the fray.

Have fun, and make some progress.

Introducing Net Grommit

2009-02-07T08:37:00.000-08:00

I have just begun a new blog called Net Grommit.

It is a place that I can go off and explore my experiences with technology and being a 21st century parent. I will get into anything from video games, kids programming, tech toys, education, and whatever else appeals to me. As a father of two boys, and an obsessive technologist, I have plenty of personal experience to talk about, and I really look forward to the participation of some other people, as well.

So, if you are a parent, or just find this as interesting as I do, please come over and join me at Net Grommit.

And now, back to our irregularly scheduled programming topics.

Los Angeles + Ruby = LARubyConf 2009

2009-01-31T09:27:00.000-08:00

It is with great excitement that I want to tell all of you about the first annual Los Angeles Ruby Conference. Thanks to the fine efforts of a bunch of LA's hardworking Rubyists, and the generous financial support of AT&T Interactive, we are preparing to bring the noise to Los Angeles on April 4, 2009.

Too many of these wannabe cities have been smoking us with their regional conferences, but no more! Time for we Angelinos to step up and be counted among the world's important Ruby communities. All joking aside, it really is time for us to share our enthusiasm and experience, and grow beyond just the local scene.

I am helping out with the organization of the conference, which means a good chance to showcase other people. We are recruiting presenters, sponsors, and most importantly attendees! There will for sure be something for everyone, at all levels of experience.

Here is what you should do:

Want to attend? Register now for the early bird price of $75
Have something cool to talk about? We want your proposal
Want to sponsor the conference? Yes, we want you, click here

We need your support, people. Let's have a great time, learn from each other, and make it a big success. 'Nuff said.

Getting Async With Sinatra And Passenger Using spork

2009-01-28T10:31:00.000-08:00

Everyone wants it all. In the case of upload processing, this means we want both the convenience of Sinatra coding, plus the performance that a nice process/threading model gives us. Go off and do some work I just assigned you, and don't make me wait around to see you do it.

If you are using Passenger, it already is buffering your uploads, and not calling your Sinatra application until the upload is complete. That saves system resources for more important things, like even more uploads. Pretty nice.

But a lot of work often has to take place after the upload is completed. A common example these days would be a video transcoding or image processing server. Once your user has uploaded their file, it would be considerate to let them go off and do something else. Sites like YouTube and Flickr have been doing exactly this for some time.

For certain highly transactional system, a solid message queue would be essential. But more humble needs, like an upload/transcoding/image thumbnail server, does not really need all of those extra moving parts.

Thanks to the power of the underlying Ruby language itself, it is very simple to fork off blocks of code into a separate process. Combining this with Passenger's already pretty robust process handling, and you get a pretty well performing solution with not much coding effort.

But wait... it does not work?! Sure enough, there are issues in the underlying interaction between Passenger and Rack, that you must address, or your long-running code will be just that, long-running, making the user sit there and wait for it to complete before Passenger allows Sinatra to return a response.

Enter spork... a blatant ripoff of both the spoon and the fork... I mean of the Spawn plugin for Rails, except with no actual Rails stuff in it, of course. spork handles the monkeypatching required to get things working the way they should be, and also lets you run under thin with no code changes should you so wish, like when running in development mode locally.

Anyhow, it may not be a glamorous implement, but it is a useful one. Have a look at spork if you are using Sinatra on Passenger, and you want to get all asynchronous on your users.

The Verdict Is In

2009-01-28T08:21:00.000-08:00

For the last week or so, I have been serving on a jury in Department 45 of the Stanley Mosk Courthouse in downtown Los Angeles. Now that the trial is over, I am able to talk/blog/pontificate about my experience.

This was my first time actually serving on a jury, despite having come close a couple of times in the past. The case itself was a canonical example of how ridiculous big companies can be when fighting over money. The dispute was about billings between two very large insurance companies. Like you probably just did, my first comment was "a plague on both their houses." However, after executing the mental gymnastics for days of maintaining equipoise, once I had heard all of the evidence presented by both sides, I felt that the plaintiff had actually been wronged by the defendant. Both attorneys did an excellent job arguing their clients' positions, but the facts seemed reasonably clear to me from the evidence and testimony. At least as clear as anything can be almost ten years after the events had originally transpired.

Anyhow, this post is not about the case. It is actually an ode to my fellow jurors, and the legal system. I have had a slightly negative view of our legal system, like many citizens. But a duty is a duty, and so when the summons arrives, we all must answer. However, most people view it as a kind of 'reverse lottery', and dread jury service like an impending root canal.

The experience of actually serving on a jury, and seeing the legal system from the inside, has altered my perspective quite a bit. Despite coming from very different walks of life, all my fellow jurors had a real dedication to our task, as absurd as we all felt it to be. Nonetheless, we fell to our work with an amazing level of morale. We tried to smile at each other, and remain happy and attentive, even as the hours of hearing and seeing the evidence in the case wore on.

Here I have to remark on an odd, and to me extremely entertaining, coincidence. Ruby programmer and father of Rubinius Evan Phoenix was on the same jury. We had known each other superficially from RubyConf etc., but had never had a chance to hang out together, despite living in the same city. Lunching every day together really relieved the boredom of enduring the testimony, and bad PowerPoint, of the trial. Thanks for putting up with my loud comments!

The lawyers and judge projected a calm friendliness, even to each other, that defined 'professional courtesy'. Even after the cross-examination starting getting bloody. Yes, this was a kind of trial by combat, albeit one of words instead of arms, as I was slightly surprised to discover, after the pomp and ceremony of the opening statements, where the opposing counsel were merely thumping their chests and circling each other.

When the time came that we jurors could finally discuss the facts in the case, after not being able to discuss it with anyone, including each other, it was an incredible release. This was the logic in not permitting us to speak openly until the deliberation had begun. We each had already really defined where we stood relative to the evidence we had seen. The level of mutual respect that we had during this special conversation known as deliberating, was remarkable.

Credit to Evan who served as our lead juror. He did a great job keeping the forum open, and also moving us along to the final verdict.

As I leave my jury experience behind me, and return to my normal slew of programming stuff, I am reminded of Churchill's quote about democracy "the worst form of government except all the others that have been tried." E Pluribus Unum!

In My Hood

2009-01-25T09:26:00.000-08:00

I was taking my morning constitutional with my youngest son, when I noticed my neighbor who programs at JPL had added a new placard to the artistic decor attached to his fence.

And who says there is no software development in Los Angeles?

Get A Grep On TextMate

2009-01-23T08:58:00.001-08:00

I really wanted to be all cool and retro by going back to EMACS, but my usual workload has kept me from figuring out the right environment for my Mac. In the meantime, programming pal and fun blogger Daniel Fischer turned me on to just one more reason to stay lazy.

TextMate is really quite a cool editor. However, one problem that annoys, especially when working on larger projects, is TextMate's "Find In Project". If you've ever literally had your entire machine lock up while doing a search, you know the pain.

Why do such a crappy search, when grep is just waiting to serve you? You like the pretty colors and do not wish to leave TextMate? Lucky for us, a wonderful human being in Sweden named Henrik has created a bundle that replaces TextMate's madness for a nice quick clean grep.

You need to get this right now, if you are using TextMate. You will save a LOT of time and frustration.

Sinatra and Passenger Back On The Rack

2009-01-21T08:16:00.000-08:00

There is a new release of the Sinatra gem, version 0.9.1. This happens to work out well, since the newest Passenger 2.0.6 has a strong dependency on Rack 0.9, and the previous Sinatra 0.3 simply did not work at all due to dependency on Rack 0.4. The last few days have had some of us apoplectic over Ruby gem version hell.

And lots of interesting and even important things require all three. For example, I have been messing around with Integrity, a very cool little continuous integration server which is built using Sinatra. Foca and the rest of Team Integrity are still struggling with upgrading a morass of DataMapper gems, but this should help.

The point here, is getting everything up on Rack 0.9 seem to have made these problems go away, at least for my own Sinatra-based projects. Thanks to everyone who worked on these new releases, and special thanks to the IRC dwellers that provide support for the community.

RubyConf 2008 - Day 1 - Testing Heresies - Francis Hwang

2008-11-13T09:07:00.000-08:00

The sessions for RubyConf 2008 day one were nearly over, when I attending this highly entertaining and informative one. I had missed the chance to see a previous presentation by Francis Hwang, and being as test-obsessed as I am, I was not going to make the same mistake twice.

Many people are confused over mocks, stubs, and other fakes. Here are the real definitions.

Gerard Meszaros - Test Doubles
- dummies - passed around but never used
- fakes - work but unstable in production
- stubs - provide canned answers to calls made during test
- mocks - mostly concerned with surface interaction

Classical TDD use real objects.
Mockist TDD will always use a mock for any object with interesting behavior

Problem #1 with mocks
Cannot figure out problem and solution at same time

Problem #2 with mocks
Bugs can slip between layers

Francis Hwang, TDD Clasiscist
- test setup creates records in the db
- no sqlite in test
- when you use test doubles, it's always a big deal

Real world example #1
Ensure that an optimization is actually being used
Redefine the actual surface interaction to 'scream' on error

Real world example #2
Faking a real web service (not written by us)
Best to use stub, to duplicate functionality
Need test, stub, AND test for stub

Dogma - if you tests do not run quickly, there's something wrong with you
Heresy - I'd like my test to run 30 sec. I'd also like a pony

Francis does not use autotest, because he does not want to run all of the tests every time he changes something. Eric Hodel chimed in from the audience says that it can run only what changed. Only if it passes, will it run ALL tests. Me, I loves my autotest... with growl notifications, it's all good.

devver
They say it is a ec2 based distributed test system...

testjour
uses bonjour to trigger a farm of machines in your LAN to perform distributed tests

Dogma - you code should always be well-structured
Heresy - under-structuring can be freeing

Uses example of constantly cleaning small NYC apt. It was living on a boat, where you spent all your time putting things away.

Dogma - Fine grained focus leads to high quality code
Heresy - Never forget the unknown unknowns

System design 101
How likely fail
What will fail look like
How can I find out quickly
How can I limit damage

Dogma - testing makes my job easier
Heresy - testing makes it easier for others to do my job - unless my job keeps growing

Rails has made it so simple to implement simple business logic, so way more is expected now.

RubyConf 2008 - Day 1 - rush, a shell that will yield to you - Nicholas Schlueter

2008-11-11T06:54:00.000-08:00

I had previously blogged a bit about Rush, the Ruby shell. Well, now Rush must now be its own force to be reckoned with, because this presentation was NOT by Adam Wiggins, the creator of Rush, but was instead by Nicholas Schlueter, the creator of RushMate. If you've not heard of it, RushMate allows you to use Rush from within TextMate, which translates to being able to write custom commands much more easily.

He showed some neat examples, including one that auto-generates an HTML page with links to all of the RDocs for all of the RubyGems you have installed locally. I have wanted something like that for times that I was not able to connect to the InterTubes, but still wanted to do some development.

The entirety of his presentation is online here with a whole bunch of code examples that will be better than my ramblings, so you should rush over and check it out.

RubyConf 2008 - Rubinius - Evan Phoenix

2008-11-10T21:21:00.000-08:00

The Ruby VM wonkiness continued at RubyConf 2008 - Day 1. Next up we heard the "State of Rubinius" address on the status of the anointed heir to the Ruby federation. Evan Phoenix is a fun guy, and it proved to be a both informative and enjoyable session.

Evan says Rubinius is "meta-circular-ish", and slow, but getting faster. They have a new VM that is written in C++, which is a sharp tool, and as such, it is possible to poke your eye out. However, he says it allows them to better model work. The three main advantages are:

- type safety
- organization
- architecture

What is type safety, may ask many Rubyists. Well, sometimes a duck actually IS a duck. The Rubinius code just feels better with OO nature of C++, as opposed to just plain C. Also, it is possible to make the VM class hierarchy that mirrors the Ruby class hierarchy.

Exporting Methods - a day in the life of a primitive operation

- things you cannot do in Ruby
- small example


class String : Object {
  Fixnum* size();
}

class String
  def size
    Ruby.primitive :string_size
  end
end

There was interesting question about why flatten out ala :string_size aka no namespacing. Evan did not seem to give a definitive answer...

Method Dispatch - From resolution to execution

Resolution
- going from receiver and method name to actual method
- three mechanisms
- hierarchical lookup
- global cache
- inline cache

Execution
- every method provide an execute function pointer
- each primitive is an executor
- Ruby methods use executor specialization

Ruby Methods
- Specialized executors based on argument info
- Very simple fast version for majority of cases
- Slower fallback case for all cases

Critical Path
- Populate message object
- Call rep over trampoline
- Fill in more of message object
- Call method executor

Execution/Framing
- Method context objects store info about each method
- Chained together using refs (spaghetti stack)
- MethodContext creation speed is crucial

- Default contexts are allocated in special section of memory
- Normal execution follows simple stack pattern
- We can exploit to allow for fast creation and deallocation

Calling to C
Can you imaging life without ruby gems that use C extensions? For example:
- hpricot
- mongrel
- mysql
- sqlite
Not a problem. A simple recompile == usable in Rubinius

Tricky
- GC interaction requires indirection
- Uses 'quantum leap" stack jumping
- Allows for semi-graceful recovery from extension segfaults

Performance
Rubinius is doing well on micro-benchmarks
- method dispatch
- GC
- OK on macro-benchmarks
- Poorly on mega-benchmarks

Solutions
- LLVM
- JIT
- AOT
- Improve algorithmic efficiency

Rubinius is concentrating on compatibility right now, but will then address performance. Just like Koichi-san said, it is a curve of improvements.

Evan ended up with a nice quote from Woody Allen - "If you're not failing every now and again, it's a sign you're not doing anything very innovative."

RubyConf 2008 - Future of RubyVM - Koichi Sasada

2008-11-10T10:13:00.000-08:00

The first session I attended at RubyConf 2008 was called "Future of RubyVM", and it began my ascent into "VM Wonkiness" as I started to call it. Koichi-san is responsible for leading the team of Ruby 1.9 implementers, and is the authoritative source of info on its status. I was hoping to learn a few things, as well as to run into new acquaintance Nobuyoshi Nakada, one of the core team members.

Techniques for VM Performance
- Simple techniques
- C level VM

- Advanced techniques
- Dynamic code generation
- Native compilation
- JIT compilation
- Polymorphic inline cache
- Selective inlining

- Online feedback opt.
- Hotspot JIT
- Tracing JIT

Pros/Cons of JRuby/IronRuby
- Using an awesome VM
- Pros
- Many clever people working on the project
- No code is good code
- Many libraries already exist on each environment
- Easy to use parallelism
- Cons
- Not only focused on ruby, so there is a semantics gap.
- Cannot use C ext direct

Pros/Cons of Rubinius
- Most code in ruby
- Pros
- Ruby in Ruby
- Says this is the *best way* to improve performance in the long run because we can easily analyze and make specific performance improvements.
- Cons
- Still a LONG way to get a high performance VM

Pros/Cons of C Ruby
- Pros
- Portability, using GCC
- C is well known language already
- Extensibility
- Performance improvement
- Easy to write simple extensions.
Cons of C Ruby
- Extension libraries are written in C
- GC problem
- Inlining problem
- Limitations on program analysis

Our Performance Policy
- CRuby is not best solution, but it is a GOOD one
- They continue to improve the CRuby implementation
- It is a pragmatic, practical selection
- It will be the main implementation still for at least several years

Keywords for Success
- Embedding
- Parallelism

Research
The team is working on taking advantage of C some projects are running
- Hidden optimization techniques on YARV
- Ricsin

Hidden Optimization Techniques
- turned off in 1.9.1 by default
- Tail call optimization
- Optimizing using unification
- Stack caching

- Left easy optimizations
- Efficient method caching
- Efficient fiber implementation using platform dependent way suvh as makcontext()
- These optimizations will be merged in 1.9.2

Ricsin; Mix-in C Ruby
- Embed part of a C program into ruby
- Like a RubyInline, but directly embedded
- Usage
- Use C libs direct
- Replace all built-in classes and methods
- Test Ruby C API
- Continuous performance improvements

Ruby to C AOT Compiler
- Translate Ruby scripts to C code ahead of time

Related work
- ruby2c
- yajit
- yarv2llvm

Atomic-Ruby project
- Issue: ruby is too fat
- EMBEDDED
- Embedded system can have special problems, such as resource limitations
- Application embedded ruby
- Some applications need just a Ruby DSL engine, not the full ruby distribution.
- We need a slim Ruby interpreter
- Utilize CRuby portability
- 3 subprojects
- Plugin/out
- Precompilation
- Swift core features

Auto-write barrier detection
- write barrier needed for several GC algorithms.
- automatic WB detection system

Generational GC
1 but ref count

Multi-VM (MVM) Project
- Multi VM in 1 process
- Watch runs in parallel
- High speed inter VM communication
- Sponsored by Sun

Summary
- CRuby/YARV is not "best solution", but it is best for right now. But eventually, Rubinius is the future. Which just so happened to be the next session I attended...

RubyConf 2008 - Day 1

2008-11-10T09:58:00.000-08:00

The crowd of over 400 was happily awaiting the kickoff in Orlando of RubyConf 2008. After brief introductions from Rich Kilmer and David Black, Matz took the stage for his opening keynote.

Opening Keynote - Matz

Reasons Behind Ruby
8 years ago was the 1st International Ruby Conference, and there were only 33 people there. Obviously since then, the community has grown. Matz said he was not going to talk about technical stuff.

Question - Why Ruby?
Matz joked that it is his beloved masterpiece. Really though, Matz loves languages, and also loves freedom. His reason for creating Ruby was to maximize freedom.

There are still a few problems. Ruby is imperfect, slow, lacks features, and is complex. And there's more. Ruby is inconsistent, has a poor implementation, has memory issues, and embedding issues.

So why do we love Ruby? "Programmers Strange Love, Or How I learned to Stop Worrying and Love Ruby"

The greatest Reason behind Ruby is Love. It made programming fun again. Again? It used to be fun.

In 1980, Matz began programming in BASIC. He liked that he could make intelligence, and felt a sense of creation. But there was also great pain.

The Basic Aristocracy
There was a big difference between language designers, vs. lowly programmers. In BASIC you cannot add to the language, and there is no abstraction.

Lisp
Everything is opposite in Lisp from BASIC. In Lisp, there is no discrimination and extreme abstraction.

As Eric Raymond said, "Lisp is worth learning for the profound enlightenment experience you will have when you finally get it; that experience will make you a better programmer for the rest of your days..."

However, Lisp was great in a book, not in reality. Programming in Lisp did not make him happy. Was it all the parentheses? No. Was it the macros? Partially.

It came down to the fact that smart people just underestimate the "ordinarity" of ordinary people.

Aristocracy is OK, as long as I have POWER. Matz wants to be between "owelty" and freedom (none of us in the audience knew what he had meant by "owelty" either, but Matz was on a roll, so we went with it).

BASIC gives you no power, Lisp gives you all power. But at both ends we lose popularity. If you stay in BASIC you are a coward, and if you go extreme, you are brave but you risk your life (or popularity).

We all love power. You see, policy matters. We need philosophy and attitude.

The community cannot live by love alone, power matters.

Then along came Rails which turned Ruby into a web DSL. You see, Ruby is about vocabulary. It has rules, or syntax. There are others, for example Rake and RSpec.

As Dave Thomas said - "Programming is a process of designing DSL for your own application"

Ruby is really like a meta-DSL. It provides a foundation for DSL's, which increases our overall productivity.

But beware of commercial success, because they will try to take your freedom away.

Now there are better implementations or Ruby than MRI - YARV, JRuby, Rubinius. Matz says that the various implementations are competing, which is a good thing.

Matz then mentions the 4X growth expected in Ruby programmers from the current 1 million to over 4 millions. There will be NEW people coming into the community, so welcome them and nourish them. Ruby can have the power to raise up people in the programming field, so feed them, and love them.

He loves us all.

We love you too, Matz.

Sessions - Day 1

I attended a number of interesting sessions throughout RubyConf 2008 day 1. Per previous reader feedback (keep each post to a manageable size!), I will put each into a separate post. Here is what I will be covering:

Future of RubyVM - Koichi Sasada
State Of Rubinius - Evan Phoenix
rush, a shell that yields to you - Nicholas Schlueter
Testing Heresies - Francis Hwang
Lightning Talks

I missed a few of the lightning talks while having a nice dinner with David Black, Jim Freeze, and Paul Brannan. Hanging out with really smart people is like a continuous lightning talk, so I didn't feel deprived by any means. Thanks, guys!

After the lightning, as it were, we had the second ever Ruby Jam Session. This time, about 5 people brought guitars, including Ruby Jams veterans like Jim Weirich and Forrest Chang, and lots of new friends showed up as well. I was better prepared this time around, and had brought about 10 harmonicas with me. But the tastiest part of the jam, was at the very end when David Chelimsky and Diego Scataglini played some really great jazz standards. A couple of tasty beers followed the music, and then it was off to rest up for RubyConf Day 2.

My Vote Was Easy

2008-11-04T13:50:00.000-08:00

Despite a massive line of people waiting for the polls to open this morning, when I went to vote just now I discovered... no line whatsoever. What a letdown! I wanted to do some sacrificing like hard struggling adherents to the cause of freedom the world over have done, just to get to the polling place, let alone cast their vote. But no, everything had to go quickly and smoothly.

After accepting my offering in Halloween chocolate (stolen from my son, I might add) the friendly poll-workers ushered me to my individualized voting booth, and afterward counted my vote extra-specially well before bidding me farewell.

Democracy is great, and kudos to the people that humbly and with no applause watch over our freedoms at the ground level... your friendly neighborhood poll-workers.

If you haven't done it yet... go! Now!

I Think I'm Comatose

2008-11-04T10:50:00.000-08:00

This last weekend, I upgraded a pretty complex application to the latest Ruby on Rails, and consequently had to upgrade a bunch of the plugins. In the case of Comatose 2.0, this was well timed, as there is a new release.

If you have never heard of Comatose, it is a very cool mini-CMS that you add to your Rails app as a plugin. For cases where you have an applcation, and you want to put the 'static' pages under the control of the users, while still integrating with your application, Comatose is the best option I have discovered.

If you are just discovering Comatose, just go check it out. Now. If, on the other hand, you are running into any problems going to the new version, read on.

Since this application had been running the 0.8 version of Comatose, I ran into a little problem, which has been documented on Google Groups here.

If you are running the older version of Comatose, you have a migration that contains this:


 module Comatose
   class Page < ActiveRecord::Base
     set_table_name 'comatose_pages'
     acts_as_versioned :if_changed => [:title, :slug, :keywords, :body]
   end
 end

The new version of Comatose, however requires this:


 class ComatosePage < ActiveRecord::Base
   set_table_name 'comatose_pages'
   acts_as_versioned :if_changed => [:title, :slug, :keywords, :body]
 end

What I did was, go and change the original migration to match the newly required format. This will allow a new clean installation to work correctly. Then, to handle upgrade installs, I created a separate rake task to perform the migration steps as described here.

Voila. Now this app is happily running the latest Rails, Comatose, ActiveScaffold, ActiveMerchant, and a few other plugins too. Life is good.

Oh, The Horror

2008-10-31T09:24:00.000-07:00

I was forwarded a link to the original, left on the cutting room floor ending of the Roger Corman classic horror film "Little Shop of Horrors" and not only was it a timely tidbit for Halloween, but it was also strangely appropriate for the election.

Enjoy your candy...

The Great Git Migration

2008-10-26T10:15:00.000-07:00

I have been using git for a while, not quite as long as the really cool kids, but long enough to have become a git snob. As such, having to work on existing projects that were using that nasty old subversion was just like, a drag, man.

I kept saying, "I'm going to migrate everything over to git." Somehow, I just never seemed to find the time. Another little problem was the underwhelming level of support for git on Windows. Yes, a couple of projects I am involved with have Windows code for client applications in there.

So when the msysgit project was announced, I followed it with great interest. The most recent versions are actually quite usable. I was able to install on a Vista VM (I use VMWare Fusion on OS X to keep the Windows thing to a minimum) and connect to several repositories on both github and Unfuddle with ease.

A couple of hours waiting on git-svn to import the old repositories (under OS X, cause mysysgit does not appear to support git-svn), and it was all git all the time. No more svn on any of the projects I am actively working on. And I was able to preserve the entire version history of each repo, too.

Cool Like Cucumber

2008-10-25T16:15:00.000-07:00

Ever since I discovered Fitnesse a few years back, I have been increasingly obsessed with the idea of executable specifications and testable requirements. As far as I am concerned, Behavior Driven Development (BDD) is a very appealing way to get the "people who need the software" and the "people who build the software" to understand each other.

For about 2 years, give or take, many Ruby on Rails developers have been switching to RSpec, or at least paying lip-service (giving rspec?). This is simply because, as far as automated testing, ye olde test::unit is getting a bit long in the tooth. The somewhat less ambitious shoulda project takes a much more minimal approach to RSpec, preferring to be a better test:unit and not going all the way to the executable specification. Maybe they did not drink the whole glass of kool-aid? Well, I did... a while back. But I digress.

There have been various attempts to introduce a more readable, less "code-oriented" way to work with the users themselves to define the requirements for their solution. This led to RBehave, later merged into the RSpec story runner. Now, there is an even better tool emerging for human-readable yet testable requirements, called Cucumber.

Cucumber is the brainchild of RSpec contributor and super smart fellow Aslak Hellesoy. Taking the basic idea of the StoryRunner and, ahem, "running with it", Aslak has not only rewritten the entire thing using the treetop parsing engine, but he has also improved on the paradigm both for expressing the user needs, as well as the coding required to turn the requirements into executable tests.

Here is a small example:

Feature: Log-in
  In order to view my profile
  As a Registered member
  I want to be required to log-in

  Scenario: Proper login
    Given I am the registered member "quire"
    And I am on the login page
    When I fill in "email" with "quire@example.com"
    And I fill in "password" with "quire69"
    And I press "Login"
    Then I should see "Account Activity"

  Scenario: Bad password
    Given I am the registered member "quire"
    And I am on the login page
    When I fill in "email" with "quire@example.com"
    And I fill in "password" with "bad password"
    And I press "Login"
    Then I should see "Simply enter the credentials you signed up with."
    And I should see "Couldn't log you in as 'quire@example.com'"
  
  Scenario: Bad email
    Given I am the registered member "quire"
    And I am on the login page
    When I fill in "email" with "quire@example.co"
    And I fill in "password" with "password"
    And I press "Login"
    Then I should see "Simply enter the credentials you signed up with."
    And I should see "Couldn't log you in as 'quire@example.co'"

Now THAT is a nice clear way to define what a system is supposed to do. Furthermore, actually implementing the "steps" that exercise the system functionality is very simple as well:

Given /I am the registered member "quire"/ do
  @user = User.new({ :login => 'quire', :email => 'quire@example.com', :password => 'quire69', :password_confirmation => 'quire69', :terms_of_service => true, :primary_talent_id => 1 })
  @user.save!
end

Given 'I am on the login page' do
  visits "/login"
end

Then 'I should be taken to my dashboard page' do
  response.request.path.should == user_path(@user)
end

Since WebRat is fully integrated into Cucumber, and there are already a group of useful steps included, you can get surprisingly far while defining fairly few steps of your own. And when you do have to, working with Cucumber really is quite superior to the old StoryRunner.

I am working on two projects right now that are using BDD and Cucumber, and we are very happy with it. We now have a lot of visibility into how complete a feature is, and it helps everyone understand what a feature does, and how it is supposed to work.

Once again, great work from the RSpec crew. Thanks, guys!

Going To RubyConf 2008

2008-09-10T16:54:00.000-07:00

Yes, it has been quite a while since I last posted. I have been busy with several interesting code things I will be talking about soon, along with my family's annual trip to Spain (lots of good times, but no posting about that).

Busy, yes. but not TOO busy to make my plans to go to RubyConf 2008. November in Florida, I can just imagine it now... hard core Ruby coding, jam sessions in the hotel, and maybe even some post-election riots. Plus roller disco. Damn, I'm there!

And if you want to be there, better get it together and register now.

RailsConf 2008 - Day 2

2008-06-08T21:14:00.000-07:00

Day 2 of RailsConf 2008 began with a keynote from Jeremy Kemper, uber-committer and now main man behind Ruby on Rails. According to the brief introduction by DHH, Jeremy did around one third of the Rails 1.0 release himself, cares deeply about the "whole thing", and has been busy helping get RoR running on Ruby 1.9 and other platforms.

That must explain why Jeremy kept yawning incessantly during his presentation... he is the reason DHH is able to get more sleep! There was some speculation that Jeremy was actually suffering from a hangover... I have no idea, I was asleep the night before.

Anyhow, Jeremy gave an overview of going from Rails 1.0 to the new 2.1 release. Basically, in his opinion 1.2 was "out for a while and got stale". In 2.0, we got all these foxy and sexy things, plus shed a lot of fat and gained speed. But the real story was the growth of the movement behind Rails from a fairly small group of core people, to a massive community of interested contributors. Here are some amazing stats:

From the 2.0 release in Dec 2007 to the 2.1 release in June 2008

> 1400 contributors (Trac, Lighthouse, and Github users combined... obviously a bunch are duplicates, but even 450+ is a large number!)
> 1600 patches
> 9000 comments

I cannot imagine the chaos of having THAT MANY people harassing me over getting their pet patches into the Rails trunk. No wonder they seem a little preoccupied... they are actually thinking about how many patches that will have to reviewed by later tonight! Much respect to the team just for dealing with that many crazed developers. This is to a large extent the key benefit of moving RoR on to Github: the much easier process of managing all these people's contributions.

Here are a few brief highlights of the new 2.1 features:

* New merged migrations - helps deal with keeping migrations synched throughout the development team. It does this by a new naming system, and now making sure that all unapplied migrations have been handled, instead of the old simple schema_info table, RoR now keeps track of all migrations, and makes sure that all migrations have been applied

* Time zone support - there were several plugins for time zone support. RoR has now included a fairly simple way of handling this with some baked in logic that extends both the Time class as well as some new ActiveSupport hotness.

* Ruby gem dependencies - now you can specify exactly which gems and versions are needed for your Rails app, instead of playing "one of these gems is not like the others" when trying to install or deploy a Rails app.

* Memcache - support for the ultra powerful memcache is now baked in, making fragment caching work nicely... made memcache a first class member of framework, and the memcache client is now bundled with the code distribution.

* ActiveRecord Dirty - now a way of track what has changed in a AR instance. This improves performance slightly by eliminating unnecessary database writes, but even more importantly results in much cleaner log files.

* Smarter :include - avoids combinatorial explosion with out of control includes, that result in too many left outer joins on every DB access. Instead, will just perform two queries to DB, with resulting database performance improvements. For example:


Msg.find(:all, :include => :user) # two queries
Msg.find(:all, :include => :user, :order_by => 'users.created_at) # uses outer join

* Named scopes - with complex models, you will have repeated convenience associations. For example:


user.recent_messages

You would prefer:


user.messages.recent

especially if you can pick it up "for free". Here is a tiny example:


class User
 has_many :messages
 ...
end

class Messages
 named_scope :recent, :order => 'created_by DESC'
 ...http://www2.blogger.com/img/gl.link.gif
end

An Embarrassment Of Riches In Ruby VM-land
Rails is now running in (milestone achieved just before the conference, thanks to heroic efforts by John Lam, and the rest of what is a very under-appreciated team within the Ruby community at-large). Rails has been running in JRuby for months. Also, RoR has been running in Rubinius since about two weeks before RailsConf.

And adding to the fun, Ruby 1.87 is now released, which includes a bunch of performance related backports from the once and future Ruby 1.9 release.

Just to show all of this multi-platform hotness, Jeremy did a little Rails 2.1 demo, showing the a minimal Rails app running on JRuby, Rubinius, and Ruby 1.9. That was amazing really, that we are actually here. Sure, Rubinius was kinda slow at starting up, but it worked, that is what counts.

Ruby 1.9 on Rails
Yes, you can actually run Rails on Ruby 1.9, at least somewhat. ruby-1.9 script/server

In Ruby 1.9, string encoding is huge change. According to Jeremy, 1.9 looks really fast... despite no formal benchmarks. He sees modest performance improvements on typical pages, and really notices it on bigger ERB pages. The typical page is about 20% faster. He says all rails tests are running on ruby 1.9.

Despite all of the yawning, I was not put to sleep at all by seeing some cool new Rails features, and even more importantly a few huge milestones in the Ruby VM race.

"Getting Git" - Scott Chacon
The keynote was good, but it had only whetted my appetite for even more substance. After a tip from a friend, I went to check out Scott Chacon's presentation on git. If you knew nothing about git, this was not the place for you. Scott really got into the internals of git, which many of us found captivating.

The room was pretty packed, and the crowd was held in rapt attention. Several of us remarked afterwards that it was the most organized presentation we had ever seen. It somewhat mimicked the format used during Dick Hardt's presentation on "Identity 2.0" at OSCON 2005, if you have seen that one. Each section of the presentation was precisely timed, and amount of material covered was massive. I took copious notes, but there was no way to try to capture that much info. Check it out online here, if you want to dive deeply into the glory that is git.

"Advanced RESTful Rails" - Ben Scofield

Next, I went to Ben Scofield's presentation on REST. Surprisingly a lot of people are not yet using REST. Or if they are, there appears to be some confusion over exactly how to implement things correctly. It was a decent presentation, but "Intermediate" would have been slightly more accurate. I mean, where was the discussion about polymorphic RESTful controllers? Now THAT would have been advanced.

It was still a good, if not great presentation. Ben knows what he is talking about, I would just suggest raising the bar as far as use of the word "advanced". That suggestion applies across the board to all RailsConf presentations.

"StoryRunner" - David Chemlisky

David Chelimsky is the closest thing we have to a test guru in the Ruby community. Here I am using the word guru in a precise way, "any person who counsels or advises; mentor". No single person has had as much impact on introducing best test practices like behavior-driven development to the Ruby community. Even the way that plain old test-driven development is done, is heavily influenced by RSpec. Not to mention the clones like Test::Spec and shoulda. All of the cooler projects I work on have test specs, and not just unit tests. The projects that I work on that are not as precisely thought out, turn out to not have test specs, but only unit tests.

Coincidence? I think not. The reason for going through the process of really defining what is to be done is to provide a common understanding between the people in the project. But if that was the whole story, BDUF might work. The problem is you need to apply test-driven development principals to the specifications themselves in your project, not just the code. And you need to be doing this as the project goes on, not just once in the beginning.

Behavior-driven development is about closing the gap between the "customer" (the system/site/application user, if you work in something other than a consulting firm) and the "developer" (that would be you). And that is where StoryRunner comes in.

StoryRunner makes it easy to create a kind of simple parser for a very english language oriented, simple text based way to write specs, requiring far less structure that the current RSpec format. It is really like a DSL that customers can use themselves to define their needs. As David put it, "Ruby code is very readable, but you will not be getting customers to send you emails with new requirements in the form of Ruby code."

So BDD is about helping the customer/user formalize their thoughts about exactly what solution they need. Can you imaging a future, where an initial RFP is put out in the form of test specs? That might provide a lot better basis for estimating, than the pages and pages of static docs that seem to still be getting cranked out inside large development organizations.

Have the requirements been changing? Just do a diff on the specs. There are so many advantages to this approach, that once you get the customer "spec-obsessed", just like developers who become "test-obsessed" the game is totally changed. You can't go back, at least willingly.

David talked about a couple of other tools that fit nicely in the RSpec stack. One very cool way to use it for full-stack integration testing is using Webrat. Another is doing true browser testing using Selenium-RC. David showed a demo of this, and remarked on how happy it made customers to be able to actually see their web site being tested, not just get the results back.

Note that RSpec is for use with Ruby, not just Rails, so if you are using Merb or Sinatra, or whatever, you can still write test specs using RSpec and StoryRunner.

Keynote - Kent Beck
During Chad Fowler's introduction, Chad said he was "like one of the superdelegates, getting to cast his vote about what we would get for presentations regardless of what we thought we wanted." All I can say is, Chad, keep it up. Some of these people might not get it, but if they continue to stay in this game long enough, they will.

At last, the moment I had been waiting for, and the biggest attraction for me personally in the RailsConf 2008 schedule: Kent Beck, one of the extremes, the Saturday night keynote.

Kent's presentation on the surface appeared highly informal, just a bunch of stories. It actually consisted of a serious of overlapping arcs on three topics that he knows a little about, because he helped create them: test-driven development, design patterns, and Extreme Programming. These stories took place over a 20 year timeline, stretching back to the origins of the work that the whole Rails community indirectly rides on today. This was before a bunch of RailsConf attendees were born, or at best were still in diapers.

I was pretty immersed in Kent's storytelling. I was too young to have been part of that original movement, but I'm old enough to want to learn all I can from the great masters, before they are all retired or just inaccessible. Amazing to have access to a couple of the extremes, like Ward Cunningham who was just hanging out on Day 1, at a fairly small conference like RailsConf, where most of the people do not even know who they are. For me, it is like a dream come true. If a bunch of other people are passing up the equivalent of intellectual caviar to stuff themselves with potato chips, well, good for them.

Enough ranting, back to Kent's "Tales to His Grandson". The unifying theme that emerged to me was that Kent is a true 60's radical who chose software as his weapon of choice to use in order to achieve social change. Like many great thinkers, Kent is a reader of books. Yes, those old things. Here are a few that he specifically mentioned that were instrumental in the development of his philosophies:

Christopher Alexander - "The Timeless Way Of Building"

Robert Mankoff - "The Naked Cartoonist: A New Way to Enhance Your Creativity"

Dr. Frank Luntz - "WORDS THAT WORK: IT'S NOT WHAT YOU SAY, IT'S WHAT PEOPLE HEAR"

The last question in the Q&A was really telling. When asked "do you think that you accomplished the political changes you set out to?" he responded that "No, he had not. And that he really did not fully understand why". Oh, those idealistic baby boomers! Kent, you may not have achieved all you set out to do directly, but the forces you set in motion are still going. Even though many of the attendees at RailsConf do not know you, they have grown up in a world where they expect more freedom and openness, and that is in part thanks to you, and fighting the good fight back then. Thanks, Kent, for staying true to the movement.

RailsConf - The Jam Session
After that very high level of intellectual brilliance, what could possibly top it off? How about a fantastic musical jam session. Chad had arranged the Musical BoF, and somehow Jim Weirich had talked a very generous and cool local music store into loaning him a perfectly playable acoustic guitar. Chad had his ukulele, Andrea Wright a recorder, and Dan Tripp, a portable folding guitar. I had brought a couple of my harmonicas with me, though not enough, of course. There were a bunch of other fun musical things, like a flute, and an impromptu Brazilian song or two from actual Brazilians. Not to mention a very creative approach toward an impromptu percussion setup by Forrest Chang.

Anyhow, great fun was had by all! We are already planning the RubyConf 2008 Jam, so it you will be coming, bring your ax! I will commit right now to the Dead Programmer Society helping sponsor having some instruments and/or PA system... plus some beer!

As I headed back to my room, I felt the smug contentment that comes from having had a really good time. Thanks everyone, for having made RailsConf 2008 "worth it" by the end of Day 2. Day 3 was still coming, and would have a few surprises of its own...

RailsConf 2008 - Day 1

2008-06-04T13:49:00.000-07:00

As RailsConf 2008 - Day 1 began, there was no avoiding a sense of just how large the community was growing. With over 2000 attendees, the physical presence was bigger than ever. But, unlike the first two conferences, the growth of the community had triggered inexorable changes. Unbeknownst to most attendees at RailsConf 2008, a rebel alliance had formed itself. The caboo.se crew, longtime Ruby on Rails committers, had decided to stick it to the MAN. Or as Courteney put it:

In May, if you don't want to cough up almost a thousand dollars to go learn things you already know, but want to come to Portland to network with skilled Rails coders from all over the world, instead of hanging out in the lobby at the Lloyd Center like I did last year, come to Caboose Conf.

The spawning impulse must have been too great for the cool kids. They were heavily conflicted when O'Reilly took over for RailsConf 2007. Not being one of them, I can only surmise how they felt. But no one likes to pay for something that once was free. And it is unpleasant to have a bunch of strangers running around what was once their private garden. But this was deeper, a sense that something had shifted, and would never return.

You can take the caboose out of the ghetto, but you cannot take the ghetto out of the caboose. Many people at RailsConf felt your absence, comrades, if any of you are reading this! As I mentioned earlier, the vast majority of attendees, including repeat ones, were totally unaware of the specifics. But they sensed something missing, some crucial element lacking.

At the same time, the community is now larger. And there are tremendous advantages to the incursions of the money people into the private reserve. The people who were saying to each other in 2006, "I wish I could program in Rails full-time, and quit programming in Java/C#/WTFLang" were now saying "I wish recruiters would stop harassing me!"

That all said, the Rails community needs the caboose crew, like we need our gall bladders. It is just a little organ compared to the overall size of the body, but without it, you will not survive long, and if you do, you will be feeling so well.

It would appear that the rebels had struck their first blow. But more about that would be revealed on Day 3.

Chad Fowler still had the his job leading off the proceedings in the new, more corporate RailsConf. As such, the ukelele would be banished from the main stage, along with any other musical instruments during presentations. Joey, we miss your accordion, too.

Chad mentioned that there were over ten times more proposals for RailsConf 2008, than attendees at RailsConf 2006. That certainly explains why my proposal was not accepted... of course, the title being "Pedal To The Metal: Alternatives To Rails" might also have had something to do with it.

Anyhow, Chad was almost apologetic as he explained how people were pissed off about RailsConf 2006 because it had sold out so quickly, and now that they had increased the size enough, people complained that the conference was "losing it's luster." You can't win, dude! But I still support what you're doing.

At that point, he introduced Joel Spolsky. Now, when I had read that Joel would be the opening keynote for RailsConf 2008, I had been fairly surprised. Although the penchant of the Ruby/Ruby on Rails community IS to reach out to others for ideas and technology, with the well-publicized spat between Joel and DHH, the choice seemed a curious one.

Joel was funny, and entertaining. He had slides, and seemed like he was leading up to some big point. But after the dog and pony show was over, I felt like I had drunk an extra-large zero calorie soda: a feeling of fullness, but no actual nutritional value to it. I was not exactly disappointed, but I was far from impressed.

Next, I headed into the Entrepreneurs on Rails session by Dan Benjamin. The presentation was probably worthwhile if you have never formed a business before, but my attention wandered, which is why I will not be writing anything else about it here.

So far, I was zero for two on my personal "RailsConf 2008 was worth it" scale. Hmmm.

I got a bit more enthused at Micah Martin's session "10 Things I Hate About Web Development", where I finally got to see the long awaited demo of Limelight, which Micah had been dropping hints about at RubyConf last year. I have a separate posting I will put up with info on that session.

After that, with my energy picking up, I went to "Rubinius: Under The Hood and Behind the Curtain" presented by Evan Phoenix, Brian Ford, & Wilson Bilkovich. This was a fun and informative session about the current state of Rubinius, although if you have been following the various blogs/IRC/github you already knew what he was talking about. Still, it was nice to see the core Rubinius team in person, and to get the sense of the crowd's excitement about Rubinius. I will be posting separately about this session.

There was a great moment that is really representative of what I get out of RailsConf. Walking between sessions, I ran into Ward Cunningham. Not only did I get to introduce myself, and take another fanboy picture with one of the extremes, but we also stood around for a few moments and chatted about his experiences migrating MediaWiki to Rails.

Afterwards, I hung out doing a little pairing with programming buddy/collaborator Ari Lerner trying to help out with pool-party before his upcoming BoF about it later that night. I know you have not yet heard about pool-party, which is an auto-scaling solution for Amazon's EC2 service, since the first version was put up on github there at RailsConf. What makes pool-party unique is that it is a pure Ruby and open source solution to the cloud computing thing. Plus it is cool.

By the time we were done, we had already missed any chance of eating dinner, and also most of Charles Nutter's talk. I walking in to see a slide "You Are The Enterprise" from his presentation. That sounded silly to me, but without any context, I probably do not know what I am talking about here.

Next were the RailsEnvy guys presenting the "Ruby Heroes Awards". I have to say, those guys from RailsEnvy are pretty funny... and brilliant self-promoters. Plenty of people have already blogged about this, so no need to go on and on about it here, except to say they made some pretty good choices with the winners:
Evan Weaver, Tom Copeland, James Edward Gray II, Ilya Grigorik, Yehuda Katz, and Ryan Bates.

Good job, guys!

Then it was time for the big David Heinemeier Hansson keynote. After my tepid enthusiasm for the opening keynote in the morning, I hoped DHH would do something exciting. Well, that was not exactly what happened... but thought provoking, nonetheless.

His speech was entitled "The Great Surplus". The salient point seemed to be that the highly leveraged productivity that Ruby on Rails gives the community (and the unreal demand) will not last forever, or perhaps for even that much longer.

His prediction is that one of three things will happen:
1. The mainstream copies Rails with something else
2. Some other dramatic alternative arrives (new paradigm) that provides massive improvement
3. Rails itself becomes mainstream

So what to do about this? Invest in yourself. Regardless what happens with Ruby on Rails, if you better yourself it will last to next wave. And do something else beside programming! You need to train more than 1 muscle... same with non programming activities.. need to engage other parts of brain. And sleep more.

It is well known that tired people take longer to do work, and that the output is not as good as a result. More importantly, it takes a toll on that individual that leads to a longer term reduction of output. So why do we consistently see investors/executives pushing their technology teams this way. And why do we allow it, when we already know better?

I had seen a friend earlier in the day during the conference, who I had not seen since last year. His startup had become involved with a well-known incubator. But far from looking energized, my friend looked worn down. "I need a month with no electricity," he croaked. And this is where many, many people in the RoR community are actually hoping and praying to get to. This is what DHH was warning about.

After DHH finished his keynote, I ran over to the BoF for pool-party. The turnout was actually pretty good! Ari did a great job with his presentation, and at the end a guy from Amazon came up and chatted us up for quite a while. I look forward to switching over my AWS application to the pool-party as soon as it is ready!

My day complete, I crawled off to bed to take DHH's advice and rest up for RailsConf 2008 - Day 2.

Going Public: From Los Angeles To RailsConf 2008

2008-06-04T11:31:00.000-07:00

With the high cost (and carbon!) of transportation, as well as the burden of the increased time it takes to fight your way through the congestion, we all need to change the way that we navigate thru our daily lives. I had decided to take public transportation to get from my house in Los Angeles, to RailsConf 2008 in Portland, OR.

When I first mentioned my plans, several friends and family members were incredulous. "It takes too long, " stated one person. "There's no way to get to airport in LA except by car, " scoffed another. I set out to prove them wrong, and by doing so began a new chapter in how I approach personal transportation.

I set out from my house in Silver Lake, walking down the hill to Sunset Blvd. I stopped at the delightful Cuban coffeehouse Cafe Tropical, which happened to be directly adjacent to the bus stop I would need to use.

I sipped a tasty cup and snacked on one of the delicious pastries while I awaited my first bus.

I jumped quickly (literally, they do not wait around) on the Route 704, which is a giant segmented affair. Despite that, the bus was almost completely filled until we reached the Evans Community Adult School (no relation), at which point almost the entire bus emptied out.

One notable feature of the ride were the video screens filled with advertising prominently situated on the bus, an attempt to subsidize the cost of the public transportation system here in L.A.

On reaching the Union Station stop, I exited the bus. I would need to walk to the FlyAway bus terminal to catch the non-stop shuttle to LAX. The way was clearly marked.

They wanted to make sure you could not make a wrong turn!

I purchased my ticket at a friendly kiosk.

The bus arrived shortly thereafter.

While I waited to board, I asked a half-dozen of my fellow travelers if they had taken the FlyAway shuttle before. All except one were first-timers, like myself.

We reached the airport quickly from the downtown Union Station. No having to drive, allowed me to check my email (GMail Mobile is still cool on my cheap phone) and daydream. Suddenly we had arrived at LAX. I noticed that my airline was in the very last terminal, and since the FlyAway shuttle would stop at each and every terminal first, I decided to get off there and cross the airport on foot.

It would seem there is plenty of parking, and available charging terminals, if you have a plug-in electric car.

I had made it to the airport in about 1 hour and 15 minutes, which was about the same amount of time that it would have taken me to drive my car to the airport parking, park, get on a shuttle bus to the airport from the parking location. Except my cost for transportation had been $5.25 instead of the fuel, plus the $60-80 for parking. With a small portion of my savings, I smugly purchased a copy of Predictably Irrational at an overpriced airport bookstore to read on the flight.

On exiting my plane in Portland, it turned out they were having a party to celebrate Portland having now become an international airport.

Politely refusing a multi-colored cupcake, I walked to the end of the terminal where the MAX, Portland's modern light rail system awaited.

I purchased a ticket from a automated kiosk, then hopped on the train. Around 15 minutes later, I had arrived at my destination.

I had successfully arrived in Portland for RailsConf 2008 only using public transportation! After checking into my hotel, I dashed across the street to the convention center. Alas, the registration had closed for the day exactly on time. Yes, these O'Reilly people are all business. More on that in my post on RailsConf 2008 - Day 1...

My luck would turn around quickly. I immediately ran into friends David Chelimsky and Micah Martin, who invited me to tag along for dinner. When you're hanging out with guys as smart as these, you absolutely cannot be bored, and you will learn a LOT! Plus, one thing about Chicagoans, we know how to pick a good place to eat. David led us on a brief journey to a great Thai place he knew of, and between the great food, beer, and conversation, we had a delightful time. Thanks guys, I appreciate it!

Converting The US To Electric Vehicles: Priceless

2008-05-27T18:40:00.000-07:00

A non-scientific, but clearly well thought-out back-of-envelope calculation by MIT CS professor Philip Greenspun is food for thought: the cost of replacing the entire fleet of U.S. passenger cars with electric ones would be zero.

Here are some of his numbers:

* total spent per year: $1 trillion
* percentage of oil consumed by passenger cars: 40
* total spent per year on oil for passenger cars: $400 billion [refining into gasoline, distributing, and retailing add even more to this]
* at 5 interest, how much we could we borrow and pay $400 billion every year in interest: $8 trillion
* number of registered cars in the U.S.: 250 million (Wikipedia)
* cost of a new electric car, if mass-produced: $20,000
* value of a used car, if exported to Latin America or China: $5,000
* cost to upgrade average existing American car to a brand-new electric car: $15,000
* number that could be converted for $8 trillion: more than 500 million cars (i.e., twice as many as we have now)

And then here is the money quote:

Instead of sending $400 billion each year to countries such as Saudi Arabia and Venezuela, we could spend it on electric car production in the U.S., Mexico, and China. At current oil prices, it wouldn’t cost us a dime extra to stop importing and burning oil for passenger cars. In fact, if the goal were to end up with the same number of cars on the road, we would have a few trillion dollars left over. One or two trillion dollars would be sufficient to build nuclear, solar, or wind electric power plants to replace all of our plants that currently burn coal and oil (note that less than 1 percent of current electricity generation in the U.S. is from oil).

Now THAT is what I talking about! Let's get on it people... and let's do better than just paying interest only on the bill while we are at it.

Zero Is Special

2008-05-27T10:00:00.000-07:00

I just read a fascinating academic paper co-authored by Dan Ariely, author of new book Predicably Irrational, a book I have heard about, but have not yet read.

The paper presents results from a series of experiments into the nature of free pricing. Here is a juicy tidbit from the abstract:

When faced with a choice of selecting one of several available products (or possibly buying nothing), according to standard theoretical perspectives, people will choose the option with the highest cost–benefit difference. However, we propose that decisions about free (zero price) products differ, in that people do not simply subtract costs from benefits but instead they perceive the benefits associated with free products as higher.

Yes, you read that correctly: people can consider something free as more valuable than something that costs money. The authors describe several interesting experiments using chocolate that appear to well prove their point.

Bottom line, so to speak: I need to get this book and read it as soon as I can! Even if it does cost money... actually here are some free excerpts.

RailsConf 2008 Better Be Good

2008-05-27T08:23:00.000-07:00

RailsConf 2008 is later this week, as if you all did not know that. Interestingly, this year it has not sold out like it did for 1.0 and 2.0. Increased capacity by O'Reilly? Or perhaps something else at work here...

If you are not aware for the cost of transport, you have been living on another planet. Just the massive increase in time required to make the pilgrimage has been substantial. Many of us have been working at home or within other time-flexible, location-unimportant arrangements for some time now. But we still gathered together unhesitatingly no matter the distance.

Two recent blog postings stood out to me. The first, from Seth Godin said it nicely:

If oil is $130 a barrel and if security adds two or three hours to a trip and if people are doing more and more business with those far afield...

and if we need to bring together more people from more places when we get together...

and if the alternatives, like video conferencing or threaded online conversations continue to get better and better, then...

I think the standard for a great meeting or a terrific conference has changed.

In other words, "I flew all the way here for this?" is going to be far more common than it used to be.

The second was simply an link from from Rails community spokesmodel Obie Fernandez to an article in the Washington Post:

But the truth is that no combination of solar, wind and nuclear power, ethanol, biodiesel, tar sands and used French-fry oil will allow us to power Wal-Mart, Disney World and the interstate highway system -- or even a fraction of these things -- in the future. We have to make other arrangements.

The public, and especially the mainstream media, misunderstands the "peak oil" story. It's not about running out of oil. It's about the instabilities that will shake the complex systems of daily life as soon as the global demand for oil exceeds the global supply.

Already, I was a little bummed out about not being able to eat a delicious Pacific salmon dinner this year. But these two postings really got me to thinking. It would be easy to expect the conference organizers to do all the work. But it is really up to us as attendees to put our hearts into it. Whether it be code drives, or Werewolf sessions, that does not really matter to me personally, although I will be attending the former.

So, all I can say is, this better be good people. We better go up there and really make this thing count, to justify the time, expense, and environmental impact. The times they are a changing, and we need to do something special.

As for me, I will be taking public transport to and from the conference... which is not easy to achieve from down here in L.A.

The Phoenix Has Landed

2008-05-25T16:58:00.000-07:00

Earthlings, we have successfully landed another probe on the red planet. Congratulations to the team at JPL, University of Arizona, and everyone else who made this one happen!

Man, I just love space exploration...

Got An Eye On You

2008-05-22T07:40:00.000-07:00

Read this piece from Rolling Stone magazine on 24-7 surveillance coming soon to a neighborhood near you.

Then be afraid. Very afraid.

In Search Of Search

2008-05-19T15:09:00.000-07:00

My client was having troubles. The site was crashing, seemingly at random. QA was baffled, management concerned. After hunting thru logs, the culprit was found: the ferret search service.

Despite having configured acts_as_ferret to use the DRb server option, the "whole thing" would fall apart on a regular basis whenever the ferret daemon would crash... which was pretty often. The ruby on rails plugin acts_as_ferret would revert to a local mode of operation, which would only make matters worse when competing mongrels would corrupt the index files on disk.

Much has been written already about this by more famous Rubyists then myself, so the time had come to make the switch to Sphinx like all the rest of the cool kids. However, there were a few tricky issues that awaited.

One is that the client site was using PostgreSQL instead of the more common MySQL. This meant you had to have already downloaded the pgsql client libs, and also when compiling the sphinx server use the --with-pgsql flag. However, what they didn't mention was that you would need to also download and include the MySQL client libs AS WELL, even if you were not using MySQL. Good to know... seems like a big of baggage to carry, but whatever.

That was one obstacle avoided. The next would prove to be more subtle and complex. There are four different choices of Ruby on Rails clients for Sphinx:

acts_as_sphinx

Sphincter

UltraSphinx

Thinking Sphinx

I looked briefly at acts_as_sphinx, but my superficial prejudices against anything named "acts_as_" caused me to continue my search for search. Next, I tried Sphincter, just to prove that I could be accepting about project names. However, I was not successful as getting it working correctly, and project development seemed like it had slowed to near non-existence. The search moved on.

My search was getting desperate as half of the options had been eliminated. I briefly perused UltraSphinx and I started to get a little excited. Development was active, and proper functionality with Postgres was claimed. But there was just one little problem: this client's site is still Rails 1.2.x.

Now, before you all start throwing various disparaging comments my way, consider how much work some clients are willing to pay far vs. something that they can actually see. Multiply that by how long since they started their project, and you are starting to get the idea of where this thing is at. I probably should have MADE them upgrade at that point, but I'm just not that kind of guy.

So that left Thinking Sphinx. I dove in with the manic desperation of someone looking to avoid major amounts of work. That rarely works out well, but in this case it wasn't so bad. The latest trunk of thinking sphinx had comments that said "basic postgres functionality" so perhaps it was possible after all.

Several hours and some major hacking of the original plug-in to actually handle proper psql syntax later, and something was working! I will try to come up with patch to thinking-sphinx for freelancing-god, cause his work was a great help.

Lastly, the delta index. By adding a single field (boolean for MySQL, integer for Postgres) you can remove the need to reindex all data every time a record is updated. Sphinx allows storing the changed records in a separate index (delta) that it will search along with the main index. This way, you do not have to wait for reindexing to be able to search, but you also do not have to reindex the entire data set every time any data within it is changed. You still need to do a complete reindex, but that can occur at off-peak hours, instead of constantly.

The code was deployed, and now all was well with the world. My search was complete.

Ain't No Stopping Us Now

2008-05-13T21:05:00.000-07:00

According to a recent eWeek article, there will be four times more Ruby programmers within five years. Before all the haters start making cracks about it going from one thousand to four thousand professional Rubyists, here is a juicy bit from Mark Driver, Gartner analyst:

"There are under one million professional Ruby developers now and we're projecting there will be four million plus by 2013."

And oh yeah, what about that whole "Ruby is for hobbyists" misinformation campaign? Officially debunked by Gartner:

Driver later told eWEEK that Gartner's research shows "strong interest" in Ruby and that the percentage of developers that will be creating commercial systems versus those that are hobbyists will be even greater for Ruby than for other languages. "Ruby will be more interesting to commercial developers"

Just imagine the power of one million people out there, right now. Developing, sharing, recruiting others... to use their favorite programming language! What did you think I was talking about, politics?

Modern Web Design

2008-05-02T14:29:00.000-07:00

This one pie chart says it all...

From ChangeTheThought.com

Goldilocks and The Three Icons

2008-04-23T18:50:00.000-07:00

Once upon a time, there were three icons. A Poppa Application icon, a Momma Folder icon and a Baby Document icon. Life was good, and people were easily able to tell the difference between them, although very few visitors were known to go to that part of the forest.

But then along came the Trashcan icon... and the dark times began. If you dragged a folder to the trash it went away back into the void. But if you dragged a Special kind of folder called a Disk icon, it was EJECTED. The trash had discovered "Free will" and their eden was never the same again.

Suddenly, there were icons everywhere. Icons here, icons there. And all different kinds of icons, too. The icons not only overran the desktop, but they began to appear inside of applications. And they did so many different things! Where the poppa had once used WORDS to describe actions by way of MENUS, the icons stubbornly sat there silently.

So then along came Goldilocks. She had wandered onto the desktop innocently enough, but suddenly was propelled into an incomprehensible world of mysterious iconography... in other words, she couldn't make heads or tails of the icons by which she was surrounded. The only icon she thought she recognized was that pesky trashcan that had started all the trouble, and she figured she wanted to stay away from that one.

It was around this time that poor Goldilocks ran into the original Three Icons. "How could you have let this happen!" raged the girl. "It's not our fault, " moaned the icons. "We were never supposed to be a replacement for anything, we were just another way of looking at things!" they grumbled.

"100% data compression means 0% information," howled the Baby Document. "That was the beauty of it. Don't you get it, 0% information. The trashcan is behind all of this. It was the trashcan!"

Just then the trashcan showed up. "Hey kid, don't believe the hype. I deal with all of the things that no one wants, and look how they treat me."

Goldilocks was impressed by the smooth animation of the trashcan, as well as its nice 3-D appearance. "Well, I wouldn't quite put it that way," she said politely. The Trashcan didn't seem at all like she had expected.

"You wouldn't believe the things that people throw away. I'm talking some good stuff here!" said the Trashcan in a friendly tone. "Why just take a look at this..."

Goldilocks' curiosity was piqued. She leaned closer to take a look, and before the warning cries of the Three Icons could reach her, she was swallowed up, and then before you could say "Empty Trash" the poor thing was gone.

"How could you have deleted that poor innocent sweet girl!" sobbed the Momma Folder. "What girl?" burped the Trashcan.

Later, the Poppa Application was alone with the Momma Folder. "Don't worry, baby," he smiled. "I have a backup and restore function. If I can just figure out which icon it is..."

The End?

April Is For AOL

2008-04-23T08:38:00.001-07:00

Last night was the always fun Los Angeles web application developer Meetup, this time at AOL in Beverly Hills. After navigating a veritable maze of cubes, we found ourselves in a fairly large screening room chock full of AV equipment, accompanied by a large pile of pizza boxes. A few moments later, the throng of LA-based developers shuffled in, ready for the latest and greatest. The prerequisites fulfilled, we got down to some serious techno-babble.

First up was Omer Singer from DigiTrust, talking about Web Application Firewalls. Although it strayed a little closely to a sales pitch, Omer was able to tie it together with his definition of "defense in depth", which is simply enough having multiple layers of overlapping protection. One good point that he raised which is often overlooked is the cost of "brand-damage" in the marketplace from a web application being hacked. There were several good security related question asked afterwards, so yes the crown was paying close attention.

Next, was Mitch Hashimoto recent hire at CitrusByte, with a brief overview of the newly beta released Google App Engine. Mitch took us on a brief tour of the main capabilities of the Python-based system, like a Datastore API (based on BigTable), and the Users API (based on the Goog's own account authentication database). If you like Python, and don't mind doing everything the Google Way and no other, then the App Engine might be for you.

After Mitch, I did a presentation about Frankie, my newly released Ruby gem for easy development of Facebook applications. If you don't yet know, Frankie uses the Sinatra web server and the Facebooker gem to allow you to create a basic Facebook application with only 10 lines of Ruby. I also did a brief demo of a new, super-secret web thing that I have been working on for a little while... more will be revealed soon!

Last up was programming chum and Ruby expert Ari Lerner, talking about Amazon's EC2 service, and showing a little of another upcoming pet project of ours called "Pool Party", which makes it easy to deploy an auto-scaling application to the EC2 cloud.

It was a really great meetup, and the turnout was really good. Thanks to the persistent efforts of organizer Will Jessup, the community in LA is really thriving, and we were even able to keep the annoyance of non-technical recruiter sales pitches away. Great job, everyone!

The Archimedes Codex

2008-04-17T07:51:00.000-07:00

Archimedes was a very cool dude. When you think of inventor/intellectuals these days, we have been trained to imagine a fairly wimpy, bookish kind of person. Not so for Archie, he was also a bad-ass who defended his beloved city of Syracuse to the death at the hands of Roman invaders. Archimedes invented what for the day, were some freaky advanced technologies.

Well, new information has been discovered about him, thanks to the painstaking work by Will Noel at the Walters Art Museum of New York. A dilapidated tome that was recycled by a 13th century monk, has turned out to contain a treasure trove of amazing information regarding this famous but actually little-known giant.

The Christian Science Monitor has a fascinating article about it. Here is a taste:

Most significant among the discoveries was the knowledge that "Archimedes was the first to calculate with actual infinity in the mathematics of the West." That is to say, he was operating at an intellectual level that didn't become common in the mathematical world until the 17th century, nearly 2,000 years after his time. The Archimedean texts, Noel writes, make the mathematics of Leonardo da Vinci "look like child's play."

The CS Monitor article is fun, and a short read. I really recommend it, if you want to get a sense of the "Original Hacker" was all about.

Frankie Goes To Facebook

2008-04-08T22:04:00.000-07:00

UPDATED 4/12/2008: Now compatible with Sinatra 0.2.0
So you want to create a small Facebook application... seems like it should be a small thing, right? But creating an entire Ruby on Rails application just for a tiny little Facebook application is, at the very least, a bit wasteful. In the case of a overly popular Facebook app you could end up with, as Marc Andreessen put it, a "self-inflicted denial of service attack", unless you have both a pretty serious infrastructure to support it, as well as lots of cash to keep that data center running.

Wouldn't you rather be able to create a highly scalable "hello, world" Facebook application in around 13 lines of Ruby code? Say hello to Frankie:

require 'rubygems'
require 'frankie'

configure do
  set_option :sessions, true
  load_facebook_config "./config/facebooker.yml", Sinatra.env
end

## facebooker helpers
before do
  ensure_authenticated_to_facebook
  ensure_application_is_installed_by_facebook_user
end

## the site
get '/' do
  body "<h1>Hello #{session['facebook_session'].user.name} and welcome to frankie!</h1>"
end

Frankie is a plugin for the minimalist and very fast Sinatra web framework that allows you to easily create a Facebook application by using the Facebooker gem. Why would you want to use it? If your Facebook application needs to be highly scalable, is fairly small, or is really a mashup of other web-available resources, than Frankie could be a good solution.

Frankie is available now for your enjoyment. Here is how to get started:

- Install the Frankie gem, which will install the Sinatra and Facebooker gems if you do not already have them.

sudo gem install frankie

- Create the application directories for your new app:

mkdir myapp
cd myapp
mkdir config

- Put your facebooker.yml file into the /myapp/config directory, and set the values to your information. Here is a simple example of the file:


-------
development:
 api_key: apikeyhere 
 secret_key: secretkeyhere
 canvas_page_name: yourcanvashere    
 callback_url: http://localhost:4567
test:
 api_key: apikeyhere 
 secret_key: secretkeyhere
 canvas_page_name: yourcanvashere    
 callback_url: http://localhost:4567
production:
 api_key: apikeyhere 
 secret_key: secretkeyhere
 canvas_page_name: yourcanvashere    
 callback_url: http://yourrealserver.com

- Make sure you have setup your facebook application on the facebook site. Google "setup new facebook application" if you are unsure how to do this. I recommend starting with an IFrame application, so that you can point a development version of your Facebook application to your local machine. This, like our example here, would use "http://localhost:4567" as the callback URL when configuring the Facebook app.

- Create your application, based on the sample above, and run it:

ruby sample.rb

This will start the Sinatra Ruby web server running:

== Sinatra has taken the stage on port 4567!

- Test it by pointing your browser to http://apps.facebook.com/yourappname. If you have things setup correctly then you should see your application appear inside of Facebook's site.

Facebook is now your playground... have fun with Frankie!

Excuse Me, While I Kiss The Sky

2008-03-13T17:42:00.000-07:00

The demo that Microsoft showed at TED of Robert Scoble crying, ahem, I mean their future Microsoft Worldwide Telescope or whatever, looks really neat. In the meantime, those provocateurs over at Google have just put live Google Sky, a fully browser based version of their Google Earth software that started this whole "star wars" virtual astronomy competition thing in the first place.

I suppose since so many people cannot see the stars anymore, that we need virtual astronomy more than ever. In the meantime, it is pretty incredible to have such an educated introduction to the heavens. I could easily spend hours just like, playing with the universe, man.

The U.S. Navy Goes Open Source

2008-03-13T12:30:00.000-07:00

That unwitting center of global innovation known as the United States Navy has made a bold announcement: from now on, they are only going to accept open source.

I just read about this on CNET, where they were linking to a recent story in Federal Computer Week. When I read this, I just about fell out of my seat. According to Vice Adm. Mark Edwards, deputy chief of naval operations for communications:

“The days of proprietary technology must come to an end,” he said. “We will no longer accept systems that couple hardware, software and data.”

Really. I am not making this up. Here is some more juicy stuff:

“We can’t accept the increasing costs of maintaining our present-day capabilities,” Edwards said. “In the civilian marketplace, it’s just the opposite. Some private-sector concerns are cutting their costs by 90 percent while expanding their performance.”

Edwards noted that the Navy has cut the number of databases and applications it maintains and has reduced its networks by 40 percent. “But it is not enough,” he added. “We would have to double our IT budget over the next several years just to run in place.”

By using an open network architecture, the Navy could rapidly upgrade its capabilities and handle increases for demand, Edwards said.

So what will be next? The U.S. Navy vs. The US Patent and Trademark Office? Wow, talk about getting bogged down in a conflict! Still, my money is on the Navy... I played on a soccer team with a Navy SEAL and a couple of Marines (also not kidding) and those guys really ARE tough!

The Telepathy Machine Demo

2008-03-13T11:00:00.000-07:00

I just saw an amazing demo of communication without speaking. No, not instant messaging, or sign language. Instead, scientists at Ambient Corporation have succeeded in tapping into the nervous system that control speech, and translate that into digitized speech. Michael Callahan showed off their demo at the Texas Instruments developer conference. Wow, this is a long way from my TI-99/4!

Applications for secret agents, people who suffer from ALS, and people who cannot shut up when on the subway will be forthcoming. The quality is "at the same level of recognition as the early days of speech to text", so it will be a little while till all those people staring off into space at cafes are actually talking to someone else.

In the meantime, I would like to scan all of you in this room one at a time...

Merb On The Virge

2008-03-05T15:17:00.000-08:00

Merb, the mighty challenger to the current Rails domination of the Ruby web framework conversation, races towards "getting very close to being stable api wise". That sounds like a 1.0 release to me!

Ezra says that the 0.9.1 developer release is ready to take thru its paces. One area where Merb really differs from Ruby on Rails is how Merb is based off of RubyGems. The refactored Merb architecture takes the Merb modularity to a massive 16 gems. You can really pick and choose what you want to include in your application.

Luckily for the very lazy, or the undecided, there is a Merb gem that will install everything for you. All you have to do is:


sudo gem install merb -y --source http://merbivore.com

And away you go! Unfortunately, "you should uninstall all of your old Merb gems before you install the new version" but that is not so bad. If you have been Merbing for while, you are not afraid to get your hands a little dirty. And if not, Merb 1.0 (dare I say it?) is right around the corner.

Thanks For Paying Attention

2008-03-01T16:51:00.000-08:00

This last week has brought some very nice and favorable attention to the Dead Programmer Society.

The ever-popular "I'd Rather be A Jazz Programmer" post popped up again, this time in the stodgy old school publication Computerworld. They ran an article on the "Rock Star Programmer" meme, and decided that my little post was part of the "Kill Rock Stars" contrarian perspective they were looking for. Thanks for making me look so good, guys!

Actually it really is pretty amazing to make an article that also mentions Linus Torvalds, Joel Spolsky, Paul Graham, Obie Fernandez, Zed Shaw, and "why the lucky" (sic). Hey, I told you Computerworld was stodgy. Sorry they mangled your handle, why!

On to more recent work. Ari Lerner and I have been getting some good attention to our introduction of the EC2 ProcessorPool gem. We managed to get covered in both Rails Envy and Ruby Inside this last week. Thank you to the editors for helping us let people know about this quite useful tool.

Anyhow, that is about enough self-congratulation for one post. Next time, back to our irregularly scheduled programming.

Rush The Shell

2008-02-20T14:20:00.000-08:00

I just read the latest posting from Adam Wiggins, very cool dude, hard-core Rubyist, and friend. He has a new project called Rush, the Ruby shell. Adam is one of the guys behind Heroku, which is a pretty amazing pure web browser-based IDE/deployment/hosting solution for Ruby on Rails, if you haven't seen that yet.

Anyhow, like many of us, Adam seems like he just wants to use Ruby for everything. And when I say everything, I mean EVERYTHING. Writing bash scripts.... sucks. Destroying the sanctity of your Ruby code with

`bash-shell-command /ugly /madness`

is not hardly any better. My man is trying to help us do away with all that, by applying a friendly familiar Rubyness, not just caging the ugliness away.

Here is a little taste of Rush, from the Rush site:

How about killing those pesky stray mongrels? Before:
kill `ps aux | grep mongrel_rails | grep -v grep | cut -c 10-20`
After:
processes.each { |p| p.kill if p.command == "mongrel_rails" }
But rush is more than just an interactive shell and a library: it can also control any number of remote machines from a single location. Copy files or directories between servers as seamlessly as if it was all local. bash and ssh, we love you, but your era is past.

Example of remote access:
local = Rush::Box.new('localhost') 
remote = Rush::Box.new('my.remote.server.com')
local_dir = local['/Users/adam/myproj/'] 
remote_dir = remote['/home/myproj/app/']
local_dir.copy_to remote_dir 
remote_dir['**/.svn/'].each { |d| d.destroy }

I have not had a chance to play with Rush yet, but I'm sure I will be. Unix shell scripting is not syntactical sugar, that is for sure, and Rush looks pretty sweet.

Everybody Into The Pool

2008-02-20T11:18:00.000-08:00

A few weeks back, I indulged my current obsession with grid computing, and Amazon's EC2 and S3 services in particular, with a lengthy post. I mentioned how friend Ari Lerner and I had built a really neat solution, and were going to put all of our processing farm goodness into a nice neat gemified package. Well, mostly thanks to Ari, we now have a new Ruby gem called ProcessorPool that uses EC2 and S3 to do all the hard work for you.

Based on Sinatra, with a bunch of S3 goodness mixed in, the ProcessorPool gem makes it staggeringly easy to exploit the power of the EC2 grid. More info about it is available at the Blog @ CitrusByte, and you can also check it out on RubyForge here.

All kinds of fun applications await you in the pool... image processing, video/audio transcoding, heavy computational stuff, long crawls of content... and it is fun to delegate to a pool of worker machines in the EC2 cloud.

Come on in... the pool is fine.

Getting To Usable

2008-02-19T07:52:00.000-08:00

I read a great posting last night from Matt Rogers, founder of startup Aroxo, on a strategy for usability testing. Suddenly, I was flashing back a few years to the early salad days of web usability...

When I first read Steve Krug's tiny masterwork "Don't Make Me Think", I was very excited; at last, a nice practical approach! After the pedantic and overly academic work of guys like Jakob Neilson, Krug's book hit me like Dr. Seuss taking out Dick, Jane, and Spot.

Somehow, Rogers' post reminded me of that sudden flash of clarity in the fog. Yes, we can DO this! Gentlemen and ladies, start your testing...