Thursday, November 13, 2008

RubyConf 2008 - Day 1 - Testing Heresies - Francis Hwang

The sessions for RubyConf 2008 day one were nearly over, when I attending this highly entertaining and informative one. I had missed the chance to see a previous presentation by Francis Hwang, and being as test-obsessed as I am, I was not going to make the same mistake twice.

Many people are confused over mocks, stubs, and other fakes. Here are the real definitions.

Gerard Meszaros - Test Doubles
- dummies - passed around but never used
- fakes - work but unstable in production
- stubs - provide canned answers to calls made during test
- mocks - mostly concerned with surface interaction

Classical TDD use real objects.
Mockist TDD will always use a mock for any object with interesting behavior

Problem #1 with mocks
Cannot figure out problem and solution at same time

Problem #2 with mocks
Bugs can slip between layers

Francis Hwang, TDD Clasiscist
- test setup creates records in the db
- no sqlite in test
- when you use test doubles, it's always a big deal

Real world example #1
Ensure that an optimization is actually being used
Redefine the actual surface interaction to 'scream' on error

Real world example #2
Faking a real web service (not written by us)
Best to use stub, to duplicate functionality
Need test, stub, AND test for stub

Dogma - if you tests do not run quickly, there's something wrong with you
Heresy - I'd like my test to run 30 sec. I'd also like a pony

Francis does not use autotest, because he does not want to run all of the tests every time he changes something. Eric Hodel chimed in from the audience says that it can run only what changed. Only if it passes, will it run ALL tests. Me, I loves my autotest... with growl notifications, it's all good.

devver
They say it is a ec2 based distributed test system...

testjour
uses bonjour to trigger a farm of machines in your LAN to perform distributed tests

Dogma - you code should always be well-structured
Heresy - under-structuring can be freeing

Uses example of constantly cleaning small NYC apt. It was living on a boat, where you spent all your time putting things away.

Dogma - Fine grained focus leads to high quality code
Heresy - Never forget the unknown unknowns

System design 101
How likely fail
What will fail look like
How can I find out quickly
How can I limit damage

Dogma - testing makes my job easier
Heresy - testing makes it easier for others to do my job - unless my job keeps growing

Rails has made it so simple to implement simple business logic, so way more is expected now.

Tuesday, November 11, 2008

RubyConf 2008 - Day 1 - rush, a shell that will yield to you - Nicholas Schlueter

I had previously blogged a bit about Rush, the Ruby shell. Well, now Rush must now be its own force to be reckoned with, because this presentation was NOT by Adam Wiggins, the creator of Rush, but was instead by Nicholas Schlueter, the creator of RushMate. If you've not heard of it, RushMate allows you to use Rush from within TextMate, which translates to being able to write custom commands much more easily.

He showed some neat examples, including one that auto-generates an HTML page with links to all of the RDocs for all of the RubyGems you have installed locally. I have wanted something like that for times that I was not able to connect to the InterTubes, but still wanted to do some development.

The entirety of his presentation is online here with a whole bunch of code examples that will be better than my ramblings, so you should rush over and check it out.

Monday, November 10, 2008

RubyConf 2008 - Rubinius - Evan Phoenix

The Ruby VM wonkiness continued at RubyConf 2008 - Day 1. Next up we heard the "State of Rubinius" address on the status of the anointed heir to the Ruby federation. Evan Phoenix is a fun guy, and it proved to be a both informative and enjoyable session.

Evan says Rubinius is "meta-circular-ish", and slow, but getting faster. They have a new VM that is written in C++, which is a sharp tool, and as such, it is possible to poke your eye out. However, he says it allows them to better model work. The three main advantages are:

- type safety
- organization
- architecture

What is type safety, may ask many Rubyists. Well, sometimes a duck actually IS a duck. The Rubinius code just feels better with OO nature of C++, as opposed to just plain C. Also, it is possible to make the VM class hierarchy that mirrors the Ruby class hierarchy.

Exporting Methods - a day in the life of a primitive operation

- things you cannot do in Ruby
- small example


class String : Object {
Fixnum* size();
}

class String
def size
Ruby.primitive :string_size
end
end


There was interesting question about why flatten out ala :string_size aka no namespacing. Evan did not seem to give a definitive answer...

Method Dispatch - From resolution to execution

Resolution
- going from receiver and method name to actual method
- three mechanisms
- hierarchical lookup
- global cache
- inline cache

Execution
- every method provide an execute function pointer
- each primitive is an executor
- Ruby methods use executor specialization

Ruby Methods
- Specialized executors based on argument info
- Very simple fast version for majority of cases
- Slower fallback case for all cases

Critical Path
- Populate message object
- Call rep over trampoline
- Fill in more of message object
- Call method executor

Execution/Framing
- Method context objects store info about each method
- Chained together using refs (spaghetti stack)
- MethodContext creation speed is crucial

- Default contexts are allocated in special section of memory
- Normal execution follows simple stack pattern
- We can exploit to allow for fast creation and deallocation

Calling to C
Can you imaging life without ruby gems that use C extensions? For example:
- hpricot
- mongrel
- mysql
- sqlite
Not a problem. A simple recompile == usable in Rubinius

Tricky
- GC interaction requires indirection
- Uses 'quantum leap" stack jumping
- Allows for semi-graceful recovery from extension segfaults

Performance
Rubinius is doing well on micro-benchmarks
- method dispatch
- GC
- OK on macro-benchmarks
- Poorly on mega-benchmarks

Solutions
- LLVM
- JIT
- AOT
- Improve algorithmic efficiency

Rubinius is concentrating on compatibility right now, but will then address performance. Just like Koichi-san said, it is a curve of improvements.

Evan ended up with a nice quote from Woody Allen - "If you're not failing every now and again, it's a sign you're not doing anything very innovative."

RubyConf 2008 - Future of RubyVM - Koichi Sasada

The first session I attended at RubyConf 2008 was called "Future of RubyVM", and it began my ascent into "VM Wonkiness" as I started to call it. Koichi-san is responsible for leading the team of Ruby 1.9 implementers, and is the authoritative source of info on its status. I was hoping to learn a few things, as well as to run into new acquaintance Nobuyoshi Nakada, one of the core team members.

Techniques for VM Performance
- Simple techniques
- C level VM

- Advanced techniques
- Dynamic code generation
- Native compilation
- JIT compilation
- Polymorphic inline cache
- Selective inlining

- Online feedback opt.
- Hotspot JIT
- Tracing JIT

Pros/Cons of JRuby/IronRuby
- Using an awesome VM
- Pros
- Many clever people working on the project
- No code is good code
- Many libraries already exist on each environment
- Easy to use parallelism
- Cons
- Not only focused on ruby, so there is a semantics gap.
- Cannot use C ext direct

Pros/Cons of Rubinius
- Most code in ruby
- Pros
- Ruby in Ruby
- Says this is the *best way* to improve performance in the long run because we can easily analyze and make specific performance improvements.
- Cons
- Still a LONG way to get a high performance VM

Pros/Cons of C Ruby
- Pros
- Portability, using GCC
- C is well known language already
- Extensibility
- Performance improvement
- Easy to write simple extensions.
Cons of C Ruby
- Extension libraries are written in C
- GC problem
- Inlining problem
- Limitations on program analysis

Our Performance Policy
- CRuby is not best solution, but it is a GOOD one
- They continue to improve the CRuby implementation
- It is a pragmatic, practical selection
- It will be the main implementation still for at least several years

Keywords for Success
- Embedding
- Parallelism

Research
The team is working on taking advantage of C some projects are running
- Hidden optimization techniques on YARV
- Ricsin

Hidden Optimization Techniques
- turned off in 1.9.1 by default
- Tail call optimization
- Optimizing using unification
- Stack caching

- Left easy optimizations
- Efficient method caching
- Efficient fiber implementation using platform dependent way suvh as makcontext()
- These optimizations will be merged in 1.9.2

Ricsin; Mix-in C Ruby
- Embed part of a C program into ruby
- Like a RubyInline, but directly embedded
- Usage
- Use C libs direct
- Replace all built-in classes and methods
- Test Ruby C API
- Continuous performance improvements

Ruby to C AOT Compiler
- Translate Ruby scripts to C code ahead of time

Related work
- ruby2c
- yajit
- yarv2llvm

Atomic-Ruby project
- Issue: ruby is too fat
- EMBEDDED
- Embedded system can have special problems, such as resource limitations
- Application embedded ruby
- Some applications need just a Ruby DSL engine, not the full ruby distribution.
- We need a slim Ruby interpreter
- Utilize CRuby portability
- 3 subprojects
- Plugin/out
- Precompilation
- Swift core features

Auto-write barrier detection
- write barrier needed for several GC algorithms.
- automatic WB detection system

Generational GC
1 but ref count

Multi-VM (MVM) Project
- Multi VM in 1 process
- Watch runs in parallel
- High speed inter VM communication
- Sponsored by Sun

Summary
- CRuby/YARV is not "best solution", but it is best for right now. But eventually, Rubinius is the future. Which just so happened to be the next session I attended...

RubyConf 2008 - Day 1

The crowd of over 400 was happily awaiting the kickoff in Orlando of RubyConf 2008. After brief introductions from Rich Kilmer and David Black, Matz took the stage for his opening keynote.

Opening Keynote - Matz

Reasons Behind Ruby
8 years ago was the 1st International Ruby Conference, and there were only 33 people there. Obviously since then, the community has grown. Matz said he was not going to talk about technical stuff.

Question - Why Ruby?
Matz joked that it is his beloved masterpiece. Really though, Matz loves languages, and also loves freedom. His reason for creating Ruby was to maximize freedom.

There are still a few problems. Ruby is imperfect, slow, lacks features, and is complex. And there's more. Ruby is inconsistent, has a poor implementation, has memory issues, and embedding issues.

So why do we love Ruby? "Programmers Strange Love, Or How I learned to Stop Worrying and Love Ruby"

The greatest Reason behind Ruby is Love. It made programming fun again. Again? It used to be fun.

In 1980, Matz began programming in BASIC. He liked that he could make intelligence, and felt a sense of creation. But there was also great pain.

The Basic Aristocracy
There was a big difference between language designers, vs. lowly programmers. In BASIC you cannot add to the language, and there is no abstraction.

Lisp
Everything is opposite in Lisp from BASIC. In Lisp, there is no discrimination and extreme abstraction.

As Eric Raymond said, "Lisp is worth learning for the profound enlightenment experience you will have when you finally get it; that experience will make you a better programmer for the rest of your days..."

However, Lisp was great in a book, not in reality. Programming in Lisp did not make him happy. Was it all the parentheses? No. Was it the macros? Partially.

It came down to the fact that smart people just underestimate the "ordinarity" of ordinary people.

Aristocracy is OK, as long as I have POWER. Matz wants to be between "owelty" and freedom (none of us in the audience knew what he had meant by "owelty" either, but Matz was on a roll, so we went with it).

BASIC gives you no power, Lisp gives you all power. But at both ends we lose popularity. If you stay in BASIC you are a coward, and if you go extreme, you are brave but you risk your life (or popularity).

We all love power. You see, policy matters. We need philosophy and attitude.

The community cannot live by love alone, power matters.

Then along came Rails which turned Ruby into a web DSL. You see, Ruby is about vocabulary. It has rules, or syntax. There are others, for example Rake and RSpec.

As Dave Thomas said - "Programming is a process of designing DSL for your own application"

Ruby is really like a meta-DSL. It provides a foundation for DSL's, which increases our overall productivity.

But beware of commercial success, because they will try to take your freedom away.

Now there are better implementations or Ruby than MRI - YARV, JRuby, Rubinius. Matz says that the various implementations are competing, which is a good thing.

Matz then mentions the 4X growth expected in Ruby programmers from the current 1 million to over 4 millions. There will be NEW people coming into the community, so welcome them and nourish them. Ruby can have the power to raise up people in the programming field, so feed them, and love them.

He loves us all.

We love you too, Matz.

Sessions - Day 1

I attended a number of interesting sessions throughout RubyConf 2008 day 1. Per previous reader feedback (keep each post to a manageable size!), I will put each into a separate post. Here is what I will be covering:

Future of RubyVM - Koichi Sasada
State Of Rubinius - Evan Phoenix
rush, a shell that yields to you - Nicholas Schlueter
Testing Heresies - Francis Hwang
Lightning Talks

I missed a few of the lightning talks while having a nice dinner with David Black, Jim Freeze, and Paul Brannan. Hanging out with really smart people is like a continuous lightning talk, so I didn't feel deprived by any means. Thanks, guys!

After the lightning, as it were, we had the second ever Ruby Jam Session. This time, about 5 people brought guitars, including Ruby Jams veterans like Jim Weirich and Forrest Chang, and lots of new friends showed up as well. I was better prepared this time around, and had brought about 10 harmonicas with me. But the tastiest part of the jam, was at the very end when David Chelimsky and Diego Scataglini played some really great jazz standards. A couple of tasty beers followed the music, and then it was off to rest up for RubyConf Day 2.

Tuesday, November 04, 2008

My Vote Was Easy

Despite a massive line of people waiting for the polls to open this morning, when I went to vote just now I discovered... no line whatsoever. What a letdown! I wanted to do some sacrificing like hard struggling adherents to the cause of freedom the world over have done, just to get to the polling place, let alone cast their vote. But no, everything had to go quickly and smoothly.

After accepting my offering in Halloween chocolate (stolen from my son, I might add) the friendly poll-workers ushered me to my individualized voting booth, and afterward counted my vote extra-specially well before bidding me farewell.

Democracy is great, and kudos to the people that humbly and with no applause watch over our freedoms at the ground level... your friendly neighborhood poll-workers.

If you haven't done it yet... go! Now!

I Think I'm Comatose

This last weekend, I upgraded a pretty complex application to the latest Ruby on Rails, and consequently had to upgrade a bunch of the plugins. In the case of Comatose 2.0, this was well timed, as there is a new release.

If you have never heard of Comatose, it is a very cool mini-CMS that you add to your Rails app as a plugin. For cases where you have an applcation, and you want to put the 'static' pages under the control of the users, while still integrating with your application, Comatose is the best option I have discovered.

If you are just discovering Comatose, just go check it out. Now. If, on the other hand, you are running into any problems going to the new version, read on.

Since this application had been running the 0.8 version of Comatose, I ran into a little problem, which has been documented on Google Groups here.

If you are running the older version of Comatose, you have a migration that contains this:


module Comatose
class Page < ActiveRecord::Base
set_table_name 'comatose_pages'
acts_as_versioned :if_changed => [:title, :slug, :keywords, :body]
end
end


The new version of Comatose, however requires this:

class ComatosePage < ActiveRecord::Base
set_table_name 'comatose_pages'
acts_as_versioned :if_changed => [:title, :slug, :keywords, :body]
end


What I did was, go and change the original migration to match the newly required format. This will allow a new clean installation to work correctly. Then, to handle upgrade installs, I created a separate rake task to perform the migration steps as described here.

Voila. Now this app is happily running the latest Rails, Comatose, ActiveScaffold, ActiveMerchant, and a few other plugins too. Life is good.

Friday, October 31, 2008

Oh, The Horror


I was forwarded a link to the original, left on the cutting room floor ending of the Roger Corman classic horror film "Little Shop of Horrors" and not only was it a timely tidbit for Halloween, but it was also strangely appropriate for the election.

Enjoy your candy...

Sunday, October 26, 2008

The Great Git Migration

I have been using git for a while, not quite as long as the really cool kids, but long enough to have become a git snob. As such, having to work on existing projects that were using that nasty old subversion was just like, a drag, man.

I kept saying, "I'm going to migrate everything over to git." Somehow, I just never seemed to find the time. Another little problem was the underwhelming level of support for git on Windows. Yes, a couple of projects I am involved with have Windows code for client applications in there.

So when the msysgit project was announced, I followed it with great interest. The most recent versions are actually quite usable. I was able to install on a Vista VM (I use VMWare Fusion on OS X to keep the Windows thing to a minimum) and connect to several repositories on both github and Unfuddle with ease.

A couple of hours waiting on git-svn to import the old repositories (under OS X, cause mysysgit does not appear to support git-svn), and it was all git all the time. No more svn on any of the projects I am actively working on. And I was able to preserve the entire version history of each repo, too.

Saturday, October 25, 2008

Cool Like Cucumber

Ever since I discovered Fitnesse a few years back, I have been increasingly obsessed with the idea of executable specifications and testable requirements. As far as I am concerned, Behavior Driven Development (BDD) is a very appealing way to get the "people who need the software" and the "people who build the software" to understand each other.

For about 2 years, give or take, many Ruby on Rails developers have been switching to RSpec, or at least paying lip-service (giving rspec?). This is simply because, as far as automated testing, ye olde test::unit is getting a bit long in the tooth. The somewhat less ambitious shoulda project takes a much more minimal approach to RSpec, preferring to be a better test:unit and not going all the way to the executable specification. Maybe they did not drink the whole glass of kool-aid? Well, I did... a while back. But I digress.

There have been various attempts to introduce a more readable, less "code-oriented" way to work with the users themselves to define the requirements for their solution. This led to RBehave, later merged into the RSpec story runner. Now, there is an even better tool emerging for human-readable yet testable requirements, called Cucumber.

Cucumber is the brainchild of RSpec contributor and super smart fellow Aslak Hellesoy. Taking the basic idea of the StoryRunner and, ahem, "running with it", Aslak has not only rewritten the entire thing using the treetop parsing engine, but he has also improved on the paradigm both for expressing the user needs, as well as the coding required to turn the requirements into executable tests.

Here is a small example:
Feature: Log-in
In order to view my profile
As a Registered member
I want to be required to log-in

Scenario: Proper login
Given I am the registered member "quire"
And I am on the login page
When I fill in "email" with "quire@example.com"
And I fill in "password" with "quire69"
And I press "Login"
Then I should see "Account Activity"

Scenario: Bad password
Given I am the registered member "quire"
And I am on the login page
When I fill in "email" with "quire@example.com"
And I fill in "password" with "bad password"
And I press "Login"
Then I should see "Simply enter the credentials you signed up with."
And I should see "Couldn't log you in as 'quire@example.com'"

Scenario: Bad email
Given I am the registered member "quire"
And I am on the login page
When I fill in "email" with "quire@example.co"
And I fill in "password" with "password"
And I press "Login"
Then I should see "Simply enter the credentials you signed up with."
And I should see "Couldn't log you in as 'quire@example.co'"


Now THAT is a nice clear way to define what a system is supposed to do. Furthermore, actually implementing the "steps" that exercise the system functionality is very simple as well:
Given /I am the registered member "quire"/ do
@user = User.new({ :login => 'quire', :email => 'quire@example.com', :password => 'quire69', :password_confirmation => 'quire69', :terms_of_service => true, :primary_talent_id => 1 })
@user.save!
end

Given 'I am on the login page' do
visits "/login"
end

Then 'I should be taken to my dashboard page' do
response.request.path.should == user_path(@user)
end


Since WebRat is fully integrated into Cucumber, and there are already a group of useful steps included, you can get surprisingly far while defining fairly few steps of your own. And when you do have to, working with Cucumber really is quite superior to the old StoryRunner.

I am working on two projects right now that are using BDD and Cucumber, and we are very happy with it. We now have a lot of visibility into how complete a feature is, and it helps everyone understand what a feature does, and how it is supposed to work.

Once again, great work from the RSpec crew. Thanks, guys!

Wednesday, September 10, 2008

Going To RubyConf 2008

Yes, it has been quite a while since I last posted. I have been busy with several interesting code things I will be talking about soon, along with my family's annual trip to Spain (lots of good times, but no posting about that).

Busy, yes. but not TOO busy to make my plans to go to RubyConf 2008. November in Florida, I can just imagine it now... hard core Ruby coding, jam sessions in the hotel, and maybe even some post-election riots. Plus roller disco. Damn, I'm there!

And if you want to be there, better get it together and register now.

Sunday, June 08, 2008

RailsConf 2008 - Day 2

Day 2 of RailsConf 2008 began with a keynote from Jeremy Kemper, uber-committer and now main man behind Ruby on Rails. According to the brief introduction by DHH, Jeremy did around one third of the Rails 1.0 release himself, cares deeply about the "whole thing", and has been busy helping get RoR running on Ruby 1.9 and other platforms.

That must explain why Jeremy kept yawning incessantly during his presentation... he is the reason DHH is able to get more sleep! There was some speculation that Jeremy was actually suffering from a hangover... I have no idea, I was asleep the night before.

Anyhow, Jeremy gave an overview of going from Rails 1.0 to the new 2.1 release. Basically, in his opinion 1.2 was "out for a while and got stale". In 2.0, we got all these foxy and sexy things, plus shed a lot of fat and gained speed. But the real story was the growth of the movement behind Rails from a fairly small group of core people, to a massive community of interested contributors. Here are some amazing stats:

From the 2.0 release in Dec 2007 to the 2.1 release in June 2008

> 1400 contributors (Trac, Lighthouse, and Github users combined... obviously a bunch are duplicates, but even 450+ is a large number!)
> 1600 patches
> 9000 comments

I cannot imagine the chaos of having THAT MANY people harassing me over getting their pet patches into the Rails trunk. No wonder they seem a little preoccupied... they are actually thinking about how many patches that will have to reviewed by later tonight! Much respect to the team just for dealing with that many crazed developers. This is to a large extent the key benefit of moving RoR on to Github: the much easier process of managing all these people's contributions.

Here are a few brief highlights of the new 2.1 features:

* New merged migrations - helps deal with keeping migrations synched throughout the development team. It does this by a new naming system, and now making sure that all unapplied migrations have been handled, instead of the old simple schema_info table, RoR now keeps track of all migrations, and makes sure that all migrations have been applied

* Time zone support - there were several plugins for time zone support. RoR has now included a fairly simple way of handling this with some baked in logic that extends both the Time class as well as some new ActiveSupport hotness.

* Ruby gem dependencies - now you can specify exactly which gems and versions are needed for your Rails app, instead of playing "one of these gems is not like the others" when trying to install or deploy a Rails app.

* Memcache - support for the ultra powerful memcache is now baked in, making fragment caching work nicely... made memcache a first class member of framework, and the memcache client is now bundled with the code distribution.

* ActiveRecord Dirty - now a way of track what has changed in a AR instance. This improves performance slightly by eliminating unnecessary database writes, but even more importantly results in much cleaner log files.

* Smarter :include - avoids combinatorial explosion with out of control includes, that result in too many left outer joins on every DB access. Instead, will just perform two queries to DB, with resulting database performance improvements. For example:

Msg.find(:all, :include => :user) # two queries
Msg.find(:all, :include => :user, :order_by => 'users.created_at) # uses outer join


* Named scopes - with complex models, you will have repeated convenience associations. For example:

user.recent_messages

You would prefer:

user.messages.recent

especially if you can pick it up "for free". Here is a tiny example:

class User
has_many :messages
...
end

class Messages
named_scope :recent, :order => 'created_by DESC'
...http://www2.blogger.com/img/gl.link.gif
end


An Embarrassment Of Riches In Ruby VM-land
Rails is now running in (milestone achieved just before the conference, thanks to heroic efforts by John Lam, and the rest of what is a very under-appreciated team within the Ruby community at-large). Rails has been running in JRuby for months. Also, RoR has been running in Rubinius since about two weeks before RailsConf.

And adding to the fun, Ruby 1.87 is now released, which includes a bunch of performance related backports from the once and future Ruby 1.9 release.

Just to show all of this multi-platform hotness, Jeremy did a little Rails 2.1 demo, showing the a minimal Rails app running on JRuby, Rubinius, and Ruby 1.9. That was amazing really, that we are actually here. Sure, Rubinius was kinda slow at starting up, but it worked, that is what counts.

Ruby 1.9 on Rails
Yes, you can actually run Rails on Ruby 1.9, at least somewhat. ruby-1.9 script/server

In Ruby 1.9, string encoding is huge change. According to Jeremy, 1.9 looks really fast... despite no formal benchmarks. He sees modest performance improvements on typical pages, and really notices it on bigger ERB pages. The typical page is about 20% faster. He says all rails tests are running on ruby 1.9.

Despite all of the yawning, I was not put to sleep at all by seeing some cool new Rails features, and even more importantly a few huge milestones in the Ruby VM race.

"Getting Git" - Scott Chacon
The keynote was good, but it had only whetted my appetite for even more substance. After a tip from a friend, I went to check out Scott Chacon's presentation on git. If you knew nothing about git, this was not the place for you. Scott really got into the internals of git, which many of us found captivating.

The room was pretty packed, and the crowd was held in rapt attention. Several of us remarked afterwards that it was the most organized presentation we had ever seen. It somewhat mimicked the format used during Dick Hardt's presentation on "Identity 2.0" at OSCON 2005, if you have seen that one. Each section of the presentation was precisely timed, and amount of material covered was massive. I took copious notes, but there was no way to try to capture that much info. Check it out online here, if you want to dive deeply into the glory that is git.


"Advanced RESTful Rails" - Ben Scofield

Next, I went to Ben Scofield's presentation on REST. Surprisingly a lot of people are not yet using REST. Or if they are, there appears to be some confusion over exactly how to implement things correctly. It was a decent presentation, but "Intermediate" would have been slightly more accurate. I mean, where was the discussion about polymorphic RESTful controllers? Now THAT would have been advanced.

It was still a good, if not great presentation. Ben knows what he is talking about, I would just suggest raising the bar as far as use of the word "advanced". That suggestion applies across the board to all RailsConf presentations.

"StoryRunner" - David Chemlisky

David Chelimsky is the closest thing we have to a test guru in the Ruby community. Here I am using the word guru in a precise way, "any person who counsels or advises; mentor". No single person has had as much impact on introducing best test practices like behavior-driven development to the Ruby community. Even the way that plain old test-driven development is done, is heavily influenced by RSpec. Not to mention the clones like Test::Spec and shoulda. All of the cooler projects I work on have test specs, and not just unit tests. The projects that I work on that are not as precisely thought out, turn out to not have test specs, but only unit tests.

Coincidence? I think not. The reason for going through the process of really defining what is to be done is to provide a common understanding between the people in the project. But if that was the whole story, BDUF might work. The problem is you need to apply test-driven development principals to the specifications themselves in your project, not just the code. And you need to be doing this as the project goes on, not just once in the beginning.

Behavior-driven development is about closing the gap between the "customer" (the system/site/application user, if you work in something other than a consulting firm) and the "developer" (that would be you). And that is where StoryRunner comes in.

StoryRunner makes it easy to create a kind of simple parser for a very english language oriented, simple text based way to write specs, requiring far less structure that the current RSpec format. It is really like a DSL that customers can use themselves to define their needs. As David put it, "Ruby code is very readable, but you will not be getting customers to send you emails with new requirements in the form of Ruby code."

So BDD is about helping the customer/user formalize their thoughts about exactly what solution they need. Can you imaging a future, where an initial RFP is put out in the form of test specs? That might provide a lot better basis for estimating, than the pages and pages of static docs that seem to still be getting cranked out inside large development organizations.

Have the requirements been changing? Just do a diff on the specs. There are so many advantages to this approach, that once you get the customer "spec-obsessed", just like developers who become "test-obsessed" the game is totally changed. You can't go back, at least willingly.

David talked about a couple of other tools that fit nicely in the RSpec stack. One very cool way to use it for full-stack integration testing is using Webrat. Another is doing true browser testing using Selenium-RC. David showed a demo of this, and remarked on how happy it made customers to be able to actually see their web site being tested, not just get the results back.

Note that RSpec is for use with Ruby, not just Rails, so if you are using Merb or Sinatra, or whatever, you can still write test specs using RSpec and StoryRunner.

Keynote - Kent Beck
During Chad Fowler's introduction, Chad said he was "like one of the superdelegates, getting to cast his vote about what we would get for presentations regardless of what we thought we wanted." All I can say is, Chad, keep it up. Some of these people might not get it, but if they continue to stay in this game long enough, they will.

At last, the moment I had been waiting for, and the biggest attraction for me personally in the RailsConf 2008 schedule: Kent Beck, one of the extremes, the Saturday night keynote.

Kent's presentation on the surface appeared highly informal, just a bunch of stories. It actually consisted of a serious of overlapping arcs on three topics that he knows a little about, because he helped create them: test-driven development, design patterns, and Extreme Programming. These stories took place over a 20 year timeline, stretching back to the origins of the work that the whole Rails community indirectly rides on today. This was before a bunch of RailsConf attendees were born, or at best were still in diapers.

I was pretty immersed in Kent's storytelling. I was too young to have been part of that original movement, but I'm old enough to want to learn all I can from the great masters, before they are all retired or just inaccessible. Amazing to have access to a couple of the extremes, like Ward Cunningham who was just hanging out on Day 1, at a fairly small conference like RailsConf, where most of the people do not even know who they are. For me, it is like a dream come true. If a bunch of other people are passing up the equivalent of intellectual caviar to stuff themselves with potato chips, well, good for them.

Enough ranting, back to Kent's "Tales to His Grandson". The unifying theme that emerged to me was that Kent is a true 60's radical who chose software as his weapon of choice to use in order to achieve social change. Like many great thinkers, Kent is a reader of books. Yes, those old things. Here are a few that he specifically mentioned that were instrumental in the development of his philosophies:

Christopher Alexander - "The Timeless Way Of Building"

Robert Mankoff - "The Naked Cartoonist: A New Way to Enhance Your Creativity"

Dr. Frank Luntz - "WORDS THAT WORK: IT'S NOT WHAT YOU SAY, IT'S WHAT PEOPLE HEAR"

The last question in the Q&A was really telling. When asked "do you think that you accomplished the political changes you set out to?" he responded that "No, he had not. And that he really did not fully understand why". Oh, those idealistic baby boomers! Kent, you may not have achieved all you set out to do directly, but the forces you set in motion are still going. Even though many of the attendees at RailsConf do not know you, they have grown up in a world where they expect more freedom and openness, and that is in part thanks to you, and fighting the good fight back then. Thanks, Kent, for staying true to the movement.

RailsConf - The Jam Session
After that very high level of intellectual brilliance, what could possibly top it off? How about a fantastic musical jam session. Chad had arranged the Musical BoF, and somehow Jim Weirich had talked a very generous and cool local music store into loaning him a perfectly playable acoustic guitar. Chad had his ukulele, Andrea Wright a recorder, and Dan Tripp, a portable folding guitar. I had brought a couple of my harmonicas with me, though not enough, of course. There were a bunch of other fun musical things, like a flute, and an impromptu Brazilian song or two from actual Brazilians. Not to mention a very creative approach toward an impromptu percussion setup by Forrest Chang.

Anyhow, great fun was had by all! We are already planning the RubyConf 2008 Jam, so it you will be coming, bring your ax! I will commit right now to the Dead Programmer Society helping sponsor having some instruments and/or PA system... plus some beer!

As I headed back to my room, I felt the smug contentment that comes from having had a really good time. Thanks everyone, for having made RailsConf 2008 "worth it" by the end of Day 2. Day 3 was still coming, and would have a few surprises of its own...

Wednesday, June 04, 2008

RailsConf 2008 - Day 1

As RailsConf 2008 - Day 1 began, there was no avoiding a sense of just how large the community was growing. With over 2000 attendees, the physical presence was bigger than ever. But, unlike the first two conferences, the growth of the community had triggered inexorable changes. Unbeknownst to most attendees at RailsConf 2008, a rebel alliance had formed itself. The caboo.se crew, longtime Ruby on Rails committers, had decided to stick it to the MAN. Or as Courteney put it:

In May, if you don't want to cough up almost a thousand dollars to go learn things you already know, but want to come to Portland to network with skilled Rails coders from all over the world, instead of hanging out in the lobby at the Lloyd Center like I did last year, come to Caboose Conf.


The spawning impulse must have been too great for the cool kids. They were heavily conflicted when O'Reilly took over for RailsConf 2007. Not being one of them, I can only surmise how they felt. But no one likes to pay for something that once was free. And it is unpleasant to have a bunch of strangers running around what was once their private garden. But this was deeper, a sense that something had shifted, and would never return.

You can take the caboose out of the ghetto, but you cannot take the ghetto out of the caboose. Many people at RailsConf felt your absence, comrades, if any of you are reading this! As I mentioned earlier, the vast majority of attendees, including repeat ones, were totally unaware of the specifics. But they sensed something missing, some crucial element lacking.

At the same time, the community is now larger. And there are tremendous advantages to the incursions of the money people into the private reserve. The people who were saying to each other in 2006, "I wish I could program in Rails full-time, and quit programming in Java/C#/WTFLang" were now saying "I wish recruiters would stop harassing me!"

That all said, the Rails community needs the caboose crew, like we need our gall bladders. It is just a little organ compared to the overall size of the body, but without it, you will not survive long, and if you do, you will be feeling so well.

It would appear that the rebels had struck their first blow. But more about that would be revealed on Day 3.

Chad Fowler still had the his job leading off the proceedings in the new, more corporate RailsConf. As such, the ukelele would be banished from the main stage, along with any other musical instruments during presentations. Joey, we miss your accordion, too.

Chad mentioned that there were over ten times more proposals for RailsConf 2008, than attendees at RailsConf 2006. That certainly explains why my proposal was not accepted... of course, the title being "Pedal To The Metal: Alternatives To Rails" might also have had something to do with it.

Anyhow, Chad was almost apologetic as he explained how people were pissed off about RailsConf 2006 because it had sold out so quickly, and now that they had increased the size enough, people complained that the conference was "losing it's luster." You can't win, dude! But I still support what you're doing.

At that point, he introduced Joel Spolsky. Now, when I had read that Joel would be the opening keynote for RailsConf 2008, I had been fairly surprised. Although the penchant of the Ruby/Ruby on Rails community IS to reach out to others for ideas and technology, with the well-publicized spat between Joel and DHH, the choice seemed a curious one.

Joel was funny, and entertaining. He had slides, and seemed like he was leading up to some big point. But after the dog and pony show was over, I felt like I had drunk an extra-large zero calorie soda: a feeling of fullness, but no actual nutritional value to it. I was not exactly disappointed, but I was far from impressed.

Next, I headed into the Entrepreneurs on Rails session by Dan Benjamin. The presentation was probably worthwhile if you have never formed a business before, but my attention wandered, which is why I will not be writing anything else about it here.

So far, I was zero for two on my personal "RailsConf 2008 was worth it" scale. Hmmm.

I got a bit more enthused at Micah Martin's session "10 Things I Hate About Web Development", where I finally got to see the long awaited demo of Limelight, which Micah had been dropping hints about at RubyConf last year. I have a separate posting I will put up with info on that session.

After that, with my energy picking up, I went to "Rubinius: Under The Hood and Behind the Curtain" presented by Evan Phoenix, Brian Ford, & Wilson Bilkovich. This was a fun and informative session about the current state of Rubinius, although if you have been following the various blogs/IRC/github you already knew what he was talking about. Still, it was nice to see the core Rubinius team in person, and to get the sense of the crowd's excitement about Rubinius. I will be posting separately about this session.

There was a great moment that is really representative of what I get out of RailsConf. Walking between sessions, I ran into Ward Cunningham. Not only did I get to introduce myself, and take another fanboy picture with one of the extremes, but we also stood around for a few moments and chatted about his experiences migrating MediaWiki to Rails.

Afterwards, I hung out doing a little pairing with programming buddy/collaborator Ari Lerner trying to help out with pool-party before his upcoming BoF about it later that night. I know you have not yet heard about pool-party, which is an auto-scaling solution for Amazon's EC2 service, since the first version was put up on github there at RailsConf. What makes pool-party unique is that it is a pure Ruby and open source solution to the cloud computing thing. Plus it is cool.

By the time we were done, we had already missed any chance of eating dinner, and also most of Charles Nutter's talk. I walking in to see a slide "You Are The Enterprise" from his presentation. That sounded silly to me, but without any context, I probably do not know what I am talking about here.

Next were the RailsEnvy guys presenting the "Ruby Heroes Awards". I have to say, those guys from RailsEnvy are pretty funny... and brilliant self-promoters. Plenty of people have already blogged about this, so no need to go on and on about it here, except to say they made some pretty good choices with the winners:
Evan Weaver, Tom Copeland, James Edward Gray II, Ilya Grigorik, Yehuda Katz, and Ryan Bates.

Good job, guys!

Then it was time for the big David Heinemeier Hansson keynote. After my tepid enthusiasm for the opening keynote in the morning, I hoped DHH would do something exciting. Well, that was not exactly what happened... but thought provoking, nonetheless.

His speech was entitled "The Great Surplus". The salient point seemed to be that the highly leveraged productivity that Ruby on Rails gives the community (and the unreal demand) will not last forever, or perhaps for even that much longer.

His prediction is that one of three things will happen:
1. The mainstream copies Rails with something else
2. Some other dramatic alternative arrives (new paradigm) that provides massive improvement
3. Rails itself becomes mainstream

So what to do about this? Invest in yourself. Regardless what happens with Ruby on Rails, if you better yourself it will last to next wave. And do something else beside programming! You need to train more than 1 muscle... same with non programming activities.. need to engage other parts of brain. And sleep more.

It is well known that tired people take longer to do work, and that the output is not as good as a result. More importantly, it takes a toll on that individual that leads to a longer term reduction of output. So why do we consistently see investors/executives pushing their technology teams this way. And why do we allow it, when we already know better?

I had seen a friend earlier in the day during the conference, who I had not seen since last year. His startup had become involved with a well-known incubator. But far from looking energized, my friend looked worn down. "I need a month with no electricity," he croaked. And this is where many, many people in the RoR community are actually hoping and praying to get to. This is what DHH was warning about.

After DHH finished his keynote, I ran over to the BoF for pool-party. The turnout was actually pretty good! Ari did a great job with his presentation, and at the end a guy from Amazon came up and chatted us up for quite a while. I look forward to switching over my AWS application to the pool-party as soon as it is ready!

My day complete, I crawled off to bed to take DHH's advice and rest up for RailsConf 2008 - Day 2.

Going Public: From Los Angeles To RailsConf 2008



With the high cost (and carbon!) of transportation, as well as the burden of the increased time it takes to fight your way through the congestion, we all need to change the way that we navigate thru our daily lives. I had decided to take public transportation to get from my house in Los Angeles, to RailsConf 2008 in Portland, OR.

When I first mentioned my plans, several friends and family members were incredulous. "It takes too long, " stated one person. "There's no way to get to airport in LA except by car, " scoffed another. I set out to prove them wrong, and by doing so began a new chapter in how I approach personal transportation.

I set out from my house in Silver Lake, walking down the hill to Sunset Blvd. I stopped at the delightful Cuban coffeehouse Cafe Tropical, which happened to be directly adjacent to the bus stop I would need to use.


I sipped a tasty cup and snacked on one of the delicious pastries while I awaited my first bus.



I jumped quickly (literally, they do not wait around) on the Route 704, which is a giant segmented affair. Despite that, the bus was almost completely filled until we reached the Evans Community Adult School (no relation), at which point almost the entire bus emptied out.

One notable feature of the ride were the video screens filled with advertising prominently situated on the bus, an attempt to subsidize the cost of the public transportation system here in L.A.




On reaching the Union Station stop, I exited the bus. I would need to walk to the FlyAway bus terminal to catch the non-stop shuttle to LAX. The way was clearly marked.

They wanted to make sure you could not make a wrong turn!

I purchased my ticket at a friendly kiosk.


The bus arrived shortly thereafter.

While I waited to board, I asked a half-dozen of my fellow travelers if they had taken the FlyAway shuttle before. All except one were first-timers, like myself.

We reached the airport quickly from the downtown Union Station. No having to drive, allowed me to check my email (GMail Mobile is still cool on my cheap phone) and daydream. Suddenly we had arrived at LAX. I noticed that my airline was in the very last terminal, and since the FlyAway shuttle would stop at each and every terminal first, I decided to get off there and cross the airport on foot.

It would seem there is plenty of parking, and available charging terminals, if you have a plug-in electric car.



I had made it to the airport in about 1 hour and 15 minutes, which was about the same amount of time that it would have taken me to drive my car to the airport parking, park, get on a shuttle bus to the airport from the parking location. Except my cost for transportation had been $5.25 instead of the fuel, plus the $60-80 for parking. With a small portion of my savings, I smugly purchased a copy of Predictably Irrational at an overpriced airport bookstore to read on the flight.

On exiting my plane in Portland, it turned out they were having a party to celebrate Portland having now become an international airport.



Politely refusing a multi-colored cupcake, I walked to the end of the terminal where the MAX, Portland's modern light rail system awaited.



I purchased a ticket from a automated kiosk, then hopped on the train. Around 15 minutes later, I had arrived at my destination.

I had successfully arrived in Portland for RailsConf 2008 only using public transportation! After checking into my hotel, I dashed across the street to the convention center. Alas, the registration had closed for the day exactly on time. Yes, these O'Reilly people are all business. More on that in my post on RailsConf 2008 - Day 1...

My luck would turn around quickly. I immediately ran into friends David Chelimsky and Micah Martin, who invited me to tag along for dinner. When you're hanging out with guys as smart as these, you absolutely cannot be bored, and you will learn a LOT! Plus, one thing about Chicagoans, we know how to pick a good place to eat. David led us on a brief journey to a great Thai place he knew of, and between the great food, beer, and conversation, we had a delightful time. Thanks guys, I appreciate it!

Tuesday, May 27, 2008

Converting The US To Electric Vehicles: Priceless

A non-scientific, but clearly well thought-out back-of-envelope calculation by MIT CS professor Philip Greenspun is food for thought: the cost of replacing the entire fleet of U.S. passenger cars with electric ones would be zero.

Here are some of his numbers:

* total spent per year: $1 trillion
* percentage of oil consumed by passenger cars: 40
* total spent per year on oil for passenger cars: $400 billion [refining into gasoline, distributing, and retailing add even more to this]
* at 5 interest, how much we could we borrow and pay $400 billion every year in interest: $8 trillion
* number of registered cars in the U.S.: 250 million (Wikipedia)
* cost of a new electric car, if mass-produced: $20,000
* value of a used car, if exported to Latin America or China: $5,000
* cost to upgrade average existing American car to a brand-new electric car: $15,000
* number that could be converted for $8 trillion: more than 500 million cars (i.e., twice as many as we have now)

And then here is the money quote:


Instead of sending $400 billion each year to countries such as Saudi Arabia and Venezuela, we could spend it on electric car production in the U.S., Mexico, and China. At current oil prices, it wouldn’t cost us a dime extra to stop importing and burning oil for passenger cars. In fact, if the goal were to end up with the same number of cars on the road, we would have a few trillion dollars left over. One or two trillion dollars would be sufficient to build nuclear, solar, or wind electric power plants to replace all of our plants that currently burn coal and oil (note that less than 1 percent of current electricity generation in the U.S. is from oil).


Now THAT is what I talking about! Let's get on it people... and let's do better than just paying interest only on the bill while we are at it.

Zero Is Special

I just read a fascinating academic paper co-authored by Dan Ariely, author of new book Predicably Irrational, a book I have heard about, but have not yet read.

The paper presents results from a series of experiments into the nature of free pricing. Here is a juicy tidbit from the abstract:

When faced with a choice of selecting one of several available products (or possibly buying nothing), according to standard theoretical perspectives, people will choose the option with the highest cost–benefit difference. However, we propose that decisions about free (zero price) products differ, in that people do not simply subtract costs from benefits but instead they perceive the benefits associated with free products as higher.

Yes, you read that correctly: people can consider something free as more valuable than something that costs money. The authors describe several interesting experiments using chocolate that appear to well prove their point.

Bottom line, so to speak: I need to get this book and read it as soon as I can! Even if it does cost money... actually here are some free excerpts.

RailsConf 2008 Better Be Good

RailsConf 2008 is later this week, as if you all did not know that. Interestingly, this year it has not sold out like it did for 1.0 and 2.0. Increased capacity by O'Reilly? Or perhaps something else at work here...

If you are not aware for the cost of transport, you have been living on another planet. Just the massive increase in time required to make the pilgrimage has been substantial. Many of us have been working at home or within other time-flexible, location-unimportant arrangements for some time now. But we still gathered together unhesitatingly no matter the distance.

Two recent blog postings stood out to me. The first, from Seth Godin said it nicely:

If oil is $130 a barrel and if security adds two or three hours to a trip and if people are doing more and more business with those far afield...

and if we need to bring together more people from more places when we get together...

and if the alternatives, like video conferencing or threaded online conversations continue to get better and better, then...

I think the standard for a great meeting or a terrific conference has changed.

In other words, "I flew all the way here for this?" is going to be far more common than it used to be.


The second was simply an link from from Rails community spokesmodel Obie Fernandez to an article in the Washington Post:


But the truth is that no combination of solar, wind and nuclear power, ethanol, biodiesel, tar sands and used French-fry oil will allow us to power Wal-Mart, Disney World and the interstate highway system -- or even a fraction of these things -- in the future. We have to make other arrangements.

The public, and especially the mainstream media, misunderstands the "peak oil" story. It's not about running out of oil. It's about the instabilities that will shake the complex systems of daily life as soon as the global demand for oil exceeds the global supply.


Already, I was a little bummed out about not being able to eat a delicious Pacific salmon dinner this year. But these two postings really got me to thinking. It would be easy to expect the conference organizers to do all the work. But it is really up to us as attendees to put our hearts into it. Whether it be code drives, or Werewolf sessions, that does not really matter to me personally, although I will be attending the former.

So, all I can say is, this better be good people. We better go up there and really make this thing count, to justify the time, expense, and environmental impact. The times they are a changing, and we need to do something special.

As for me, I will be taking public transport to and from the conference... which is not easy to achieve from down here in L.A.

Sunday, May 25, 2008

The Phoenix Has Landed

Earthlings, we have successfully landed another probe on the red planet. Congratulations to the team at JPL, University of Arizona, and everyone else who made this one happen!

Man, I just love space exploration...

Thursday, May 22, 2008

Got An Eye On You

Read this piece from Rolling Stone magazine on 24-7 surveillance coming soon to a neighborhood near you.

Then be afraid. Very afraid.

Monday, May 19, 2008

In Search Of Search

My client was having troubles. The site was crashing, seemingly at random. QA was baffled, management concerned. After hunting thru logs, the culprit was found: the ferret search service.

Despite having configured acts_as_ferret to use the DRb server option, the "whole thing" would fall apart on a regular basis whenever the ferret daemon would crash... which was pretty often. The ruby on rails plugin acts_as_ferret would revert to a local mode of operation, which would only make matters worse when competing mongrels would corrupt the index files on disk.

Much has been written already about this by more famous Rubyists then myself, so the time had come to make the switch to Sphinx like all the rest of the cool kids. However, there were a few tricky issues that awaited.

One is that the client site was using PostgreSQL instead of the more common MySQL. This meant you had to have already downloaded the pgsql client libs, and also when compiling the sphinx server use the --with-pgsql flag. However, what they didn't mention was that you would need to also download and include the MySQL client libs AS WELL, even if you were not using MySQL. Good to know... seems like a big of baggage to carry, but whatever.

That was one obstacle avoided. The next would prove to be more subtle and complex. There are four different choices of Ruby on Rails clients for Sphinx:



I looked briefly at acts_as_sphinx, but my superficial prejudices against anything named "acts_as_" caused me to continue my search for search. Next, I tried Sphincter, just to prove that I could be accepting about project names. However, I was not successful as getting it working correctly, and project development seemed like it had slowed to near non-existence. The search moved on.

My search was getting desperate as half of the options had been eliminated. I briefly perused UltraSphinx and I started to get a little excited. Development was active, and proper functionality with Postgres was claimed. But there was just one little problem: this client's site is still Rails 1.2.x.

Now, before you all start throwing various disparaging comments my way, consider how much work some clients are willing to pay far vs. something that they can actually see. Multiply that by how long since they started their project, and you are starting to get the idea of where this thing is at. I probably should have MADE them upgrade at that point, but I'm just not that kind of guy.

So that left Thinking Sphinx. I dove in with the manic desperation of someone looking to avoid major amounts of work. That rarely works out well, but in this case it wasn't so bad. The latest trunk of thinking sphinx had comments that said "basic postgres functionality" so perhaps it was possible after all.

Several hours and some major hacking of the original plug-in to actually handle proper psql syntax later, and something was working! I will try to come up with patch to thinking-sphinx for freelancing-god, cause his work was a great help.

Lastly, the delta index. By adding a single field (boolean for MySQL, integer for Postgres) you can remove the need to reindex all data every time a record is updated. Sphinx allows storing the changed records in a separate index (delta) that it will search along with the main index. This way, you do not have to wait for reindexing to be able to search, but you also do not have to reindex the entire data set every time any data within it is changed. You still need to do a complete reindex, but that can occur at off-peak hours, instead of constantly.

The code was deployed, and now all was well with the world. My search was complete.