Showing posts with label software architecture. Show all posts
Showing posts with label software architecture. Show all posts

Monday, November 05, 2007

RubyConf 2007 - Day 1 - "Advanced Ruby Class Design" - Jim Weirich

Jim Weirich is a very smart guy. So smart that his presentation was not about complicated classes, but about Ruby itself and how to take advantage of it to simplify things. Jim's background in computing is long and varied, something like a history lesson in programming languages. Here is the abbreviated list:

FORTRAN > C > Modula 2 > C++ > Eiffel > Java
and in parallel to this:
LISP > FORTH > TCL > Perl

Jim got started programming taking an introduction to FORTRAN class that just so happenened to be taught by Daniel Friedman, author of "The Little LISPer". As such, they studied LISP for the first half of the semester. They did finally get around to FORTRAN, where Jim learned firsthand the adage:

"A real programmer can write FORTRAN (Java) in any language"

Jim took us thru three examples that illustrate some very cool techniques to use as part of elegant class design in Ruby.

Box 1. Rake::FileList
Everyone knows and loves Rake, right? Rake::FileList is like an array but:
- can init with GLOB (an array of filenames, created using a pattern match expression)
- has a specialized to_s
- has extra methods
- uses lazy evaluation

In his first cut, Jim derived from Array:

class FileList > Array
...
end

Here is the first implementation for FileList:

class FileList > Array
def initialize(pattern)
super
@pattern = pattern
@resolve = false
end

def resolve
self.clear
Dir[@pattern].each do |file|
...
end
end
end

The problem is this:

f1 = FileList.new("*.c")

won't work, cause we never call "resolve". What we need to do is to do some lazy loading to "auto resolve", like this:

def [](index)
resolve if ! @resolve
end

Lots of methods need to call resolve in this manner... which is a problem that Jim solves later using a little metaprogramming.

But there is a another problem with the current implementation. This is OK:

f1 = FileList.new('*.rb')
f1 + ['thing.rb']

But this is NOT:

f1 = FileList.new('*.rb')
['thing.rb'] + f1

Why? Because the + method requires passing a literal array to it as an argument, not an object of class Array. If only there was a way for an arbitrary object could indicate that it wants to be treated like an array... ah, but there is! The "to_ary" method was designed to do exactly this. The problem is you cannot call the to_ary method on an Array. The solution is not to derive the FileList class from Array, but instead to use the "to_ary" method to access an Array that is encapsulated into the FileList class.

Another shortcut is instead of calling resolve on each method, using some metaprogramming to add the "resolve" call to every method that might need it, like this:

RESOLVING_METHODS = [:this, :that]
RESOLVING_METHODS.each do |method|
...
end

The big lesson here is when trying to mimic a class, use "to_ary" and "to_str" rather than inheritance.

Box 2 - The Art of Doing Nothing
Builder is a very cool library which is part of the standard Ruby library to create XMl files, but using a friendly Ruby syntax. Did I mention that Jim is the original creator of Builder? Here is an example of Builder in use, if you are not familiar with it:

xml = Builder::XmlMarkup.new(:indent =>2)
xml.student {
xml.name("Jim")
xmp.phone_number
}

Builder uses method_missing to construct tags. This works really well... except what happens if you try to use it with a predefined method? method_missing will not work anymore, since the method is actually there.

Jim solution is to "inherit from Object without actually inheriting from Object", a class he calls BlankSlate. We want to have our Builder class inherit from BlankSlate, instead of Object.

class BlankSlate
instance_methods.each do |method|
undef_method(method)
end
end

That does indeed get rid of all of the methods on the class. But that does not work, because there are some internal methods that Ruby needs that we do not want to get rid of, so we extend the class to this:

class BlankSlate
instance_methods.each do |method|
undef_method(method) unless name =~ /~__/
end
end

This is not the only problem, however. Since in Ruby classes are open, we can also extend them using Modules.

require 'blank_slate'

module Kernel
def name
"hi"
end
end

xml.name("Jim")

class BlankSlate
def self.hide(method)
...
end

instance_methods.each do |method|
undef_method(method) unless method =~ /^__/
end
end


module Kernel
class << self
alias_method :original_method_added, :method_added

def method_added(name)
result = original_method_added(name)
BlankSlate.hide(name) if self == Kernel
result
end
end
end

We will need to do something similar for Object, in order to avoid the same problem. So are we done? Not quite, there is still one more thing... but I didn'y quite catch what it was! I will update this when Jim puts his slides up:

require 'blank_slate'

module Kernel
def name
"hi"
end
end

class Object
include Name
end
...
xml.name("Jim")

Hint: Use #append_features to modify open classes.

Box 3 - Parsing Without Parsing
ActiveRecord is a wonderful abstraction to use, but why is it that we use such a non-Ruby-like technique to select records. For example, we would use this in Rails:

User.find(:all, :conditions => ["name = ?", 'jim'])

Versus using a more standard Ruby language enumerable approach, like this:

user_list.select { |user|
user.name == "jim"
}

Wouldn't it be nicer to do something like this:

User.select { |user|
user.name == "Jim"
}

Jim starts out with a naive implementation, like this:

class User
def self.select(&blk)
find(:all).select(&blk)
end
end

There are several problems, not the least of which is that the above code is far from efficient. A large set of results will use an enormous amount of memory.

Jim then goes on to show a properly "magic" implementation:

class User
def self.select(&blk)
cond = translate_block_to_sql(&blk)
find(:all, :conditions => cond)
end
end

How to implement the magic translate_block_to_sql method?
- write a parser yourself
- Use Parse Tree (Ambition project)
- Just execute the code in the block

As one might suspect, the answer is of course "just execute the code". In other words, create appropriate methods that are called when the block is executed to return the correct SQL conditions clause for the query. Let me warn you, dear reader, that I typed as fast as I could, but some of the code examples here are a little incomplete. As soon as Jim posts his slides online, I will revise and complete them.

class User
def self.select(&blk)
cond = translate_block_to_sql(&blk)
find(:all, :conditions => cond)
end

def translate_block_to_sql(&blk)
instance_eval(&blk) # this is not exactly the code, but I couldn't type fast enough!
end
end

class MethodNode < Node
def to_s
...
end
end

OK, now what about the "==" operator?

class Node
def ==(other)
BinaryOpNode.new("=", self, other)
end
end

class BinaryOpNode < Node
def initialize(operand, left, right)
@operand = operand
@left = left
@right = right
end

def to_s
"#{@left} #{@operand} #{@right}"
end
end

class LiternalNode < Node
...
end

class StringNode < Node # puts quotes around the string for SQL query
...
end

Do not use a case statement to differentiate between types. Instead open core classes, like this:

class Object
def as_a_sql_node
...
end
end

class String
def as_a_sql_node
...
end
end

Be careful how you name methods to avoid collisions.

Possible problems? Literals on left, because "==" is commutative. The solution is to use "coerce" method to handle numeric operators.

More possible problems?
"&&" and "||" operators cannot be overridden in Ruby cause they have short-circuit semantics in the Ruby language interpreter itself. Perhaps & and | instead? Too bad, because we have to write "special" semantics. Also "!" and "!=" cannot be overridden in Ruby

The ruby "criteria' lib already implements some of these ideas.

Conclusion
One important lesson to take away, is that programming languages really shape the way we approach problems. Learn the corners of whatever language you are using to take full advantage of it. Don't be afraid to think outside the box of past experience...

Jim was an amazing and dynamic speaker. All I can say is Joe O'Brian and the people at EdgeCase are very fortunate to have him around.

Wednesday, March 14, 2007

I Speak For The Code

Recently I was in a client meeting with a bunch of managers to discuss some very complex changes that were being proposed for an existing system. I was invited to the meeting so I could "speak for the code", which to them meant that I could validate the ideas of the group based on my knowledge of the code base.

After the meeting ended, I kept mulling over that phrase "speaking for the code". The more I did, the more I realized the parallels between "speaking for the code" as an architect or lead developer, and the Lorax "speaking for the trees" in the Dr. Seuss story of the same name.

So here goes my attempt to illustrate some lessons about best practices in software development, inspired by the story of the Lorax...


At the far end of town where the Grickle-grass grows and the wind smells slow-and sour when it blows and no birds ever sing excepting old crows...is the Street of the Lifted Lorax.
And deep in the Grickle-grass, some people say, if you look deep enough you can still see, today, where the Lorax once stood just as long as it could before somebody lifted the Lorax away.

Our story begins looking at what remains of a once thriving software application, now gone to ruin. What went wrong?

What was the Lorax?
Any why was it there?
And why was it lifted and taken somewhere from the far end of town where the Grickle-grass grows?
The old Once-ler still lives here.
Ask him. He knows.

Perhaps the Lorax was the original system architect. But he is gone now, leaving behind Once-ler who is now in charge of the project.

You won´t see the Once-ler.
Don´t knock at his door.
He stays in his Lerkim on top of his store.
He stays in his Lerkim, cold under the roof,
where he makes his own clothes
out of miff-muffered moof.
And on special dank midnights in August,
he peeks out of the shutters
and sometimes he speaks
and tells how the Lorax was lifted away.
He´ll tell you, perhaps... if you´re willing to pay.
Then he hides what you paid him away in his Snuvv,
his secret strange hole in his gruvvulous glove.
Then he grunts, I will call you by Whisper-ma-Phone,
for the secrets I tell you are for your ears alone.

The Once-ler is the swamp guide project manager or developer who remains behind, holding on tightly to the secrets of the system.

Now I´ll tell you, he says, with his teeth sounding gray,
how the Lorax got lifted and taken away...
It all started way back... such a long, long time back...

Way back in the days when the grass was still green
and the pond was still wet
and the clouds were still clean,
and the song of the Swomee-Swans rang out in space...
one morning, I came to this glorious place.
And I first saw the trees! The Truffula Trees!
The bright-colored tufts of the Truffula Trees!
Mile after mile in the fresh morning breeze.

And under the trees, I saw Brown Bar-ba-loots
frisking about in their Bar-ba-loot suits
as they played in the shade and ate Truffula Fruits.

From the rippulous pond came the comfortable sound
of the Humming-Fish humming while splashing around.

The original system was a fine merger of art and science. It provided for all of the inhabitants of its software ecosystem (users, developers, testers, etc.) by using practices that conserved the ecology of the system. It was able to successfully grow from its origin to a useful level of functionality while preserving the environment of the code base.

In no time at all, I had built a small shop.
Then I chopped down a Truffula Tree with one chop.
And with great skillful skill and with great speedy speed,
I took the soft tuft. And I knitted a Thneed!

The instand I´d finished, I heard a ga-Zump!
I looked.
I saw something pop out of the stump
of the tree I´d chopped down. It was sort of a man.
Describe him?...That´s hard. I don´t know if I can.

He was shortish. And oldish.
And brownish. And mossy.
And he spoke with a voice
that was sharpish and bossy.

Mister! he said with a sawdusty sneeze,
I am the Lorax. I speak for the trees.
I speak for the trees, for the trees have no tongues.
And I´m asking you, sir, at the top of my lungs--
he was very upset as he shouted and puffed--
What´s that THING you´ve made out of my Truffula tuft?

The Once-ler took the original application in a direction that it was not architected to take, in a way that disturbed the equilibrium enough to attract the ire of the Lorax, who is the guardian for the system's logical integrity.

Look, Lorax, I said. There´s no cause for alarm.
I chopped just one tree. I am doing no harm.
I´m being quite useful. This thing is a Thneed.
A Thneed´s a Fine-Something-That-All-People-Need!
It´s a shirt. It´s a sock. It´s a glove. It´s a hat.
But it has other uses. Yes, far beyond that.
You can use it for carpets. For pillows! For sheets!
Or curtains! Or covers for bicycle seats!
The Lorax said,
Sir! You are crazy with greed.
There is no one on earth
who would buy that fool Thneed!

The Once-ler is utterly convinced of his own ideas regarding the directions for the system, and is not interested in listening to the warnings of the Lorax. He thinks he knows what the users need, and how to make the system fill that need. Or perhaps he is using the wrong design patterns, or too many different patterns, or some classic anti-patterns.

And, in no time at all,
in the factory I built,
the whole Once-ler Family
was working full tilt.
We were all knitting Thneeds
just as busy as bees,
to the sound of the chopping
of Truffula Trees.

Then...
Oh! Baby! Oh!
How my business did grow!
Now, chopping one tree
at a time
was too slow.

So I quickly invented my Super-Axe-Hacker
which whacked off four Truffula Trees at one smacker.
We were making Thneeds
four times as fast as before!
And that Lorax?... He didn´t show up any more.

Now the Once-ler is scaling up the project, using all of his old buddies. Now four times more staffing than before. Not only that, but with such confidence in his vision that he doesn't miss the fact that the crucial architectural knowledge of the Lorax is no longer guiding the project.

But the next week he knocked on my new office door.
He snapped, I´m the Lorax who speaks for the trees
which you seem to be chopping as fast as you please.
But I´m also in charge of the Brown Bar-ba-loots
who played in the shade in their Bar-ba-loot suits
and happily lived, eating Truffula Fruits.
NOW...thanks to your hacking my trees to the ground,
there´s not enough Truffula Fruit to go ´round.
And my poor Bar-ba-loots are all getting the crummies
because they have gas, and no food, in their tummies!

They loved living here. But I can´t let them stay.
They´ll have to find food. And I hope that they may.
Good luck, boys, he cried. And he sent them away.

I, the Once-ler, felt sad as I watched them all go.
BUT... business is business! And business must grow
regardless of crummies in tummies, you know.

The Once-ler is willing to do anything at all to the code base, including losing the architectural consistency of the system, in order to implement his new vision for the project. Or lose anyone at all on the project team, for that matter, to maintain control over the project. Too bad Once-ler doesn't realize that once the tipping point of architectural instability is reached, a system can become unreliable and unmaintainable very quickly.

And then I got mad. I got terribly mad.
I yelled at the Lorax, Now listen here, Dad!
All you do is yap-yap and say, Bad! Bad! Bad! Bad!
Well, I have my rights, sir, and I´m telling you
I intend to go on doing just what I do!
And, for your information, you Lorax, I´m figgering on biggering
and BIGGERING and BIGGERING and BIGGERING,
turning MORE Truffula Trees into Thneeds
which everyone, EVERYONE, EVERYONE needs!

And at that very moment, we heard a loud whack!
From outside in the fields came a sickening smack
of an axe on a tree. Then we heard the tree fall.
The very last Truffula Tree of them all!

Did you see THAT coming? Of course you did! Once the technical debt of the system exceeded the available resources, the application went bankrupt.

No more trees. No more Thneeds. No more work to be done.
So, in no time, my uncles and aunts, every one,
all waved me good-bye. They jumped into my cars
and drove away under the smoke-smuggered stars.

Now all that was left ´neath the bad-smelling sky
was my big empty factory... the Lorax... and I.

Once the project's future prospects are bleak, project funding or company revenues dry up. As a result, key resources leave. When this vicious cycle begins, there may be no stopping it.

The Lorax said nothing. Just gave me a glance...
just gave me a very sad, sad backward glance...
as he lifted himself by the seat of his pants.
And I´ll never forget the grim look on his face
when he heisted himself and took leave of this place,
through a hole in the smog, without leaving a trace.

And all that the Lorax left here in this mess
was a small pile of rocks, with one word...
UNLESS.

If you are an architect or dev lead, you are probably sympathizing with the Lorax as you sometimes feel like you are "defending" your system. If you are a hiring manager or recruiter reading this blog, perhaps at this point in the story you are getting a idea about why it is hard to recruit and retain the best system architects and developers to work on toxic projects.

But now, says the Once-ler,
Now that you´re here, the word of the Lorax seems perfectly clear.
UNLESS someone like you cares a whole awful lot,
nothing is going to get better. It´s not. SO...
Catch! calls the Once-ler. He lets something fall.
It´s a Truffula Seed. It´s the last one of all!
You´re in charge of the last of the Truffula Seeds.
And Truffula Trees are what everyone needs.
Plant a new Truffula. Treat it with care.
Give it clean water. And feed it fresh air.
Grow a forest. Protect it from axes that hack.
Then the Lorax and all of his friends may come back.

System architects, let a thousand flowers bloom. Fight for your systems.

RIP, Dr. Seuss. We miss your wisdom.

Friday, February 23, 2007

Gazing Into The ORM

Jeremy Miller has an interesting post today regarding Object Relation Mapping (ORM). Droves of developers are now flocking to ORMs via Ruby on Rails, Castle, or one of the other many projects in this space, so it is good to have people like Jeremy exploring from a real implementation perspective.

Anyhow, his post really is concerned with applications that use an existing database, have a large amount of logical processing. Simply applying an ORM without considering the logical implications of the data, not only fails to take advantage of the power of ORMs, but falls into a bit of a quagmire, with potentially a very non-DRY result.

Some of the specific examples he cites are:


  • Big tables don't map to a single object.  I don't think it's possible that a class with a 100 different properties can possibly be cohesive.  We'd be much better off in terms of writing business logic if that 100 column table is modelled in the middle tier by a half dozen classes, each with a cohesive responsibility.  It may make perfect sense to have only one table for the entire object hierarchy, but big classes are almost always a bad thing.
  • Data Clump and Primitive Obsession code smells.  A database row is naturally flat.  I want to do a bigger post on this later, but think about a database table(s) with lot's of something_currency/something_amount combinations.  There's a separate object for Money wanting to come out.  If you make your business objects pure representations of the database you could easily end up with a large amount of duplicate logic around currency and quantity conversions.
  • Natural cases for polymorphism in your object model.  I think the roughest part of O/R mapping is handling polymorphism inside the database.  Check out Fowler's patterns on database mappings for inheritance.



He then goes on to make an important point regarding the role of the database in the software architecture of your application. Basically there are only two perspectives that make any sense:

  • The database is paramount, and the system is expressed and understood in terms of the tables and rows in the database.  The application code and even user interface is just a conduit to get information back and forth into the database.  You design the database first and then build the business and data layers to match the database.  In the .Net world we might just consume raw DataSet's in the application, effectively just working with the database tables offline.
  • The behavior of the system, primarily in the middle tier and user interface, is paramount, and the database is "just" a means to persist the state of the system.  The database is either built to match the business classes or designed somewhat independently.



So in the case of an application that has significant business logic, it would seem to me that the second case is quite preferable. However, in the case where a legacy database is involved, how can we really avoid the first? I have seen many applications get stuck in that quagmire myself, and one real problem is that the business logic becomes a lot harder to represent cleanly when having to preserve the existing database schema's problems. Especially when trying to achieve a very DSL like approach for business service layers. The less that the domain experts have to understand the database, and can just speak in terms of domain concepts, the better!

Thursday, October 12, 2006

Architect Is Not An Honorary Title

The job title "architect" seems to mean something different inside every organization. At far too many companies, it is almost an honorary title, being reserved as a promotion for long-term company loyalists who "know the business", instead of actually meaning anything related to the overarching technical design required for the successful release of quality software.

Knowing the business is great. In fact, knowing the business is essential. I hope that business experts abound within your organization. But "architecting" is not what these people are doing.

Here is a list of some of the knowledge, skills, or qualities that I think are required for anyone who wants to be called "architect". In no particular order:

- Speaks in terms of design patterns
- Uses open source/commercial off the shelf(COTS) software
- Can determine the least cost solution for a particular business need
- Desire for a career path in technology which is NOT management
- Knowledge of both current and next-generation technologies, so that development efforts can be synched up with other developers in the industry
- Knows how to avoid or get out of anti-patterns
- Embraces simplicity as a fundamental design principle
- Can embrace the good ideas of others
- Unafraid of change

Do you know who the architects are in your organization? Are these the qualities that they are known for?