Amazon's S3 storage service is a really cool way to store and serve up massive quantities of data. Interestingly, it is more like a database than like a file system. Despite the convenience of referring to an object like "/path/to/thing.jpg" S3 really does not have any separate objects for "/path" or "/path/to". In fact, neither "/path" nor "/path/to" even exist.
This is why, when you store an S3 object, the name it is given is called "key" instead of "filename". It functions like a database key, returning a file. However, as very "ultra hipster Web 2.0" as this may be, it is still convenient to browse a large collection of files by using a familiar directory/sub-directory/etc.
In fact, all of the S3 utilities that I use allow you to create a folder inside of a bucket, and sub-folders inside of that etc. But I did not see any programmatic facility for creating a folder independent of any object inside of S3. If you have been paying attention, you remember that in S3 there is no such thing as a directory! So how do these nice GUI browsers do it?
I did some Googling, but no luck, beyond "it couldn't be done". Since I had two different programs that already did it, I knew THAT wasn't the case. It took a little old-fashioned investigation... I looked directly at the output from a query to a bucket that had the little folders in them already.
Lo and behold, it turns out, they cheat, by creating a specially named object for each "directory". For a directory named "/path", you would create an object with the key "path_$folder$", and for a directory named "/path/to", you create an object with the key "path/to_$folder$". Then to get a directory listing for "/path" you just do a query on S3 for all object whose key starts with "/path". Ignore any objects that end with "_$folder$" and there you have it: S3 folders.
I decided that it would be nice if the aws/s3 gem would support this foldering the same way that copying a file within a file system does: if the enclosing directories do not exist, they are created before the file copy.
Thanks to the beauty of modern dynamic languages, I was easily able to put together a little monkeypatch for aws/S3 to the S3Object class, that handles this.
module AWS
module S3
class S3Object
class << self
alias :original_store :store
def store(key, data, bucket = nil, options = {})
store_folders(key, bucket, options) if options[:use_virtual_directories]
original_store(key, data, bucket, options)
end
def store_folders(key, bucket = nil, options = {})
folders = key.split("/")
folders.slice!(0)
folders.pop
current_folder = "/"
folders.each {|folder|
current_folder += folder
store_folder(current_folder, bucket, options)
current_folder += "/"
}
end
def store_folder(key, bucket = nil, options = {})
original_store(key + "_$folder$", "", bucket, options) end
end
end
end
end
Sure, you can have the best of both worlds: massive virtual storage, and a convenient directory-like structure. And now you can have it with your favorite Ruby S3 library.
Happy storage!
Get ready, get set, go!! The annual frenzy of registration for RailsConf 2008 has begun. 
Software patents... as, Obi-Wan warned, "You will never find a more wretched hive of scum and villainy. We must be cautious." Well, the dark side has struck back, people. A massive salvo has been launched, this time aimed directly at the open source community in the form of Trend Micro vs. Barracuda Networks.
I just ran into a really great resource for open source API documentation. Called gotAPI, it links in a delightfully searchable way all of the APIs we love, but turn out to not actually know. Well, we know them, just not well enough to have their numbers setup for speed dial on our cell phones. Or our code able to call them directly without looking up a reference first. 
Apparently the Merb team was not yet satisfied with just having created a better Rails than Rails itself. Now, they are going after even more conceptual high ground, and creating an even more flexible web development paradigm that incorporates many of the great ideas percolating around the mongrel community. 
Lately, I have had a chance to get seriously into Amazon's Elastic Compute Cloud (EC2) service. I doubt any reader of this blog is unfamiliar with the existence of EC2, but many of you have probably not even gotten to the tire kicking stage, like I had been until recently.
Amazon's S3 storage service is a really cool way to store and serve up massive quantities of data. Interestingly, it is more like a database than like a file system. Despite the convenience of referring to an object like "/path/to/thing.jpg" S3 really does not have any separate objects for "/path" or "/path/to". In fact, neither "/path" nor "/path/to" even exist.
I read with great excitement that Engine Yard, my favorite of the Ruby-centric hosting companies, was taking in a round of $3.5 million from Benchmark Capital. I am a big fan of Ezra, and the rest of the crew, and the whole waiting list thing for getting a new slice setup has been a big inhibitor of their growth.
I keep thinking about Zed Shaw's vitriolic (and newly updated) posting, and one particular aspect about it that seems to bother me. One way of looking at a ghetto is that is keeps people in. Another, is that is keeps outsiders away. In either case, it is an economy, like most any other. And like any economy, there are winners and losers.
Zed Shaw has always been a rather curmudgeonly figure. I find him smart and hilarious, but of course many are rather insulted by his "tell it like he $%#@! well sees it" style. To me, Zed is like the Hunter S. Thompson of our community. The more your believe your own BS, the more he pisses you off.