Brainspl.at

We’ve got a few job openings I wanted to float out there for your consideration. Here are the listings:

Ruby Developer

Systems Engineer in Sacremento

Systems Engineer

We need another team member for the Vertebra project. You know ruby like the back of your hand and are interested in distributed systems as well as learning erlang.

Systems engineers work on our cluster systems and will help with our custom Gentoo linux build as well as be involved with the Load Balancers, Switching Fabric, Linux, Xen, Coraid and helping design the latest and greatest platform for running Ruby applications in the cloud.

If you think you have the chops for any of these jobs then we want to hear from you. Please send a resume and a short email describing who you are and why you kick ass to jarnold@engineyard.com and cc ezra@engineyard.com

I gave a talk on Vertebra today at Pivotal Labs. This time it was a bit more technical then my last talk at railsconf as the system has evolved since then. We’re working furiously on this thing and hope to have the first release very soon.

We will be opening up an early beta for a few folks so if you have some wicked cool idea you want to use Vertebra for get in touch and a few people can get added to the early access.

Video was taken today of the talk and should get posted online within a few weeks.

Slides are on slideshare here if you cannot view the embedded version below.

Valery Kholodkov has written a very cool nginx module for handling uploads.

The way this works is that you specify a location block to handle the uploads. So if you are using the standard nginx.conf for rails apps then you would add this in your server block right above your “location /” block:

    # Upload form should be submitted to this location
    location /upload {
      # Pass altered request body to this location
      upload_pass   /internalupload;

      # Store files to this location
      upload_store /tmp;

      # Set specified fields in request body
      upload_set_form_field $upload_field_name.name "$upload_file_name";
      upload_set_form_field $upload_field_name.content_type "$upload_content_type";
      upload_set_form_field $upload_field_name.path "$upload_tmp_path";
    }

    # Pass altered request body to a proxy
    location /internalupload {
        proxy_pass   http://mongrel;
    }

Then make a simple upload form that does a multipart POST to /upload. Now you can have your Rails or Merb app on the backend with a route called /upload. In the action of your app that responds to the /upload route you will get a set of params that look like this(assume the name of your upload fields is called ‘file1’ and ‘file2’):

{"file2.path"=>"/tmp/0000123459", "file1.path"=>"/tmp/0000123458",
"file2.content_type"=>"image/png",  "submit"=>"Upload",
 "file2.name"=>"Picture 2.png", "action"=>"index", 
"file1.name"=>"Picture 1.png",  "controller"=>"test", 
"file1.content_type"=>"image/png", "test"=>"value"}

What this is doing if parsing the multi-part mime boundaries in C in the nginx plugin, putting the parsed files into files in /tmp and then stipping the multipart stuff out of the POST body and replacing it with the info you need to get the name and location of the file on disk.

This means that by the time the request hits your application, the expensive mime parsing is already done and you simply move the file to it’s final resting place. This is a huge win since now the hard work is done in C in nginx before your app ever gets involved.

Of course this is a fresh new module so do your own testing and deciding whether or not this is a fit for your needs. But I think this is a great plugin and have verified it works as advertised.

Man it seems like yesterday that Engine Yard was a small 3 person startup with big ideas and little cash. People seemed to like what we were offering and the business grew into a thriving startup.

In the beginning we only had plans to be the best fully managed rails/ruby hosting company. And I think we met this goal early on. But it became apparent that we had a strong brand and we started kicking around ideas of becoming an open source software company as well.

We believe in Ruby as a platform, and we’ve put our money where our mouth is when we sponsored Rubinius and Merb, hiring many developers to work on both projects.

Engine Yard has become much more then just a managed rails hosting company, we want to strengthen the Ruby ecosystem for everyone by providing the infrastructure and open source software for the next wave of Ruby deployments in the cloud.

We’ve also been delving into the cloud computing arena as I think the next 5 years are going to see huge transition from standard hosting models into the cloud. Our upcoming Vertebra project is a new application programming platform for building distributed cloud applications with XMPP. You can expect to see the first open source release of Vertebra this summer, I think this is a truly unique and very fun project to work on. I think a lot of folks out there will have tons of different use cases for it.

So fast forward to right now, Engine Yard now has more then 80 employees worldwide. Half of these people are Application Support and SysAdmins working to support all the awesome applications we host. Providing 24/7 support. But we also have a growing Engineering team working on all kinds of exciting stuff. I consider myself lucky to work with such a talented team.

So with all that being said, I’m excited to annouce that we have just closed our Series B round of VC funding totalling $15million dollars(insert austin powers joke here)! This round includes investment from NEA, Amazon and Benchmark.

We’re going to use this money towards making Ruby the platform of choice for cloud computing and web development in startups and the enterprise alike.

Watch this space, we have lots of exciting announcements in the coming months. I’d like to thank all of our customers and users of our open source software and the Ruby community in general. It has been a wild ride these last 2 years and I expect many more exciting things to come!

Chris Matthieu and Steven Bristol interviewed me for the Rubyology Podcast a few days ago and they have posted the interview here. I talk about my early days of computing, merb, vertebra, rubinius and the story behind how Engine Yard got started and has grown. I’ll warn you though it’s a long interview, 120 minutes! Have a listen if you want to hear the story.

I’m going to be speaking at Oreilly’s Velocity conference next week on Monday June 23rd. Velocity is a conference about Web Scale operations and performance, lot’s of stuff about infrastructure and cloud computing.

I’ll be giving a short talk on Vertebra at 4:45pm on Monday and I will also be on a panel called “Getting into the Clouds” with some guys from Amazon, 3terra, Rackspace and joyent at 3:45pm. Should be fun.

Come see it if you’re in the area or already coming to the conference.

Velocity Conference

I’d like to point folks to our new Engine Yard Express VMWare image. This is a replica of an EY slice that you can download and use locally whether you are an EY customer or not. When you boot it up it will autotune mysql to use the proper amount of ram, boot up a merb and a rails app along with nginx and monit et all. It will give you a user/pass and show you the IP address as part of the boot process. So just boot it up and peek at the IP address on port 80 and port 81, the index pages of the rails and the merb app have detailed instruction on how to get a deploy.rb and deploy your own code.

Enjoy!

Here are the slides from my talk at RailsConf 2008. I’m too tired to write much more right this instant so I’ll let the slides talk. Rest assured you will here more about vertebra in the near future…

| View | Upload your own

There is a great tutorial/screencast about a new merb feature that I really like a lot called merb-slices. This is now part of merb-more, contributed by a long time friend of mine, Fabien Franzen(worked on ez-where with me)

merb-slices are “Little slices of MVC cake”. These are self contained merb apps with models, controlers, views and assets that you can distribute as rubygems. You can mount a merb-slice at a specific point in your router definition and you can override any part of the slice up in your main app. So in a way these are similar to what Rails-Engines promise, except merb-slices are built into the framework and will not break when merb itself is updated.

Check out the tutorial/screencast for a peek at how merb-slices work.

It seems like a lot of folks out there want there to be a battle royal between Merb and Rails. I just wanted to dispel this myth.

Merb was written because it scratched an itch that I and many of my hosting customers had, mainly memory footprint, speed, concurrent requests and simple maintainable code. This does not mean that Merb has to be thought of as a Rails competitor. Merb stands on its own merits just as Rails does.

I feel that both of these frameworks have great strengths and weaknesses and that they compliment each other in many ways. Face it Merb would never be around if Rails had not come first. And I see Merb as a great experimentation playground for exploring web framework functionality while learning a great many things from Rails and building a fresh view based on Rails strengths and weaknesses.

There is no reason that a lot of the goodness in Merb cannot be applied back to Rails, but it is no small feat to trim down rails size without breaking backwards compatibility.

I view it like this, more choices make the Ruby ecosystem a better place. So let’s just stop with the Rails VS Merb stuff. How about people choose what framework they want to use based on the frameworks merits and features rather then religious arguments about how my framework can beat up your framework.

The Ruby community in general has been exceedingly nice as long as I have been a part of it. I wrote Merb for a reason, if Merb’s philosophies agree with your tastes then great, use Merb. If Rails is your thing then that’s great too! We do not need a monoculture.

Rails has been Ruby’s killer app and has brought Ruby to the masses. But Ruby is growing up, there are many alternate Ruby implementations now and they all work together for the common good of Ruby.

I’d like the same thing to happen with Rails, Merb and all the other ruby web frameworks. Let’s all work together to make the Ruby ecosystem stronger. We do not need to fight amongst ourselves when there are plenty of other Programming language communities to fight with :P

I guess what I am saying is that I see the proliferation of ruby web frameworks as a sign that Ruby itself has “arrived” and is here to stay. So can’t we all just get along in Ruby web framework land?

My Deploying Rails Applications Book has shipped in print form! This book has been two years in the making and was rewritten a few times as the Rails deployment landscape shifted. I’m so glad it is finally in print \m/ Big thanks to all my co-authors and beta readers who gave valuable early feedback.

And in other amazingly beautiful news, Rails runs on Rubinius as of last night!. A huge round of applause for Evan, Wilson, Brian, Ryan, Eric and Eero, the rubinius core team and also to all 150 rubinius contributors. This is a huge milestone for the project. Being able to run rails, which is one of the largest and most complex ruby programs out there is huge. This means that rubinius is now ruby compatible for the most part. Now the team can start to steer their focus on speed improvements like JIT , polymorphic inline caching, LLVM and various other techniques for making rbx smoking fast.

Michael Klishin made a sweet mapping of a merb server’s boot process. Hopefully we can get stuff like this made for other parts of the framework as well as I think it can really help you find your way around the source code of merb-core.

Merb 0.9.3 is mostly (but not only) a bugfix release. We’ve not only added several features very useful for developers who want to use Merb for web services, but fixed many bugs, greatly improved spec coverage and quality, made documentation better, and did some performance-related work.

diffstat shows 95 files changed, 3359 insertions(+), 837 deletions(-)

Engine Yard is looking for one or two kick ass Erlang programmers to help work on our next generation cloud computing platform. Knowledge of ruby/xmpp/ejabberd/linux is a plus.

If you think you fit the bill then please email me directly at ezra@engineyard.com.

So i’ve spent this week hacking on Rails, specifically going spelunking in ActionPack and porting Merb’s rack machinery to rails. I figure that merb is a very nice experimentation ground and decided it was time to give some love back to the framework that inspired merb.

While still not complete, I have made significant headway on racking up rails in my github fork of Rails. I’ve added rack adapters for mongrel, eventedmongrel, thin, ebb and webrick. All of this is controlled via ./script/rackup in a rails app. So to start a cluster of 5 thin servers you would run this command:

./script/rackup -a thin -c 5 -e production

I do have to say that parts of ActionPack haven’t been touched in a long time and had accumulated some cruft, with the rails rack adapter that thin and ebb use there was a dogpile of wrappers going on. A web request was wrapped in many layers like this(the → means ‘wrapped in’)

raw request -> rack env -> Rack::Request -> CGIWrapper -> CgiRequest

That was far too many wrappers, it also turned out that some of these wrapper were making duplicates of all the CGI headers, meaning quit a bit of wasted memory on every request. I’ve slimmed it down to this now:

raw request -> rack env -> ActionController::RackRequest

With no more duplication of CGI headers, win!

I’ve also changed the giant mutex lock to be much smaller, it used to lock around the dispatcher callbacks, route recognition, controller instantiation, filters and action dispatch. Now it only locks around filters and action dispatch, Dispatcher callbacks, route recognition and controller instantiation all happen outside of the lock. This makes for a nice little speed boost with standard mongrel under concurrent load.

Of course this doesn’t work so hot in development mode when you have concurrent requests, it falls apart due to route set reloading and class reloading done in dev mode. But if you are just using your browser to test the app in dev mode you won;t ever notice since you won;t be making concurrent requests. In production mode the route recognition appears to be thread safe, more extensive testing and stressing will be needed to be 100% certain though.

All in all I feel like these are some big wins for Rails. And I’m not done yet, I plan on beating up ActionPack quite a bit more until it submits ;)