Merb

Apologies for the delay in posting about this. I was enjoying the fantastic Ajax Experience and jQuery Camp out in Boston, where I got a chance to hang with my jQuery compatriots as the huge Microsoft and Nokia announcement broke.

Now that I’m back, I want to talk about the cool new features we’ve been adding to Merb’s server. Let’s take a look at them one at a time:

• When a Merb server boots up and tries to bind to existing ports, it doesn’t crash. Instead, it waits for each port to become available and binds when they do. As a result, you can start up a new merb, and then gracefully kill the old one. This should make restarting clusters of merbs significantly less painful, and with almost no downtime.
• Merb’s clusters have been rewritten to take advantage of Ruby Enterprise Edition, which makes it much easier to share memory between workers in a Merb cluster. Using Ruby Enterprise with Merb 1.0 should yield around the same 30% memory improvement that Phusion Passenger yields.
• Merb clusters are now controlled by a master process, which can be told to gracefully kill all of its children, or reload the application code (but not init.rb or gems).
• Sending an INT to the master process (directly or via merb -K all) tells each of the worker processes to gracefully die.
• Sending a HUP to the master process tells each of the worker processes to gracefully die, but to start up a new cluster with reloaded code. This bypasses reloading Merb, gems, and init.rb, so the restart is much quicker than doing a full reload. This is also the same internal code used by development mode code-reloading, which has now been made 100% foolproof.
• Killing a worker process (either via INT, which is graceful, or KILL, which isn’t) will cause a new process to be respawned instantly. We’re talking basically no time at all, since no code needs to be reloaded (the spawner process forks right before binding to a port)

All of these improvements are just harbingers of even more improvements, making Merb’s master process even more powerful and smart.

The most important thing to keep in mind is that Merb has been significantly tuned for Ruby Enterprise, and with regular Ruby, there is a bit of overhead for the master processes. All of the features above will work correctly with standard Ruby, but you’ll get them for free (memory-wise), as well as quite a bit of improved memory overall in a cluster by using Ruby Enterprise.

Please please please check it out.

MiniGems is a lightweight drop-in replacement for rubygems to facilitate faster loading of gems as well as reducing memory consumption considerably. Depending on the amount of gems you have installed about 10-20 MB less RAM will be used, compared to the full rubygems library version, which keeps a cache of all gems and files referenced by them.

Minigems handles loading of required gems from your scripts. If however, other functionality is needed, the full rubygems library will be loaded automatically to continue normal operation. Therefor, it's completely backwards compatible with RubyGems.

There's currently a pending patch on RubyForge that integrates minigems into the standard RubyGems system. You can find more information at the tracker page; if you like minigems, please post a vote/followup there.

That said, the current standalone release of MiniGems is based on the same concepts, but implemented in such a way that it can be used without touching RubyGems itself. We've decided to release it in the wild, in the advent of the patch being included in the forthcoming RubyGems release. It's mainly meant for people to play around with it, so we can find bugs and other issues.

Also, the way you use the standalone version is quite different to what the RubyGems patch provides; see the RubyForge tracker for the actual use. For now, you will mostly use the following features, exposed by the minigem command.

To get started run: sudo minigem setup

Then, to let the 'merb' command utilize minigems.rb run: sudo minigem prepare merb-core

In case you want to revert back to plain rubygems: sudo minigem remove merb-core

If you're on a Mac, you can easily check out the difference in memory consumption between the two in Activity Monitor.

The standalone minigems gem can be found on Github and you can use rake install to install it.

Highlights of this release are:

• The sometimes-on, sometimes-off merb-freezer has been replaced with built-in bundling functionality, which stabilizes the local gem-cache feature that has been in Merb since 0.5. Fabien Franzen did a great job with it and written a tutorial you can find at Merbunity: http://merbunity.com/tutorials/18
• Thor is now an official Merb depenedency. It is used instead of Rake for the bundler, as it supports complex command-line options. Expect Thor usage in Merb grow over time ;)
• Fixed several reported bugs in the new sessions implementation released in 0.9.6.
• Fixed plugin generators.
• Fixed some test helpers.
• Fixed an issue in 0.9.6 that had libxml and memcached required at runtime.
• Session store that uses memcached now works with both memcached and memcache-client libraries.

We did a short (5 days) release cycle to fix all the problems after some large merges right before 0.9.6.

What to expect in Merb 0.9.8.

Merb 0.9.8 will feature a new, very powerful but backwards incompatible router and (hopefully) a new merb-cache.

You can read about the at the wiki:

• http://github.com/carllerche/merb-core-enterprise-edition/wikis/home
• http://github.com/carllerche/merb-core-enterprise-edition/wikis/whats-new-with-the-router

Also, read about the new cache at Merbunity:

• http://merbunity.com/tutorials/15

API freeze.

Merb 0.9.8 is expected in about a week, around September 20th. Merb 0.9.9, expected to be released approximately September 27, will entail an API freeze. After 0.9.9, we will be adding additional spit-polish and fixing bugs only.

Merb 1.0.

Merb 1.0 is going to be released at MerbCamp on Oct 11th to 12th. More information at merbcamp.com.

If you want to help out with polishing merb, it’s right about time to give gems in merb-more some love. ;)

As we approach 1.0 and work toward locking the API, we’re trying to figure out what sorts of things drive people nuts in Rails.

Basically, we’re looking for your top one or two gripes that drive you crazy. Because Merb borrows so heavily from Rails’ philosophy of convention over configuration and MVC model, it’s entirely possible that we’ve inadvertently borrowed some of the things that frustrate people.

So, gripe away!

Over the past few releases, there have been a number of breaking changes, which has frustrated a few people who have been working with Merb.

Let me explain exactly what’s going on. As I posted previously, we’re currently working towards Merb 1.0, which we intend to release on October 11, at Merb Camp. As part of Merb’s underlying philosophy about our public API, we intend to avoid breaking the public API, except for extremely urgent reasons, before the next major release (2.0).

As a result, it’s extremely important that the API that we expose for 1.0 is actually an API we’re comfortable standing by for the long haul. There’s still quite a few parts of Merb that were dragged in from Merb 0.5, and possibly even from Rails. As we march toward 1.0, we’re doing a full audit of the merb codebase and shoring things up.

Sometimes, as in the case of our ground-up rewrite of the form helpers or router, this requires some API changes. In other cases, as in the case of our ground-up rewrite of the dispatcher or sessions, this has virtually no public API effects.

So again, while it may seem as though a lot of things are changing quickly, those changes are in the service of post 1.0 stability. Once we get there, feel free to judge us for how successful we are at remaining stable. For now, use 0.9.x if you’re willing to put up with some changes. Otherwise, it’s a short jaunt to 1.0.

Also keep in mind that you can freeze to a specific version of Merb, and wait to upgrade all at once in October.

Following up with our 2-week release cycle, we’ve released Merb 0.9.5. In anticipation of the next release, “Time Machine,” which will allow for fool-proof code reloading and instant deploy restarts via Kernel.fork, we’re calling this release Spoons and Knives.

We know that some of you guys have been a bit frustrated with the more common API changes than there have been in the past. By way of explanation, we’re hoping to get Merb 1.0 released by MerbCamp on October 11, so we’re focusing on breaking what needs to be broken now, so that Merb can remain relatively stable after 1.0.

Some changes in 0.9.5:

• The helpers have been completely refactored:
  • form_for should now look like <= form_for … %>...< end =%> as we discussed in a previous post
  • helpers like check_box_field and check_box_control have been merged into a single helper: check_box, which will automatically determine whether they should be bound or not based on the current context and parameters (check_box(:awesome) is a bound check box while check_box(:checked => true) is an unbound check box).
  • All form helpers, including unbound helpers, are executed inside a builder context. You can completely ignore this fact, but if you’re interested in having more control over your forms, we’ll be posting more about this in the next few weeks.
  • If you don’t want to have to change helpers yet, you should be able to stay on merb-helpers 0.9.4, while working on upgrading.
• Merb’s deferred actions, a way to handle requests in threads on event-driven servers (so the event loop is not blocked by long requests) now works on Thin with the default Merb Rack application (read: out of the box).
• We’ve added clear_content to the controller context, which allows you to clear out previous throw_contents.

Over the past week, Merb has been working on merging our core extension library with DataMapper’s, to provide a single non-conflicting library for both Merb and DataMapper to use. Unfortunately, it has caused some confusion for people who use edge merb but do not really dive into development process.

Here’s what changed and why:

• Merb core extensions and support classes were extracted into merb-extlib.
• merb-extlib was merged with DataMapper’s extlib on a separate branch.
• merb-core, merb-more, merb-plugins all have a branch where the merged extlib is already used. For instance, Merb::SimpleSet has become Extlib::SimpleSet.
• Extlib, Merb and DataMapper release rake tasks were unified; the extracted release task was added to extlib/tasks and rakefiles just need to override a few constants for the framework in question.
• once the Rakefiles are updated we will merge migration branches into Extlib’s master and merb subprojects.
• Finally, we will move Merb’s inflection code into extlib

In the meantime, expect edge to be a bit unstable as we make this transition.

Recent changes in Merb.

New merb-gen.

The Merb generator suite has been rewritten using the Templater library by Jonas Nicklas. This means that Merb generators now:

• have a better description of options
• have named arguments instead of positional only
• can be easily unit tested
• have much cleaner code
• have no ActiveSupport dependency :-D

The new generator has (almost) the same user interface as previous one but uses named arguments. Run merb-gen to see the list of generators and merb-gen to see its options and description. If you want to take a look at the new generators, make sure you’re operating against HEAD, as the released gems do not yet include the Templater-based generators.

Merb core extensions library.

Merb’s extensions of Ruby core classes now live in a separate library, called merb-extlib (github.com/wycats/merb-extlib). This is the first step towards merging Merb and DataMapper’s core extension libraries into a single small, fast library. Because Merb profiles and benchmarks extensively, our core extensions library is extremely fast. Even though Merb and DataMapper do not have significant overlap in our extension libraries, we believe that it’s important to create a single library that developers can work against, and avoid duplicating work (or, even worse, having two different implementations of the same extension).

Road to 1.0.

Once Merb hits 1.0, no serious changes or breakages will be allowed (at least until 2.0). As a result, Merb’s contributors decided to make all necessary breakages in the 0.9 series, which is a developer release. Between 0.9.0 and 0.9.3, we have made significant enhancements to the framework, with only minor breaking changes to the public API, but there are some changes to come in the upcoming months.

Some things on the roadmap between 0.9.3 and 1.0:

• Rewritten merb-gen (basically done)
• merb-extlib extraction (first step done)
• Improvements to merb-cache
• Rewrite of merb-helpers (currently in a branch; almost ready to merge in)
• New awesome merb-notifier (extending merb-mailer to support other forms of notification like Jabber)
• Getting rid of load_dependency quirks
• Significant updates to our (already very powerful) router. The New router will be able to generate URLs from regexp routes (in most common cases) and will use a tree to work even faster
• Support for multibyte characters and localization in merb-more
• Modification of the Sessions API to make it less brittle and easier to extend
• Finish the move over to YARD documentation syntax and YARD documentation generation

We also plan to do a complete code audit as well as performance benchmarks before 1.0 to be sure that the codebase is as solid and reliable. Finally, we will lock down the public, plugin, and private API differences so that the 1.x series will be unlikely to cause major API breakage.

If you’re interested in getting involved or helping out, now’s the time!

Stay tuned.

If you're looking for more Merbness, there's a couple places to go.

Planet Merb aggregates RSS feeds from active members of the Merb community. It's a great way to see what everyone is up to, whether related to Merb or not. If you'd like to have your blog included, email Ivey with your name and RSS feed.

If you have some cool Merb-related links to share, you can post them to the Merb Reddit. It's just getting started, but expect to see more links to Merb mentions on the rest of the net.

If you haven't joined Merbunity yet, what are you waiting for? Come be a part of a community for Merb developers, currently featuring screencasts and tutorials, but with more content on the way.

Do you have other places to keep up with Merb? Share them in the comments!

The latest change to edge Merb doesn’t so much change the end-user functionality as it fixes a long-standing issue with how inheritable accessors work. If you’re mostly interested in how Merb work for an end-user, and not so much with Merb’s internals, it’s safe to skip this post.

Let’s start easy, with regular accessors:

class Foo
  attr_accessor :bar
end

@foo = Foo.new
@foo.bar = 7
@foo.bar #=> 7

Regular attr_accessors are part of Ruby proper, and allow you to easily create accessor methods for instance variables in a class. The above code roughly converts to:

class Foo
  def bar
    @bar
  end

  def bar=(obj)
    @bar = obj
  end
end

This works great for simple instance variables. However, Ruby also has class variables, and Rails (as well as Merb) provides a simple way to bind methods to class variables:

class Foo
  cattr_accessor :bar
end

Foo.bar = 7
Foo.bar #=> 7

That works great too, subject to the normal rules of class variables. Specifically, class variables are shared by a class and all of its subclasses, so they cannot be used to implement behavior specific to different subclasses (for instance, specific Controllers). Here’s a quick demonstration:

class Merb::Controller
  cattr_accessor :blinking
end

class First < Merb::Controller
  self.blinking = true
end

class Second < Merb::Controller
  self.binking = false
end

First.blinking #=> false

As you can see, cattr_accessors cannot be used for controller-specific behavior. The next thing to try would be class instance variables:

class Merb::Controller
  class << self
    attr_accessor :blinking
  end
end

class First < Merb::Controller
  self.blinking = true
end

class Second < Merb::Controller
  self.binking = false
end

First.blinking #=> true
Second.blinking #=> false

At first glance, this seems like the ticket. Sadly, this is not the way these sorts of accessors are usually used. What Merb (and Rails) wants to do is be able to set a preference on Merb::Controller and have it automatically inherit onto the child controllers unless specifically overridden. Like:

class Merb::Controller
  attr_inheritable_accessor :blinking
  self.blinking = false
end

class First < Merb::Controller
  self.blinking = true
end

class Second < Merb::Controller
end

First.blinking #=> true
Second.blinking #=> false

In particular, we use this for things like layout location, which we want to allow to be declared in the Application controller, and have automatically percolate down to other controllers. This is also used in plugins quite a bit, where Merb::Controller will get a default inheritable attribute, which can be overridden at a lower level.

The Rub

Rails comes with an inheritable accessor, which, until now, Merb used. Unfortunately, it has the following behavior:

class Merb::Controller; end

class First < Merb::Controller; end

class Merb::Controller
  class_inheritable_accessor :blinking
  self.blinking = true
end

First.blinking #=> nil

To make a long story short, the reason for this is that the accessor gets inherited at the same time as the class gets inherited, which is before the accessor was defined (in this case). In practice, this leads to extremely brittle load order, as it is fairly common practice for plugins to add inheritable_accessors, and if the plugin is loaded in after the application’s controllers, very difficult-to-debug behavior can result.

A survey of #merb revealed that this enigmatic issue had struck quite frequently, so it was time to resolve the issue.

The New Behavior

In attacking the issue, we decided that the new inheritable accessor had to support the following behaviors:

• Resolve the load-order issue. Child classes should continue to look up the inheritance chain until a superclass has stored an accessor.
• Support ||= to store a child accessor. if Parent.foo nil, Child.foo ||= "Hello" will result in Parent.foo remaining nil, but Child.foo “Hello”
• Support << to modify a parent object. If Parent.foo [1], Child.foo << 2 will result in Parent.foo remaining [1], but child.foo [1,2]
• Support modifying child objects in place. If Parent.foo [1,2,3], Child.foo.reverse! will result in Parent.foo remaining [1,2,3], but Child.foo [3,2,1].
• Support similar semantics for a Hash. If Parent.foo {:x => "y"}, Child.foo.merge!(:z => "a"} will result in Parent.foo remaining the same, but Child.foo {:x => “y”, :z => “a”}

We were able to achieve all of the above, with one caveat:

Parent.foo = "Hello" 
Child.foo
Parent.foo = "Goodbye" 
Child.foo #=> "Hello" 

This is because the way the above behavior is implemented is by storing the parent value on the first READ, which turns out to be much more versatile and correct that storing the parent on inherit. As it turns out, it’s almost always the case that the first read of an inheritable attribute happens at runtime, while the writes happen at boot-time, which makes this a perfectly reasonable tradeoff.

Of course, as I said above, this is a fairly esoteric change that doesn’t affect the average user of the framework, but it does dramatically reduce certain strange behavior from plugins.

We try hard not to break public APIs in Merb, but in the lead-up to 1.0, we’ll probably be doing a bit more breakage than we have in the past in an effort to reduce breakage post 1.0.

Today is one of those days. In order to improve the way ERB handles blocks, we were forced to make a substantial change that I will detail here.

First, some background.

The Dirty Truth

Frequently, you’ll want to write a helper that takes a block. An example of such a helper is the form_for helper, which looks something like this in Rails:

<% form_for :foo do |f| %>
  <%= f.text_field(:zoo) %>
<% end %>

The semantics you want are something like this:

1. Capture the contents of the block (<%= f.text_field(:zoo) %>)
2. Create some boilerplate (<form ...><%= contents of block %></form>)
3. Concatenate the results of the entire thing into the buffer

But wait! As it turns out, those are only the semantics you want if you are being called from ERB. Let’s take another example, the gentle content_tag helper.

In ERB, it works like this:

<% content_tag(:div, :id => "foo") do %>
  Hello, my name is Foo
<% end %>

The above semantics still work just fine, so all is well. But lots of people use content_tag in regular Ruby code, as in:

  def link(text, url)
    content_tag(:a, :href => url) { text }
  end

which will be called from ERB as:

  <%= link("A Link", "http://example.com/foo") %>

Now look back over the semantics below. Step 3 is “Concatenate the results of the entire thing into the buffer”. But wait. What buffer? If we do a concat onto the ERB buffer, the results of the #link function will still get concatenated, as well as the results of the content_tag function.

I won’t go any deeper, but suffice it to say that the fundamental problem here is that helpers that take a block cannot simply return a string. In other words, the following doesn’t work:

<%= form_for :foo do |f| %>
  <%= f.text_field(:zoo) %>
<% end %>

The reasons for this are kind of arcane, but the bottom line is that since it doesn’t work, we’re forced to use the concat hack, which then has the problem of having to deal with figuring out whether it was called from ERB (and thus should concat) or a regular helper (and thus should return a String).

How Does Rails Do It?

When confronting issues such as these, the first step is usually to investigate how Rails handles the problem. In general, they do tackle the problem in some way or other, and so it’s usually worth figuring out what their approach is.

Let’s whip up the latest tag_helper.rb on Rails edge and pull up content_tag

def content_tag(name, content_or_options_with_block = nil, options = nil, escape = true, &block)
  if block_given?
    options = content_or_options_with_block if content_or_options_with_block.is_a?(Hash)
    content_tag = content_tag_string(name, capture(&block), options, escape)

    if block_called_from_erb?(block)
      concat(content_tag)
    else
      content_tag
    end
  else
    content_tag_string(name, content_or_options_with_block, options, escape)
  end
end  

The important part is the middle section, beginning with block_called_from_erb?. Let’s take a look at the implementation of block_called_from_erb?:

BLOCK_CALLED_FROM_ERB = 'defined? __in_erb_template'

# Check whether we're called from an erb template.
# We'd return a string in any other case, but erb <%= ... %>
# can't take an <% end %> later on, so we have to use <% ... %>
# and implicitly concat.
def block_called_from_erb?(block)
  eval(BLOCK_CALLED_FROM_ERB, block)
end

So effectively, every time you use content_tag, or any helper that uses content_tag, Rails executes an eval inside the block to check whether __in_erb_template is defined. This is sort of an acceptable solution, but it means that any helper that does a concat needs to handle this case.

The solution

Feel free to skip this section if you just want to kno

I alluded to the solution above:

<%= form_for :foo do |t| %>
  <%= f.text_field(:zoo) %>
<% end %>

If this would compile, then all would be well. Unfortunately, Erubis (like ERB) doesn’t support multiline. As I dug into why, it became messier and messier. To get a sense, the following two cases need to be treated differently:

<%= form_for :foo do |t| %>
  <%= f.text_field(:zoo) %>
<% end %>

<% if @foo %>
  <%= @bar %>
<% end %>

The first case needs to be compiled into something like @_out_buf << (form_for(:foo) {|f| @_out_buf << f.text_field(:zoo)} ).to_s. The second case gets compiled into if @foo; @_out_buf << (@bar).to_s; end.

The bottom line is that with a lot of hacking, I was able to get Erubis to compile blocks correctly, in the following form.

<%= form_for :foo do |t| %>
  <%= f.text_field(:zoo) %>
<% end =%>

Note the =%> on the last line. That tells Erubis that this end is the end of a block opened up via <%= previously. This provides a bit of symmetry from multi-line expression in Erubis (they open in <%= %> and close in <% =%>).

The Bottom Line

Effective today’s edge, helpers that take blocks are to be instantiated via the <%= %>...<% =%> pattern. If you make a new helper that takes a block, use #capture to get the contents of the block, and simply return a String. Let that sink in for a moment. Helpers that take blocks simply return a String.

Another Note

Also coming very soon to edge are completely rebuilt Form Helpers. The new helpers were refactored with modularity in mind, making it very easy to create your own Form classes to use in your application with as little code as possible. Stay tuned for more on this.

Merb 0.9.3 is out. Check Ezra’s announcement for the details.