Nuby on Rails

This is the way the world ends / Not with a bang but a whimper. —T. S. Eliot

BIBERACH, GERMANY—Late last week, Christian Neukirchen’s influential tumbleblog Anarchaia breathed its last.

Anarchaia took the well established blog form and turned it into a multimedia stream of consciousness featuring photographs, poetry, lyrics, and links. Topics ranged from the merely curious to the highly technical. It managed to maintain the same visual theme throughout its lifespan which included a half-dozen posts almost daily.

The term tumblelog was coined in mid-2005 by Why the Lucky Stiff on his RedHanded blog.

Anarchaia also inspired the popular projectionist tumblelog, which launched a few months later. Entire startups were subsequently built around the form.

It is survived by the other Christian Neukirchen projects Rack, test/spec/bacon and his own blog. There are rumors that it has already reincarnated as Trivium.

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

I’ve had to rely at times on silence and on talking quick / Defending myself with nothing but my walking stick. —Buck65

Here are nine easy tips that will help you communicate better at your next conference.

Dan Grigsby’s presentation at RubyFringe was an intentional example of this. All the titles were at the top, with humorous stock photos below.

Keep it in the top third, if possible.

Giles Bowkett is such an entertaining speaker that people once skipped the first 20 minutes of lunch to hear the remainder of his presentation at RubyFringe (which involved more than 400 slides).

He also used only the typefaces included with Mac OS X, including Futura Condensed Medium and Condensed ExtraBold, which work really well in bright colors on black. So even if you don’t choose to buy a single typeface, you can assemble a great-looking presentation.

• Giles Bowkett Videos – Start 5 minutes in to see the slides.

It’s easy with either:

Copy as RTF – A TextMate plugin. You can paste the syntax-highlighted text and even edit it afterward in Keynote.

pygments – A command-line syntax highlighter written in Python. It’s used at GitHub to emit HTML but can also emit RTF from any source file. The resulting rich text can be pasted into Keynote.

pygmentize -f rtf -o out.rtf code.rb

Choosing just the right transition can soak up a lot of time and adds absolutely nothing to the content that people remember afterward.

Dan Grigsby also noted that transitions and multi-step builds make it difficult to go back and forth in the presentation since you have to wait for the transition to finish. Unless…

Useful Keynote shortcuts (while the presentation is playing).

Key	Description
/	Show a list of keyboard shortcuts.
H	Pause the presentation and go to the last used application (useful for demos). Command-tab back to Keynote to resume the presentation.
= or -	Show a thumbnail menu that can be used to jump forward or backward to a specific slide. Use the arrow keys to select and the enter key to jump.
B	Pause and show a black screen.

I love live coding but often it goes awry, creating an awkward situation for both the presenter and the audience.

Give yourself some insurance and either record a short screencast that you can narrate during the presentation, or take screenshots that you can refer to.

Extra Credit!

If you’re speaking at a conference, you’re probably doing it to promote yourself, your projects, or your business. Make it stick in people’s minds by distinguishing yourself with a color scheme and a typeface that communicate the attitude you want to be remembered for.

Choose a color scheme and use it for all your presentations. Ideally, it would be the color scheme of your company or personal blog. If you’ve paid for a corporate identity, use it!

Resources

• Color Burn Widget – A Mac OS X dashboard widget with a new color scheme every day.
• ColourLovers.com – Tons of color combinations for every attitude.

Again, buy a typeface and use it on your blog and in your presentations.

They’re not as expensive as you might think! You can get a single font for $20.

Here are some nice condensed ones as mentioned above:

• Antenna
• Caslon
• Block Gothic
• Garage Gothic

Or try these shops:

• FontShop
• Sudtipos
• Vllg
• T.26
• Process Type Foundry
• H&FJ
• Typographica

I saved my favorite for the end…

A presentation remote gives you the freedom to step away from the lectern and talk directly to the audience. The remote that comes with Mac laptops doesn’t count! It only works if you have a direct line of sight to the infrared receiver on the front edge of the laptop.

A radio frequency transmitter works much better. The Kensington Presentation Remote can be bought for about $40. It works out of the box without the need to install any drivers, and it’s less distracting than phone-based options.

See you in Berlin!

I’ll be in Berlin at RailsConf starting this Sunday. Find me and get a free PeepCode t-shirt!

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

For the third year in a row, it’s the Seattle.RBBQ at Geoff’s house in Seattle!

• Tuesday, August 26, 7-9 pm.
• I’ll provide pizza. Bring any other beverages or food that you want to eat.
• Directions and Bus Numbers

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

PeepCode is hiring! I interviewed two part-time editors last week and am now looking for a virtual office assistant. I’m offering $18/hour to start for 2-3 days a week. This will probably turn into a full-time position with increased pay and benefits within the next few months.

This is a non-programming position. Duties include:

• Internet research
• Customer service
• Promotion via email and web
• Data entry
• Various other creative tasks

Preferred skills are:

• Ability to use the web and email
• Familiarity with a Macintosh
• Willingness to learn new things
• Regular communication with Topfunky headquarters
• Ownership of a business license (sole proprietor is OK, we can help you fill out the paperwork if you don’t have one already)

None of these tasks are time-sensitive, so you can work from home at whatever time of day (or week) is best for you. Send your resume via email to peepcode@topfunky.com with VA in the subject.

Dublin

I had a great time in Dublin at the end of June. I spoke at an open source conference and met several other Rubyists while I was there.

I’ve frequently felt that I should bring home some tchotchke from each of the places I visit. While in Dublin I realized that it’s not extra things that I really want, it’s people!

If you’re traveling for business or pleasure, I highly suggest that you find the local Ruby group or find a local co-working space and meet people there. I had breakfast with Ana Nelson, who introduced me to Paul Campbell of Contrast, who is working on an error reporting webapp for Rails developers.

He introduced me to Eamon Leonard and David Coallier who happen to be a few musically inclined PHP developers. We had a great time exploring the pubs of Dublin, and I learned that Guinness beer has a different taste in Dublin than anywhere else in the world.

Oh, and a few members of Ruby Ireland hosted a nice dinner in the city.

So the next time you find yourself in an unknown city, meet the locals!

Elsewhere

I just published a new PeepCode screencast about Phusion Passenger (technical editing by Phusion staff).

I’ll be speaking about podcasting and entrepreneurship at BizJam Seattle on Wednesday, July 9.

Jim Freeze wanted me to mention that the Lone Star Ruby Conference is open for registration. I’ll be at RailsConf in Berlin at that time, but I’m sure it will be a great conference if you can get yourself to Texas. I frequently post other worldwide conference and workshop events at Ruby on Rails Workshops.

Also, I’ve helped to launch a Deck-like ad network for Ruby blogs. We have several top quality publishers and have a few spots for advertisers. If you have a product, website, or service that would be of interest to Rubyists, check out Ruby Row.

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

I truly feel that I have the coolest job in the world.

Two weeks ago, I was asked to speak at the IOTC Conference in Dublin on June 18-20. Not only that, but they will be streaming live video of the event as it happens!

Few conferences record even the audio of their lectures, so I’m not sure how they managed to pull this off. They must be using some of this new open source software that everyone is talking about!

They tell me that you’ll be able to view it via a link that will be posted here, next Thursday at 10:25am Dublin time.

I’ll also be hanging out with members of Ruby Ireland in Dublin on Thursday night.

Then I’ll be hurrying back home to be with my son, who aspires to be the next mini me.

At PeepCode

The ever diligent Ryan Daigle has updated our Rails 2 PDF for Rails 2.1. If you purchased the first copy, you can login or revisit the link originally sent to you for the free update.

If not, now is a great time to buy a copy!

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

Some of the things I’ll be bringing to RailsConf 2008 in Portland next week:

Item	Qty	Purpose	Location
PeepCode T-Shirts	45	Free!	Find me!
Mannequin	1	Bodyguard	Heroku booth
Utilikilt	1	↑
Samson Microphones	4	Podcasting	Heroku booth at lunch and afternoon break
Life-size poster of Peter Cooper	2	Why not?	Conference walls
PeepCode Coupons	?	Try it!	Heroku booth

Live Podcast Interviews!

My friends at Heroku have offered me a couch at their booth in the exhibit hall to perform a few podcasts. Come see a Rails Podcast recorded live during lunch and afternoon breaks!

Time	Event
Friday, Lunch	Adam Keys interviews me
Friday 3:40	GitHub founders
Saturday, Lunch	Phusion Passenger (mod_rails)
Saturday, 3:40	?

Sessions

I’ll be moderating one panel and attending an evening session.

Time	Event
Friday 2:50pm	The Profitable Programmer
Friday 9-10pm	Podcasting BOF

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

We need new inventions that reveal people’s true intentions, a portable pride protector, affordable lie detector…—Buck65

I was initially skeptical about mod_rails. The developers had the cojones to start promoting it weeks before it was ready, and they managed to generate a lot of hype around it before anyone had used it.

And we all know that hyped products never deliver!

Mostly, I was probably one of those developers described when they said

...we believe that experienced Ruby on Rails developers don’t consider deployment to be a problem; probably annoying at most…

Which is true. I was pretty happy with my current setup of Nginx and Mongrel. My only problem was that Nginx would often fail to create a pid file when the logs were rotated every night.

I still use Nginx and Mongrel at PeepCode, but I’ve switched a few smaller apps on a separate VPS to mod_rails. It turns out that there are many benefits to using mod_rails, both in development and production.

Great for development

It’s easy to ignore the pain of development. I didn’t realize how tedious it was to start and stop Mongrel while in development. It was also the reason I keep so many terminal tabs open at once.

With mod_rails, apps are always ready whenever I need them. If I need to watch the logs, it’s an easy tail -f log/development.log.

I have many pages that access dynamically generated graphs. A single Mongrel would process each serially, which would often take quite a while. With mod_rails, it’s nearly instant since new processes are spawned to handle the extra requests in parallel.

Configuration

I followed Manfred’s article to start using mod_rails in development. Two tweaks were to edit /etc/hosts and to point Apache to a local config file in my home directory.

First, edit /etc/hosts with fake names for the applications you’ll be developing against.

127.0.0.1    localhost peepcode.local podcast.local

Next, in /etc/apache2/httpd.conf, I set the RailsEnv to development and myself to the RailsDefaultUser. I also included a config file in my home directory (stored in source control). This config file includes all my virtual hosts and is the same on both my laptop and my desktop machine.

RailsEnv development
RailsDefaultUser topfunky
Include /Users/topfunky/repos/configs/mod_rails_vhosts.conf

The mod_rails_vhosts.conf file has several VirtualHost sections like this:

ServerName podcast.local
DocumentRoot /Users/topfunky/repos/podcast/public

Easy to debug visual plugins or graphs

Sometimes I need to visually debug code located in plugins or frozen gems. For example, I worked on some enhancements to my sparklines gem while viewing graphs on a reporting page. Normally, this would be impossible since Rails doesn’t reload itself for changes in the vendor directory.

Rstakeout comes to the rescue!

rstakeout "touch tmp/restart.txt" "vendor/**/*"

When I edit a plugin or gem in the vendor directory, mod_rails reloads the whole application and I can see the result immediately.

Rotate Logs

Other production tasks become very easy with mod_rails. Rotating the Rails logs is as easy as touching the restart.txt file.

Here’s /etc/logrotate.d/rails:

/var/www/apps/rubyonrailsworkshops.com/shared/log/production.log {
  daily
  rotate 7
  compress
  missingok
  sharedscripts
  postrotate
    touch /var/www/apps/rubyonrailsworkshops.com/current/tmp/restart.txt
  endscript
}

Coexist

Many people are sad that mod_rails doesn’t yet support rack. No problem. I just followed Coda Hale’s classic article and am running Merb under Mongrel through Apache, alongside a few mod_rails apps.

Problems

The only initial problem I’ve experienced is that Apache seems to use much more memory than Nginx did. A few days after switching to Apache, the memory usage skyrocketed and killed all my mod_rails apps. I had to restart Apache to get everything back.

I’m working on a god config file to keep tabs on everything. Does anyone have a good one yet?

At PeepCode

Scott Chacon recently published a Git Internals PDF that Jamis Buck called required reading for all Git users. He also launched GitCasts.

Mike Mondragon and Luke Francl finished a book on MMS2R. If you’ve ever tried to receive email into a web application, you know it can be quite a pain. MMS2R is a gem that makes it super simple to receive both plain text and multimedia attachments. It’s made me want to write an email-powered application just so I can use the concepts they show in the book!

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

I just realized that it’s been almost two months since I’ve blogged! Some saw the announcement on Twitter, but for the rest, here’s what I’ve been up to:

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

David Letterman often asks: Is this anything? And now I ask you.

• Newer MacBook Pros and MacBook Airs can detect trackpad gestures (three finger swipe up, down, left, right).
• A new preference pane called MultiClutch makes it easy to call key commands when gestures occur.
• ThisService turns command line scripts into services with key commands.

The end result?

• Swipe up to git push
• Swipe down to git pull
• Swipe left to see git status
• Others are also possible!

Easy setup

Download my pre-packaged services and drop in Library/Services (local or systemwide…your choice).

Download MultiClutch and set commands for multitouch gestures that call the services.

You can try it at home!

Use ThisService to make a service that calls any shell script or even a Ruby script (start autotest, run rake, etc.).

I used a simple shell script that echoes a command to the Terminal.

#!/bin/sh
echo "git pull"

Product placement

See the PeepCode Git Screencast to learn how to use Git.

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

When I suggested that you write your own blog software, I had no idea that some people would actually take me seriously!

So here’s another tidbit to help you on your way: custom Textile tags.

The Feature

As a developer, it’s nice to use Textile or Markdown for text entry. Simple tags will be converted to HTML, which will save typing and automatically apply some typographic rules.

But it can go one step further if you’re using RedCloth. You can subclass RedCloth and add methods that define your own tags. This is a great way to add shortcuts for complicated sets of HTML or inject hooks for CSS and Javascript.

To create Figure A, I used a technique mentioned by Garrett Dimon in a Digital Web article. It involves a div enhanced with a few CSS classes, enclosing a caption and an image. In addition, I have the power to make it small, medium, or large. A more complete solution would even allow for alternate backgrounds and placement.

But that’s a lot to type every time I want to reference a captioned image! Instead, I wrote a custom Textile tag to do it for me.

Usage

Here’s the custom tag I use:

figure(med). RedCloth 3 Logo | redcloth3-title.jpg

Let’s break it down:

• figure is the custom tag name. It’s a hook into a textile_figure method in my application’s subclass of RedCloth.
• med is a CSS class that will be applied to the div that wraps the figure. This gives me easy control over the size of the figure, and could even be used to pass multiple classes for other effects.
• Next comes the caption. I chose to use a pipe as the delimiter between the caption and the image name.
• It ends with the name of the image.

In that single line, I have all the information I need to construct a figure. Those 50 characters will result in over 200 characters of HTML. I can also use a few lines of Ruby to assign an auto-incrementing figure letter (“Figure A”) and reuse the caption as an alt tag for the image.

Implementation

First, choose a color for your subclass. CaneSugarBrownCloth? MintJulepGreenCloth? It could take hours.

Next, create a file for your subclass. I put yellow_cloth.rb in the lib directory and required it from environment.rb.

unless RedCloth
  require 'RedCloth'
end
class YellowCloth < RedCloth
  # ...

Next, define your custom method. I have a simple ruby tag that applies a CSS class for use with my code highlighter.

def textile_ruby( tag, atts, cite, content )
  # TODO Use the arguments to build some HTML 
  #      and return a string.
end

The arguments passed in can be reused or ignored.

• tag is the name of the tag passed in (ruby in this case).
• atts is a pre-assembled string with passed CSS classes, IDs, hard-coded styles, and language identifiers. If you want to augment this, you’ll need to parse the string and append your own CSS classes (I did this for the figure tag).
• cite is rarely used.
• content This is the actual code, text, or paragraph that follows the tag. In the figure tag, I decided to parse this further and use the elements as the caption and image name.

The figure tag is more complicated.

• I add a figure class to any CSS classes passed in. In order to do this, I need to parse the atts and extract existing CSS classes.
• I split the content on the pipe and use the elements as the caption and image name. You could prepend a filepath onto the image name if all your images are stored in the same place.
• The figure letter is dynamically incremented (Figure A, Figure B, etc.).

def textile_figure(tag, atts, cite, content)
  span_class = "" 
  if atts =~ /class="([^\"]+)"/
    span_class = $1
  end
  (caption, img_url) = content.split("|").map { |w| w.strip! }
  figure_name = "Figure " + @figure_counter.chr
  figure_id   = figure_name.downcase.gsub(" ", "-")
  @figure_counter += 1
  # TODO Construct the HTML
end

Finally, here’s the code to setup the counter:

def initialize(*args)
  @figure_counter = ?A
  super
end

To use your new class, you can override the built-in textilize helper or write a before_save callback on your model to convert the text to HTML with your custom subclass.

before_save :textilize_body
def textilize_body
  self.body_html = YellowCloth.new(self.body).to_html
end

The entire code can be downloaded: yellow_cloth.rb

Bonus

If you use TextMate, you may be aware of the “Show Web Preview” menu item.

It can pipe the text of the current document through its own textile.rb script or through your custom script. I wrote a simple script that uses my subclass instead of RedCloth.

# script/textile.rb
require 'rubygems'
require 'yellow_cloth'
textile = ARGV[0] ? File.read(ARGV[0]) : STDIN.read
puts YellowCloth.new(textile).to_html

Then use the full path to this script as the argument to ‘Pipe text through’:

/Users/topfunky/repos/funkyblog/script/textile.rb

Product Placement

I’ll be speaking at both RailsConf Portland and RubyFringe.

After many requests, I’m proud to introduce the new PeepCode Unlimited plan that gives you a year of access to all PeepCode content. On sale for only $139 until March 21.

Merb is a great compliment to Rails or a capable web framework in its own right. A draft of the PeepCode PDF on Merb is now available (includes a free upgrade when the final is published in the next few weeks).

PeepCode has been growing quickly and we have a ton of new content in the works. Ryan Daigle’s Rails 2 PDF recently became the best-selling PeepCode product of all time. Cody Fauser’s ActiveMerchant PDF is already being called a must read.

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

Three pens for five dollars, black, red and blue, recommended by scholars. —Buck 65

Two events from January, unrelated but relevant:

1. I toured the offices of Highgroove Studios and viewed their upcoming Scout reporting tool. They’ve written their graphing tool to allow easy comparisons between any group of measurements on the system, such as hits to their blog vs. new signups to their webapp. Comparing seemingly unrelated datasets often results in unexpected conclusions.
2. Information design guru Edward Tufte posted a video review of the iPhone. One of his widget-related suggestions was to include sparklines to show months of history instead of just a single number.

A third event preceding either of these was a redesign of my internal reporting page to use sparklines extensively.

The Problem

Previously, I used Gruff to generate one or two 400×300px graphs to show a few important metrics. But it kept bugging me that I had all kinds of information in the database that I couldn’t visualize.

I like to use a single reporting page with as much information as possible. I also want to be able to apply a different stylesheet and view these pages on my iPhone or a laptop with a small screen.

The problem with large graphs is that adding more of them means occupying a lot more space on the page. Having a bunch of large graphs makes it hard to get a quick idea of where things are at.

Books such as Information Dashboard Design observe the fact that graphs are best at showing general trends, not specific numbers. Tables are great at showing specific numbers, but not general trends. So why not omit all the labeling and just focus on the graphic?

So I scrapped all the labeled, numbered, scaled graphs and replaced them with purely graphical sparklines that each show about three months of data in 300×30px. When I need to know the exact numbers, I can look at a nearby table that lists the data for the current day or the past week.

Go Small

I started with a graph of the number of dynamic hits to the site, as reported by the Rails analyzer and stored in the database.

This isn’t very useful alone. The graph isn’t scaled from zero, so it’s impossible to make specific conclusions. But it is valuable for finding out about relative changes from day to day.

I also added a white line as a target value for comparison. Specific numbers aren’t relevant, but I can tell whether or not I’ve hit the target for the day.

Where it really starts to get interesting is when you add other values to compare to.

Here’s a whisker graph marking the days when products were released. Today is on the far right. You can see a few correlations between product releases and the number of hits to the site. The conclusion may be obvious (product releases result in hits to the site), but now the graph of hits makes a little more sense. A few recent traffic spikes on the right side can be explained by a corresponding product release.

Next, I added a graph of overall daily performance.

Performance is pretty steady except for some recent fluctuations. This statistic isn’t too meaningful since it goes all the way from the delivery of action cached pages to slower pages that call remote APIs.

In order to track a specific page, I added another sparkline.

Many more could be added: daily revenue, user signups, coupon redemptions, referrers from a specific site, downloads.

Putting these all together, I get a graph that is about half as tall as the single graph shown at the beginning, but shows several hundred numbers.

They can be stacked on a web page, viewed on an iPhone, and compared easily.

Tips

In order to make the most of stacked sparklines, I’ve found it helpful to:

• Use a similar horizontal scale
• Add a text label
• Cache when possible

To get you started, here’s some sample code.

Make a controller that will render the graphics. Include an image_tag in your view that calls the controller.

<%= image_tag(formatted_graph_url('hits_by_day_past_three_months', :format => 'png')) %>

I like to use the show action and check params[:id] for the name of a recognized report (/graphs/hits_by_day_past_three_months.png). And of course, add a before_filter to restrict access!

Most graphs will have similar options, so I wrote a method that returns a hash of default options.

def default_sparkline_options
  {
    :background_color => 'transparent',
    :step             => 3,
    :height           => 30,
    :line_color       => "#6699cc",
    :underneath_color => "#ebf3f6",
  }
end

Another method generates the graph and caches the binary result in memcached for an hour.

def hits_by_day_past_three_months
  graph = Cache.get("Sparklines:hits_by_day_past_three_months", 1.hour) {
    records = LogAnalysisRequestTime.find_recent_by_resource("All")
    data    = records.map { |r| r.quantity.to_f  }
    graph   = Sparklines.plot_to_image data, default_sparkline_options.merge({
      :target => 1_000_000
    })
    annotate(graph, "Daily Dynamic Hits")
    graph = graph.to_blob
  }
  send_data graph, :type => "image/png", :disposition => "inline" 
end

The annotate method adds a white box and label to the bottom of the sparkline, using a pixel font.

Two things confused me at first:

• The data is displayed with the newest items at the right, but only shows the most recent three months. So the SQL query needs to sort DESC but use Ruby to reverse the results.
• The production_log_analyzer reports the fraction of a second that it took to serve a request, but I like to view the data as requests per second. So I wrote a method in the model to invert the performance values (1/seconds_per_request).

Resources

• Sparklines
• Tidy Table gem
• Pixel fonts
• Book: Information Dashboard Design by Stephen Few.
• Book: Show Me the Numbers also by Stephen Few.
• Book: Beautiful Evidence by Edward Tufte.

Recently Published at PeepCode

---

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

I’ll be in San Francisco next week (January 15-18)! I have quite a few interviews planned for the Rails Podcast and apparently there’s a computer conference going on, too.

I’m planning to be at Osha Thai, 149 2nd St (map) on Tuesday night at about 7pm. If you’re in the area, I’d love to see you there!

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

Figure A Fox tall and fox small

In the past few months, I’ve been on the receiving end of several conversations to the effect of “Why do you use the word ‘bacon’ in some of your code examples? Are you a flaming anti-vegan, or just a pork lobbyist?”

I learned Ruby from the classic Why’s Poignant Guide to Ruby. Or rather, I read The Poignant Guide cover to cover and then decided to learn Ruby.

Adrian Holovaty reviewed it by saying

I attempted to [read it] then I started sticking a pen in my eye because I was so annoyed.

Unfortunately, The Poignant Guide is required reading for ALL Ruby programmers! Remove all pens from your pockets, then read at least…

Chapter 1—a very short intro Chapter 3—The most classic chapter of this classic book. I would be willing to bet that 90% of readers have never made it past this chapter, but have subsequently lived meaningful and productive lives.

Then go back to Chapter 2 and follow it through to the end. There is also a soundtrack.

Remember, we need to keep up with other developers who are much funnier than us!

This has been a public service announcement.

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

Figure A Messaging queues are a tool for executing code without taxing your web application processes.

Web developers often get into the rut of thinking about every programming task in the context of a request and a response. A request comes for a URL, content is retrieved and converted into useful output, then sent back to the client. Lather, rinse, repeat.

But there are also other types of programming tasks that don’t fit into that cycle. What about tasks that need to happen

at a certain time of day? after the response has been sent back to the client? at some point in the future after another event has happened?

Here are some examples from my applications:

Time of day: Log parsing, sales reports After a response: State-based payment transactions with other servers that need to complete before moving to the next stage, spam-filtering of blog comments against an external API server In the future: A task to checkup on an order to make sure it has been delivered within an hour after purchase

Previously, I approached most of these problems with a few rake tasks and a cron job that ran every minute. While it worked, it wasn’t as fast as it could be and felt a bit hackish (a delay of even one minute is too slow sometimes).

For a while, I’ve wanted to learn more about messaging queues. I love tools that don’t only enhance something I’m already doing, but completely change the way I think about designing an application.

The Problem Figure B If one process calls a resource-intensive method, it won’t be available to answer web requests.

Queues are a great tool for some tasks. Having the ability to send something off to a queue can solve some of these problems and also give you another option for optimizing the speed of normal HTTP responses, too.

Initially, I decided to try this out on my blog. I use the remote Akismet service to check comments for SPAM. To be honest, Akismet is usually fast enough that I could make the call in the middle of the request without any problems, but I wanted to try out the message queue before deploying a similar idea at PeepCode.com.

In the application, every comment starts out with a received state (using acts_as_state_machine). The Comment controller will fire off a job and a separate worker will handle the SPAM-checking so the web process can respond quickly and get back to work responding to other web requests.

Figure C Sending jobs to a queue keeps Rails processes available to do what they do best…answer web requests. Messaging Servers

There’s been some fresh activity even just over the last few weeks in this area. Ara Howard compared some of these recently. I haven’t evaluated all of these packages, but here are a few I’ve looked at:

Product Features Drawbacks beanstalkd Fast, simple, designed to mirror the style of memcached. Rails plugin available, or usable with a simple Ruby-based API. Server written in C, but is very easy to install. Memory only…jobs are not persistent. New, so the internal protocol may change. Workers may be difficult to manage. bj Rails plugin. Self-spawning. Can only send shell commands. Jobs start a full copy of your Rails app on every execution. BackgroundRB Ruby-based. Can be polled for incremental feedback on the progress of a job. Recently rewritten. Amazon SQS Runs on Amazon’s cluster, so it can handle a ton of traffic. Operated by Amazon, so it doesn’t run locally. Not open source. Apache ActiveMQ Well-known. Persistent. Requires several installation steps and database tables. ActiveMessaging Rails plugin. Works with ActiveMQ and others. Requires external job server. BBQ Nothing to install…involves only a single line of code! Doesn’t work on Windows NT4.

For this blog, I chose to try beanstalkd.

It is not persistent and exists only in memory, but my application uses acts_as_state_machine, so I can see if any job failed to run. It’s very fast and was made to help with scaling a multi-million person Facebook app running on Rails. There are some nice features like delayed jobs. Put a job in the queue and it can show up immediately, or after a period of time. There is a Ruby client and a Rails plugin. Installation

Download the beanstalkd server and compile it. Use make for production or make debug for your development copy (to print out extra messages as it’s working). There’s no task to install it, but you can just copy the executable to /usr/local/bin.

Start the server (use -h to see other possible arguments):

% beanstalkd
beanstalkd: net.c:90 in unbrake: releasing the brakes

Install the beanstalk-client gem. For this blog, I chose to use the gem directly.

sudo gem install beanstalk-client

In merb_init.rb (or config/environment.rb), I setup a connection to the beanstalk server.

BEANSTALK = Beanstalk::Pool.new(['localhost:11300'])

In the Comments controller, I put a comment job into the queue, using the id of the new comment.

# Comments controller
def create
  @comment = Comment.new(params[:comment])
  if @comment.save
    BEANSTALK.yput({:type => "comment", :id => @comment.id}) rescue nil
    # Then redirect and return

The yput method uses YAML to serialize any arguments and put them into the queue.

Finally, I wrote a rake task to function as the worker.

loop do
  job = BEANSTALK.reserve
  # ybody unserializes the job
  job_hash = job.ybody
  case job_hash[:type]
  when "comment" 
    if Comment.check_for_spam(job_hash[:id])
      job.delete
    else
      job.bury
    end
  else
    puts "Don't know what type of job this is: #{job_hash.inspect}" 
  end
end

In the future, I’d like to look into using daemonize or some other method for running the worker. In the meantime, I’m using god to start the worker and keep it running.

The details are a bit of a hack, but here is the god.conf if you want to try it. The benefit is that god keeps the worker running and daemonizes it so it runs in the background.

sudo god start -c /var/www/apps/mysite.com/current/config/god.conf

I can also call god restart beanstalk-worker from a Capistrano task to restart it and keep the code fresh.

Results Figure D The comments screen shows the state of comments.

In practice over the past week or so, this has been very reliable. The message passing is so fast that sometimes it actually runs the SPAM check before the redirect back to the article page is done!

It was fairly simple to setup and now provides me with a tool for accomplishing tasks that don’t need to be completed in the scope of an HTTP response.

Tips Keep queue items small. Put an id and some kind of identifier, not an entire model. I could have stored the entire contents of the comment, but it’s more efficient to just pass the id of the comment and let the worker get a current copy from the database. Keep worker code small: By calling methods on the Comment model, I keep the code in one place (even though it will be executed in different contexts). Future

Now that it’s working smoothly here, I hope to use it on PeepCode. Some possibilities:

Google Checkout pings: I need to ping Google after their server notifies me that an order has passed a security check. It can’t be done in a model callback because I need to complete an XML response to Google before moving to the next state. This kind of setup will be perfect since I can put a job in the queue and give it a small delay before it runs. Order followup in the future: I currently have some cron jobs that check recent orders to make sure they have completed succesfully. It would be easy to use this system to put several jobs in the queue after an order is placed. They would run after 30 minutes, 1 hour, or 2 hours. The job would send a notification if the order has not been completed, or ignore it if it’s done by the time the job runs. Finally, PeepCode in Italiano!

Ryan Daigle’s Rails 2 PDF is now available in Italiano as well as English and Español.

PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.

An interview at RubyInside recently suggested that Rails developers who want to be hired should maintain a Rails-related blog.

I’ll add to this and say that every beginning Rails developer should write their own blog software. It’s a great learning experience and you can try things that aren’t possible with just an app running on localhost. It’s also a great environment for learning without the pressure of a mission-critical app. When you’re working for a client and deploying an important application, you’ll have made all the beginner mistakes on your own time (hopefully).

This blog started on shared hosting with Typo software. I later switched to a VPS with 128 MB RAM and currently run on a RailsMachine VPS with 512 MB RAM.

In the process, I’ve learned a lot about

Running Rails on a shared host (don’t do it!) Deployment and automation with Capistrano (do it!) Unix server administration Process monitoring and uptime Rails development Merb development Application design and asynchronous processes Optimizing for speed Reporting and log parsing

If you need a blog for your business or for a client, I would definitely check out Mephisto, Radiant, or Simplelog. But if you’re thinking about starting a blog for yourself, you should write your own from scratch. It only requires 2 controllers (Articles and Comments) and apparently some guy even filmed a screencast showing you how to start one in only 15 minutes.

Every developer’s favorite topic: Performance

A Wordpress lecture mentioned the fact that a standard Wordpress install on a given server only achieves about 8 requests per second. The way they get it up to 300 req/sec is not by profiling loops and optimizing method calls, but by caching as much as possible.

The biggest speed benefit you can get is from page caching. However, that entails writing a bunch of code to expire the cached pages when articles are edited, comments are made, or templates are changed.

Expiration by Key Name with Memcached

Recently, I decided to try out some caching strategies I had read about1. The idea is to use memcached to store objects with keys that will automatically expire when the item changes.

For example, this page you are reading needs to expire if the article is edited or if new non-SPAM comments are posted.

So I’m caching the entire rendered page in memcached, with the article’s ID, timestamp, and number of comments in the key. In memory, that’s something like

"Articles:show:11097:1197661574:11"

This way, there’s no need to explicitly delete cached items. The application will ask for a new page when the data changes, and if memcached is full it will clear out the older, unused items.

Implementation

NOTE: You’ll need to have a memcached server running. I wrote a previous article about how to do that.

I’m using this technique both on my blog and at PeepCode.

First, I’ve put most of the key logic into the Article model.

class Article < ActiveRecord::Base
  def cache_key
    "#{self.class.name}:#{id}:#{updated_at.to_i}:#{ham_comments.length}" 
  end

For this blog (using Merb), the controller is pretty simple since render returns the HTML that was generated.

# Articles controller
def show
  @article = Article.find params[:id]
  Cache.get("Articles:show:#{@article.cache_key}", 1.hour) do
    render
  end
end

I’ve also wrapped it in some extra logic that doesn’t use the cached version if there is a flash message to display.

For PeepCode (using Rails), I wrote a render_cached method in the ApplicationController that handles more of this automatically. It takes a key, an expiration time, and a Hash of options that will be sent to the Rails render method.

Here’s how it’s called:

render_cached(@article.cache_key, 1.hour, :action => "show")

The render_cached method does several things:

It builds a key that includes the controller name, action name, and format being rendered. This ensures that HTML is cached separately from XML. It uses the render_to_string method to generate the template as a string, or render :text to send back the cached version. If checks a should_cache? method to see if caching should happen. This is different from Rails’ internal development/production caching. Instead, it’s a controller-specific method that can turn off caching in some circumstances. By default, I use no caching if someone is logged in or if there is a flash message to display.

def render_cached(key, expiration, render_options)
  return unless perform_caching
  if should_cache?
    combined_key = [
      'controller',
      controller_name,
      action_name,
      params[:format] || 'html',
      key
    ].join(':')
    output = Cache.get(combined_key, expiration) do
      render_to_string render_options
    end
    render :text => output
  else
    render render_options
  end
end

The should_cache? method looks like this:

def should_cache?
  (current_user.nil? && current_order.nil? && flash[:notice].blank?)
end

Summary

Overall, this has worked quite well. The most frequently accessed pages turn out to be pretty responsive. The cached versions perform much faster and there’s not a single line of cache-expiring code.

Raw performance numbers from pl_analyze (requests per second, controller and action names omitted):

And a word from our sponsor…

I’ve got several authors working on some great PeepCode PDF books. They will be published in the next few months.

Currently, many people have found Ryan Daigle’s Rails2 PDF to be useful for getting up to speed with Rails 2.0.1. It’s even available in Español and will soon be available in a few other languages, too.

I refilmed the Capistrano 2 screencast from scratch. It’s mostly the same content as the first Capistrano screencast, but is up to date for the new namespaces, callbacks, and other features of Capistrano 2.

The Rails from Scratch series has been updated with code and notes about Rails 2.0.1 compatibility. It’s a free update if you purchased it in the past.

Finally, I published a screencast on Git a while ago. I’m using Git wherever possible and really love the speed, easy branching, and flexibility in areas that were frustrating in Subversion.

Resources

1 Articles about auto-expiring memcached keys:

The secret to memcached Clever caching Caching makes your brain explode Intelligent fragment cache PeepCode Screencasts – Learn Ruby on Rails and Javascript! Hour-long screencasts for $9.