Cobalt Edge

[Update: I resolved the Git issue, and have now switched to using Hudson. The advantages of it's in-progress display, ability to more closely monitor and/or kill a build, and my impression of it being more stable, won the day.]

After switching our CI server to Integrity, there were a few blips, one of which is that we were hitting swap (memory). Hitting swap is not surprising and I'm surprised it hasn't been a problem sooner, since memory is our #1 battle. I figured if I was going to up the memory on our CI server, I might also try Hudson, as that was the main reason for not trying it previously.

The short story is that I tried it, and we're back to Integrity, but I learned some interesting things. The following is both some notes on installation, as well as some reasons why it didn't work out...

Installing Hudson

Our CI box is a slice at Slicehost, running Ubuntu 8.04 LTS, and I'd just upped it from 256MB RAM to 1GB. Also of note, I've built and install Git 1.6.4 for this system. The following are brief notes on getting Hudson going as it wasn't quite as simple as most folks made it out to be...

First, I followed the directions on the Hudson site for installing Hudson from the Ubuntu package. This amounts to adding the package repository, updating apt-get, and then doing:

sudo apt-get install hudson

This installs all the Java 1.6 stuff you need (and mine seems fine with 64bit Java), and other dependencies. After the install, Hudson is running, on default port 8080. Next up I added a server block/configuration in Nginx for Hudson (and unlinked/removed Integrity). I then went to Hudson in my browser. What I got was an error message about the xstream library. Fixing that was easy, as it turned out: downloaded the latest (1.320 at the time) hudson.war from the Hudson site, and replaced /usr/share/hudson/hudson.war with it. Restarted Hudson, and voila, now it was up and running.

Also, Hudson runs as the user "hudson", so I needed to add an SSH key for that user, and then add the public key to GitHub. And, setup a ~hudson/.gitconfig as needed. Finally, as I found later, do a git clone or an SSH to github so that you get past the whole SSH authenticity question when you first SSH to an unknown server. Note, the Hudson user is not an interactive user, i.e. you can't directly login as that user, so to gen the SSH key, you'll need do something like su to root and do, sudo -u hudson ssh-keygen -t dsa.

Configuring Hudson

Before adding a project, I needed to config some plugins. I went to the Manage page, clicked on Installed, and turned off the Subversion plugin and restarted. Next was going back in to manage plugins, and installing the Git, Github, and Rake plugins, and again restarting. Both restarts showed an Nginx bad gateway error, but simply refreshing got it back (probably just needed more time for Hudson to restart).

Then to configure a build, from the main Hudson page after a fresh install, click the "create new jobs" link. In the ensuing form, enter a project name, and select "Build a free-style software project".

Next:

• checked "Discard Old Builds" which then shows you options (so you can put in keep for X days, or X number of builds)


• Added URL for Github project, http://github.com/yourproject/yourproject


• Select Git as the SCM, and entered by git@github.com:yourproject/yourproject.git URL for the repository


• Turned on merge after build option. This will supposedly add tags for the build to your code base and then merge them back in. More on this in a bit.


• Next I configured the build steps for my project. All I really did here was take the same build steps I used for Integrity, and added them as individual shell and rake tasks.

At this point, I fired off a build (truth be told, I started with just a single build step to vet it), and things worked, with the exception of the very last step, where I push the Git tags back to GitHub. This is what I saw:

[workspace] $ /bin/sh -xe /tmp/hudson1444107192962944065.sh
+ /usr/bin/git push --tags
XML error: syntax error
error: Error: no DAV locking support on https://github.com/dealbase/dealbase/
error: failed to push some refs to 'https://github.com/dealbase/dealbase'

I've now been searching for answers to this, and haven't yielded anything. I've tried the git push directly from a shell, with the same result. If I do this as a different user (e.g. under my user) it works fine. This git push is attempted both as part of a rake task (the ci_tag task), as well as I tried making it just a straight shell command in Hudson.

After a lot of googling, and asking, and no resolution in sight, I've gone back to Integrity...

Comparisons and Notes

First off, I think Hudson looks pretty stellar. There are a TON of plugins for it, and it seems quite mature and polished. The essentially 100% configuration via the GUI is slick. Install, despite a few hoops above, was actually pretty painless. So, here's just a few notes/opinions:

Hudson

• You will need some memory to run Hudson, more than with Integrity or CruiseControl.rb. From what I can tell, you probably want a system with 1GB or more. Various other folks I talked to all had 2GB or more systems, and their Hudson processes were taking 1.5GB or more. This is partly just a Java thing. It should be noted, the others were running more than one build with Hudson, and mine seemed to work fine on a 1GB total memory system (didn't seem to hit swap).


• Hudson allows you to kill a build while it's running (nice!).


• Hudson works with CCMenu/CCTray out of the box.


• The git integration has more options in terms of picking branches, doing merging, and various more involved operations, but doesn't have GitHub post-receive hook support out of the box (there are plugins up on GitHub for it, but not listed in the standard plugin list).


• Hudson runs as user "hudson", which is a user that has no shell. You could change this, although the idea is you shouldn't have to. However, this complicates setting up SSH keys and various things. And, of course, I had the issue with Git as mentioned above. I could probably change this to run as my user and so on, but part of all this for me is not having to change a lot of defaults and start messing with core configurations/designs of the system. In part, I just don't have time to do that, and to maintain it (these kinds of changes often cause problems when you upgrade versions, etc. - also known as you may forget to redo these changes if an upgrade undoes them :).


• The UI and web app itself is quite nice, understandable, well done in general for something like this.

Also, thanks to the Tea-Driven blog for motivating me to try Hudson (and for some tips on Testjour - more on that in another post).

Integrity

I really only have one main complaint about Integrity, and that has to do with indication of a build in progress. It essentially doesn't indicate it - it says it hasn't been built yet, but a build may actually be running. The CCMenu/integritray plugin will show you that a build is running, so this mostly solves my problem, but this seems like a core failure. I may have to look at fixing this. And that is a win for Integrity in that it's Ruby, and thus I'll be more apt to go fix this (while I spent MANY years doing heavy Java work, I don't have interest in working in that code anymore for a task like this).

CruiseControl.rb

One thing that remains an advantage for CC.rb is that it has "build artifacts" - i.e. you can create files and such as part of your build and have those known to CC.rb, where it then links to them in the summary of your build. You may or may not need it, but it's also very handy for simply showing you the Git tag you created on a successful build. I just touch a file in the build artifacts dir with the name of the Git tag, and then I don't have to dig through the output of the build to find my Git tag.

That's all the time I have for today, hopefully this is also helpful to others...

Loading mentions Retweet

After discovering that autospec was taking up a lot of CPU while it was "idle", I looked for alternatives. I found RSpactor, which doesn't take as much CPU, and is better since it's a dedicated window with nice GUI results and so. My only gripe was that I'd gotten used to the spoken results output I'd rigged up for Autospec.

I really prefer the spoken results because it is not visually distracting, and doesn't require me to be paying attention to the area of the screen where they pop up (I use a 30" monitor, plus the 17" laptop monitor, so I'm not always looking at the right spot for the Growl notices, and I don't like the monitor-wide growls). Lucky for me, RSpactor is open source and is up on GitHub.

I thought it was an Objective-C Cocoa app, but as it turns out it's a RubyCocoa app. I'd built RubyCocoa apps before, so was familiar with that, plus of course know Ruby. It wouldn't have mattered either way (I'm fine working in ObjC as well), but this did make things a slight bit faster.

Anyway, did a quick bit of work and got a new preferences panel for Speech added, and then rigged that up to test results, so that I now have my desired spoken results. A slight improvement comes along in that it (optionally) speaks the number of passing/failing/pending tests - just insert a question mark in the string/phrase you want spoken for each and it'll say the number at that spot.

I've sent a pull request to RubyPhunk, but no guarantees it will get added to the main line. In the mean time, if you're interested, grab it from my RSpactor fork on GitHub. Update: RubyPhunk integrated my changes into the main RSpactor code.

Loading mentions Retweet

When testing web apps for a general/wide-ranging audience, one must test across a slew of different browsers and operating systems (and different versions of each). This is a real pain, even if you have an army of testers at hand (which most of us don't). I employ various tools to help me with this (CrossBrowserTesting.com, BrowserCam, VM Ware with different VM setups, etc.). But, the reality is that I don't feel like these really stack up to what I need. More specifically, BrowserCam, and its competitors do not. CrossBrowserTesting is actually pretty awesome and does what it advertises quite nicely, it's a favorite tool right now.

What I'm after is a way to see what a survey of pages on my site look like on different browsers and OS's. This is by no means a definitive test of a site, but more of a quick visual inspection of appearance, without having to fire up a dozen different pages across maybe 30 different browsers/OS configurations. We use an automated test suite to test the bulk of other things, but appearance can't be tested that way.

BrowserCam helps, but honestly I find it quite lacking as a tool in this arena. Note, I'm picking on BrowserCam, as that's the one I use, but the others (BrowserShots, Litmus, etc.) all seem to suffer some or all of these problems, and further, they don't seem to have projects or ways to establish a standard suite of tests. Getting on with it, all of this really boils down to two primary things:

• Ability to edit the settings on a project's images. As far as I can tell, once you select the URL's, image size, browsers, etc. for a project, there is no way to change any of that! What if I just want to change the window size? Can't, I have to create a new project (or add new images to the existing one, replicating all my previous work. This seems like an obvious hole.


• Automation. I want to be able to automate regeneration of the project. I'd like something like a simple URL/HTTP API to do this, so that I can, from a command line, use curl or similar to issue a single HTTP request that will regenerate a project's images. Thus, the API would need to use HTTP authentication or similar, and specify which project to regenerate. With this, I would be able to automate requesting a regeneration of the project as part of/after deploying my application.

Finally, if anyone knows of a service similar to BrowserCam that solves these, even if it supports a few less browsers, do let me know. At very minimum it would need to handle Internet Explorer and Firefox, preferably also Safari, and cover Windows XP, Vista, MacOS X 10.5, and one popular Linux flavor. Nice to haves would include Opera and various others.

Loading mentions Retweet

Uh, ya, another great blog title, but we'll get over it. We use geocoding in our app, and that's a relatively costly operation time wise, especially when you may be doing it hundreds or thousands of times when your test suite runs. I can't stub out the objects that use it in many cases, so I wanted to stub out the actual geocode call unless I truly needed real geocoding (which is only when I'm testing the actual geocoding itself, and thus is a very small part of the test suite).

We use RSpec Specs and Stories and I wanted to mock out the geocoding by default, but unmock it in a few places. I asked about this on the mailing list, Googled and so on, but didn't find a solution that was working. So here is what I wound up doing...

In my spec_helper.rb file, I added:

Spec::Runner.configure do |config|
  config.before(:each) do
      # Setup fake geocoding unless told not to
      unless @do_not_mock_geocoding
        fake_geocode = OpenStruct.new(:lat => 123.456, :lng => 123.456, :success => true)
        GeoKit::Geocoders::MultiGeocoder.stub!(:geocode).and_return(fake_geocode)
      end
  end
end

What this does is mock the geocoding unless a test has set the @do_not_mock_geocoding variable to true. One caveat, at least from what I've found, is that you need to set that to true in a before(:all) block in your tests, so that it happens before the before(:each). This is minor, as you can just have something like:

describe "with real geocoding" do
  before(:all) do
    @do_not_mock_geocoding = true
  end
  # your tests that want real geocoding
end

The impact this has had on our test suite is tremendous. I had already had some partial mocking of the geocoding in place, but was sweeping the system to put it in because the time it took to run our test suite was out of hand at about 13 minutes! Now that I've got this in, it runs in 2 minutes! Geocoding is used in two of our most core objects, which is why it has such a big impact on the test suite. This is one place mocking has really proved to be a massive value!

Loading mentions Retweet

I've been using RSpec exclusively on my latest project. I'd say I'm still fairly new to it, but it has won me over for the time being. However, the view testing has been a real problem. You can test views quite easily and nicely with RSpec, however, when you do something subtly wrong, it can cause RSpec to fail without any visible error! In fact, Autotest doesn't report a failure at all, and the only way I know it is failing is either I pay close attention to the number of examples it said it ran, or more likely, my cruisecontrol.rb CI server will actually fail (because it detects the commands exit status).

For example, while Autotest says everything is fine, the CI server will show something like this (note for easier reading I've trimmed this a bit):

/widgets/index.html.erb
- should render list of widgets
/bobbles/index.html.erb
Finished in 4.977586 seconds
175 examples, 0 failures
rake aborted!
Command /usr/bin/ruby1.8 -I"/var/cruisecontrolrb/projects/myproject/work/vendor/plugins/rspec/lib"  "/var/cruisecontrolrb/projects/myproject/work/vendor/plugins/rspec/bin/spec" "spec/controllers/bobbles_controller_spec.rb" ... --options /var/cruisecontrolrb/projects/myproject/work/spec/cruisecontrol_rcov.opts failed
(See full trace by running task with --trace)

As you can see, RSpec is reporting 0 failures, yet rake fails. This is because in reality RSpec is returning an exit code of 1 instead of 0. But, looking at the output, it's certainly not revealing. Running with --trace is of no help either.

What I now know to pay attention to in situations like this is the fact that that last spec it ran, the /bobbles/index.html.erb one has no examples listed under it. That is thus the culprit (you can also argue that it's the last thing to "run" and then rake fails, so it's likely in this, etc.).

The real pain comes when you try to figure out what the heck is causing this. You have zero feedback, and no way that I know of to somehow debug or inspect the test to see what's failing. In my experience to date with these, it boils down to some problem in your mocks and stubs, but this can be difficult to figure out. I admit, the one that prompted me to write this blog entry is one I've still yet to figure out, and finally just punted on.

I've been hearing a lot about not testing views, or testing very little of the views. I agree in general on this, and am now looking into using Webrat to do integration testing to really test "view" functionality, and leave the rest of my view testing mostly to testing my helpers, controllers, and models. Here are a couple of blog entries related to all this:

• Err the blog (via comments) discussion on view testing techniques. This is a good read.


• Rubby Russell's blog entry on testing your views with RSpec.


• Integration Testing in Ruby with RSpec's Story Automation Framework by David Chelimsky (PDF)
  

Loading mentions Retweet

Update: the first .autotest I had in here was bogus, sorry about that. The .autotest file contents below work properly with Rspec tests at least.

I make extensive use of ZenTest's Autotest to constantly watch my test suite and ensure my app's tests are passing on my dev box/during development. Historically I've used Growl/growlnotify to get little popup notices indicating if my tests passed or failed. That's nice, and I've done the enhancements that add graphics and style it nicely, etc. But, in reality, I'm not always looking and sometimes don't see the messages. Plus, they can be somewhat distracting.

So, I've switched to using the handy say tool on the Mac (on Linux I think you could use "espeak", no clue on Windows, but then, uh, well, why are you doing dev work on Windows?! ;-)

The speaking is nice - I hear it, but don't get visually distracted. I use different voices for tests passed vs. failed too. This may not work great if you work in a cube farm, or even a cafe, but here at home, or in your own office, I think it's great. Here's my .autotest file as an example:

require 'autotest/redgreen'
module Autotest::Growl
  def self.growl(title, msg, img, pri=0, stick="")
    system "/usr/local/bin/growlnotify -n autotest --image #{img} -p #{pri} -m #{msg.inspect} #{title} #{stick}" 
  end
    Autotest.add_hook :ran_command do |at|
    results = at.results.last
    unless results.nil?
      output = results[/(\d+)\s+examples?,\s*(\d+)\s+failures?(,\s*(\d+)\s+pending)?/]
      if output
        failures = $~[2].to_i
        pending = $~[4].to_i
      end
      if failures > 0
        `/usr/bin/say -v Zarvox "you broke the code"`
      elsif pending > 0
        `/usr/bin/say -v Alex "Tests passed, with some pending"`
      else
        unless at.tainted
          `/usr/bin/say -v Victoria "all tests passed"`
        else
          `/usr/bin/say -v Victoria "tests passed"`
        end
      end
    end
  end
end

To see what voices your system has available, open the Speech system preference pane, pull down "System Voice" and select "Show More Voices" to see the full list.

Loading mentions Retweet

Like some other folks working with Rails, I've been a bit frustrated with Fixtures. Foxy Fixtures, Rathole, and such things, including what is in Rails 2, have helped a lot. However, the two biggest frustrations for me come down to the fragility of fixtures, and knowing what fixtures you have and how they relate to their associated fixtures. I would find myself thinking, "hmm, which user do I use when I want the one with X associations" or what not. Naming your fixtures well helps, but only so much.

Lately, I've come across two plugins that I am really loving. These are Shoulda and object_daddy. Shoulda seems to be gaining in popularity in the Rails community, which doesn't surprise me. It gives you some of the best syntax of RSpec, without having to use RSpec (which I am not all in love with, unlike various others), as well as it gives you some nice "should" methods, and other features I'll get into in a minute.

object_daddy I think is rather obscure. I only found out about it due to Tammer Saleh's presentation (video link) on Shoulda from the 2008 MountainWest RubyConf. object_daddy has improved in its short life, and is quite useful today. What it does, is provide a factory/generator mechanism for creating model objects. I've done this in the past with object constructors or factories, etc., but object_daddy organizes all this, and provides a slick mechanism, called "exemplars", that specify how model attributes are defined when generating objects, and more importantly, when generating multiple objects. Tammer covers this issue in his presentation, which I highly recommend watching.

It's taken me a bit of time/use of Shoulda to get really into it, but like so many things, use it a bit and then the light bulb not only goes off, but seems to erupt with light. The big one here for me was how to leverage contexts, and by that I really mean nested contexts. My tests now not only read better, but can be written in a much nicer fashion, as well as organized in a great way. On the organization front for example, I now often have two top level contexts in a functional test: one for cases where I'm testing actions without a user being logged in, and the other for when a user is logged in. Great way to group them.

But, what's got me most excited lately is the combination of these two testing tools, and what it's done to my tests. First, I've darn near eliminated fixtures on the project I'm using this most with so far. This has removed the fragility, as well as it's just FAR easier to understand a test case's setup/scenario. I use Shoulda's contexts and setup, combined with generated objects from object_daddy to create "scenarios" (to steal the term from the plugin that provides this kind of thing for fixtures). The benefit is that you have all of the info about your test right there in one place in front of you. You don't need to bounce between likely multiple fixture files and your test code file to ascertain what's being used in your test. Plus, you can be very specific about the data being used in that particular test (or tests).

I heartily recommend you try this out. Two other things of note:

• Shoulda can be mixed in with existing TestUnit. So, you can slowly convert to it, or just build new tests with it, etc. Very nice. And, it doesn't require anything special to run the tests (it really is just a method generator for building TestUnit tests).


• Check out Shoulda's "should_eventually" method. I'm making more and more use of this, as I use a Test First approach. So, I go in, and build lots of tests, and do a lot of "should_eventually" as I think of things to test and functionality I need, etc. Then as I determine how to write those tests, and following on from that, write the implementation, you simply remove the "_eventually" and let it rip.

Note, if you are an RSpec fan, you can of course achieve the same as Shoulda for contexts and such (I think anyway), so just pull in object_daddy and leverage that aspect.

And last, but not least, I've forked object_daddy to make one tiny change (a single line, actually, a single method call name change!) that's made a big difference for me (comments very welcome). This change is to, by default, call create in the generate method that object_daddy adds to your ActiveRecord objects, instead of calling new. This avoids what I found to be the common case of doing:

my_new_object = SomeModel.generate
my_new_object.save
my_new_object  # or some use of it

p.s. For those that wonder why I've "all but eliminated fixtures", as in, what do I still have in fixtures? The only things are standard data, which in my case amounts to a couple specific users, and a couple of Roles and Permissions. These normally get setup in the DB migrations, but I'm working through what Rails does when you run tests and it wipes the DB clean (and thus doesn't pick up seed data from migrations), and other such issues.

Loading mentions Retweet

Historically, I've used migrations to set up standard data that my database must contain in a Rails app. This would be things like standard Roles for the system, or maybe country codes or such things. However, it appears this simply won't work in Rails 2.x, because as far as I can tell, when you run something like "rake test", it blows away ALL data in your database (not just fixture data). If I'm wrong about that, please correct me. This makes sense given that it seems the drive is towards schema.rb being the official way to create a DB from scratch, and that you have the equivalent of Foxy fixtures which do lots of magic to make creating your fixtures easy (but likely quite painful to figure out how to explicitly clean up those fixtures in certain cases - so it's easier just to wipe the DB clean).

There are various solutions for creating seed/standard/structured data for your app. However, from what I've seen none address this problem that that data will get wiped out when testing. For many people that may not matter, their tests may not hinge on it. But, I like to stay DRY, and when you have standard roles, or similar types of data, there is no reason I should have to recreate those in fixtures (and risk being out of sync), or leave them out, etc. I likely have app functionality that directly depends on such things, and thus I need this during testing as well.

My solution as of now is a simple one, and one that does not scale well for large amounts of data, but for the five records I need at this point in the particular app I'm working on, it's an approach (I very much welcome better approaches!)... I simply created a "seed_data.rb" file in my config/initializers directory. Within this file I have code that does a create_or_update (or similar) of the standard data I need. This seems to work out quite well.

Update: the above breaks things like "rake db:reset", because when the initializers run, as part of the Rake environment, and the DB has been dropped, the initializer fails, and thus fails rake.

Loading mentions Retweet

Facebook's terms of service don't allow "fake" users, or user's who aren't associated with a human. This makes it harder to test Facebook applications which aren't published. However, they do provide a mechanism to mark an account as a test user account (see the Test Accounts wiki page). It falls under a bunch of different rules, some of which make things slightly challenging. For example, non-test user's cannot be friends with test users. This means in order to use a test user, you will have to create at least two of them, so they can be friends. Doesn't exactly parallel real life does it? :)

So, the next problem that crops up, is that you will want to register your developer app using a real account, but if real accounts and test accounts can't be friends, how do you share that app? This is not obvious, but you can extract how to do it from the questions at the bottom of that Test Accounts wiki page. Here's how I set it up (these are streamlined instructions as I had to futz around a bit before seeing how to work with the limitations):

1. Create two new user accounts on Facebook. You'll obviously need a valid email to go with this. I run various domains so just setup users in that, but you could use a Yahoo account or whatever.


2. Add the Developer application to each of the test user accounts. This is required in order for them to use your application which is still in development and not public.


3. Have these users become friends now if you like (they can become friends later as well). Also, have them be friends of the user you have that registered your Facebook application.


4. In your real user account that has the registered app, make the test users developers.


5. In one of the test user accounts, add the application. I start with just one, as I often want to test the Invite mechanism for the application.


6. Now mark the test accounts as test accounts using the URL provided on the Test Accounts wiki page.

Now you're set to have your test user's be what you use to test your application, allowing you to do invites, post messages to the mini-feed, and so on, and see how this works across friends. Hopefully Facebook will improve this in the future, as per various suggestions on the wiki page.

Loading mentions Retweet

I'm doing some benchmarking of a Rails based web app (technically, a web service) to try to establish some baselines and to use in assessing build outs/hardware deployment, etc. To start off, I wanted to establish a baseline by using pure static content. The performance I got didn't seem quite right to me. Below is a message I sent to the mongrel-users mailing list. I figured this was a good blog item, and would like to solicit help on this...

I'm trying to do some initial benchmarking of our setup, mainly just to establish baselines. I'm essentially using the process Zed outlines in a previous message:
http://rubyforge.org/pipermail/mongrel-users/2006-May/000200.html

What I'm running into is that Mongrel appears only half as fast as Apache when serving a small static HTML file. If I then add in Apache with mod_proxy_balancer, going to a single Mongrel, it drops down to nearly about a third of what pure static Apache will do. This seems bogus to me, and I suspect I have either some configuration problem, or something. My understanding from what I've read is that Mongrel should be fairly close to Apache when serving static content (at least not only 50% as fast). Is that right as a generalization?

Here's some info to back this up.
- Server: HP DL360 dual 3.0GHz Xeons, 4GB RAM, 10k RPM SCSI disks
- OS: Fedora Core 6, up to date as of a few days ago
- Apache 2.2.3-5, with mod_proxy_balancer
- Mongrel 1.0.1, mongrel_cluster 0.2.1
- serving a static HTML file from the Rails app's public directory
- Testing using an identical server, that sits above this one in the rack, connected to a switch (so it's one hop)
- Using httperf for testing, with the following command:
httperf --server lab05 --port 80 --uri /mongrel_alive.html --num-conns 10000
The number of connections I vary to be near the 10 second mark...

Here's the results:

- Just Apache, num-conns=15000, ~1400 req/sec
- Direct to Mongrel on port 5000, num-conns=8000, ~740 req/sec
- Apache mod_proxy_balancer to a single Mongrel, num-conns=5000, ~475 req/sec

So, I'm incurring a massive penalty for the balancer/cluster setup. In production we will be using hardware load balancers, but even still, my understanding was that instead of a drop from 1400 to 740, it should be somewhat closer (say at least over 1000).

What would folks suggest, or what comments do you have?

• Nginx with one Mongrel: ~612 req/sec
• Pen with one Mongrel: ~670 req/sec

Loading mentions Retweet