Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

LABS
Brown Bag: Blaine Cook on Starling

This week we were treated to a lunchtime tech-talk by Blaine Cook of Twitter. He came to talk to us about Starling, the all-Ruby message queue system that runs much of Twitter. Blaine spoke about the history and motivation for creating Starling, then showed how it worked, and talked about possible future enhancements and directions for further development.

Starling looks quite simple to use. The Starling server speaks the memcache protocol, so to talk to it you just need to load up the memcache-client gem and create a client instance. Note, the Starling server doesn’t use memcached for its implementation at all; it just speaks the protocol.

Some interesting bits about why Blaine built Starling. It basically comes down to that every other solution had some problem that made it unsuitable for them to use. Here’s the list:

  • rq (by Ara Howard) – nfs/disk based, high latency
  • DRb – not robust under load
  • Rinda – very slow! O(n) for take operations
  • Apache ActiveMQ – super complex
  • RabbitMQ – Erlang dependency

In the last few months we’ve seen a lot of Starling-like things appear, some inspired by Starling itself.

  • beanstalkd – uses memcached for storage, not persistent or recoverable
  • bj – database backed
  • thruqueue – uses Thrift protocol, ugly
  • sparrow – Starling imitator
  • ap4r – full-featured

Interesting new directions for Starling… Currently Starling has some overhead from polling on both client and server sides. Kevin Clark and Chris Wanstrath have hacked it to run using EventMachine to eliminate polling. Not sure what happens if clients die while request is waiting to be filled. Also, some issues with load balancing and starvation need to be looked at. And there are opportunities to build a richer client API.

Comments
  1. Joann says:

    This is another case of NIH syndrome. I can’t believe Apache ActiveMQ was dismissed as “super complex”. It is actually very simple to use and start implementing it. And as you mentioned in your last para about the missing features in the current version of Starling, it will become equally complex once those feature are baked in. That itself will take time to implement and test. Whereas ActiveMQ is already here and tested and deployed in a number of production environments.

  2. Chad Woolley says:

    It would be interesting to hear what is super complex about ActiveMQ. Installation, maintenance, API, all of the above?

    The Spring configuration for ActiveMQ doesn’t look too bad: http://activemq.apache.org/spring-support.html

    What Ruby support for ActiveMQ already exists? Is it comparable to Spring’s, in richness and complexity?

    Then again, if Twitter doesn’t currently have Java as part of their architecture, that can be a compelling reason to Invent it Here. If Starling is clean, well-tested, stable Ruby that they understand and can easily support (because they wrote and tested it), and they can avoid a dependence on the Java ecosystem, that could save a lot of long-term cost – maybe even enough to offset Inventing it Here.

    — Chad

  3. alexis says:

    I feel obliged to join the chorus of ‘unfair’ here :-)

    RabbitMQ does not require any knowledge of Erlang to use. Please try one of the many language clients. Hey there is a STOMP client now too.

    For installation we have many packages designed to make your life easy:
    http://www.rabbitmq.com/download.html

    Most of these installs are trivial. Probably the hardest ‘mainstream’ install is on MacOSX, because we have not fully packaged for that O/S yet, and even that is just a few shell commands. Here is a well known Java person describing his experience with MacOSX: http://rossmason.blogspot.com/2008/02/erlang-on-os-x.html

    And – free free to get in touch!

    alexis

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *