Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

LABS
Build Your Own Rails Plugin Platform with Desert

This post was originally written by Brian Takita.

While it is easy to include plugins in your Rails projects, it isn’t easy to extend and customize the plugin for your own application. Desert solves that limitation/complication by making it just as easy to extend or modify a plugin class as it would be with any other class. In this post we will go over how Desert provides an easy way to manage and extend your plugins.

At Pivotal, we offer an integrated platform of Rails plugins named Socialitis.

Socialitis is an internal project that grew out of the observation that many of our start-up clients needed to build the same non-differentiating features; user management, friends/contacts, activity feeds, on-site messaging, etc.

The Socialitis platform is broken up into number of plugins that extend the Rails app in specific ways. These plugins may have dependencies on other plugins.

One of the major design goals of Socialitis is easy, drop-in, integration into existing Rails apps. This means using convention over configuration and removing as much integration responsibility from the user of the plugin as possible.

Another design goal was to provide sensible defaults and make each plugin easy to customize for your app.

We used Desert to achieve these goals, and so can you for your own platform.

The major features that Desert provides are:

  • Defining a Rail’s like directory structure into your plugin (models, views, controllers, helpers)
  • Plugin dependencies
  • Seamless overriding of classes and modules defined by parent plugins
  • Plugin migrations
  • Plugin routing

Desert provides a similar feature set to the Radient plugin system and the now defunct Appable Plugins framework.

For a simple example, lets say you have two plugins, name User and Messaging. The User plugin provides basic authentication and login features, and the Messaging plugin allows Users to send Messages to each other. The Message plugin depends on the User plugin.

The directory structure of the full Rails app looks like:

  |-- app

  |   |-- controllers

  |   |   |-- application.rb

  |   |   `-- blogs_controller.rb

  |   |-- helpers

  |   |   |-- application_helper.rb

  |   |   `-- blogs_helper.rb

  |   |-- models

  |   |   `-- user.rb

  |   `-- views

  |       |-- blogs

  |       |-- layouts

  |       |   `-- users.html.erb

  |       `-- users

  |           |-- index.html.erb

  |           `-- show.html.erb

  |-- db

  |   `-- migrate

  |       `-- 001_migrate_users_to_001.rb

  |-- lib

  |   `-- current_user.rb

  |-- spec

  |   |-- controllers

  |   |   `-- blogs_controller_spec.rb

  |   |-- fixtures

  |   |-- models

  |   |-- spec_helper.rb

  |   `-- views

  |       `-- blogs

  `-- vendor

      `-- plugins

          `-- user

              |-- app

              |   |-- controllers

              |   |   |-- logins_controller.rb

              |   |   `-- users_controller.rb

              |   |-- helpers

              |   |   |-- logins_helper.rb

              |   |   `-- users_helper.rb

              |   |-- models

              |   |   |-- login.rb

              |   |   `-- user.rb

              |   `-- views

              |       |-- logins

              |       |   |-- edit.html.erb

              |       |   |-- index.html.erb

              |       |   |-- new.html.erb

              |       |   `-- show.html.erb

              |       `-- users

              |           |-- edit.html.erb

              |           |-- index.html.erb

              |           |-- new.html.erb

              |           `-- show.html.erb

              |-- config

              |   `-- routes.rb

              |-- db

              |   `-- migrate

              |       `-- 001_create_users.rb

              |-- init.rb

              |-- lib

              |   `-- current_user.rb

              |-- spec

              |   |-- controllers

              |   |   `-- user_controller_spec.rb

              |   |-- fixtures

              |   |   `-- users.yml

              |   |-- models

              |   |   `-- user.rb

              |   |-- spec_helper.rb

              |   `-- views

              |       `-- users

              `-- tasks

          `-- message

              |-- app

              |   |-- controllers

              |   |   `-- message_controller.rb

              |   |-- helpers

              |   |   |-- message_helper.rb

              |   |   `-- user_helper.rb

              |   |-- models

              |   |   |-- message.rb

              |   |   `-- user.rb

              |   `-- views

              |       `-- messages

              |           |-- edit.html.erb

              |           |-- index.html.erb

              |           |-- new.html.erb

              |           `-- show.html.erb

              |-- config

              |   `-- routes.rb

              |-- db

              |   `-- migrate

              |       `-- 001_create_messages.rb

              |-- init.rb

              |-- spec

              |   |-- controllers

              |   |   |-- message_controller_spec.rb

              |   |   `-- user_controller_spec.rb

              |   |-- fixtures

              |   |   |-- messages.yml

              |   |   `-- users.yml

              |   |-- models

              |   |   |-- message_spec.rb

              |   |   `-- user_spec.rb

              |   |-- spec_helper.rb

              |   `-- views

              |       `-- messages

              `-- tasks

The User plugin introduces the various User and Login Rails objects. The Message plugin introduces its respective Message objects.

Notice that the Message plugin also reopens some of the User objects to insert functionality.

For example, vendor/plugins/users/app/models/user.rb looks something like:

class User < ActiveRecord::Base

  has_many :logins

end

The Message plugin would then reopen User in vendor/plugins/message/app/models/user.rb:

 class User < ActiveRecord::Base

   has_many :messages_received

   has_many :messages_sent

 end

Meanwhile, the main application can also reopen User in app/models/user.rb

 class User < ActiveRecord::Base

   def custom_app_method

     # custom app logic #

   end

 end

Desert allows you to utilize Ruby’s ability to repoen classes to layer on functionality in your plugins and application.

At Pivotal, we have had success in sharing code across multiple client applications using this technique.

Another thing to note is normally the Message plugin would be loaded before the User plugin. Desert allows you to create plugin dependencies. So in vendor/plugins/message/init.rb:

require_plugin 'user'

require_plugin 'will_paginate'

This means you no longer need to define plugin load order inside of environment.rb. Your plugins can take care of that. Desert works with practically all plugins. That means you can have a plugin dependency on any existing Rails plugin.

To see more examples & documentation, take a look at the Desert project at http://github.com/pivotal/desert.

Comments
  1. Adam says:

    You mention that Desert can manage plugin dependencies. In the example you give, the Messages plugin would seemingly depend on the User plugin. How would you specify that so that Desert ensures the User plugin gets loaded first?

  2. Brian Takita says:

    @Adam – Thanks for the comment.

    Inside of vendor/plugins/message/init.rb you would include:

    require_plugin ‘user’

    I updated the article.

  3. Dave Duchene says:

    So, I assume you had a reason to go this route rather than use Rails Engines. What would that reason be?

  4. Simon says:

    Looks interesting — but we’re using Engines currently; just wondering if you’d be able to compare it to that? It does look broadly similar — it’s the details that are important to me I guess.

  5. Brian,

    This is somewhat similar to what Django does with apps. e.g. Admin feature in django is a separate app and can be installed in other apps.

    I have been looking for such a feature in Rails for long.

  6. Luis Lavena says:

    Nice article Brian!

    (just spotted it from the Radiant mailing list).

    One question, though: it also manages the overloading/overwriting of plugin views and controllers into the main application?

    A silly question, but will look into the code later today :-D

    Thanks for your hard work and for sharing this with the community!

  7. Brian Takita says:

    @Luis – Thank you.

    One question, though: it also manages the overloading/overwriting of plugin views and controllers into the main application?

    Yes, Desert does support overloading/overwriting of plugin views and controllers.

  8. Brian Takita says:

    I was [informed](http://www.ruby-forum.com/topic/157253) by James Adam that Rails Engines does not support plugin dependencies, since he recommends using gemified plugins.

  9. Gabe V says:

    I have played with desert a bit and am having trouble figuring out the best way to bootstrap it for unit testing. What is your recommendation for making the plugins easy to test by themselves and within another project?

  10. Brian Takita says:

    @Gabe – You can create a test suite within the plugin itself and integration tests within your application.

    If the plugin you are making is simple, you may not need it to be within the context of a Rails application.

    If the plugin is more complex and requires the Rails application to work, it would be easier to use a test Rails application that uses the plugin.

    There are a couple of common patterns that I know of:

    * Embed a Rails application within the plugin’s test directory and have your suite symbolically link or copy the plugin main directory into vendor/plugins
    * Have a test project that has an external to your plugin in its vendor/plugins directory

    Now your applications that use the plugins should have logic dependent on the plugin working correctly. If this is the case, you are implicitly testing the integration of the plugin withing your applications.

    In rare cases, you would need to have explicit integration tests for the plugin.

  11. Simon says:

    Thanks for the reply; I think Engines did do the stuff you wanted — we’ve been using it since about March 2007. I’ve actually already written a plugin to do model sharing, and I’ve submitted a patch. I’m not sure what happened with that; it may have been taken up in a recent version, but we’re sort of using different versions in different apps and I can’t honestly remember.

    Desert looks interesting though, I might have to evaluate it a bit more. Engines doesn’t seem to play as nicely with Rails 2.0 as it used to with 1.2, so I’d be interested to see how Desert works.

    You don’t happen to know how the autoloading stuff interacts with Passenger/mod_rails do you?

  12. Christian Seiler says:

    Just wanted to give it a try, but it failed. I created a new Rails 2.1 project and followed the instructions of the readme. After adding the “require ‘desert'” to environment.rb Rails won’t startup any longer (e.g. script/console). It keeps on doing something with the filesystem, seems like scanning my whole disk (used lsof to track what’s going on).

    I’d really like to try it out since I have class-reloading issues with Rails Engines.

    Any clue?

  13. Brian Takita says:

    @Christian – Its possible that there is a circular dependency with the constant loading.

    For example:

    # user.rb
    class User < ActiveRecord::Base
      include SomethingSpecial
    end
    
    # something_special.rb
    module SomethingSpecial
      User.a_class_method
    end
    

    The User constant is not added to Object when it is evaluated in something_special.rb. When User is encountered in the SomethingSpecial module, Object.const_missing is invoked and user.rb is loaded again. Also SomethingSpecial did not yet get added either, so something_special.rb will get loaded again.

    A fix for this issue is something like:

    # user.rb
    class User < ActiveRecord::Base
    end
    
    User.class_eval do
      include SomethingSpecial
    end
    
    # something_special.rb
    module SomethingSpecial
      User.a_class_method
    end
    

    This would work because the User constant is added to Object when SomethingSpecial is loaded.

  14. Brian Takita says:

    @Simon – I havn’t tried Desert with mod_rails yet so I’m not sure about all of the details.

    One thing to note is Desert will lazily load your files and mod_rails uses fork. This may cause a performance issue if the files are not loaded in environment.rb, because each forked process would need to do the same work to load the files over and over again.

    To fix that issue (if there is one), you would need to show the constants or require the files within environment.rb.

    For example:

    # environment.rb
    
    require 'config/boot'
    require 'desert'
    
    Rails::Initializer.run do |config|
      # ...
    end
    
    User
    AnyOtherModule
    require_dependency 'some_controller'
    
  15. Mattias Ottosson says:

    Hi.

    I just started to look into desert, cause it really looks like a great thing to use for sharing code between projects. Especially when you can mange them as a plugin with git/svn and piston, pretty neat.

    However, i ran into problems pretty fast. I’ve followed the few install instructions, but I can’t get the desert_plugin generator to work.
    “script/generate desert_plugin foobar” just give me the errormsg

    “Couldn’t find ‘desert_plugin’ generator”

    I can manually copy an application to the plugin-folder, and it works as it should, but it would nicer to use the generator to have things set up automaticly. Any thoughts?

    /MAttias

  16. Dan Fitch says:

    I am also getting the following error.

    “Couldn’t find ‘desert_plugin’ generator”

    Dan

  17. Thom says:

    This is really great plugin and I’d like to use it instead of Rails Engines as I need to re-open model classes in other plugins.

    But there are 2 things that keep me away from it:

    1) Automatic migration: Engines has a nice generator (script/generate plugin_migrations) that automatically creates a migration file. Is there anything like this planned for Desert?

    2) Plugin assets: Engines copies all files in public/ or assets/ directory to public/plugin_assets/plugin_name/. I saw the desert_assets plugin on github, but it’s not really working. Will there be an automatic copying mechanism available?

    Also I think it would be great, if either Engines would be able to re-open models or Desert would add the missing features from Engines so that we have to support only one uber-plugins plugin for Rails.

    I have to say that i envy the Merb guys as they have slices built into core…

  18. "Couldn't find 'desert_plugin' generator" says:

    Maybe use desert_tools (http://github.com/etrangedev/desert_tools/tree/master) to generate plugin.

  19. N says:

    @Gabe – hey, I needed the same thing (plugin specific unit tests), so I modified the original Appable Plugin’s plugin_tests and stuck it up here: http://github.com/natlownes/desert_plugin_tests

  20. Eric says:

    I am having some trouble getting desert to work on Windows or Ubuntu 8.04 with Rails 2.1.

    I installed the desert gem, created a new rails app, and then put the require in environment.rb like the readme says.

    The first time I tried starting the server or generating a desert plugin it gave me the error, ‘Dependency is not a module’. After prefixing all of the Dependency usages in lib/desert/rails/dependencies.rb with the ActiveSupport module, starting the mongrel server was successful, but took quite a bit of time to load — Is that normal?

    Now, when I try to create a desert plugin it says the desert_plugin generator can’t be found.

    I would love to start using desert. Anyone else have the same problems?

  21. Levi says:

    @Eric — Having the same problem here. Trying to decide if I want to wade into the code :~

  22. Levi says:

    FWIW; I was able to get desert working with my app by cloning from git-hub, and building a new gem. Seems to be working fine.

  23. dnew says:

    wondering if anyone has made this work with passenger? i am not sure if this is my problem, but i have been building a TOG app (which uses desert. And deployed it on slicehost via passenger. i get errors – you can see them if you hit http://www.rareseen.com/posts. Maybe someone has an idea?

  24. Brian Takita says:

    I’m sorry for not responding sooner. I did not have an atom listener to my comment feed. Bad, bad.

    Fyi, there are two email groups for Desert.

    * desert-devel@rubyforge.org
    * desert-users@rubyforge.org

    I will be spending some effort in getting the desert_tools enhancements incorporated into Desert.

    @Thom: Both script/generate plugin_migrations and plugin assets look good. I’ll add them as a feature requests.

    @N: I’ll take a look at what you have and see if any of it can be added to Desert.

    @Eric and @Levi: This may be a bug. I’ll investigate.

    Desert is slower than a normal Rails app because it does more file searching and inclusion. When require is called in normal Ruby, the file search ends with the first match and the file is loaded.

    In Desert, all files that match the require argument in your project are loaded. If no matching file is found in your project, the normal ruby require is invoked.

    @dnew: Does your plugin have a init.rb file or a lib directory?

  25. dnew says:

    do you mean the frozen desert plug in? it seems to have an empty init.rb file …and then a lib directory.

  26. Andrei Erdoss says:

    I am glad that I found your plugin. I like it that over Rails Engines, it can easily override models. I noticed your comment on mod_rails. I am trying to make the decision to use desert over rails engines. Since then, did you get a chance to test mod_rails with desert? If not, can you provide some pointers on how to test this, in order to see if there would be a problem using desert with mod_rails.

    Thanks,

  27. JellyCatz says:

    Anyone tested the Performance of desert vs engine??

  28. Kevin Marsh says:

    Is desert supposed to work with Rails 2.3? I started down the path of using Engines to encapsulate some common functionality and ran into issues with model overloading in my main App. desert seems like the solution for this, but I couldn’t get past routes. “undefined method `namespace’ for main:Object” when eval’ing the routes file from the plugin.

  29. Joseph Palermo says:

    Hey Kevin,

    Desert does not work with 2.3 yet. We have every intention of making it work with 2.3 (especially since we need it for nearly all of our projects), but we haven’t had the time to work on it yet.

    We hope to find time before the end of the week, but it could be a few weeks until we find the time.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *