Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

LABS
Experience Report: Engine Usage That Didn't Work

On the project I’m currently working on we have a main portal that provides a user registration system and a generic billing mechanism. It also has several sub applications which need to know some information about the user and be able to publish billing events. With a fairly easy to articulate boundary, we thought it might make sense to be deliberate in how we organized our code – we came up with three main solutions:

  1. One big app, just use namespaces
  2. Create the portal and expose API endpoints over HTTP to get user data and set billing data.
  3. Create the portal, and have each sub application contained in a Rails Mountable Engine.

Our Boulder office had been making some noise about Rails Mountable Engines for some time, gave a presentation in the SF office, and I had experience working with engines both in the dark times of engines in Rails 2.x and the markedly improved days of Rails 3.x, and even better still in 3.1+. We had need to scale one sub-component of one of the applications independently, but not entire apps as the primary usage of the system would be low-volume. We set off down the engines path…

It worked pretty well. Two months in to the project we had a retrospective specifically about how engines were working. We agreed that we had felt some pain, but overall they drove out interesting decoupling and that the cost to pull them out could potentially outweigh the little pain we might encounter in the future.

Then our team size doubled. We had multiple epics that had high priority and deadlines associated with them. We also needed to ramp up four new people which led me to reassess our choice of engines once again, not to mention that we eventually will be handing over the code to developers who haven’t seen much Rails.

The conclusion I came to was that engines had to go. Here’s the list of why:

  • Asset pipeline already feels magical, with engines you have to think even harder about what you’re doing and what to include in your application.js or application.scss.
  • Testing. Writing an RSpec request spec didn’t make sense in the engines where certain styles were expected, or a specific layout – we found ourselves stubbing out a whole lot, or putting more and more into a gem that contained shared code between the portal and the engines.
  • Running Tests – We had to have multiple instances of RubyMine open to get the correct load paths for the engine and the portal, constantly trying to remember “Is this in the shared gem, or in the engine?” or “Oh wait, this is specific to the engine, but it is a request spec so we need to go back to the portal to re-run that” was not unsolvable, but felt like a tax on our choice every time I fumbled.
  • Migrations everywhere – You write the migration in the engine, then it has to be copied to the dummy app for testing, and also back up to the main app for running request specs, now you have three copies of the same migration, all with different timestamps. Not fun when you realize you also needed to change that string column to a text column.
  • Everything still had to be namespaced anyway for the engines so we had the folder explosion we were trying to avoid from the One Big App solution.
  • Confidence – It got to the point where I found myself asking “Is this broken because I wrote it wrong, or because there’s something I misunderstood about engines?”. I could never be sure of why something had gone wrong.
  • Documentation – Rails Mountable Engines documentation is a small subset of the knowledge available on StackOverflow for example. If we want to make the ramp up and handoff as smooth as possible, we want to do the most vanilla thing possible so that things are intuitive or at least Googleable.

None of this was impossible, or even difficult to solve. It’s just that it wasn’t intuitive, not what a well seasoned Rails developer was expecting out of the box. It would have been a waste of our client’s time to bring a whole new team up to speed on all that we had learned about engines, rather than having one big application.

This is especially true since the only semi-real-payoff was that it made us isolate our sub-application code from the portal code. I say “semi-real” because the boundary was artificial – the reality was that the sub applications needed to know about the user and his account, and anything we built out for billing was really a dependency of each of the apps. This was different from a great engine like rails_admin that really is a drop in and has no domain-specific dependencies. Here we had nothing but domain-specific dependencies and now, by removing engines, our code and our domain are back where they belong: together.

  • UPDATE *
    Engines really are great and there’s lots of situations where they can be really powerful. Boulder Pivot, Stephan Hagemann, had these additional tips that I wanted to share with you.

  • Regarding RubyMine and running specs: there is a simple way to make RubyMine run all specs in all engines. It will make all engines modules with their own “RubMine root”, which fixes spec runs. http://blog.pivotal.io/users/shagemann/blog/articles/intellij-modules-in-rubymine-

  • Regarding migration duplication: You can have an engine or app that requires others run their migrations. Check out the migrations run by the main app in this sample app: https://github.com/shageman/the_next_big_thing

  • Regarding testing: If an engine is relying on some layout or style to be around, it should depend on it and include it (potentially by way of another engine).

Comments
  1. Thanks for the report. Real world experience is valueable.

    You seem to have two separate issues: wanting to quickly train a new team of lower experience, and the architectural issues around engines. I wish you had had time to address the later separate from the former. I think the issues you report with engines themselves would either make great github issues for rails engines, or have been worth learning how to efficiently address.

  2. ara.t.howard says:

    “I also must confess to a strong bias against the fashion for reusable code. To me, “re-editable code” is much, much better than an untouchable black box or toolkit. I could go on and on about this. If you’re totally convinced that reusable code is wonderful, I probably won’t be able to sway you anyway, but you’ll never convince me that reusable code isn’t mostly a menace.”

    – Donald Knuth

  3. Will Read says:

    @Michael, we had appropriate technical solutions for the work we were doing with engines, but they acted as stumbling blocks for new folks. So much of what we do at Pivotal is about being able to hot-swap people around, those stumbling blocks had to go.

    @ara if I understand you correctly, you’re supporting the removal of engines. Engines do have their place. When done well, they’re just specialized gem libraries that include assets, controllers, and views. Reuse is what a library is designed to do. In our case, we aren’t making a library, we’re making one-use code that, as you point out, needs to be easy to modify. Thanks for the great quote.

  4. Mike Barinek says:

    The use of engines doesn’t necessarily == having controllers, views, and dealing with the asset pipeline. The intent may also not be simply reuse across applications. The main benefit is that for larger projects you’re able to test in isolation areas within your domain with confidence because you have individual builds. A gem would actually work just fine although, you might miss some of active record’s wiring and configuration, a slight benefit that engines provide over standard gems. I’ve found that namespacing actually works great but slightly difficult to test modules in isolation with confidence.

    A concrete example might be a travel site that depends on a payment service. Both the payment service and the travel site are internal to the travel company. In addition, the travel site is the only client to the payment service. This would be a nice candidate for an [unbuilt gem](http://pivotallabs.com/users/mbarinek/blog/articles/2022) or engine (i.e. the payment service doesn’t need it’s own git repo or versioning, it’s a just module within the larger application – developed and tested in isolation, positioned for reuse).

  5. Austin Putman says:

    Thanks for the writeup. Having to switch into different dev environments to run tests, and in some cases having to write two sets of tests for the same code in the different environments, was a major drag on engine use in 2.x. The issue of almost all code needing access to user data is blocking us on separating out some engines or even apps in my current project.

  6. Fiona Tay says:

    Great writeup, Will. I was on this project for a bit and the biggest problem was writing and running tests. Every time we wanted to write a test, we had to stop and think about where it would go.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *