We'll respond shortly.
Developers need to do stuff that you will never ask them to do, and if they asked you if it was ok to do it, you would tell them, “Let’s do something else.” I’m not talking about “gold-plating” an app, or the kind of thing an “architect astronaut” would cook up, I’m talking about real value-adding tasks that are near-impossible to assign a direct business value.
Tasks that fall into this category are things like code-refactoring which will allow a sustainable development pace, or crating monitoring charts or other monitoring tools which allow developers to see how a system is behaving, but may be Greek to a non-developer. These are the things that are important to developers, that help us exact our expertise on a daily basis instead of guessing. It is equivalent to market research before making a decision on how to proceed on the business side.
However, our time is to be accounted for. We insist that the client uses a backlog with priorities, and the only thing that keeps those priorities useful is that we as developers honor them. If we venture off the reservation, then the client sees his tasks sitting around in the backlog while we do as we please.
The workarounds look like this:
Each one has its own flaw. Doing tasks outside of work hours goes against the “sustainable pace” principle. It also usually means that someone is soloing, meaning that knowledge isn’t transferred, and now you’ve got code-ownership whether you wanted it or not. Slicing out a chunk of the day and working on things in stealth way hides things from the customer, which leads to a breakdown of communication and trust. Talking to the customer sounds great, but as mentioned, it can be hard to assess value of such tasks, and ultimately, the customer will either simply trust you that this is important, or he’ll decide that you’re over engineering something and deny you the freedom to do technically critical tasks.
You could argue for the developers to be assigned a “budget” each iteration for development tasks. That’s not half bad in principle, because now the developers have some freedom, and they also have to prioritize what is and is not important. But then, what do you bill against that budget? Does all testing go against it? What about the time spent setting up a server in a repeatable way? Where is the line, who draws it? This system gets complex and contentious very quickly.
The only solution I’ve found: People. These tasks have to get done because they “fell” important to the developers. The developers experience pain points, and can anticipate when certain design choices will impede future progress. They will solve this problem on their own, but one situation solves the problem more than most: Have someone develop with the team who owns the backlog. He may be a “junior” developer, and he may spend a significant portion of his time tending to business tasks as well, but he will feel the same development pain as the rest of the team if he is in it, pairing, and hearing the groans of “this should be better” or “this will bite us in the future”.
No methodology, be it Agile, Waterfall, or anything else I’ve heard of, accounts for this person. It is a treacherous line this developer-business hybrid walks. Too much development, and he risks becoming out of touch with the business goals. Too much business, and he risks being alienated from the development team.
Developers need this person. They need someone who will hear and understand their technical needs, and still be able to temper their work with the business goals in front of the company. Without him, you create complexities too great for any system to handle, or you run the risk of breaking down the lines of communication.