Process as a Substitute for Competence
If you work in a sotfware development organization, you've probably heard a lot about "alternative" software development and project management methodologies, such as Agile, SCRUM, Extreme Programming (XP), Test-Driven Development, etc. These are, of course, all alternatves to the traditional waterfall model (wonderfully parodied in the preceding link -- the conference linked to is, of course, not real. I particularly like the session entitled "Pair Managing: Two Managers per Programmer.")
Most of my development experience came at Microsoft, which has its own software development life cycle (SDLC) and, for all its problems, still has a large number of very smart developers and a management corps consisting almost entirely of former coders. As a result, I operated under the assumption that this SDLC was more or less typical for the industry, as it sort of vaguely lined up with the traditional waterfall SDLC.
And then I came to my current job. Yes, I have found the IT shop that performs development on the unmodified waterfall model -- welcome to 1979. The great majority of our software development work is outsourced to a contracting firm in India, and thus worked on overnight, largely out of contact with our management, and the amount of process involved is staggering.
On one recent project, the software development managers insisted on a 5-page requirements definition document, two management reviews of that document, a functional specification with four contributors, a peer review and a management review of that document, a detailed design document, a management review and sign-off of that document, and a test plan before development could begin on the code.
The code in question? Five lines of SQL.
Yes, a 2-hour project took over a month. Actually, it's not quite done yet, due to the repeated test cycles. You might think all that documentation would have rendered bugs impossible, but if so, you'd be wrong.
Why on earth is so much process present? Why does it take three documents and six meetings to write five lines of SQL? There are twice as many pages of documentation as lines of code! Well, having worked with the people involved on these documents, plus the development, and the testing, not just on this project but several others, I think I've figured out the purpose: process is being used as a substitute for competence.
The thing that's surprised me, though, is that process actually turns out to be a pretty good substitute for competence. With enough steps, documents, design reviews, and test plans, I think that the proverbial 1,000 monkeys with typewriters really could produce a functional IT system. The incredible amount of mutual reinforcement breaks the task down into such minute pieces that each piece is comprehensible and completable by anyone with even the slightest modicum of coding or testing ability. It turns what is inherently a massively complex endeavor -- software engineering -- into little more than rote following of directions. Each step is so long and involved as it must be performed in so much detail that the next step follows necessarily, without putting much thought into it. The waterfall model enforces this level of detailing by making it nigh-impossible to change decisions made early on. It also helps project management -- after years of this sort of development, the project managers know to a pretty good degree of accuracy how long each minute task will take, and can thus do a pretty good job coming up with a (shockingly long) timeline for a project in any stage of development.
The problem with this model, though, is that anyone who is halfway competent finds himself spending more time on process than on actual, productive development activity. Everything is drawn out over a huge period of time. I think this is the main thing that's been driving alternative methodologies like Agile and SCRUM -- the desire by skilled developers to sweep away all that process that's "in their way," and get rid of documentation and meetings.
This has led to the principles of the Agile Manifesto:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
This is pretty much the complete opposite of the waterfall model, and developers often enjoy working in such a model quite a bit more. (This said, it can be done badly, as discussed in Steve Yegge's blog here, though that post eventually turns into Google-worship.) Good developers, on the whole, hate process, don't like writing documentation, have no concept of or interest in business concerns, and like to code by the seat of their pants anyway. A methodology that says "what you want to do anyway, that's good!" will naturally get a lot of traction. Nothing gets developers excited like telling them that they don't need to produce documentation, the code is the documentation.
However, I can see that as much as I hate the process-bound development process I work with here, an Agile methodology would be a total failure with these developers. They are simply paralyzed by the slightest bit of uncertainty, entirely unwilling to just figure out for themselves what will work. They entirely lack both the ability and the desire to experiment. In short, while they're programmers, they're not geeks, and that idea still seems somewhat foreign to me -- when I think about developers, I still think about people who program for fun. But these developers approach programming like it's on an assembly line, and they need to be told the right way to do everything before they're willing to touch it.
So if you're faced with a very involved, complicated process to do something, stop and consider why that process is there -- it's possible, likely even, that it's there to remove the responsibility of making decisions from some set of people. Sometimes that's because someone higher up is (or was) a petty tyrant and doesn't care for decisions being made by anyone but himself -- in that case, the process may be ripe for a change. But sometimes it's there because the people involved are unwilling or unable to act without it, and in that case you change it at your peril. While a more agile process would be better for me, and is better for the development teams at Microsoft and Google, it's not necessarily better for everyone -- not everyone is a self-motivated geek.
Unfortunately for me, I don't think this is a process I can change here. It turns out that outsourcing development to India is so much cheaper that even if it takes four times as long, it's still a net savings.