Jul 162011

Large projects are typically so large because they try to solve many and complex problems at a time.
People seem to feel that the natural way of getting a handle on such complexity is to increase the level of ceremony: The more is predefined and regulated, the more things seem controlled.
Agile methods in general take the opposite route. Therefore one can often hear and read that agile is a good thing for smaller projects, larger and more complex ones however require a more formal process, may be even a waterfall.
Can that be substantiated? What is the most productive approach for larger projects?

Managing the unknown
Complex situations are best resolved by isolating problems from one another and tackling them separately. Examples for such a “divide and conquer” approach in a software system would be splitting it by different functional areas (e.g. online and batch functions) or by functions per user group. This kind of structuring narrows the scope, allows people to focus and (because of smaller problems) makes solutions more likely. Governance is required for scope management and to control the interfaces between the areas.
Of course there are always interdependencies, and decomposition may not be straightforward. But taking such decisions is a first step towards reducing the complexity and getting a large initiative on a defined (and therefore controlled) track.

What the process models are saying
The agile mindset in fact leads to exactly this kind of segmentation: It’s a logical consequence of short iterations required to result in executable and usable software. While the agile methods (pair programming, collective code ownership etc.) don’t scale well by headcounts, the agile philosophy (identify value, decompose into smaller problems, go stepwise) is applicable and value-adding to any project size – and in particular helpful for large ones. Having smaller subprojects allows in turn to use more of the agile methods again, and benefit from their positive impact on team productivity.

Traditional processes don’t forbid such decomposition, but normally they aim for one big unified solution. Instead, they segment the work into disciplines (e.g. requirements engineering, architecture) and suggest to scale by headcounts.
There are three problems with that approach:
1. Tayloristic separation of roles into disciplines makes communication and knowledge transfer within the team more difficult and therefore less effective. This problem even grows with the team size.
2. Going for one big solution does not favor the more challenging nature of large and complex problems. It is more risky, because it’s an “all or nothing” approach and you may not know until rather late if there is one solution found that will do the job.
3. The governance is focussed on the process with its disciplines, not on the outcome.

Problem solving gone large
Large projects are more complex and therefore require a highly effective approach for problem solving and a proper model for governance. One can say:

  • The larger a project, the more it should follow the agile mindset. Isolating problems reduces the risk, increases manageability and  leads to earlier results.
  • Governance needs to focus on the outcomes and their alignment.

Traditional processes appear so controllable because they define in detail how the whole job should get done. However, that does not help in developing a solution to a complex problem. Instead, the segmentation into smaller sub-problems helps to gain the required control and to implement an effective governance.