The 6 hidden costs of large waterfall projects

I was speaking with the Head of Delivery at a company recently and he said that he wants to move to more agile releases but he can't justify the cost. He has been pushing his teams to split big projects into smaller releases so that value can be delivered quicker but every time they plan out a project in multiple phases the costs become prohibitively high.

For context, this company had limited automated testing so every release needed to undergo expensive manual testing. Smaller releases, therefore, multiplied the cost of this manual testing effort. The company decided that they would hold off on trying to reduce project sizes until automated testing was in place.

I agree with the decision to invest in test automation but I disagree with the conclusion that they should hold off on smaller releases in the interim because I don't think it takes into account all of the real costs to the business. The assumption of it being more expensive is based on one metric - the cost of delivery of a single project. But nothing happens in isolation, so to measure the real cost you need to factor in the costs across the portfolio.  Which brings me to the 6 hidden costs of large projects.

1. Building the Wrong Thing

Context: Universal

We'll start with the biggest cost. Yes, it might cost a bit more to release more frequently, but how much could you save by learning that you are on the wrong track and shifting your focus before you build the wrong thing. Up to 66% of features fail to deliver the expected value so, even without any of the further hidden costs, this alone is enough to warrant the extra investment.

2. Holding Cost

Context: Universal

Holding cost refers to the revenue / value that the business is losing by not releasing product features or fixes that are ready to be released. When a project is approved it typically has a return that is a multiple of the investment - the business value far outweighs the implementation cost. So even if you have to increase the cost of delivery slightly the early release will likely mean that the business is better off.

3. Merge Costs

Context: Monolithic code base. Code branching.

Projects don't happen in isolation. In the company I am currently in, for example, there are over 60 projects running concurrently. This means that there is a constant need to merge code changes from projects that are delivering now into long-running projects. Each merge carries risk but bigger merges have exponentially larger risk which results in merge costs increasing exponentially with project size and duration.

4. Delaying Costs

Context: Monolithic code base. Code branching.

Another issue that can occur is premature dependency enforcement. Let's say you want to start a new project and you think it will take 6 months to deliver. There is already another project in flight that will go live in 5 months time so it makes sense to build your changes on a branch of the earlier projects code. This reduces the merge cost above but if the earlier project gets delayed then you're delayed too.

5. Blocking Costs

Context: Monolithic architecture

With monolithic architectures, which are still the norm, there is only one configuration of code and infrastructure in production so only one project can be on the path to production at a given time. This means that other projects must queue up behind it. Larger projects have longer path to production durations which block other projects from releasing.

6. Planning Costs

Context: Monolithic architecture (single path to production)

When an organisation runs a lot of projects they are often scheduled very close to one another to take advantage of the limited release windows across the portfolio. This causes any project slippage to have a cascading effect on the following projects. I refer to the resulting chaos as circular planning; align all of the variables to generate a plan that works, start again when one dependency slips and repeat continuously until you finally release. Once you factor in the number of projects in the portfolio and the planning, resource management and communication overhead the cost of a single project slippage multiplies quickly.

7. (ish) Change Requests

I've included a 7th cost because people will, rightly, call out one cost that I've left a major cost out: Change Requests. This is an obvious, but contentious cost. The larger the project the more change requests will be required along the way. This is so accepted that many companies include change request contingency into their project planning. The problem with this cost is that people assume that they will need this anyway, regardless of methodology. And since nobody runs the same project in two methodologies it's hard to prove definitively. I've wasted a lot of time arguing that the removal of the overhead of a change control process justifies an agile implementation approach but it just isn't effective.

Now that we are aware of the costs the most important question now is why do these costs stay hidden?

Because this the way things work so there is no point fighting it.

What can we do to make these costs visible?

Most of the costs are related to the size of the project. Smaller projects reduce these costs as they release features as soon as they are developed, merges become smaller and less frequent, the risk of delays reduces and quicker releases do not block the pipeline for as long a time.

Project processes that are ingrained in many organisations have a bias towards larger projects, as is the case in the company that I mentioned at the start. If we want to enforce smaller projects we need to introduce forcing functions that break the habits of old. There are two types of forcing functions that we can use:

1. Regulation

Imagine if you could only spend one week on your path to production. What impact would that have on your projects? To test all changes in a shorter time you need to do less changes. Alternatively it may incentive innovation such as automation or performing risk-based testing. Both of these are great outcomes and also further enables for smaller projects as they reduce the cost of releasing. Once these innovations are delivered you can change your forcing function to be 2 days on the path to production triggering further innovation or size reduction.

2. Internalise the externalities

It might not be possible for projects to shrink their timelines or split into smaller pieces as quickly as you would like. Often there are technical and architectural tasks that must be completed to enable smaller projects. These enabling changes compete with the functional work that needs to be done for both funding and priority and they often loses out.

In such cases you need to internalise the externality, which is just a fancy way of saying that you need to make the costs of the bad practice felt by everyone. Taxes are a great way of achieving this goal. A sliding scale tax that increases exponentially in line with the length of a project will both expose the hidden costs of the project and provide the funding to do the enablement work. Either outcome enables smaller projects.

Enforcing change is hard. Practices have been embedded and hardened over years and people are used to working in particular ways so any rocking of the boat is often unwelcome. By exposing the hidden costs of the current way of working and proposing solutions you can start to work on the motivation element of introducing change. But motivation alone won't work. You need to focus on the personal ability to adopt new practices and the organisational ability to enable the change as well. If you're interested to learn more check out UXDX which is focused on helping companies shift from traditional waterfall projects to cross-functional product teams.

there any other costs that you are aware of? What strategies for exposing them have worked for you? I'd love to hear your comments and thoughts.