Why do projects get slower over time? Part 2

This is part two of a three part series on why projects slow down over time.

How many times have you heard - or thought to yourself - things used to be much better before? We used to be able to release quickly but now there is so much overhead and things move at a snail's pace.

You're not crazy - things do slow down over time. There are many causes but I'm going to delve into three of the main causes over the next few articles.

• Predictability
• Complexity
• Scaling

Systems tend towards Complexity

Every company starts out with a simple process for delivering software and every system starts out with a great design that is as elegant as it is beautiful. But then there are the edge cases that need to be solved and the extra requirements that break some assumptions that you had baked into the product. And on top of that, the system is worked on by dozens, if not hundreds of people, of differing skill levels and opinions on design, who evolve the system in different directions. Like Newton's second law of thermodynamics - the energy in a software product is gradually moving towards disorder.

This disorder is manifested as product, design, technical or process debt which, left unchecked, continue to slow down the pace of delivery over time.

Product debt includes the features of the product that aren't performing as well as expected. These extra features are cluttering up the product, they are not returning the expected benefit and they contribute to the slowdown in development and design by absorbing time, and money, to maintain.

Design debt includes the inconsistencies in the interface or user experience due to the incremental nature of adding new features which break the original flow of the application. Design debt hurts the standing of a brand because it looks unprofessional but it also slows down a designers ability to come up with a new feature design because there is no consistency in the current system.

Technical debt is commonly thought of as the bugs in a system. But the bigger problem is the features or dependencies that are ingrained in the system and are a nightmare to edit because they always break in unexpected ways. There is a double hit on productivity when these areas need to be updated. Firstly, it takes the developer a longer time to make the initial changes and, secondly, it takes test and bug fixing time to fix the fallout from the changes.

Process debt refers to the ways of working that slow down your ability to release software. This level of documentation required by the process, the internal meetings to get approval to move onto the next stage, or the manual environment management processes that lead to instability and lost time moving code through environments all contribute to a slowdown in delivery.

Paying down the debt

Debt in product development is taken out at loan shark rates and the collectors always show up at the worst possible time. The best time to pay down the debt is as soon as possible.

Product Debt

The definition of done is contentious. Developers use it to refer to the feature being developed. Continuous Delivery teams refer to it as being ready to deploy. DevOps teams refer to it as the feature being deployed. But the best definition of done is when the expected outcome is delivered. This means that you are not finished working on a feature until it is a success. By changing the definition of done teams can't leave those under-performing features alone - they are either improved or removed.

Design Debt

There will always be a push to release the next new feature asap. Investing early in a design system will help to mitigate against inconsistencies in the design but they also help to speed up decision making on design by introducing a level of constraint into the process.

Technical Debt

Some people view code as an asset, but it is better to think of all code as a liability. Every line of code has assumptions that can be proven wrong at any time. If a team recognises this the primary goal of the design of the system will be for disposability - the ease of deleting code. One example is the shift from a previously held core belief such as DRY (Don't Repeat Yourself) with freely copying and pasting code where necessary. This can make maintenance harder in some cases but early abstractions are one of the leading causes of spaghetti code - they make it harder to reason on a problem and have unintended consequences.

Process Debt

The release process debt can be tackled through automation. DevOps practices have shown us that automation in testing and deployment can increase quality and reduce time and cost at the same time.

As for the development process debt shifting from the overhead of projects to more nimble cross-functional product teams removes a lot of the unnecessary overhead. I've covered this in more detail in part 1 of this series.

Who is responsible for paying down debt?

Here is the kicker. Nobody wants to pay for this.

In a project view of the world, there is no requirement to leave the code in a good condition. Every project has tight deadlines that lead people to cut some corners and leave a little debt behind. This is like the Tragedy of the Commons, where self-interest in a shared good (the product) can lead to a spoiling of the good for everyone.

Some companies set up separate teams to do on-going management of the product, but these are often more focused on fixing the obvious bugs in the system or keeping up to date with dependency upgrades.  The hard, and expensive, work of paying down debt is often a secondary concern, if that.

I've also come across the problem in companies where paying down debt is seen as an OpEx expense whereas new features are CapEx. And these companies have much larger CapEx budgets than OpEx budgets.  The incentives to fix the debt just aren't aligned with how the budgets are allocated.

The end outcome is the inevitable "big re-write" that happens every few years. New development is put on hold while everyone focuses on creating a new system, where all of the same problems can be repeated again. But the big rewrite is funded by CapEx so at least there is budget for that.

What can we do?

We need to shift the incentives towards paying down the debt as early as possible. In solving the Tragedy of the Commons dilemma there are three main ideas proposed:

1. Introduce regulation. Reducing usage would work for something like a field which will grow back if left alone but a software product degrades if left alone so regulation isn't the answer.
2. Taxation. Introducing a tax on usage limits people's behaviour and raises funds to implement corrective measures. From a company perspective, this would mean slowing down the pace of change which is not a great strategy when other companies are iterating quickly.
3. Make the commons private. The new owner will have an incentive to ensure that it is well maintained. This is actually the approach that the most efficient companies are taking to solve the challenges of paying down their debt.

Product Teams

Product teams own a part of the product are responsible for it from cradle to grave. By having a dedicated long-lived team people can become specialists in the product and get a real understanding of where the problem areas are and how to fix them. On projects, people are dipping in an out across a wider product domain so they aren't able to generate the same level of detailed knowledge. This structure also solves the process problems that were discussed in part 1 so product teams can move even quicker.

But shifting from projects to products is not easy. I go into more detail in part 1 but the key challenges are around setting the scope for the product teams, ensuring that each of the teams are working towards the same goals, upskilling people in the changes to the ways of working and enabling the supporting management, HR and finance functions to adapt to the new structure.

If you want to learn more about how to build the necessary alignment and execution skills check out the UXDX conference where we have stages dedicated to each of these topics. The Vision stage focuses on the business, customer, process and team alignment whereas the Execution stage focuses on the skills teams need to deliver.

With predictability and complexity covered its time to move onto part 3 - scaling. Often things move quickly when you are starting out because things start small but growth leads to a slow down. Is this inevitable or can we do something about it?