The Evolution of Software Delivery

I often get asked what is THE agile process. And I give that annoying answer of "There isn't a single process. It depends on the context within each team". To try to give a better answer to people I've created this (faux) history of the evolution of software delivery. I'm going to outline a linear progression even though I know that it wasn't as neat or as linear as I am about to make out. But I find it useful to reference something to give more context to my "It depends" answer. Hopefully, it will be of value for you too.

The beginning

When software development started there were no best practices, or ideas, on how best to run development projects. Starting with first principles people identified some key assumptions about software development and then mapped them to an existing, understood process.

The assumptions that people started with were:

1. Software development takes a long time
2. Software releases are infrequent
3. Software is hard to change after you build it, so make sure you get it right first time
4. Software development requires a number of distinct, expensive, skill sets

Another industry, construction, had very similar assumptions. Buildings take a long time and you can't easily add floors or expand the footprint after the foundations are poured. Construction also involves many different specialities from architects to developers, to QA inspectors as well as labourers, electricians and plumbers among many others. As you can see from the names of the roles, software development copied a lot more than just the construction process.

Construction follows a process where they break the end-to-end project into distinct phases with different specialists owning each stage of the process. This alignment of specialised roles with each phase enables the most efficient use of expensive people.

This seemed like a great process to follow. Waterfall was adopted around the late 60's but it wasn't until the mid 80's that it became the de facto standard for software delivery. But something went wrong. According to the 2014 Chaos report 31% of waterfall projects were cancelled after significant spend and a further 52.7% of projects needed to nearly double their budget.

Given the poor success rates a number of people, independently, started coming up with new and better ways of delivering projects to overcome some of the shortfalls of the Waterfall process.

Agile Development

People don't like seeing their work get thrown away so one of the first ideas was to break up large projects into smaller pieces and build a solution iteratively over time. This approach would give greater visibility on the progress of a product over time and would enable the team to gather feedback to see if their product solves the customers' problems or not. That early feedback enables teams to course-correct as necessary.

But shifting from waterfall to iterative delivery is hard so a lot of organisations translated iterative development into incremental delivery of a solution over time. Agile as a term is very flexible and, by design, there is no one Agile process - the focus is on small iterations, continuous learning and reactivity. But these are seen as soft skills, which are hard to implement, whereas processes are easy to follow and implement. The lack of clarity on a process allows people to avoid some of the more difficult changes and implement the agile process that "works for them" with catchy names like Wagile or WaterScrumFall.

The greater transparency limits the risk of major overruns but it doesn't solve the underlying problems that led to the high failure rates of waterfall projects. You can still get projects which take longer to build than expected and some will be cancelled when it becomes apparent that they won't deliver the expected value.

To make the leap to iterative delivery we need to change our assumptions about software delivery. We need to accept that we can't know what to build upfront, and therefore we need to replace the Big Design Up Front (BDUF) stage with incremental design.

This gives the team the ability to learn as they proceed and change the design if necessary. The challenge here is that the design phase is critical for ensuring alignment within and outside of the project team and for gathering estimates on how much time is required to build the solution. No process exists in isolation in a company so if you try to change one process it will impact on other, seemingly unconnected, processes. The estimates coming from the BDUF, for example, are required by the supporting processes of funding, governance, resource allocation, environment allocation and more.

The problem with these "agile that works for us" adaptions are that they fail to acknowledge or address the conflict that now exists in our assumptions about software delivery.

Assumptions

1. Software development takes a long time
2. Software releases are infrequent
3. Software is hard to change after you build it so make sure you get it right first time
4. It is not possible to know exactly what to build upfront so break up large development into pieces and learn as you go
5. Software development requires a number of distinct, expensive, skill sets

If it is not possible to know exactly what to build upfront then how do we solve the problem that we have to get it right the first time? The big design and the end product in software are often quite different - unforeseen challenges arise that require a shift from the design and new requirements continuously appear. What this demonstrates is that, unlike construction, software is actually not that hard to change after you build it so there isn't the same need for upfront rigour as you have with a physical building. You can easily add (software) floors, change a level or expand the footprint without having to re-pour foundations. So assumption 2 is actually incorrect.

(Real) Agile Development

If the design can change as we learn through the iterations it makes sense to only design what we need in each iteration to limit the amount of re-work that may be needed.

This is actually one of the hardest steps to make because it requires a change in the way of working for a number of functions within the project team as well as changes in the supporting functions.  

1. There aren't enough architects to put one in each team to guide them through the continuous architectural choices. So instead of defining the fixed architecture, architects need to define the practices, principles and forcing functions that ensure that designs stay within identified constraints.
2. A lot more responsibility is handed down to the developers. Instead of building to a specification they are tasked with identifying the best solution for a problem. Because the architect is removed from the challenges of delivery their assumptions can often be incorrect. The developers at the coal-face can often identify better solutions to challenges encountered.
3. Without the BDUF, giving estimates for time and cost are difficult. In a Waterfall project Scope, Time and Cost are fixed based on estimates at the start of a project. With incremental development, Scope has to become flexible because you need to be able to course correct if you learn something new. Time and Cost estimates can still be provided based on a high-level understanding of the delivery and team size, but the ROI and benefits realisation schedule can't be provided upfront because the Scope can change.

These aren't easy challenges to overcome but they are solvable. I'll go into the ways to solve these challenges in future articles.

Let's update our assumptions again.

Assumptions

1. Software development takes a long time
2. Software releases are infrequent
3. Software is hard to change after you build it so make sure you get it right first time
4. It is not possible to know exactly what to build upfront so break up large development into pieces and learn as you go
5. Software development requires a number of distinct, expensive, skill sets
6. Implementation teams need to take responsibility for the design, within defined boundaries

Continuous Integration

A large part of the benefit of iterative delivery is the ability to gather early feedback about the product that you are building. In order to get this feedback though you need a working version of the software, which addresses the customer problem, to be able to demo and get true feedback. And there are two main challenges that teams need to overcome to deliver this - how software development is prioritised and the ways of testing it.

In a traditional waterfall project the most efficient way to build is by component - build the perfect wheel, the perfect chassis and assemble into your car at the end. Context switching costs time so less switching makes the most efficient use of developer time. This component development approach is embedded within teams so even if they adopt agile development the way in which they build can remain the same. The problem is if a customer wants to get from A to B they won't be able to provide any valuable feedback when you show them a finished wheel. It doesn't solve their problem. We need to shift from component-based development to iterative development which requires a shift in mindset within the team to focus on the customer problem instead of the solution. This shift is only viable once you have replaced the BDUF and acknowledged that you can't know what you need to build up front. But even so, it will require training for the team as it impacts on how requirements are written and development is done.

Developer time is a significant cost in software development so is the cost of additional context switching worth it? In addition to the 31% failure rate, analysis from Microsoft highlights that up to 66% of features fail to deliver the expected business value. Given the failure rates, validating that we are building the right thing is more important than developer efficiency. We'll come back to this point shortly because it is going to influence our next assumption.

The second challenge is around testing. To get the real value from the feedback you need to get working software in front of customers, not just demos. But this requires testing, which takes a lot of time and effort to complete. Automation of test cases was deemed uneconomical in the past when you only performed one test pass per (large) release. But two things have changed the economics on testing. First, software is now required for nearly all aspects of a business so the days of one release per year are long gone. Second, like with developer time, since the biggest waste on a project is developing the wrong thing the benefit of automating testing so that we can test ideas with customers more frequently quickly outweighs the cost. Automated testing cycles enable teams to develop "production-ready" code continuously throughout the development process. 100% automation is a tricky challenge so there is still the need to do a smaller manual acceptance test of the software before release but the quality is enough for validating the solution.

In the Waterfall process, the ensuring the high efficiency of specialised teams was a critical assumption in order to keep costs down. In software though, because the frequency of releases has increased the economics of efficiency have changed as well. Instead of ensuring the efficiency of each function the focus becomes on ensuring the efficiency of the end-to-end process. This isn't actually something new to software - the approaches and techniques used are borrowed directly from the Lean manufacturing movement.

The requirement for near-continuous software development has shifted the analogous industry from construction to manufacturing!

Continuous Delivery / Deployment (DevOps)

The benefit of continuous integration is that the code is ready to be deployed at any stage. Should you find out that the project is going to be shut down at least you can deploy the code that you have ready. But like testing previously, deploying code is very costly and time-consuming.  

Even if you have your software ready to go it can take a while to get the people to move the code through the necessary environments and deployment steps. And you run the risk of finding new problems with your software such as poor performance under load. So code that you think is ready to release, actually isn't.

The earlier that you find problems in your software the cheaper it is to fix them. So ideally teams should find out as soon as they finish each iteration whether there are any of the non-functional requirements that need to be fixed. Manual deployments take too long so the deployments need to become automated as well.

Continuous delivery shrinks the amount of time to release your code but there are times when you might not want to release. You might be only partway through a feature so it might make sense to release every second iteration, for example.

But going through the deployment phase can help us to find issues so we should try to do it more frequently. This is where feature toggles / feature flags come in. By building new features behind flags we can release partially built features but keep them turned off. The customer won't even know it is there but we get the benefit of testing that the code hasn't broken something else.

Feature flags also give other benefits. By having the feature behind a dynamic toggle we can start to do A/B testing and multivariate testing in production to more accurately quantify the effect of our changes which enables us to more accurately validate our assumptions about our customers and course correct if necessary. And since we can dynamically turn on and off features we can do our user acceptance testing in production, saving time and cost.

After all of these changes, let's revisit our assumptions.

As teams improve delivery times shrink from months to weeks, to days to multiple times per day. We can't really say that software development takes a long time. And if it doesn't take a long time can we justify the specialisation of roles into distinct stages - how do you manage multiple stages per day? We haven't reduced the number of skill sets required though so we still need architects, business analysts, manual testers, automated testers, developers, operations, security, pipeline developers etc. Each person only contributes a small amount so we simply can't afford to keep all of these people on the team as it would be too expensive. But equally, we can't have them outside the team because it would be too slow. As we described with the architect role earlier, by specifying the boundaries and constraints certain roles no longer need to be heavily involved. However, others on the team need to take on more responsibility. These types of people are referred to as T-shaped people. They have deep specialisation in one area, "|", and some broad understanding of other areas "--". The broad coverage allows them to pick up some of the areas where full time involvement is not required.

Assumptions

1. Software development takes a long time
2. Software releases are infrequent
3. Software delivery can be quick and frequent
4. Software is hard to change after you build it so make sure you get it right first time
5. It is not possible to know exactly what to build upfront so break up large development into pieces and learn as you go
6. Software development requires a number of distinct, expensive, skill sets.
7. T-shaped people are the most efficient for continuous delivery
8. Implementation teams need to take responsibility for the design, creation and release, within defined boundaries

Continuous Value

If we are releasing quickly and iteratively what is the definition of done? It was easy when we had a scope and a set time. Now that we can deploy the code at any time what prevents us from doing so? How do we know when enough is good enough?

If software no longer takes a long time to deliver then it opens up a world of experimentation and validation that we can do.

And this, in turn, allows us to properly verify the effects that each small iteration that we deliver has on both our customers as well as our business' bottom line.  

The business case process no longer makes sense so it is replaced by alignment teams with business outcomes that they must deliver. The teams experiment and iterate on the ideas to be developed to ensure that they achieve the target outcome.  

This structure enables continuous delivery of validated experiments but there are still inefficiencies because there can be a lack of alignment between the Product / UX / Design team and the implementation team. This separation is also commonly referred to as the "Business", who are defining the scope of work to be developed, and "IT", who are delivering what they are told. This means that only half of the team are really accountable for the targets, the other half remaining as a delivery vehicle. There is still a need to shift from thinking about delivering outputs to delivering outcomes for the business.

Assumptions

1. Software development takes a long time
2. Software releases are infrequent
3. Software delivery can be quick and frequent
4. Software is hard to change after you build it so make sure you get it right first time
5. It is not possible to know exactly what to build upfront so break up large development into pieces and learn as you go
6. Specialisation of roles will enable the most efficient use of expensive people
7. Cross-functional teams with T-shaped people are the most efficient for continuous delivery
8. Implementation teams need to take responsibility for the design, creation and release,  within defined boundaries
9. Implementation teams need to take responsibility for the outcomes that they deliver

Product Team

Finally, we have a truly cross-functional, product team working together across both discovery and delivery to build outcomes for customers and the business. By getting developers involved in ideation you get both a wider range of ideas but also input into the complexity of delivering them. By involving designers throughout the development teams can get a greater appreciation of the impact of design on the customer.  

To make the transition all of the way from the defined stages of a Waterfall project to the continuous flow of a product team it requires countless experiments and initiatives that chip away at the blockers that are encountered. But things will keep moving. Teams are growing into the marketing space as well as customer service and operations depending on the needs of their product. The evolution of the process never stops. So I'll add one last assumption:

1. The best processes arise from continuous reflection and experimentation

Where does this all lead us?

Process is fantastic. When you are doing a repeated task a clear process helps you to understand what is required and to avoid making known mistakes. In large companies, processes institutionalise more efficient ways of doing things.

But sometimes this can go wrong. Jeff Bezos highlighted this in his day 1 philosophy with the great quote:

"Good process serves you so you can serve the customer. But if you're not watchful, the process can become the thing. The process becomes the proxy for the result you want."

The problem with all processes is that they embed the assumptions and constraints of the time and situation when the process was created. Over time these starting assumptions are forgotten as the process becomes embedded. The danger is that when the assumptions and constraints are no longer relevant the process can start to work against you.

Have a look at the assumptions below and see which ones are more accurate for where you are working today.

Making the transition is not easy, but over the coming articles I will delve into the corresponding changes that are required throughout the organisation in order to support the transition to product teams, including:

• Management / Structural Changes
• Governance Changes
• Finance Changes
• UX and Design Changes
• Development Changes
• Testing Changes

Helping teams to make the transition from projects to products is the mission of UXDX so if you want to learn more you can subscribe to our newsletter and be the first to get access to the coming articles or you can check out the agenda for the conference at https://uxdx.com.

And, as always, I'd love to hear your feedback so please share your thoughts below or on twitter @roryuxdx.