Integrating Agile with Stage-Gate® – How New Agile-Scrum Methods Lead to Faster and Better Innovation

By: Dr. Robert G. Cooper / Søren Kielgast / Tomas Vedsmand

Recent experiences show that Agile project-management methods can be used in the innovation process and has a great potential to reduce development time and increase the success rate of new products. The article briefly outlines how an Agile method, such as Scrum, can be used within a structured innovation process with milestones and decision points, such as Stage-Gate®, and what benefits it provides to both manufacturers and service-providers.

The product innovation process has traditionally been undertaken using a gated model, consisting of a series of defined stages and gates or Go/Kill decisions points. Such Stage-Gate® processes were developed to deal with the random and disorganized, often chaotic, approach to new-product development, once prevalent in many major firms (and which is still a challenge for too many small and medium-sized companies). A typical gated model is shown in Figure 1.

A structured gating process for product innovation has clear benefits: Such a system allows management to select the best ideas and projects with more insight and knowledge; it also reduces the risk and costs of project failure and increases the chance of new products success.

An exciting new development in the field of innovation management combines the principles and methods of Agile project management – whose origins lie in the software development world – with the tried-and-proven Stage-Gate® method for the development of new services and physical products. Recent evidence shows that this new hybrid approach speeds up the innovation process, makes it more flexible and iterative, and ensures a measurable output early in the process.

A solid and proven model for Agile innovation for manufactured products does not exist yet (such as the traditional Stage-Gate® model). Early-adopters’ experiences, however, reveal how to practice this hybrid model for new physical product development, and also prove that it works. Such early experiences also identify some key challenges and inconsistencies that one must be aware of before implementation.

Why Stage-Gate® is  a Relevant Model for Product Innovation

Stage-Gate® is a widely-used model for product development and innovation: An estimated 75 percent of major U.S. companies have adopted Stage-Gate®, and in Europe, most major manufacturers use a version of Stage-Gate®.¹ The benefits of Stage-Gate® can be summarized as follows.² Stage-Gate® is:

• A risk-mitigating system, by breaking the investment decision-process into a series of stages, separated by Go/Kill decision points (gates). This gated and incremental commitment approach ensures that poor projects are identified and killed early, thereby reducing the chances of continuing to invest in a bad project.
• A simple decision model, where gates are based on defined criteria and pre-specified deliverables. At gates, designated gatekeepers (management) make go-forward decisions, investment decisions, and prioritize and commit resources across ideas and across development projects.
• A transparent process from idea to launch: everyone can see what the requirements are for good ideas and for projects to proceed (by way of decision-criteria and scorecards), as well as who decides (the gatekeepers).
• A comprehensive set of methods, tools and templates developed for each stage and gate, such as templates for idea description, concept description, business case, evaluation scorecards, launch plan, how to run a gate-meeting, how to score ideas, how to build in voice-of-customer, etc.
• A portfolio management system built in, to manage the entire pipeline of ideas and projects. Portfolio management in Stage-Gate® is based on economic methods such as NPV (Net Present Value), IRR (Internal rate of Return) and the Productivity Index (PI), as well as qualitative methods such as scorecards. Stage-Gate® provides evaluation methods for ideas and projects and the tools to decide stop, go, hold or go back.
• A system with proven results and built-in best practices: Companies using Stage-Gate® have more success in developing and launching new products than companies that do not use Stage-Gate®.³
• A well-known and implemented model: many people have been trained over the past 25 years and use Stage-Gate® every day.
• An adaptive and scalable method: Stage-Gate® can be scaled and applied to different types and sizes of projects and organizations (see Figure 2).⁴ There are also versions of Stage-Gate for process development and technology development projects.

Stage-Gate® thus remains a highly relevant model for managing product and service innovation.

.

In the last decade, many companies have made their Stage-Gate® system leaner by analyzing the value-stream, simplifying the process, streamlining deliverables, and removing non-value-added activities. In most cases, a more efficient and leaner innovation process has been the result. “Lean” has not solved all the challenges, however: Issues remain, for example, dealing with uncertainties and ambiguity; coping with rapidly changing customer needs and wants; and accommodating the realities of a faster-paced world where plans often change. Experience have demanded that Stage-Gate® become more adaptive, more agile, and more accelerated.

A New Approach: The Agile-Stage-Gate® Hybrid Model

A new approach – namely integrating principles and methods from Agile project management into Stage-Gate – promises to deal with these challenges and appears to yield dramatic performance results. As noted in a recent Research-Technology Management article: “Indeed, integrating Agile-Scrum methods into Stage-Gate to yield this new Agile–Stage-Gate hybrid model may be the most exciting and significant change to the new-product process since the introduction of gating systems more than 30 years ago.”⁵

Why? First, because this novel Agile-Stage-Gate® hybrid approach provides a framework for dealing with uncertainties and ambiguity in the front-end, accelerates the process through the use of time-boxed iterations, and focuses on the results via development of tangible product increments as the measure of progress.

…the new hybrid method can be used in all stages from Discovery (ideation) to Launch.

Second, the new method also has a major potential for increasing success rates of new products: the model requires that the project team interact with users and customers, starting in very early stages, thus getting valuable feedback and early market validation.

Finally, Agile-Stage-Gate is not “rocket science”. Rather, Agile-Scrum is fairly straightforward to employ as a working method as part of the stages of Stage-Gate®. Moreover, although usually initially deployed in the two technical stages, namely Testing and Development in Figure 1, the new hybrid method can be used in all stages from Discovery (ideation) to Launch, as shown in Figure 3.⁶

Agile Project Management – Some Background

Agile project management is based on principles from The Agile Manifesto (see Figure 4).⁷ Agile was developed in reaction to major failures using the dominant “waterfall” way of managing large IT projects. Waterfall focused on the big, long-term goal: the final product and its major features. But requirements tend to change rapidly in IT projects; the features and criteria defined when the project was initially planned often were no longer valid by the end of a 12- to 18-month development cycle. Committing early to features and schedule means that compromises will be needed late in the game; early commitments to large features, long schedules, long feedback loops, and the replanning inherent in traditional product development processes create inefficiencies and slow the development cycle.⁸

.

In Agile, development projects are based on the principles of self-organizing teams; close dialogue with users and customers; many short iterations with functional and visible results; interdisciplinary collaboration between business and development; and adoption of changes as the project progresses.

The results in the IT world have been very positive, in particular in reducing development time.⁹ Agile project management has therefore gained ground as the new dominant development methodology in many IT projects and IT-driven companies in the last five years. Figure 5 shows results achieved with Agile from an HP survey of over 600 IT product developers across multiple businesses.¹⁰ Nearly 50 percent of the IT professionals in the survey reported increased customer satisfaction and more than 40 percent witnessed reduced development costs and shorter times to market.

Experience shows that certain IT projects are more suited for Agile than others. For example, the largest bank in Denmark, Danske Bank, reports that more complex projects will gain more from Agile than simpler projects which can be well-defined upfront.¹¹ Danske Bank’s experience is that successful Agile projects need a clear project owner, and also project team members that are experienced in Agile development. Developers were trained initially and then helped by an Agile coach for two months during implementation. Performance results were significant at Danske Bank, with development times cut in half by using Agile. Currently the bank has more than 40 development teams using Agile project management, in parallel with the classic waterfall model, each model being used for different purposes and in different contexts.

Scrum – A Rugby Approach

Scrum is one of the more popular versions of Agile, and is indeed the version that is most often used in conjunction with Stage-Gate. Scrum was first identified in 1986 as “a flexible, holistic product development strategy where a development team works as a unit to reach a common goal” as opposed to a “traditional, sequential approach”.¹² The basic framework of Scrum consists of:¹³

• Sprints: Time-boxed development periods of 1-4 weeks, each sprint consisting of a development-and-test iteration. Several sprints can be run simultaneously or one after the other, depending on the size of the project.
• The Product Backlog: Features of a new product or IT solution, seen from the perspective of the business (value-adding). At the beginning of a project, where the product definition may be largely unknown, user stories are often backbone of the product backlog (stories about what is not working with the current product, list of user needs and how a new solution might help users). An overall vision of the product or solution provides guidance to create the product backlog.
• The Sprint Backlog: The features the project team will develop in the current sprint, that is, specific elements of the product backlog. The sprint backlog is agreed by the development team at the sprint planning meeting.
• The Sprint Planning Meeting: A meeting, lasting not more than eight hours, to kick off the sprint. The sprint is planned and the sprint goal is defined by the project team. The project team also agrees upon the definition of “done”, that is, when is a sprint finished in terms of a done increment (such as a working piece of software). The definition of “done” is important in Agile-Scrum, as, together with the sprint goal, it directs the effort of the development team during the sprint.
• The Backlog Tasks: Specific tasks to be undertaken by the development team, as defined at the Sprint Planning Meeting. Tasks are based on the development team’s estimations of what work-effort each task requires and thus defines the number of tasks that can be completed in the sprint. Note that time and resources are not flexible, but the number of tasks in the sprint are.
• Daily Scrum Meeting: Daily morning meetings, often called “daily stand-ups”, lasting not more than 15 minutes, where the development team discusses what they have done the last 24 hours, what they will accomplish today, and any challenges that may impede their work.

 

• Sprint Review and Retrospective: When the sprint is over, results are presented and demonstrated to stakeholders (management and customers). Results are considered to be a part of the Product Backlog: a working increment, such as a piece of functioning software. The Product Backlog is revisited and the team gives inputs to the next sprint. The Retrospective is a meeting to evaluate the process only.

The elements of the Scrum process are illustrated in Figure 6.

Roles and Responsibilities in Agile-Scrum

There are three major roles in the Agile-Scrum framework, which together are called “The Scrum Team”:

• The Product Owner who is responsible for obtaining inputs from the business, creating relevant user stories, and determining priorities of product features in the Product Backlog.
• The Scrum Master, the facilitator for the development team, ensures that the development team learns and uses the Scrum method; facilitates team meetings; and assists the team in removing outside obstacles (the Scrum Master does not function as the traditional project manager).
• The Development Team, typically a team of 3-9 people, responsible for the sprint planning (estimating how much will be done in a sprint), and the development, testing and delivery of a product increment (a functional outcome consistent with the definition of “done”).

Most notably is the absence of the traditional project manager role. Traditionally, the project manager’s role is to manage the project in terms of scope, cost and time using a project plan with defined deliverables, detailed budgets, a milestone plan; to facilitate project meetings, ensuring progress, communication; and sometimes to be involved in project staffing.

The project management role is divided somewhat differently in an Agile-Scrum team:

• The Product Owner plays an important role in defining what creates value and is responsible and actively involved in prioritizing the backlogs (increments to be developed) at meetings, such as the sprint planning meeting and sprint review meeting.
• The Scrum Master, in its basic form, handles only facilitation within the Scrum framework, and helps to remove impediments to the development team. In some cases, the Scrum Master may also be the communicator of progress to the Product Owner and other stakeholders during a sprint. The role may include a wider role as facilitator of ideation and innovaton throug the entire project, also after the initial stages.¹⁴

How Do Agile and Scrum Apply to Stage-Gate®?

A handful of leading manufacturers have begun to experiment with combining their traditional Stage-Gate process with the newer Agile methods. The resulting Agile-Stage-Gate® hybrid model is still new, but initial experiences from   lead users are very positive. This hybrid version of Agile (such as Scrum) and Stage-Gate® seems to be a promising new innovation model.¹⁵

The benefits of combining Agile with Stage-Gate®, based on a number of cases from Denmark, Sweden, Germany and the U.S., can be summarized in five areas, listed in Table 1.¹⁶

.

The most provocative results flow from two factors in Table 1:

• Factor 1 – Gets the products right: Numerous studies reveal that many new products fail, or don’t deliver the promised value, when introduced in the market. The major reason for failure: a lack of understanding of customer needs, and the product failing to meet needs. Even though companies that employ a new-product system such as Stage-Gate have higher success rates, there is still room for improvement. Using iterative sprints, which build in numerous user tests and market validations, and earlier in the project, greatly enhances the chances of getting the product right, and thereby greatly heightens the odds of success.
• Factor 3 – Accelerates development: Long times to market is a challenge to many companies. The reasons: too slow to pick the right projects; challenges in defining the winning product concept in the early stages; getting the market feedback in place; and a lack of focus. The highly iterative, time-boxed and speedy approach in Agile-Stage-Gate, with its focus on getting working product increments developed, is a promising method to solve these time-to-market challenges. The use of focused teams, dedicated to one project (Factor 4 in Table 1), also helps to increase speed to market.

How much Agile-Stage-Gate can increase success rates and accelerate development time is still not yet fully documented, and may vary across industries and types of organizations. But in one Danish study of five manufacturing firms, the results were extremely positive on a number of performance metrics: more adaptive (faster response to changing product requirements), better team communication, higher team morale, and most important, faster to market.¹⁷

Agile-Scrum provides a simple methodology that can be readily built into Stage-Gate® to speed up the innovation process, make it more flexible and iterative, as well as ensuring user-feedback and a measurable output during the process. By working in the sprint format, speed and fixed-time are emphasized: the tasks may vary, but not the length – a sprint cannot last longer than what was agreed by the team. This applies to all stages in the innovation process.

The greater flexibility in content inherent in Agile-Stage-Gate is well-suited to the innovation process, where much uncertainty and ambiguity may initially exist.

The greater flexibility in content inherent in Agile-Stage-Gate is well-suited to the innovation process, where much uncertainty and ambiguity may initially exist. That is, in some development projects, no matter how much voice-of-customer work or technical assessment is done in the early stages, the product definition is still not totally certain as development begins. In Agile-Stage-Gate, however, the product is not defined in detail in advance, but rather its definition is determined as part of the development process and as the team moves forward. Note that Agile-Stage-Gate® is an iterative process which builds in a user focus throughout the process: the project team builds product increments to test and learn from customers and users along the way at every stage.

Developing a physical product or service is obviously quite different than developing software, and so the notion of “product increments” created in each sprint in an Agile-Stage-Gate system must be modified somewhat from the IT-version of Agile. But early-adopter manufacturing firms have made the adjustment, and typically define “product increments” as “any deliverable that is tangible, the result of work done by the project team”. These tangible deliverables can be, for example, the results of market research (voice-of-customer work); computer generated 3D drawings of the product concept; detailed design drawings; protocepts (a product version between a concept and a ready-to-trial prototype); rapid prototypes; and early working models – anything that can be demonstrated to stakeholders, including customers, for feedback.

Using Agile-Stage-Gate® in the Front-End of Innovation

Agile-Stage-Gate® is used most often during the technical stages –  Development and Testing – but also applies to the early or “fuzzy front end” stages of the innovation process, namely Ideation (discovery), Concept Development, and Building the Business Case. Figure 7 illustrates how Agile principles and Scrum can be built into these earlier stages leading to Development in a Stage-Gate® framework.

The iterative (spiral) sprints in the early stages include the Idea Sprint and Concept Sprint, followed by a number of Development sprints. Each of these early stages typically are made up of several sprints before entering the gates in Figure 7. The sprint reviews, which follow each sprint, are in effect “mini-gates” to evaluate each increment done in the individual sprint; by contrast, each gate in Figure 7 is a more in-depth evaluation of all the increments completed in the project thus far, as well as a Go/Kill or investment decision point.

Each of the multiple sprints in the Idea and Concept stages lasts 1-4 weeks, and consists of a number of repeated actions within the Scrum framework. Early-stage sprints and detailed actions are illustrated in Figure 8, and while they may be executed in different ways, some important caveats and best practices exist, and are highlighted below:

• Prior to the first sprint, the “Reason-Why” and the Vision is defined by management; note that top management recognition and support is essential for the project to move forward.
• User-stories are important for the backlog at this early stage, when a new solution is not yet known; only ideas and hypotheses exist at this point. User-stories provide guidelines for the initial idea sprints in two ways:
• First, user-stories identify customer needs, points-of-pain or jobs-to-be-done; these needs and user stories are best derived from a Voice-of-Customer user study, such as an ethnographic observation or a personal interview study.
• Second, user-stories illustrate how users can solve their challenge using a new solution and what benefits they would derive from the solution: for example, scenarios of how a new solution might help users get their jobs-to-be-done better, faster and cheaper.¹⁸

An example

The Danish company, Danfoss, undertook a major waste-water treatment innovation project, whereby Danfoss designers and developers co-ideated new solutions and created scenarios jointly with users. Both groups worked together at the user’s waste-treatment plant: Here they followed the users’ treatment process step-by-step, and imagined how a new digital monitoring solution might help users get their jobs done better. (This case is documented by Buur and Matthews, who label the method “Participatory Innovation” to emphasize the participation of users early in the process¹⁹). Developing user-stories with this approach is ideally a part of the second idea sprint – a test of early ideas – and thus guides subsequent concept sprints.

• The Reason-Why, Vision and user-stories are summarized into an “Innovation Challenge” to scope the project.

The Sprint Planning Meeting is important to plan the sprint and define “Done” (referred to as DoD = Definition-of-Done). In the world of physical products, “Done” may be obvious at later sprints, when development is underway; less evident is “Done” in the earlier stages, where the product is undefined and nothing physical is yet available. Here are some specific guides for the early stages:

• In the Idea Sprint, DoD may be defined as: ”A short-list of rpbust ideas that have been selected from a larger pool and matured somewhat”. For example, each idea might have undergone a short analysis, incl. fast user feedback. Ideas are pre-screened, visualized and presented on an Idea Canvas to qualify for DoD.
• In the Concept Sprint, DoD may be defined as: “A visual concept description,  rapid prototype, protocept or a “pretotype”, with the business case summarized on a Business Model Canvas”.²º (A “pretotype” is similar to a protocept, namely an extremely simplified, mocked-up or virtual version of the product, used to help validate the evolving product).

An important step following each sprint is to revisit the backlog. For example, the Idea Backlog is refined after each idea sprint. This Idea Backlog ultimately becomes the Concept Backlog once it passes the first gate in Figure 7. Similarly, the Concept Backlog is updated after each successive concept sprint. The ongoing refinement of the product backlog must be transparent and visible to all project teams and team members. The Product Owner is manager of the Idea and Concept Backlog and decides what should be prioritized and included in the backlogs (for example, what product features), based on an understanding of what creates value to the customer and to the business.

Best practice visual, user-oriented and collaborative methods that are utilized within Agile-Stage-Gate® include:

• Voice-of-Customer methods, such as ethnographic studies and personal interviews.
• Visual methods, such as visual brainstorming and mind mapping.
  • Digital methods, to collect user-inputs through mobile research and digital tools for collaborative brainstorming.
  • Simple templates, for example the Idea Canvass and Business Model Canvas that facilitate team member collaboration,  and are also easy to share and communicate.
  • Protocepts, pretotypes, and rapid prototypes to get representations (or early versions) of the product in front of stakeholders (customers, users and management) for technical and market validation – excellent methods for getting early users-feedback
  • Minimum Viable Product is highly relevant method in early sprints to identify core features of a new solution and may be combined with the above.

The Future

Combining Agile project methods, such as Scrum, with Stage-Gate® has the potential to yield more productive and faster product and service innovation. Agile-Stage-Gate® gets the product right via multiple, early customer-validations; it deals with uncertainty and ambiguity regarding what the product solution might be; it focuses project teams on the one project, thus accelerating projects; and it fosters much stronger team morale. Examples of early adopters of this new system are appearing around the world where companies successfully bring these two models together – a hybrid approach which delivers the best of each model –  for innovation for both digital solutions, physical products and service products.

These are early days however. And more evidence and experience-based results are required – results evaluated in terms of objective measures such as development times-to-market and new-product success-rates. Moreover, there remain a number of inconsistencies and paradoxes that early adopters are still working out, especially in the world of manufactured products, and where outside expert help might be needed.²¹ Nonetheless, early results are very promising, sufficiently so that any firm seeking improved product development performance should investigate and consider these new Agile-Stage-Gate hybrid methods.

Are you interested in learning more about Agile-Stage-Gate®? Please feel free to contact GEMBA Innovation.

By Tomas Vedsmand, Søren Kielgast & Robert G. Cooper

References

1. www.bobcooper.ca/articles/next-generation-stage-gate-and-whats-next-after-stage-gate

2. About Stage-Gate®: see “The Stage-Gate system: A roadmap…” at www.bobcooper.ca/articles/next-generation-stage-gate-and-whats-next-after-stage-gate; and: gemba.dk/stage-gate/stage-gate-resources/

3. www.bobcooper.ca/images/files/articles/1/1-New-Products-What-Separates-The-Winners-From-The-Losers-and-What-Drives-Success.pdf

4. Source: Cooper, R.G.; see www.bobcooper.ca

5. Cooper, R. G., “Agile-Stage-Gate Hybrids: The next stage for product development” Research-Technology Management, Jan 2016, 59, 1, pp 1-9. http://www.bobcooper.ca/articles/agile-stage-gate

6. Cooper, R.G. & Sommer, A.F., “Agile-Stage-Gate: New idea-to-launch method for manufactured new products is faster, more responsive” Industrial Marketing Management, forthcoming 2016.

7. ‘See Beck et al., 2001: www.agilemanifesto.org

8. Cooper, R. G., “Agile-Stage-Gate Hybrids: The next stage for product development” Research-Technology Management, Jan 2016, 59, 1, pp 1-9. http://www.bobcooper.ca/articles/agile-stage-gate

9. Examples: https://agileinsights.wordpress.com/2013/08/11/what-evidence-is-there-that-agile-works/

10. Source: http://www8.hp.com/h20195/v2/getpdf.aspx/4AA5-7619ENW.pdf?ver=1.0

11. www.version2.dk/artikel/2000-danske-bank-udviklere-arbejder-baade-agilt-og-med-vandfald-19403

12. Schwaber, K. & Sutherland, J.: The Scrum Guide, 2013, see: www.scrumguides.org/docs/scrumguide/v1/scrum-guide-us.pdf and Schwaber, K. Agile Project Management With Scrum, Microsoft, 2003.

13. Takeuchi H. & Nonaka, I. “The new new product development game”, Harvard Business Review, 1986, 64, 1, pp 137–146. https://hbr.org/1986/01/the-new-new-product-development-game and: The Knowledge-creating Company: How Japanese Companies Create the Dynamics of Innovation, 1995, Oxford University Press.

14. Vedsmand,T.: ‘Making Ideation a Part of the ‘Innovation Project Machine’, 2013, InnovationManagement.se, staging.innovationmanagement.se/2013/02/04/making-ideation-a-part-of-the-innovation-project-machine/.

15. Cooper, R “What’s next After Stage-Gate”, Research Technology Management, 2014: www.bobcooper.ca/images/files/articles/0/3-Whats-Next-After-Stage-Gate.pdf and: Sommer et al., 2015: www.projectmanagement.com/content/attachments/jwoolcott_260515093635.pdf and Cooper, R. & Sommer, A., ”The Agile–Stage-Gate hybrid model: A promising new approach”, Journal of Product Innovation Management, 2016, on-line.

16. Cooper, R., webinar presentation May, 2016; also: Cooper, R.G. & Sommer, A.F., “Agile-Stage-Gate: New idea-to-launch method for manufactured new products is faster, more responsive,” Industrial Marketing Management, forthcoming 2016.

17. See: Sommer et al., 2015; and Cooper Sommer, 2016, in footnote 14.

18. Christensen, C.M. & Rayner, M.E., “The innovators solution”, Harvard Business School, 2003.

19. Buur, J. & Matthews, B., “Participatory innovation”, International Journal of Innovation Management,  Sept. 2008, 12, 3, pp. 255–273. Participatory Innovation involves collecting experiences based on Lead-User Innovation, Participatory Design and Design Anthropology methods.

20. Ostervalder, A. & Pigneur, Y., Business model generation: A handbook for visionaries, game changers, and challengers, 2010, NY: John Wiley & Sons.

21. See Cooper & Sommer, endnote 15, 2016.

Photo by Surface on Unsplash