By: / / /

Thanks to loads of compelling research studies and best practice cases in open innovation (OI) carried out over the last decade, several companies nowadays begin to embrace and partially apply the new principles and methods OI offers. However, when managing open innovation at the project level, even experienced managers still go blank at the question: how to assess, control, and measure the performance of these activities? In this series of articles, we will address the above issue by discussing a general framework for an open innovation performance measurement system (Part 1). Given this framework, a metricsbased management toolkit will be presented that provides a suite of key performance indicators (KPIs) for a specific set of OI methods that demonstrates the key results of our Open Innovation KPI 2012 Study (Part 2).

Obviously, since innovation by nature is a non-routine, creative and unpredictable task, metrics might seem like something for the Controller-nerds, rather than a common skill that is central to the innovator’s DNA. And while many argue that too much measurement stifles innovation, it still remains key for the survival of every business.

Assessing progress and measuring the impact of your innovation activities enables you to change your strategy before mistakes become expensive or great ideas are refused. While the development of innovation metrics in general is still an emerging discipline, there is absolutely no clear guidance on how companies should approach them in order to measure the success of their open innovation initiatives. Anyway, in these times of fast changes, there is actually a good chance that the ‘old’ systems you set out to measure innovation won’t match the challenge you’re going to face when piloting the new and emerging trends of open innovation.

Next Generation Open Innovation – What has changed?

When doing a google search for the term “open innovation” you will receive more than 3 million hits today. Henry Chesbrough who coined the term did the same 10 years ago in 2003 and got around 200 page links. It’s for sure no coincidence that at the same time the Internet facilitated the emergence of online communities and social media networks, our innovation model was beginning to change from a formerly closed to a more open one. But does this mean that open innovation is something new that has grown up with the rise of the Internet?

Many of our industry clients argue that it is merely old wine in new bottles. Since the last few decades companies such as Procter & Gamble (P&G) and General Electric (GE) have been already innovating with external partners beyond their own boundaries in order to succeed in R&D’s ever changing environment and to deliver better quality and more competitive products in a more cost effective manner. Thus, they have undertaken substantial efforts to acquire and use external knowledge from outside actors by collaborating with in technology partnerships, joint ventures or strategic alliances – to name just a few examples of traditional inter-firm co-operations.

However, whereas the existence of such external cooperation networks is not a new world phenomenon, the Internet and in particular the rise of new information and communication technologies (ICT) in fact have substantially broadened the scope and deepened our understanding of open innovation.

In this new era of open innovation companies are innovating with external actors in a very flexible and informal way beyond the traditional notion of technology partnerships or innovation alliances (Diener & Piller 2013). Just as cloud computing provides companies with virtually unlimited storage capacity and processing resources on demand, today new ICT-enabled methods of open innovation such as crowd-sourcing platforms connect companies to the ‘human cloud’ –a worldwide network of millions of individuals, ready to deliver ideas and solve problems that range from simple to the complex.

Anyway, whereas these new methods of open innovation have become an important part of many companies’ innovation strategy, they also imply a fairly high level of complexity and uncertainty that innovation teams, within their exclusively internal or even traditional inter-firm cooperation projects, have been never faced before.

Upcoming Challenges of Measuring today’s Open Innovation Practices

Teamwork will not just be cross functional, but will span across a higher number of companies, universities, governments, suppliers, customers, and individuals. While in the past traditional problem-solving processes led to perhaps a few hundred ideas, these days, a successful ideation contest– if it is directed to an external network– can easily generate thousands of insights from different sources across the globe.

In this new environment of a problem solving process, external idea or solution provider often get access to numerous online tools, such as search engines, databases, source codes, tools for creating wikis, podcasts, websites, CAD programs, and other toolkits (Habicht & Möslein 2011). With this equipment they already started to collaboratively develop successful products such as applications for mobile devices or other open source innovations that have the potential to change an entire industry.

The incorporation of such a large number of diverse insights can be challenging, confusing and apparently seems to be uncontrollable. In this context, measuring open innovation would mean that the contribution of each participating individual and their innovation tools needs to be transparently stated in a firm’s performance measurement system in order to accordingly evaluate the quantity and quality of their provided inputs.

The level of complexity of initiatives driven by open innovation far exceeds the one which corporate innovation teams in traditionally executed innovation projects have to deal with.

It is easy to see that the level of complexity of initiatives driven by open innovation far exceeds the one which corporate innovation teams in traditionally executed innovation projects have to deal with. This means, that deploying open innovation requires not only access to financial resources and the clear allocation of responsibilities. The untapped secret lies in a company’s ability to successfully measure the huge amount of knowledge– ranging from very general submitted ideas to highly complex technical solution proposals – which might include developing a list of ‘approved’ indicators for project managers to incorporate into their performance measurement systems.

Does Your Company Measure Up? Call for Open Innovation Metrics

Studies have shown that around 90 % of company’s innovation efforts never result in commercialized products or services (Cooper 2002). The low return on innovation leads to the suspicion, that innovation in practice still seems to rely on fairly random incidents, rather than being the result of clearly defined performance measurement procedures (Reichwald & Piller 2005). Other research confirms the suspicion, pointing especially to the shortcomings of coordination and underestimation of the complexity that arises in the context of open innovation processes (Hagenhoff 2008). It seems, however, that if companies start approaching open innovation in a more organized and systematic way – e.g. through the application of new innovation metrics – they could raise their return on innovation at no or small additional costs.

Historically, organizations have always measured performance– primarily to reduce process costs and improve business effectiveness. Several performance measurement systems are in use today, at which the balanced scorecard (BSC) is one of the most widely applied approaches that takes a holistic view of an organization.

Nowadays, numerous companies employ the BSC or similar tools to control and measure their internal innovation activities. However, only few recognize the need to adapt their measurement tools to the new concepts and challenges of open innovation. Given that open innovation involves innovating with others, makes it for example nearly indispensable to have a certain degree of transparency about the capabilities and characteristics of your innovation partners. The heterogeneity of a network, incentive systems or the design of tools and platforms for cooperation – to name just a few basic success factors – becomes more critical for successful innovation in an open innovation environment.

For this reason appropriate metrics need to be developed that help to quantify these new critical success factors and allow an appropriate evaluation of progress, success as well as strengths, weaknesses or even possible reasons for failure of your open innovation initiatives.

Among those companies that traditionally do measure innovation, we found out that most of them still use very generic innovation metrics that are primarily based on R&D and product-development metrics solely (i.e. number of patent filed in the past year or the number of ideas submitted by employees). Though somewhat useful, these metrics provide only little support for organizations on their innovation journey, since they do not map performance measures that instantly drive, impact or completely indicate a company’s (open) innovation performance.

Some companies readily admit their shortcomings in open innovation measurement, but none of them seem to be willing to substantially work on that vision in order to remedy this weakness.

“Poor measurement practices often result in avoidable project extension or in far too early cancellations with wasted resources and a lower return on innovation investments”

In line with the quote “you cannot manage what you cannot measure” it is not surprising, that many companies still fail on open innovation and that most of them are disappointed in their return on innovation spending – so do poor measurement practices often result in avoidable project extensions or in far too early cancellations with wasted resources and a lower return on innovation investments.

But what is the reason that innovation departments still don’t have access to the right tools and metrics that enable them to successfully control and measure their open innovation projects?

In our experience it’s not that the commitment to new innovation measurement approaches is missing. What seems to be a real challenge for companies is finding the relevant metrics for their open innovation activities and the discipline making measurement a priority in innovation management as part of a standardized process. Thus, appropriate tools and metrics are needed that empower innovation teams to properly measure open innovation in order to be able to promote the best innovation ideas and solutions and in fact to turn new knowledge into successful commercialized products or services. If our clients could raise their return on innovation with just 10-20 % through controlled and measured open innovation practices this would give them a significant competitive advantage and the potential to be true game-changers.

Framework for an Open Innovation Performance Measurement System

With our project experiences in performance measurement and the findings of desk-research we singled out three quite distinct principles that companies must consider in order to successfully implement a metrics-based performance measurement system for their open innovation projects.

A simple framework (Figure 1) is outlined, which combines our three principles on OI metrics. It provides the perspective for a suite of KPIs and provides a better idea of how to properly set up a performance measurement system that will help you to assess, control and measure your open innovation activities.

Principle 1: Use unique metrics for each open innovation method

In order to pilot open innovation at project level, you first have to select a specific OI method that suits your desired project goals. Some of the instruments are designed for the active integration of need information that usually occurs primarily in ideas or creative thoughts from external partners at the earlier stages (upstream to ideation) of an innovation. Other instruments focus on solutions provided by innovative outsiders, answering an open call for cooperation and can be used at the later stages (downstream) of the innovation process (Diener & Piller 2013).

“Method specific metrics or KPIs are needed in order to be able to properly assess and measure the progress and success of each of these activities”

This implies that measuring open innovation highly depends on your desired innovation goals and the underlying open innovation method with its fundamental features, characteristics and resources that you are going to use in your OI project. In other words, method-specific metrics or KPIs are needed in order to be able to properly assess and measure the progress and success of each of these activities.

For this purpose we deep dived into the three most prominent methods of open innovation, which cover both the various early as well as the later stages of the innovation process.

  • The lead user method identifies innovative users who are at the leading edge of important trends and benefit greatly from obtaining a solution to their needs. Thus they are motivated to discuss and tackle their innovation needs and ideas in innovation workshops.
  • In an ideation contest, a firm seeking innovation-related information posts a task-specific challenge to a population of independent, competing agents (e.g. customer, suppliers, etc.) who then submit ideas within a given timeframe. The firm awards the participants that generated the best solutions.
  • Broadcast Search involves contests that seek technical solutions rather than just ideas. Online broker companies, so-called intermediaries, such as InnoCentive or Nine Sigma provide firms access to a global pool of scientists, engineers and other professionals to help them solve primarily R&D problems they have been unable to solve through internal methods. The companies submit a problem, with a stipulated time frame and cash prize for the winning solution, and then, with the help of the intermediary, define the problem and develop criteria for picking a solution.

It is quite obvious that measuring the innovation success of a lead user project requires a different set of KPIs than those required for broadcast search. Whereas the focus of a lead user project lies primarily on evaluating the identified new needs and trends provided by innovative users, measuring the success of broadcast search requires metrics that map the potential performance of a technical solution.

Principle 2: Consider different types of measures: input, process, output and outcome (IPOO)

The second principle concerns the different types of measures that need to be tracked by a holistic performance measurement system. The framework should be designed to link the outputs or outcomes of an open innovation initiative to the inputs.

  • Input KPIs measure the input elements within a project, such as human or financial resources.
  • Process KPIs are used to transform inputs into outputs and to improve the efficiency of the innovation process: time variances, budget variances, error ratio, etc.
  • Output KPIs measure the results of the development activities within an innovation process: number of ideas, number of patents, number of publications, etc.
  • Outcome KPIs aim to determine the value of an innovation in terms of economic and market-oriented performance indicators.

“Only the combination of both input and output (outcome) metrics can provide a meaningful understanding of the cause-effect relationships of your project”

Only the combination of both input and output (outcome) metrics can provide a meaningful understanding of the cause-effect relationships of your project.

For this reason, you better constitute a frame that allows return on investment considerations, i.e. relating the input to the output (outcome) of a broadcast search project, with significant measures for efficiency.

Moreover, since the real value of the output (outcome) of an open innovation initiative is the result of more than just the resources invested (input), various measures of the processing or transformation procedures should be also integrated into the framework.

Principle 3: Think about how to effectively utilize your open innovation metrics

The mere provision of a performance measurement system through the collection of appropriate management information, per-se is no guarantee for successful innovations. The collected KPIs must be initiated by the responsible actors within your company.

Pelz (1978) proposes that metrics can be utilized on three different levels: instrumental, conceptual and symbolic.

  • Instrumental use refers to the application of information/metrics used directly for decision making. For instance, when the open innovation project is cancelled because the metric “expected sales” is below a specific threshold, the metric was used instrumentally.
  • A more indirect use is the conceptual one. The use of the information/metric does not directly lead to a concrete action, but rather provides general enlightenment and understanding. For example, when a manager recognizes that the lead time of open innovation projects is on average 30 % lower than for conventionally-run innovation projects, he is using the metric “lead time” conceptually.
  • Metrics can also be used after decisions have already been taken to legitimize and justify them. This kind of use is called symbolic. In case an open innovation project is cancelled due to cost overruns, the official reason for its termination is “quality of ideas” – this metric is used symbolically.

“The way how metrics should be utilized highly depend on your desired project goals”

The way how metrics should be utilized highly depends on your desired project goals. For instance, if you are following rather long-term goals than short-term success with your open innovation project, i.e. to facilitate a sustainable innovation culture, hard measures such as “expected sales” should be used conceptually for providing general enlightenment and understanding, and less for decision making purposes.

Outlook

So far, a simple and easy to apply framework for open innovation performance measurement has been outlined. However, while we proposed a first approach of how we should look at open innovation metrics, there is still no answer on what we actually should measure. What are the relevant key performance indicators behind that framework?

This question was the focus on our Open Innovation KPI 2012 study, in which we identified the most relevant key performance indicators from the perspective of innovation managers and performance measurement consultants.

In our next article we will discuss the key results of this study. Furthermore, based on our given framework and study results,a metric-based management toolkit will be presented that provides the most relevant key performance indicators for a specific set of OI methods.

Please refer to Measuring Open Innovation – a Metrics-Based Management Toolkit for Successful Innovation Teams, for Part 2

About the authors

Marc Erkens is an innovation and strategy consultant within the Performance Improvement department at Ernst & Young’s Advisory. Passionate about innovation, Marc supports large German companies and multinationals to improve their capabilities in innovation management. There, he specializes on open innovation controlling and the selection and integration of appropriate open innovation tools to stimulate companies’ innovation success with the focus on new business models. Prior to joining Ernst & Young, Marc was a student research assistant at the Technology and Innovation Management Group at RWTH Aachen University, where he received a master’s degree in economics.

Susanne Wosch is a Senior Manager within the Performance Improvement department at Ernst &Young’s Advisory focusing on innovation and strategy consulting services. Based on her scientific background and more than ten years of professional experience as part of Ernst& Young´s European Life Sciences Center she has a strong preference to serve clients of the life science industry (pharma, biotech, medtech). Susanne is in charge for Ernst & Young’s innovation management services within the life science industry sector. There, she focuses on Business Model Innovation, Open Innovation and the evaluation of innovation activities. Before joining Ernst & Young, Susanne studied biochemistry and obtained a PhD in neurology.

Dirk Lüttgens is an assistant professor at the Technology & Innovation Management Group at RWTH Aachen University. He also is a consulting team member at Competivation where he is responsible for the scientific link between Competivation and RWTH TIM. He has over ten years experience as a project manager in technology and innovation management and received his PhD on innovation networks. At the Chair of Technology and Innovation Management, he heads the areas of open innovation for technical problem solving, service innovation, and innovation process design. As a consultant, Dirk supports particular companies in the machinery and plant engineering in the implementation of innovation processes, the selection of appropriate innovation tools and the design of an innovative corporate culture.


Frank Piller is a chair professor of management and the director of the Technology & Innovation Management Group at RWTH Aachen University. He also is a founding faculty member and the co-director of the MIT Smart Customization Group at the Massachusetts Institute of Technology, USA. Frequently quoted in The New York Times, The Economist, and Business Week, amongst others, Frank is regarded as one of the leading experts on mass customization, personalization, and open innovation. Frank’s recent research focuses on innovation interfaces: How can organizations increase innovation success by designing and managing better interfaces within their organization and with external actors.

Photo: Business collage pen, ruler and graph from shutterstock.com

References

Cooper, R. G. (2002) Top oder Flop in der Produktentwicklung. Erfolgsstrategien. Von der Idee zum Launch

Diener K. & Piller F. (2013) The Market for Open Innovation. A market study of intermediaries, brokers, platforms and facilitators helping organizations to profit from open innovation and customer co-creation, available at www.study.open-innovation.com

Habicht, H. &Möslein, K. M. (2011) Open Innovation Maturity. Ein Reifegradkonzept zum Controlling von Open Innovation. Controlling- Zeitschrift Unternehmenssteuerung.

Hagenhoff, S. (2008) Innovationsmanagement für Kooperationen. Eine instrumentenorientierte Betrachtung. In NiedersächsischeStaats- und Universitätsbibliothek.

Pelz, D. C. (1978) Some Expanded Perspectives on Use of Social Science in Public Policy. In: Yinger, M. & Cutler, S. (1978) Major Social Issues: A Multidisciplinary View.

Reichwald, R. & Piller, F. (2005). Open Innovation: Kunden als Partner im Innovationsprozess.