Measuring Innovation part 1: Frequently Used Indicators

By: Leif Denti

One of the most common questions people ask me is how I measure innovation when conducting my research. The question echoes an underlying concern about how innovation can be captured and adequately measured. In this article I delineate the most frequently used innovation indicators, their strengths, and their drawbacks.

This article is the first part of a two part series on how to measure innovation in organizations. But let’s make it clear from the beginning. It is not easy to measure innovation since virtually all indicators risk missing some aspects of the process. For instance, patents or patent applications indicate some kind of technological progress but patents are seldom transformed into tangible products. In fact it looks bad for patents. 90 to 95 percent of all patents lack any market relevance and 99 percent fail to bring any profit to the firm (Stevens & Burley, 1997). Still, patenting is often necessary for many firms to keep their competitive advantage so clearly patents indicate something about the innovativeness of a firm. What about financial figures such as revenue? These risk being too broad since many other influences affect revenue besides innovation, including luck.

Below I summarize the most frequently used measures of innovation including their main advantages and drawbacks. For better clarity I divide them into three categories. Product/technology measures, financial measures, and subjective measures.

Product/technology measures

These indicators measure aspects of the innovation process which have the distal goal of transforming new ideas and technology into tangible products (goods or services).

Measure	Example study	Main advantage	Main drawback
New products or product improvements	Elenkov & Manev (2009)	Measures actual implementation	All products are not guaranteed to succeed
Patents or patent applications	Jung et al. (2008)	Measures technological progress	Patents are seldom realized
Patent citations	Makri & Scandura (2010)	Measures importance of patents	Patents may be self-cited
Invention disclosures or suggestions	Axtell et al. (2000)	Measures rate of idea generation	Ideas are seldom realized
Process innovations	West et al. (2003)	Measures improvements in processes and methods	Too much focus on processes is the ‘innovators dilemma’

Financial/market measures

These indicators measure aspects of the financial performance of an organization. Mainly in relationship to R&D spending and sales of new products (goods or services).

Measure	Example study	Main advantage	Main drawback
Ratio of sales of new products to total sales	Czarnitzki & Kraft (2004)	Indicator of success on market	Very broad, other factors confounds the measure
Ratio of sales of new products to R&D expenditures	Gumusluoglu & Ilsev (2009)	Indicator of R&D efficiency	Difficult to establish a valid baseline
Total R&D spending	García-Morales et al. (2008)	Easy to obtain	Does not indicate innovation efficiency
Number of employees in R&D	García-Morales et al. (2008)	Easy to obtain	Does not indicate innovation efficiency
New markets entered	Elenkov & Manev (2009)	Indicator of radical innovation	Roughly 60 percent of new products succeed

Subjective measures

Although the indicators above are among the most common when measuring innovation in firms, many of the activities that can be characterized as innovative risk being overlooked if innovation is measured solely using the broad searchlight of these quantitative measures. Activities like these can be labeled ‘dark innovation’ (Martin, 2012).  Examples include activities that are incremental (e.g., improvements in quality), involve little formal R&D (e.g. the concept of ‘skunk work’), and is seldom patented. One method to capture ‘dark innovation’ is to use subjective assessments.

Measure	Example study	Main advantage	Main drawback
Innovative work behavior (IWB)	De Jong & Den Hartog (2010)	Flexible, can measure any innovation activity	IWB’s does not unequivocally lead to tangible outcomes
Team innovativeness	Hurley & Hult (1998)	Flexible, can measure any innovation activity	Low correlations with number of implemented innovations
Organizational innovation	Chen et al. (2006)	Holistic assessment of the organization	Difficult to establish a valid baseline

Selection of measure is highly specific to the organization and depends on a number of factors. The relevance of indicators varies with technological domain and also depends on the specific product offering of the firm. Moreover, R&D functions of organizations may differ in degree of formalization which affects the degree to which ‘dark innovation’ takes place.

In my research which primarily concerns the innovation of individuals and teams in industrial (mainly automotive) R&D environments we have chosen a combination of quantitative indicators of innovation and subjective assessments of individuals’ innovative behaviors. This method covers outcomes of innovation as well as ‘dark innovation’ activities. In my next article we will look more closely at the elusive nature of ‘dark innovation’, as well as subjective assessments of innovation including what they measure and how they correlate with tangible outcomes.

By Leif Denti

 

References

Axtell, C. M., Holman, D. J., Unsworth, K. L., Wall, T. D., & Waterson, P. E. (2000). Shopfloor innovation: Facilitating the suggestion and implementation of ideas. Journal of Occupational and Organizational Psychology, 73, 265-285.

Chen, G., Tjosvold, D., & Liu, C. (2006). Cooperative goals, leader people and productivity values: Their contribution to top management teams in China. Journal of Management Studies, 43, 1177-1200.

Czarnitzki, D., & Kraft, K. (2004). Firm leadership and innovative performance: Evidence from seven EU countries. Small Business Economics, 22, 325–332.

De Jong, J., & den Hartog, D. (2010). Measuring innovative work behavior. Creativity and Innovation Management, 19, 23-36.

Elenkov, D. S., & Manev, I. M. (2009). Senior expatriates leadership’s effects on innovation and the role of cultural intelligence. Journal of World Business, 44, 357–369.

García-Morales, V. J., Mathías-Reche, F., & Hurtado-Torres, N. (2008). Influence of transformational leadership on organizational innovation and performance depending on the level of organizational learning in the pharmaceutical sector. Journal of Organizational Change Management, 21, 188–212.

Gumusluoglu, L., & Ilsev, A. (2009). Transformational leadership and organizational innovation: The roles of internal and external support for innovation. Journal of Product Innovation Management, 26, 264–277.

Hurley, R. F., & Hult, T. M. (1998). Innovation, market orientation, and organizational learning: An integration and empirical examination. Journal of Marketing, 62, 42-54.

Jung, D., Wu, A., & Chow, C. W. (2008). Towards understanding the direct and indirect effects of CEO’s transformational leadership on firm innovation. The Leadership Quarterly, 19, 582–594.

Makri, M., & Scandura, T. A. (2010). Exploring the effects of creative CEO leadership on innovation in high-technology firms. The Leadership Quarterly, 21, 75–88.

Martin, B. R. (2012). Innovation studies: Challenging the boundaries. Sussex: Science and Technology Policy Research, University of Sussex

Scott, S. G., & Bruce, R. A. (1994). Determinants of innovative behavior: A path model of individual innovation in the workplace. Academy of Management, 37, 580–607.

Stevens, G. A., & Burley, J. (1997). 3,000 raw ideas = 1 commercial success! Research-Technology Management, May-June, 16-27.

West, M. A., Borril, C. S., Dawson, J. F., Brodbeck, F., Shapiro, D. A., & Haward, B. (2003). Leadership clarity and team innovation in health care. The Leadership Quarterly, 14, 393–410.