The Relevance of Prediction Markets for Corporate Forecasting

The Relevance of Prediction Markets for

Corporate Forecasting

Thomas Wolfram

Submitted for the Degree of Doctor of Business Administration

Heriot-Watt University Edinburgh Business School

June 2021 The copyright in this thesis is owned by the author. Any quotation from the thesis or use of any of the information contained in it must acknowledge this thesis as the source of the quotation or information

i

Abstract

Prediction markets (PMs) are virtual stock markets on which shares are traded taking

advantage of the wisdom-of-crowds principle to access collective intelligence. It is

claimed that the accumulation of information by groups leads to joint group decisions

often better than individual participants’ approaches to solutions. A PM share represents

a future event or a market condition (e.g. expected sales figures of a product for a specific

month) and provides forecasts via its price which is interpreted as the probability of the

event occurring. PMs can be used in competition with other forecasting tools; when

applied for forecasting purposes within a company they are called corporate prediction

markets (CPMs). Despite great praise in the (academic) literature for the use of PMs as

an efficient instrument for bringing together scattered information and opinions,

corporate usage and applications are limited.

This research was directed towards an examination of this discrepancy by means of

focusing on the barriers to adoption within enterprises. Literature and reality diverged

and neglected the important aspect of corporate culture. Screening existing research and

interviews with business executives and corporate planners revealed challenges of

company hierarchy as an inhibitor to the acceptance of CPM outcomes.

Findings from 55 interviews and a thematic analysis of the literature exposed that CPMs

are useful but rarely used. Their lack of use arises from senior executives’ perception of

the organisational hierarchy being taxed and fear of losing power as CPMs (can) include

lower rungs of the corporate ladder in decision-making processes. If these challenges can

be overcome the potential of CPMs can be released. It emerged – buttressed by ten

additional interviews – that CPMs would be worthwhile for company forecasting,

particularly supporting innovation management which would allow idea markets (as an

embodiment of CPMs) to excel.

A contribution of this research lies in its additions to the PM literature, explaining the

lack of adoption of CPMs despite their apparent benefits and making a case for the

incorporation of CPMs as a forecasting instrument to facilitate innovation management.

Furthermore, a framework to understand decision-making in the adoption of strategic

tools is provided. This framework permits tools to be accepted on a more rational base

and curb the emotional and political influences which can act against the adoption of good

and effective tools.

ii

Dedication

The 18th century German writer and statesman Johann Wolfgang von Goethe supposedly

said “Erfolg hat drei Buchstaben: Tun” which roughly translates into ‘success is based

on doing or action’. This was impressed upon me lovingly by my spouse and never ceased

to impress and guide me whenever I needed patience and perseverance during the long-

winded path writing this thesis. Without my wife’s continuing support and

encouragement this ‘oeuvre’ would not have come to pass, and it is thus dedicated to her.

Encouragement also came from my sister who regularly lent a helping hand in improving

my haphazard English when on occasion I swayed or rambled (the German colloquial

term would be Schwurbeln) in my expression, diction or style rather than coming

succinctly to the point.

Also, my stepson took time out of his busy academic schedule to do likewise and pointed

out the necessary academic rigour when he felt I needed prodding in that direction – my

supervisor surely also needs to receive kudos for his effort to equally point out

unsubstantiated facts when they occurred.

There will be others who ‘share the blame’ of my having finished this work (a good friend

who provided me with his apartment and copious espressos to support work and

inspiration, my late godfather, the managing director of an idea & innovation software

company, and a friend of my sister’s amongst them), but not all will obviously come to

mind, I am afraid.

To all a heartfelt Thank-You!

iii

Declaration Statement

Research Thesis Submission Please note this form should be bound into the submitted thesis.

Name: Thomas Wolfram

School: Edinburgh Business School

Version: (i.e. First, Resubmission, Final)

Final Degree Sought: DBA

Declaration In accordance with the appropriate regulations I hereby submit my thesis and I declare that: 1. The thesis embodies the results of my own work and has been composed by myself 2. Where appropriate, I have made acknowledgement of the work of others 3. The thesis is the correct version for submission and is the same version as any electronic versions submitted*. 4. My thesis for the award referred to, deposited in the Heriot-Watt University Library, should be made available for

loan or photocopying and be available via the Institutional Repository, subject to such conditions as the Librarian may require

5. I understand that as a student of the University I am required to abide by the Regulations of the University and to conform to its discipline.

6. I confirm that the thesis has been verified against plagiarism via an approved plagiarism detection application e.g. Turnitin.

ONLY for submissions including published works Please note you are only required to complete the Inclusion of Published Works Form (page 2) if your thesis contains published works) 7. Where the thesis contains published outputs under Regulation 6 (9.1.2) or Regulation 43 (9) these are accompanied

by a critical review which accurately describes my contribution to the research and, for multi-author outputs, a signed declaration indicating the contribution of each author (complete)

8. Inclusion of published outputs under Regulation 6 (9.1.2) or Regulation 43 (9) shall not constitute plagiarism. * Please note that it is the responsibility of the candidate to ensure that the correct version of the thesis is submitted.

Signature of Candidate:

Date: 15.06.2021

Submission

Submitted By (name in capitals): THOMAS WOLFRAM

Signature of Individual Submitting:

Date Submitted:

15.06.2021

For Completion in the Student Service Centre (SSC)

Limited Access Requested Yes No Approved Yes No E-thesis Submitted (mandatory for final theses)

Received in the SSC by (name in capitals): Date:

iv

Table of Contents

Abstract ............................................................................................................................. i

Dedication ........................................................................................................................ ii

Declaration Statement ................................................................................................... iii

Table of Contents ........................................................................................................... iv

List of Abbreviations .................................................................................................... viii

Glossary of Terms .......................................................................................................... ix

List of Tables .................................................................................................................. xi

List of Figures ............................................................................................................... xiv

1. Introduction .............................................................................................................. 1

1.1 Background and Research Rationale .................................................................. 1

1.2 Research Focus ................................................................................................... 7 1.2.1 Research Question, Aims and Objectives ............................................................................ 8 1.2.2 Thesis Composition – Research Design ............................................................................ 13 1.2.3 Research Contribution ....................................................................................................... 14

1.3 Thesis Structure ................................................................................................ 15

2. Literature Review .................................................................................................. 16

2.1 Prelude .............................................................................................................. 16 2.1.1 Collective Intelligence ....................................................................................................... 17 2.1.2 Forecasting and PMs .......................................................................................................... 19

2.2 Why Forecast and Approaches to Forecasting ................................................. 20 2.2.1 Relevance of Forecasting and Relevance of PMs – A Decision-Making Keystone .......... 20 2.2.2 ‘Machine Prediction’ ......................................................................................................... 23 2.2.3 Managing for an ‘Unpredictable’ Future ........................................................................... 25 2.2.4 Selection of a Forecasting Approach ................................................................................. 27 2.2.5 Oracle or Evidence? Predictive Planning as a Success Factor .......................................... 29 2.2.6 Prediction: Data or Market? ............................................................................................... 30

2.3 Principles of PMs ............................................................................................. 32 2.3.1 Theory of PMs Based on Three Pillars .............................................................................. 32 2.3.2 Pertinence of PMs in the Context of Strategic Planning ................................................... 33

2.4 Traits and Attributes of PMs ............................................................................ 34 2.4.1 Advantages and Disadvantages of PMs ............................................................................. 34 2.4.2 The Three Pillars – Revisited ............................................................................................. 36 2.4.3 Alternative Approaches – Polls, Scenarios, and Non-Predictive Action ........................... 42 2.4.4 How Do Benefits and Drawbacks Balance? ...................................................................... 43 2.4.5 Implementation at Scale and Proper Presentation to Management ................................... 45

2.5 Business Use of PMs ........................................................................................ 46

v

2.6 Impact of Company Hierarchy and Management Acceptance ......................... 51

3. Literature Synthesis ............................................................................................... 54

3.1 PMs and Their Potential ................................................................................... 54

3.2 CPM Adoption .................................................................................................. 56

3.3 Obstacles and Hindrances to CPM Adoption ................................................... 58

3.4 Summarising the Literature and Linking It to the Objectives .......................... 60 3.4.1 Research Objectives ........................................................................................................... 60 3.4.2 PMs’ Low Use at Corporations – A Potential Misnomer .................................................. 61

3.5 Gaps in the Literature Regarding PMs in Corporate Forecasting .................... 63

4. Theoretical Conceptualisation .............................................................................. 65

4.1 Theoretical Framework and Research Questions ............................................. 66

4.2 Framing a Concept for CPM Adoption ............................................................ 68 4.2.1 Benefits Promoting CPM Usage – Method Effectiveness and Theoretical Underpinning 69 4.2.2 Barriers to Corporate Adoption – CPM Use and Organisational Aspects / Challenges .... 69 4.2.3 Moderation of the Challenges to Adoption – Alternative PM Use Cases ......................... 70 4.2.4 CPM Adoption – Innovation Management as a Use Case but also Method Combination 70

4.3 An additional Slant of the Theoretical Framework – a further Contribution ... 71

5. Research Methodology .......................................................................................... 73

5.1 Approach and Strategy – Philosophical Position and Research Design ........... 73

5.2 Sampling, Research Setting and Participants ................................................... 79 5.2.1 Inclusion of Secondary Data – Pre-Study Interviews ........................................................ 80 5.2.2 Research Participants ......................................................................................................... 81 5.2.3 Sampling Technique / Approach ....................................................................................... 83 5.2.4 Research Setting – Interview Guide .................................................................................. 84

5.3 Analysis Plan .................................................................................................... 85

5.4 Pilot Study – Interviews ................................................................................... 87 5.4.1 Interview Approach ........................................................................................................... 87 5.4.2 Analysis Approach to Pilot Interviews .............................................................................. 88 5.4.3 First Results Based on Pilot Interviews ............................................................................. 92 5.4.4 Pilot Study Reflection ........................................................................................................ 93

5.5 Main Study – Interviews .................................................................................. 94

5.6 Potential Methodological Constraints .............................................................. 96 5.6.1 Validity .............................................................................................................................. 96 5.6.2 Bias in Qualitative Research .............................................................................................. 97 5.6.3 Generalisability .................................................................................................................. 97 5.6.4 Reliability ........................................................................................................................... 98 5.6.5 Reflections Regarding the Validity and Reliability of the Pilot ........................................ 98

vi

5.7 Ethics ................................................................................................................ 98

6. Results / Findings ................................................................................................. 100

6.1 Data Collection via Interviews ....................................................................... 100

6.2 Interview Content, Illustration, Analysis, and Results ................................... 101 6.2.1 Transcription, Analysis and Content Portrayal – Thematic Aspects ............................... 101 6.2.2 Representation of Interview Content – Pilot Study – Thirteen Interviews ...................... 102 6.2.3 Summary of Pilot Interviews: Key Interview Points and Core Aspects .......................... 107

6.3 Overall Interview Summary and Interpretation .............................................. 108 6.3.1 Interview Topics – Key Thematic Aspects ...................................................................... 108 6.3.2 Topics Related to the Literature Review ......................................................................... 112 6.3.3 Topic ‘Accuracy’ – Analysis and Findings from the IARPA Forecasting Competition . 113 6.3.4 Topics ‘Hierarchy’ and ‘Management Acceptance’

and Merging Literature and Interview Results ................................................................ 123

6.4 Analysis of CPM Usage ................................................................................. 125 6.4.1 Modelling of CPM Adoption ........................................................................................... 125 6.4.2 Analysis of Actual CPM Applications ............................................................................. 127

6.5 Significance of CPMs and a Possible New Field of Application ................... 130

6.6 Additional Interviews – Focus on Innovation Management .......................... 133 6.6.1 Content Elicitation ........................................................................................................... 134 6.6.2 ‘Innovation Interviews’ – Summary ................................................................................ 138

6.7 CPMs’ Chances in a Nutshell ......................................................................... 141

7. Discussion .............................................................................................................. 142

7.1 Literature Reappraisal .................................................................................... 143 7.1.1 The Journal of Prediction Markets (JPM) ........................................................................ 143 7.1.2 Literature on Alternatives to PMs .................................................................................... 146

7.2 Result Appraisal – Findings in the Context of the Research Questions ......... 147

7.3 Limitations of the Research ............................................................................ 152

7.4 Research Impact ............................................................................................. 154

8. Conclusion ............................................................................................................. 155

8.1 Epilogue .......................................................................................................... 155

8.2 Recommendations for further Research ......................................................... 160 8.2.1 Recommendations to Academia ...................................................................................... 161 8.2.2 Recommendations to Practitioners .................................................................................. 163 8.2.3 Integrating Academic and Practice Research Recommendations .................................... 165

9. References ............................................................................................................. 167

Appendix A: IARPA Data and Analysis ................................................................... 217

Appendix B: Feedback on the Researcher’s IARPA Data Analysis ....................... 220

vii

Appendix C: Interviewee Information Letter .......................................................... 221

Appendix D: Interview Schedule – Pilot, Main Set, Extended Interviews ............ 223

viii

List of Abbreviations

Abbreviation Meaning CPM Corporate Prediction Market IARPA JPM PM

Intelligence Advanced Research Projects Activity Journal of Prediction Markets Prediction Market

ix

Glossary of Terms

Throughout the thesis, newly introduced or specialised terms are highlighted in bold as they first appear. Black Swan Event: defined as a rare and unexpected event with severe consequences Box Plot: or Box-Whiskers-Plot – a graphical representation of statistical values, “a standardized way of displaying a dataset based on a five-number summary: the minimum, the maximum, the sample median, and the first and third quartiles [or 25th and 75th percentiles]” (‘Box plot’, 2020), sometimes the mean is added to the representation as well. A box plot always consists of a rectangle, called a box, and two lines that extend this rectangle. These lines are called whiskers and are terminated by a dash. Outliers may be plotted as individual points below or above the whiskers. The box corresponds to the area where the middle 50% of the data is located. It is bounded by the upper and lower quartiles and the length of the box corresponds to the interquartile distance. The latter is a measure of the dispersion of the data and is determined by the difference between the upper and lower quartiles. The median is drawn as a continuous line in the box (if the mean is present it would be represented by an asterisk). This divides the entire diagram into two halves, in which 50% of the upper and lower values lie. Its position within the box gives an impression of the skewness of the underlying distribution. The whiskers usually extend 1.5 times the interquartile distance (the box size) Corporate Prediction Market (CPM): a (corporate) forecasting instrument or tool based on the concept of Prediction Markets, i.e. a company internal prediction market Crowdsourcing: defined as outsourcing tasks or decision-making to an extrinsically or intrinsically motivated group using modern information and communication systems, particularly Web 2.0 tools The Delphi method: “a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts … Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach, prediction markets. Delphi is based on the principle that forecasts (or decisions) from a structured group of individuals are more accurate than those from unstructured groups. The experts answer questionnaires in two or more rounds. After each round, a facilitator … provides an anonymised summary of the experts' forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel. It is believed that during this process the range of the answers will decrease and the group will converge towards the ‘correct’ answer” (‘Delphi method’, 2019) Enterprise 2.0 tools: a collection of web-based social technologies initially made popular by end users and now used in companies as well like Wikis, Blogs, Podcasts, etc. but also Prediction Markets Favourite-longshot bias: occurs when outcomes deemed more likely are under-priced and ‘long shots’ overvalued

x

Idea Markets: a specialised form of prediction markets using “virtual stocks to represent new product ideas … [Partakers] trade those stocks in a virtual marketplace” (Soukhoroukova, Spann and Skiera, 2012, p.110). The resulting share prices indicate the likelihood of success of various new products Ideation: also called idea creation. This stage in an innovation process, typically in a workshop setting, focuses on the generation and formation of ideas or concepts (Geissdoerfer, Bocken and Hultink, 2016) which are then filtered and evaluated into a sub-set of ideas to subsequently build prototypes to get innovative solutions into the hands of users Iowa Electronic Markets (IEM): “a limited, Internet-based market on which traders, often students, buy and sell contracts whose payoffs depend on election outcomes and interest rates“ (Hall, 2010, p.29) Net-promoter Score (NPS): The NPS principle works by posing just one question to a customer: ‘How likely is it that you would recommend the company / the product to a friend or colleague?’. Answers can range from 0 (unlikely) to 10 (highly likely). The score is based on ten meaning ‘extremely likely’ to recommend, five denotes neutral, and zero equals to ‘not at all likely’. In that way three clusters emerge. ‘Promoters’ rate the question with nine or ten; the ‘passively satisfied’ log a seven or an eight, and ‘detractors’ score from zero to six (Reichheld, 2003). The actual NPS number is computed by subtracting the percentage of detractors from the percentage of promoters Prediction Market (PM): “a group decision-making tool that uses a market mechanism to rapidly aggregate information held by large, diverse groups of participants” (Buckley, 2016, p.85), comparable in its working principles to a stock-market Superforecaster: “a person who makes forecasts that prove to be consistently higher than those achieved by others. Superforecasters use modern analytical and statistical methodologies [to] augment the art of forecasting events” (‘Superforecaster’, 2020), this claims for them to possess higher accuracy than experts not using such techniques The term Web 2.0 describes a change in the use of the Internet from a socio-technical perspective. The focus is no longer on the mere dissemination of information or products by website operators, but on the participation of users in the Web and the generation of additional benefits. The expression Web 2.0 is still current and used by McKinsey, a global consulting firm, when referring to tools like Prediction Markets or Big Data (Bughin and Chui, 2010; Chui et al., 2012; Bughin, 2015)

xi

List of Tables

Table 2-1 Managing for an Unpredictable Future – Forecasting Approaches Compared (Developed for Research from Ord, Fildes and Kourentzes, 2017; Schoemaker and Tetlock, 2016; Armstrong, 2001b) ......................................................................................................... 26

Table 2-2 PMs’ Strengths and Weaknesses (Developed for Research from Kloker et al., 2019) ..................................................................................... 43

Table 2-3 Balancing Arguments Pro and Con of PMs (Developed for Research) .................................................................................................... 44

Table 2-4 Diffusion of Social Technologies within Companies – Increase in Value-add in Percent per Level of Penetration (‘2007–15 McKinsey survey of 1,500 companies’; Bughin, 2015, researcher’s emphasis) ............................................................................... 49

Table 3-1 Obstacles to Using Social Technologies (Developed for Research) ............. 59

Table 3-2 Literature Synthesis and Research Objectives (Developed for Research) .................................................................................................... 61

Table 5-1 Adopted Research (Developed for Research) ................................................ 78

Table 5-2 Chosen Expertise Levels in Interview Partners for the Anticipated Interview Domains (Developed for Research) ........................................... 79

Table 5-3 Pilot Interview Partner’s Experience and Qualifications (Developed for Research) ........................................................................... 82

Table 5-4 Interview / Qualitative Data Collection Phases (Developed for Research) .................................................................................................... 82

Table 5-5 Interview Questions’ Fit to the Research Objectives (Developed for Research) .............................................................................................. 85

Table 5-6 Interview Partners while Piloting (Developed for Research) ........................ 88

Table 5-7 Example 1: Interview Content and Derived Conceptual Themes – Coded Transcript Sample from Enterprise Interviewee (Developed for Research) ........................................................................... 90

Table 5-8 Example 2: Interview Content and Derived Conceptual Themes – Coded Transcript Sample from Interview with Economic Professor (Developed for Research) .......................................................... 90

Table 5-9 Example 3: Interview Content and Derived Conceptual Themes – Coded Transcript Sample from Enterprise Interviewee (Developed for Research) ........................................................................... 91

xii

Table 5-10 Summary of Thematic Aspects from Three Interview Transcripts (Developed for Research) ........................................................................... 91

Table 5-11 Key Thematic Aspects from Pilot Interviews (Developed for Research) .................................................................................................... 92

Table 6-1 Interviews from Data Collection, Pre-Study, and Pilot (Developed for Research) ............................................................................................ 100

Table 6-2 Key Thematic Aspects from Interviews (Developed for Research) ............ 102

Table 6-3 Summary of Key Aspect Occurrence from Interviews (Developed for Research) ............................................................................................ 109

Table 6-4 Negative and Positive Key Aspect Occurrence in Percentages across Interview Domains and in Total (Developed for Research) .................................................................................................. 110

Table 6-5 IARPA Data – Categorisation for Analysis (Developed for Research) .................................................................................................. 114

Table 6-6 IARPA Data – Categorisation Examples (Developed for Research from Moore et al., 2017) .......................................................................... 115

Table 6-7 IARPA Traded Stocks per Question Type and Region (Developed for Research from Good Judgment Project, 2016) ................................... 116

Table 6-8 Perspectives from Researching the Literature – Cultural Acceptance in the Literature Review (Developed for Research) ............. 123

Table 6-9 Perspectives from Researching the Literature – Company Hierarchy in the Literature Review (Developed for Research) ................ 124

Table 6-10 Gauging the Size of Selected Prediction Market Vendors (Developed for Research) ......................................................................... 131

Table 6-11 Interview Partners ‘Focus on Innovation Management’ (Developed for Research) ......................................................................... 134

Table 6-12 Major Points from Interviews around PMs and Innovation Management (Developed for Research) ................................................... 139

Table 6-13 Clustering of Key Thematic Aspects (Developed for Research) ............... 140

Table 6-14 Comparing Interviews Focusing on Innovation Management with Interviews from the Main Study (Developed for Research) ............ 140

Table 7-1 The JPM and Its Coverage of PMs in the Context of Innovation (Developed for Research) ......................................................................... 144

Table 7-2 Adoption of PMs in Business – Aspects from Data Collection and Analysis in Line with the Theoretical Framework (Developed for Research) ............................................................................................ 148

xiii

Table 8-1 PMs Associated with ‘Business Management’ in a Forecasting Milieu – Literature Search 2019-2021 (Developed for Research) .................................................................................................. 155

Table D-1 Interview / Qualitative Data Collection Phases .......................................... 223

Table D-2 Interview Schedule Pilot Interviews ........................................................... 223

Table D-3 Interview Schedule Main Set ...................................................................... 224

Table D-4 Interview Schedule Extended Interviews ................................................... 224

xiv

List of Figures

Figure 1.1 Implementation of a PM – Setup of a Stock to Be Traded (Fletcher-Hill, 2019) ..................................................................................... 2

Figure 1.2 Example of a Public PM (Data Source: PredictIt – https://www.predictit.org/markets/detail/6653/What-will-be-the-Electoral-College-margin-in-the-2020-presidential-election) ............... 2

Figure 1.3 Argument Pattern to Arrive at a Possible Recommendation for CPM Uses (Developed by the Researcher from Minto, 2009) .................... 9

Figure 1.4 ‘Making Better Decisions’ – PM Attributes Based on Inkling’s Website, a PM Platform Provider Founded in 2006 (Developed by the Researcher from Inkling, 2015) ....................................................... 13

Figure 1.5 Flowchart of the Overall Research Design Sequence (Developed for Research) .............................................................................................. 14

Figure 1.6 Thesis Structure (Developed for Research) .................................................. 15

Figure 2.1 Literature Review – Conceptual Path (Developed for Research) ................. 16

Figure 2.2 Google Trends – Results from Search Terms ‘Prediction Market’ (blue line) and ‘Big Data’ (in red) in January 2021 (Data Source: Google Trends – https://www.google.com/trends) ....................... 28

Figure 2.3 Usage of CPMs until 2013 (Developed by the Researcher from McKinsey & Company, 2013) ................................................................... 47

Figure 2.4 Corporate Use of Web 2.0 / Enterprise 2.0 Technologies, Percentages – Graph on Top – and Actual Numbers – Second Graph (Developed by the Researcher from McKinsey & Company, 2013) ......................................................................................... 48

Figure 3.1 Possible Path for CPMs – Innovation Management Making Up for the Decline in Use and Interest (Developed for Research) .................. 54

Figure 3.2 A Mix of Old and New – Extent to Which Social Technologies Can Change Organisational Processes, Percentage of Respondents (Developed for Research from Bughin, Byers and Chui, 2011; Razmerita, Kirchner and Nabeth, 2014) ................................. 56

Figure 3.3 Adoption of Web 2.0 Tools (Developed by the Researcher from McKinsey & Company, 2013; Jabeen et al., 2016; Pan et al., 2016; Bughin, Chui et al., 2017; Clement, 2019) ...................................... 57

Figure 3.4 Interest in ‘Prediction Markets’: Google Trends – Results from Search Term ‘Prediction Markets’ in December 2020 (Data Source: Google Trends – https://www.google.com/trends) ....................... 62

xv

Figure 3.5 Academic Interest in PMs and PM Usage by Corporations (Developed by the Researcher from Tziralis and Tatsiopoulos, 2007; Horn, Ohneberg and Ivens, 2014; McKinsey & Company, 2013) ......................................................................................... 62

Figure 4.1 Elements of the Theoretical Framework Supporting the Overarching Research Question (Developed for Research) ...................... 67

Figure 4.2 Theoretical Framework: Inputs and Influences into PM Tool Adoption; Assess and Embed CPMs as a New Forecasting Process (Developed for Research) ............................................................. 68

Figure 5.1 Research Method – Five Steps to Form the Findings (Developed for Research) .............................................................................................. 73

Figure 5.2 Progression of the Research – from Research Objectives to Conclusion (Developed for Research) ....................................................... 77

Figure 5.3 Flowchart of the Research Stages (Developed for Research) ...................... 95

Figure 6.1 Types of Topics Derived from Interviews, Not Showing Topics Less Than Four Percent (Developed for Research) ................................. 111

Figure 6.2 Key Topics Stemming from Interviews versus the Prevalence of Major Topics in the Literature, in Ascending Order by Occurrence in Interviews (Developed for Research) ............................... 112

Figure 6.3 IARPA Trades from All Markets (Developed by the Researcher from Good Judgment Project, 2016) ........................................................ 118

Figure 6.4 Trades from All Markets per Category ‘Type’ (Developed by the Researcher from Good Judgment Project, 2016) ..................................... 118

Figure 6.5 IARPA Market Behaviour per the Three Categories and Long, Medium, and Short Markets (Developed by the Researcher from Good Judgment Project, 2016) ........................................................ 119

Figure 6.6 Comparison between PMs and ‘Surveys’ Run During the IARPA Project (Developed by the Researcher from Good Judgment Project, 2016) ........................................................................................... 120

Figure 6.7 Comparison between PMs and ‘Surveys’ per Question Type; Long, Medium, and Short Trading or ‘Survey’ Periods (Developed by the Researcher from Good Judgment Project, 2016) ......................................................................................................... 121

Figure 6.8 Comparison between PMs and ‘Surveys’ Box Plot per Question Type (Developed by the Researcher from Good Judgment Project, 2016) ........................................................................................... 121

xvi

Figure 6.9 Model of Corporate Adoption of Enterprise 2.0 Technologies in Percentage Terms (Bughin, 2015) ............................................................ 126

Figure 6.10 Decrease in CPM Usage, Actual Usage Calculated from Usage in Percent and Total Number of Participants (Developed by the Researcher from McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui and Harrysson, 2016b; Bughin, Chui et al., 2017) ........................................................................ 127

Figure 6.11 Net Promoter Score – Constant Negative Delta between ‘High’ and ‘Medium to Low’ Usage (Developed by the Researcher from Reichheld, 2003; McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui and Harrysson, 2016b) ....................................................................................................... 129

Figure 6.12 Portion of Employees Using Prediction Markets by Usage Stratum, Excluding ‘Don’t Know’ Answers (Developed by the Researcher from McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui and Harrysson, 2016b) ................... 130

Figure 7.1 Classification of Published PM Articles Based on a Google Scholar Search with the Term ‘Prediction Markets’ and ‘Sort by Relevance’ (Developed for Research) ...................................................... 145

Figure 8.1 Outline to Arrive at a Recommendation for Corporate Prediction Markets, a Path to and Factors for Success of Prediction Markets in Corporations: Support of Innovation Management (Developed by the Researcher) ................................................................ 157

Figure A.1 Trades from All Markets Plus ‘Jitter’ (Developed by the Researcher from Good Judgment Project, 2016) ..................................... 218

Figure A.2 Trades from All Markets per Category ‘Type’ plus ‘Jitter’ (Developed by the Researcher from Good Judgment Project, 2016) ......................................................................................................... 219

Figure A.3 PMs versus ‘Surveys’ Plus ‘Jitter’ (Developed by the Researcher from Good Judgment Project, 2016) ........................................................ 219

1

1. Introduction

1.1 Background and Research Rationale Forecasting is a method used by corporations to plan for an uncertain future to be able to

anticipate future challenges or risks. Planning is undertaken to improve a company’s

decision-making and often results in an adaption of strategy or even a complete change.

The purpose of forecasting is to inform the process of planning future actions. Successful

forecasting can allow for more efficient use of resources, better met customer needs,

reduced costs and enhance an organisation’s competitiveness, while unsuccessful

forecasting can be detrimental to an organisation; improved mechanisms are therefore

regularly looked for (Kauko and Palmroos, 2014). Judgemental forecasting methods such

as the use of experts, surveys, the Delphi method or the use of analogies are one family

of forecasting methods. When choosing a forecasting system, checks and evaluations

need to be in place to ensure that the system does the job for which it was designed, a step

that is often overlooked in corporations (Ord, Fildes and Kourentzes, 2017).

For forecasting to reach optimal results, knowledge of insights generated by one part of

an organisation that could impact variables to be forecast in another part need to be shared

across the company (ibid.). Predictions are a central element of decision-making under

uncertainty, and a mechanism for aggregating information and integrating dispersed

knowledge would positively contribute to such an endeavour.

Prediction markets (PMs) are exactly such a tool. PMs are a form of artificial stock

market that use the ‘wisdom-of-crowds’ (Surowiecki, 2004) to bet on future events. The

wisdom-of-crowd concept champions the idea that for certain problem types the

collective decisions and information aggregation of a large group of individuals often

surpass those of individual experts (Hosseini et al., 2015). In a PM, participants buy and

sell contracts anonymously, and the individual contracts trade between 0% and 100%.

PMs incentivise participants to disclose personal viewpoints as profits can be made by

trading the shares based on the information inherent in personal beliefs (Wolfers and

Zitzewitz, 2006; Van Bruggen et al., 2010; Richard and Vecer, 2021).

A PM thus gathers scattered information to form a share’s price as in a traditional stock

market. The trading price reflects the aggregated beliefs of the traders about the outcome

of the future event, it indicates what the crowd estimates the likelihood of the event to be.

2

Setup occurs with the help of a software platform by placing the topic of inquiry – usually

in the form of a question – on the platform where it will then be available to the

participants or traders in the market; an example can be seen in Figure 1.1.

Figure 1.1 Implementation of a PM – Setup of a Stock to Be Traded (Fletcher-Hill, 2019)

The presentation of an individual PM varies from platform to platform, be it company

internal or public, showing differing user-interfaces; Figure 1.2 shows a public PM run

by PredictIt.

Figure 1.2 Example of a Public PM (Data Source: PredictIt – https://www.predictit.org/markets/detail/6653/What-will-be-the-Electoral-College-margin-in-the-2020-presidential-election)

3

Further introductions to PMs and explanations of the concepts underpinning them are

given by Watkins (2007), Graefe and Armstrong (2011) or Ozimek (2014) which the

researcher would suggest reading when seeking more background, particularly on PMs’

‘mechanics’. This thesis elaborates PM concepts further in the literature review chapter.

Though PMs have repeatedly been hailed as a tool for forecasting (Gruca and Berg, 2007;

Vaughan Williams, Sung and Johnson, 2019), including business forecasting (Brown and

Yang, 2019; Costa Sperb et al., 2019), they remain largely overlooked by corporations,

who are uncertain about their use and question their accuracy (Graefe, 2010; Rieg and

Schoder, 2010). PMs have been shown to be more accurate than other comparable

forecasting technologies like polls, scenarios or the Delphi method; yet, perhaps because

of high profile failures in predicting political events, there has been a substantial decline

in actual use in corporations. In 2014, 54% of major corporations surveyed said they had

not considered investing in PMs for use in their companies (Bughin, Chui and Harrysson,

2015), a negative finding, particularly as only 12% reported using the technology and

only 14% said they would potentially consider it.

But applications of such corporate prediction markets (CPMs) in product screening or

similar activities to find novel ideas appeared to gain traction (Prokesch, Heiko and

Wohlenberg, 2015; Canonico, Flathmann and McNeese, 2019). The use of these digital

technologies reorganises decision-making authority in companies (Brynjolfsson, Hitt and

Yang, 2002) to include a wider swathe of the organisation through participation in the

market. As any successful tool requires significant efforts for successful implementation,

other approaches and mechanisms have emerged as potentially more suitable. It remains

to be seen whether PMs in general are a fleeting fad, a re-emergence could come about

in new areas of practical use, such as innovation management (Alfaro et al., 2019).

Web 2.0 with its participatory approaches is transitioning to the nascent Web 3.0 which

is called the machine-readable Web – here a confluence is happening from connected

people and crowdsourcing to Big Data. Data analytics in effect will become a bigger part

of ‘forecasting life’. But both Big Data and PMs are reputed to be ‘black box’

technologies. Depending on a prediction model’s design, its results can be hard to

comprehend. When leaders could not see the justification for the model’s suggestions,

they often ignored them.

4

Decision-makers were unable to put trust in the models when they outwardly resembled

a Magic 8-Ball (Babel et al., 2019). Thompson (2012) remarked that even though PMs

could overcome the inability of many companies to aggregate the views and insights of

their employees on critical issues, businesses do not understand this use of PMs.

At this juncture, both prediction markets (PMs) and corporate prediction markets (CPMs)

have now been introduced. It is important to mention that the terms are not

interchangeable. The two terms denote different – albeit interconnected/related – things.

Referring to a CPM – sometimes also termed ‘corporate PM’ when in the researcher’s

view the context demanded this phrase for clarity – is always meant to signify a market

run by and within the confines of a firm, i.e. a corporation. When using the expression

PM, the researcher referred to either the concept or a market run publicly or in a research

setting (i.e. not implemented for company forecasting purposes). This brief digression is

intended to clarify the two terms and their differences for the entire thesis.

At this stage it makes sense to take another short detour to clarify another point. PMs

belong to an area of forecasting called qualitative or judgemental forecasting where

judgement must be applied in order to forecast the effect of situations when the data

are incomplete or there are no historical precedents (Lawrence et al., 2006; Hyndman

and Athanasopoulos, 2018). Interspersed in this thesis are a range of other methods from

this prediction realm including polls, the Delphi method, the use of experts, surveys, the

use of analogies, brainstorming, nominal groups, or for instance scenario analysis which

works by pondering both optimistic and pessimistic options (Makridakis, Hyndman and

Petropoulos, 2020). Modifying forecasting data by adding black swan events can also be

considered as a type of judgemental forecasting (Prestwich, 2019). PMs are repeatedly

contrasted or mentioned in conjunction with some of these methods without, however,

defining these other methods more precisely. This discourse serves to emphasise their

membership of a ‘forecasting family’ to which PMs also belong and to explain the fact

that the thesis will not dwell on their particulars but will stay on focus for PMs.

PMs are in common use for political forecasts but have received limited interest in the

business world (Bughin, Manyika and Miller, 2008), even though some companies still

use corporate PMs to this day, like Deutsche Telekom (Ivanov, 2020) or Google (Siegel,

2018; Easley and Kleinberg, 2019). But the hierarchies of corporate structure create the

perception the lower the level the less smart the people (Lavoie, 2009), blinding

organisations to the value of input from people lower in the hierarchy.

5

Thompson (2012) argued that by entrusting employees with the future direction of the

company, CPMs might overcome dissatisfaction and the low level of their use.

PMs in general may facilitate decision-making more directly than traditional forecasting

methods. Luckner et al. (2012) argued that for traditional forecasting tools to aid for

example environmental scanning, their output needs to be further evaluated. When

appropriately set up, PMs obtain such information directly and guide decisions directly.

Also, new approaches to innovation dealing with shortened life cycles have included idea

markets which can help to innovate across disciplines and ingrained beliefs (Hamel,

2012). This useful (corporate) application of PMs relies on the ‘wisdom-of-crowds’ to

prompt innovation: a problem or product idea is posted on an appropriate platform and

participants of the platform share their view on the importance, usefulness, and/or success

of the idea or problem solution (Pisano and Verganti, 2008); in case of a PM the sharing

takes place with the help of the trading mechanism.

Technology which a company could use to get internal feedback – such as PMs and other

crowd-based functions – already exists. Existing research indicated that PMs could

succeed as a forecasting methodology for companies (Buckley, 2016; Buckley and Doyle,

2017b; Brown, Reade and Vaughan Williams, 2019; Sung et al., 2019), but only with

clearly mandated support and direction from higher echelons in a company.

The general role of information systems in companies is to take data and turn it into

information, and then transform the result into organisational knowledge with a view on

a company’s future, too. As technology has developed, this role has evolved into the

backbone of an organisation (Bourgeois, 2018) and makes it the remit of business

executives. These executives form social networks that, if heterogenous, can lead to

considering novel ideas and the possible adaptation of innovative tools, according to

Ferris, Javakhadze and Rajkovic (2019); in such a way a social component contributes to

a system’s usage (Junglas et al., 2013). If decision-makers were in a network where the

idea of PMs would be accepted, this could foster their use (Maertens and Barrett, 2013).

Such a consideration, i.e. such an exploration of corporate cultures could therefore shed

light on the acceptance of new technologies within the higher ranks of a company (Bughin

et al., 2018; Smaje, 2020) and the influence of a company’s internal structures.

6

Reflecting on the possible success of a new forecasting technology, the researcher

considered looking at some of the critiques, particularly of Big Data – hailed as

permeating almost all aspects of companies’ decision-making strategies (Griffin and

Wright, 2015) – as Big Data is also placed in the context of PMs (Desouza and Jacob,

2017).

Machine learning, aka Big Data, does not work without training data and a model to

interpret such data. But Big Data does not have any internal model of the world and so it

cannot accomplish reasoning that requires such a model (Mitchell, 2019). Unreasonable,

sometimes dangerously misogynistic, misanthropic or otherwise biased, and thus

unwanted results are often created (Turner, 2017); even though it is argued that the more

data to which an algorithm can be exposed, and the more complex the algorithm is made,

the better it would perform. A wrongly chosen model can also treat random noise in the

analysed data as a meaningful result or input to a decision – the model overfits the data

leading to wrong conclusions, as no algorithm can overcome flawed training data which

led to its creation (Bergstrom and West, 2020). Many examples bear this out and point to

the fact that Big Data’s accuracy is rather questionable (ibid.) and overall deployment is

still lacking (Herzberg et al., 2020), but the downsides are played down for a forecasting

approach ‘hyped’ to be superior (Gandomi and Haider, 2015; Aeppel, 2017).

One could argue it might be the other way around for PMs – criticism exists galore (Shrier

et al., 2016) and their real advantages are hardly known or recognised. Therefore, this

forecasting approach seems to be at a dead end. If this really is the case is still open to

debate, however, as for example Dreger et al. (2015) saw PMs well suited to the need of

quickly identifying and aggregating dispersed information and clearly outperforming

surveys. Hidden but important information can be more easily and reliably revealed in

PMs, facts which otherwise would perhaps not be disclosed (Camerer et al., 2016;

Schoenebeck, Yu and Yu, 2020).

Already in 2009, an editor at Wired magazine suggested crowdsourcing had become

mainstream (Howe, 2009). On the other hand, in 2010 according to Gartner, a research

firm, the maturity of CPMs was still at least five to ten years out (Landry, 2010),

suggesting it could be 2020 before they reach maturity. It might be nearly impossible for

CPMs to escape the ‘Trough of Disillusionment’, in Gartner’s terminology, where less

than 5% of the potential ‘audience’ has fully adopted a technology.

7

Gartner had introduced the idea of a so-called ‘Hype Cycle’ in 1995 to gauge the maturity

of a concept or product ranging and evolving from ‘Technology or Innovation Trigger’,

‘Peak of Inflated Expectations’, ‘Trough of Disillusionment’, ‘Slope of Enlightenment’

to finally the ‘Plateau of Productivity’(Fenn and Linden, 2005). In 2011, PMs in general

were taken off Gartner’s Hype Cycle for Social Software as an independent technology,

combined with crowdsourcing and moved back two phases to the very beginning of the

cycle, indicating that they were less mature than initially suggested (Mann, 2011).

Through conversations with forecasters and business executives, mostly from the service

and manufacturing industries, as well as from finance and telecommunication sectors, in

both large multinationals and medium sized companies, the researcher became interested

in why PMs had not been adopted by corporate culture. The literature, on the other hand,

mostly suggested that PMs were widely adopted. The discrepancy between the

researcher’s discussions with corporate leaders and the researched literature led to the

idea of investigating the factors that lead to or prevent PM adoption. The approach to

selecting a management tool can be as intricate as the business problems it is being

enlisted to resolve (Rigby, 2015). To illuminate the situation, the researcher collated

arguments and evidence for and against the use and implementation of corporate PMs

and to force this reasoning through a critical evaluation process.

1.2 Research Focus

Summarising rationales from the literature as to the usefulness of PMs, and summaries

from first discussions and conversations the researcher undertook gave rise to the idea to

create classifications and subsumptions to gain insights about corporate adoptability of

PMs. When assembling information about them, and seeking theoretical grounding,

presumptions or explanations around corporate PM usage seemed to have more to do with

underexplored circumstances regarding human experiences and actions rather than fact-

based reasons.

Larkins and McKinney (2012) advanced on four types of theory in understanding human

social behaviour and the patterns of interactions, among which two – theory as criticism

and taxonomic theory (the other two being theory as classics and scientific theory) –

appear to be fitting to form a first basis for the research undertaken in this thesis.

8

The former, around literary criticism (Collins, 2019), seeks explanations from a thorough

understanding of the literature. The latter, taxonomies, “are useful ways of representing

knowledge about objects in a [specific] domain … [and] classifying objects of interest”

(Muntermann, Nickerson and Varshney, 2015, p.3536) to assist explaining reasons and

to array knowledge. This encompasses the two aspects of exploring PM relevance – from

a factual and a gestalt perspective.

PMs were therefore looked at more from a perspective of knowledge generation about

their usefulness rather than from explicit theoretical considerations about them, as only

very few operating CPMs exist (cf. Ruberry, 2013; Williamson, 2014; Ovcharov, 2018).

This led to generalising about and analysing the topic to illuminate determining motifs.

Another area of potential theoretical grounding could relate to the theory-practice gap

(Elwyn et al., 2011), and whether theories around that should have taken centre point in

the research pursued in this thesis. Planners seem to pick and choose theories that fit their

purpose as a basis for forecasting and planning exercises, and with a focus on planning’s

practicalities there is often little interest in bridging the perceived theory-practice gap by

practitioners (Allmendinger, 2017). With the assumption that there is no best plan for the

future, no plan for every situation, an openness to practical decisions over preconceived

theories, receptiveness to new procedures, and a fluid framework, was advised (ibid.).

Practitioners of planning do not need to rely on (normative) theories; their common sense

or relying on sound judgement suffices when deciding upon strategic planning

instruments (Altrock, 2008).

The researcher identified a clear gap between viewpoints from scholars in the literature

and experienced reality in the use of CPMs. Following the arguments above, though,

rather than applying developed theories belonging to this particular theory-practice gap

the researcher advanced that the importance of these aspects and dealing with them could

be left to exploring evidence around this discovery.

1.2.1 Research Question, Aims and Objectives

Forecasting is often perceived as assisting strategic planning, which is consistently ranked

number one or two globally amongst 25 concepts and techniques (Rigby and Bilodeau,

2015; 2018; Thorén and Vendel, 2019). Following the advice of Rigby (2015) and Thorén

and Vendel (2019) it is sensible to fashion a concept for decision-making in management

tool selection.

9

It is possible that the heyday of PMs is already past. How to approach forecasting and

what to incorporate there (Mayer-Schönberger and Cukier, 2013) had never been

promoted with PMs and corporations did not have a valid basis to choose or reject PMs,

to support a well-rounded and accurate forecasting process. The downsides of PMs were

expressed succinctly by Gartner Vice President Carol Rozwell and captured by Landry

(2010), both cited disillusion amongst early users stemming from an overestimation of

the usefulness and accuracy of PMs and also the cost and effort to set them up.

Despite the conceptual promise of PMs, they have not been adopted in business

forecasting as anticipated, except perhaps in innovation management (O’Leary, 2020).

Understanding why that is the case was the motivating question of this research project.

Where such an understanding, such a journey, could lead, could perhaps already be seen

along a path using the argumentation triad from the ‘situation’ of PMs’ positive

underpinnings, via the ‘complication’ of corporate PMs’ lack of use, to the ‘resolution’

of new use cases driven by the concept of idea diffusion, and arriving at a core message

looking at idea markets as a successful application for CPMs (cf. Garrette, Phelps and

Sibony, 2018). Figure 1.3 portrays this triad and its conclusion.

Figure 1.3 Argument Pattern to Arrive at a Possible Recommendation for CPM Uses (Developed by the Researcher from Minto, 2009)

SITUATION01 COMPLICATION02 RESOLUTION03 CORE MESSAGE04

PMs have a strongtheoretical underpinning

and can be effective

Usage of CPMs is limited, and organisational set-up

leads to limited acceptance

Combining PMs with other methods and expanding

their use cases

One route to successful CPM adoption via

innovation management in companies

Innovation and idea markets gain importance

Meissner et al., 2017; Tello, 2018;

Alfaro et al., 2019

Gain traction with the help of idea diffusion and creating tipping points

Lundblad, 2003; Rogers, 2010; Turner et al., 2017

Corporate prediction markets struggle

Bughin, Chui and Harrysson, 2015;Shrier et al., 2016

Prediction markets have a sound basis

Snowberg, Wolfers and Zitzewitz, 2013;Buckley, 2016

10

The motivating question posed above gave rise to a research question cascade of ‘What

are the advantages and disadvantages of PMs’, followed by ‘What are the barriers to

adoption of PMs by corporations?’, and lastly ‘Are there ways to make PMs more

adoptable for corporations?’. Thus the research aim was ‘To appraise and understand

why PMs are not used as a corporate forecasting method, to assess if they can be shown

to have efficacy, and to comprehend how PMs can be adopted’.

The objectives to meet the research aim were:

• Objective 1: Analyse evidence of corporate use of PMs and identify their

advantages and disadvantages based on the study of relevant literature

• Objective 2: Explore and identify challenges in the use of CPMs, such as

company hierarchy or management acceptance of their results, and compare PMs’

performance to alternative approaches

• Objective 3: Gauge the knowledge of corporate forecasters and researchers

regarding PMs, understand how and why they chose forecasting tools, and learn

their perceptions of challenges in the forecasting process from interviews

• Objective 4: Identify steps for a framework to guide adoption, and develop

recommendations for companies on the use of PMs

Further explaining and expanding on the ‘PM research versus reality’ contrast will be the

major contribution of this thesis as forecasting is often a point of contention, especially

as predictions are frequently not very precise and hence decisions based on them can go

astray (Wallace, 2006). Developing a basis to better understand PMs, which can drive

improved accuracy and reduced bias in projecting the future, was another goal of the

thesis.

Achieving this would give companies the opportunity to become more efficient, and

improvements help to take on even more difficult forecasting challenges (Weiss, Raviv

and Roetzer, 2018). Avoiding poor performance and understanding forecasting

mechanisms helps to choose a prediction process, possibly a CPM, that drives more

efficient use of resources, enhances customer satisfaction and allows a company to

become more competitive.

The question arises if it is worthwhile pursuing research on the success of CPMs and on

conditions to choose them, as this chapter paints a potentially unflattering picture in this

regard. Research could be seen as purely historical on a concept with its prime supposedly

about ten years ago in the forecasting arena.

11

Roughly counting arguments pro and con from this introductory chapter would arrive at

circa 60% negative versus 40% positive assertions brought forward. From the

researcher’s experience the value of PMs should be considered, as their underlying

technology and user interfaces have improved to better gather, present and evaluate the

trading data, and they can be opened up to other areas in business decision-making, giving

the overall narrative a more positive slant.

And the general discussion about PMs has not abated yet (Tetlock and Gardner, 2015;

McCaffrey, 2016), recently PM setup mechanisms to make them more successful were

deliberated (Rötheli, 2017; Tai, 2020) and PM advantages highlighted (Eagar et al., 2020;

Schoenebeck, Yu and Yu, 2020).

On the business side, as corporations were the focus of this thesis, PMs positively

contributed to the success of professed so-called superforecasters (Mellers et al., 2015;

Katsagounos et al., 2020), the concept of which is also commercially used (Good

Judgment Inc., 2020). Also, innovation is a clear new area of application for CPMs (Tello,

2018; Alfaro et al., 2019) which lent credence and value to pursuing the research

objectives.

Regarding the attainment of the objectives, particularly around guidelines for CPM

adoption and implementation opportunities, it is conceived that the notion of PMs being

‘a good thing’ needs to be corroborated for the research to have merit, as for example

alternatives might be more efficient.

It has already been established though that judgemental forecasting plays a valuable role

in forecasting regarding innovation management (Rostami-Tabar et al., 2020; Bannister

and Connolly, 2020), and that idea markets – ‘CPMs for innovation’ – have gained

increased importance in business circles (Gloor et al., 2016; 2020). Existing research

would also refute the opening argument of the above paragraph and positively argue that

CPMs have started to prove their worth in this area of forecasting (Czwajda et al., 2019;

Tiberius, Siglow and Sendra-García, 2020). But even on a more general level, usage of

CPMs can make sense and recent research bears this out.

At the end of section 1.1 Big Data was briefly touched on as a potentially more superior

and/or more accepted forecasting mechanism but insights from humans combined in a

market still retain an edge when something happens without precedent. And humans can

imagine scenarios that the past has not anticipated (Kasparov, 2017).

12

But it is expected that as Big Data continues to advance and by leveraging data-driven

predictions based on algorithms, human decisions are supported in a ubiquitous way

(Abeliuk, Benjamin et al., 2020). Research showed, though, that forecasters are affected

more by their existing beliefs than by the machine learning forecasts (Abeliuk, Huang et

al., 2020,) and in that way such human predispositions will continue to distort decision-

making based on all types of forecasts, by for instance applying post-hoc corrections. Due

to their prevailing biases, individuals do not always succeed in adapting their initial

judgements to reflect newly provided information given to them, particularly from

algorithms as forecasters often recognise that “algorithms are far from perfect and do not

necessarily beat humans in some deeply uncertain settings” (Abeliuk, Benjamin et al.,

2020, p.9).

This situation around ‘negative’ individual influence on decision-making, however, is

different when deploying a PM. The same biases still apply on an individual level but are

cancelled out when participants’ individual pieces of information are aggregated with the

help of a PM. Human ingenuity can access and assess various informational sources, even

ones that machines cannot incorporate such as gossip – which can credibly contribute to

positive information forming according to a research project from Linkoping University

(Takács, Bravo and Squazzoni, 2018; Zachar et al., 2018). Thus, PMs can generate highly

accurate forecasting results (Abeliuk, Benjamin et al., 2020).

And particularly in group settings where anonymity is preserved, as fostered by a PM,

valuable predictions occur regularly. Apart from PMs, only the Delphi method

‘combines’ acknowledged positive aspects from social influence (Kloker, Straub and

Weinhardt, 2017; Flostrand, 2017; Kloker et al., 2018) as both methods provide diversity

to filter social influence enough to achieve reliable outcomes (Mavrodiev, Tessone, and

Schweitzer, 2012).

If group deliberations start further away from a good result, these would usually receive

extra benefit from social effects (Yaniv, 2004; Jayles et al., 2017) which can introduce

biases. But the diversity introduced by a PM augments information aggregation and PMs

in general mitigate biases with the help of the inherent anonymity of the market.

Even if they are perhaps not that widely used currently, corporate PMs have a positive

underpinning, an advantageous and concrete basis in their underlying theoretical aspects

(dos Santos Pinheiro and Dras, 2017; Grainger and Stoeckl, 2019) – they could be

beneficial if their ‘service’ was considered.

13

This thesis postulated neither a possible nor an ensured success of CPMs but endeavoured

to contribute a critical examination of a forecasting mechanism. A look at the website of

a PM provider illustrated the possibilities of CPMs in the form of an ‘advertising

campaign’, depicted in Figure 1.4.

Figure 1.4 ‘Making Better Decisions’ – PM Attributes Based on Inkling’s Website, a PM Platform Provider Founded in 2006 (Developed by the Researcher from Inkling, 2015)

This expression – ‘advertising campaign’ – and the sequence of images and texts were

chosen by the researcher, but the basis behind it is real: CPMs have a genuine chance of

significantly improving corporate forecasts according to Archibugi (2017), Graefe

(2019), Geurts (2020), and Sohrabpour et al. (2020).

1.2.2 Thesis Composition – Research Design

To arrive at an answer how CPMs can foster a contribution for the forecasting function

of a company the researcher took the following steps in this thesis, reflected in the

following major blocks: First reviewing, appraising, and synthesising the literature then

developing a framework, research questions and a methodology followed by interview

sessions which were thematically analysed together with quantitative data; finally, an

overall result appraisal led to the final conclusions.

A corresponding schematic is shown in Figure 1.5 below, giving a brief visual

explanation of the points sketchily expanded upon in the paragraph above.

Company CultureUse prediction markets to transform how you forecast, quantify risk, and make decisions in your organisation.

Decision-MakingQuantify the likelihood something is going to happen through real-time probabilistic predictions.

Concrete BenefitImprove your (sales) forecasting or predict the outcome of key performance metrics.

14

Figure 1.5 Flowchart of the Overall Research Design Sequence (Developed for Research)

The diagram visualises the steps taken in this thesis to make a meaningful contribution.

1.2.3 Research Contribution Theory is often derived from careful observation of the past, then by categorising those

observations and correlating them with the outcomes of interest, and eventually

understanding what causes those outcomes (cf. Christensen, 2011). Following Kuhn

(2012), the key to improving any theory is to surface anomalies.

Pulling together evidence from monographs, papers, etc. – both past and current literature

in line with the first two objectives from this thesis – the researcher strived to foster the

advent of novel discoveries (ibid.) and endeavoured to amend core PM concepts with

perspectives from other fields drawn for their pertinent literature.

The literature review in the following chapter encompasses this approach and therefore

comprises and constructs original and unique views pertaining to PMs. A particular area

of focus was on issues of acceptance by management and hierarchical problems

constituting a contribution to the canon of the academic body of knowledge about PMs.

Explaining the schism of a seemingly good forecasting instrument and its neglect in the

business world would be a further addition to knowledge, which was developed through

this research and via interviews. The researcher believed that this would consequently

contribute to transforming extant views, as there appeared to be an inconsistency between

observations in the literature and ‘conventional wisdom’ about PMs.

Frameworkincl. research

questions development

Literature Review & Synthesis

Methodology Decision

Thematic Analysisaspects from literature and

interviews

Analysis of PM Accuracy

based on the IARPA forecasting

competition

Literature ReappraisalConclusions

Qualitative Interviews

Pilot

Extended Interviews

innovation related

Result Appraisal

Analysis of CPM usage

Main Study Interviews

15

A company interested in choosing PMs as a forecasting tool could learn from this research

about the pitfalls and challenges but could also understand the advantages of PMs and

thus make an informed decision regarding the choice and implementation of a potential

forecasting instrument. In the development of knowledge, this research will hopefully

make contributions to understanding the decision-making for and implementation of

management support technologies and tools.

1.3 Thesis Structure

After having introduced the topic of PMs in the context of forecasting and

entrepreneurship and the research problem (question, aim, and objectives) in this chapter,

the structure of the overall thesis is illustrated in Figure 1.6.

Figure 1.6 Thesis Structure (Developed for Research)

After the Introduction chapter the thesis is structured to continue with a Literature Review

(Chapter 2) summarised in a Literature Synthesis (Chapter 3), leading to a Theoretical

Framework (Chapter 4), and subsequently laying out the Methodology and Methods

(Chapter 5). After presenting Results and Findings (Chapter 6), a Discussion leads to the

Conclusions (Chapters 7 and 8).

1. IntroductionBackground and Research Focus

2. Literature ReviewForecasting Context, PM

Principles, Usage and Challenges

5. Research MethodologyPhilosophical Position,

Research Design and Pilot Study

3. Literature SynthesisLiterature Summary – Focus

on PM Potential and Adoption,Link to Objectives

4. Theoretical Conceptualisation

Theoretical Framework and Research Questions

7. DiscussionBrief Literature Reappraisal, Findings about PMs put into

further Context and Limitations

6. Results / FindingsData Collection, Summary, Interpretation and Result

Appraisal

8. ConclusionWrap-Up, Directions for

Future Research and Outlook

16

2. Literature Review

In this chapter the review of the relevant literature regarding the principles, advantages,

and disadvantages of PMs with emphasis on their use in business and with special

attention to company hierarchy and management acceptance takes place. The research’s

conceptual foundations broached in the introduction follow the path below in the

literature review: Figure 2.1 shows its main blocks, depicting the individual section

headings together with a brief thumbnail sketch of their main content.

Figure 2.1 Literature Review – Conceptual Path (Developed for Research)

The sequence of sections builds a narrative around the current state of play for CPMs

and the results from the literature review are then encapsulated in a synthesis chapter.

2.1 Prelude

Google aroused corporate interest in PMs when they deployed a CPM in 2005 (Coles,

Lakhani and McAfee, 2007). Microsoft had already been using a CPM since 2003, which

they expanded in 2006 (Berg, 2007) and reported regularly about (McCaffrey, 2016).

From these high-profile examples, one could infer wide-spread use of CPMs.

But by that time the proliferation of Web 2.0 technologies perhaps prevented PMs from

attracting enough attention. One issue was the many incarnations of PMs, such as

artificial agent-based models applicable to business and finance forecasting (Horx, 2011;

Orrell and McSharry, 2014). The availability of these many alternatives prevented any

one of them from gaining traction (Bonabeau, 2009).

2.1 – Prelude

Setting the Scene, Collective

Intelligence and Forecasting

2.2 – Why forecast?Forecasting Context

2.3 – Principles of PMsTheoretical Underpinning

and Usage

2.4 – Traits and Attributes of PMs

Critiquing PM Principles

Alternatives to PMs

2.5 – Business Use of PMs

PM Prevalence in Business in

the Context of Web 2.0 Tools

2.6 – Impact of Company Hierarchy + Management

AcceptanceInfluence of Company Culture

3 – Literature Synthesis

Literature Summary – Focus

on PM Potential and Adoption,

Link to Objectives

17

CPMs could be considered ‘kaput’ even in 2008, with a market penetration of less than

1% according to Gartner (Cain and Drakos, 2008). A comprehensive study from

McKinsey reported a mere 6% usage at that time. While they continue to be used by some

companies, CPMs may have never recovered from their disappointing introduction to the

marketplace (Zinger, 2018). Whereas the literature continues to cite the same PM success

stories such as the sales forecast at Hewlett Packard and project management at Siemens

(cf. Horn, Brem and Ivens, 2014; Palak and Nguyen, 2017; Horn, Bogers and Brem, 2018;

Klingert and Meyer, 2018), the researcher learned that these CPMs had been long

abandoned through personal conversations with those companies.

2.1.1 Collective Intelligence

Particularly with forecasts made simply by extrapolating current facts into the future,

many of the long-established assumptions, trends and habits have unexpectedly lost much

of their significance, observed Dobbs, Manyika and Woetzel (2015). Perhaps this is

because in many organisations an overabundance of data and information exists which

results in misleading forecasters who tend to bring to bear understandings and notions on

how things functioned at a time when modifications were gradual and reasonably

predictable; they then obtain inaccurate results (ibid.).

New approaches which improve neutrality and pull together many informed views

promised to remedy that. Collective intelligence, as one such way forward, observes how

knowledge, creative ideas or decisions can be gained by concerned parties from

information distributed in a group (Gloor, 2011). Systems theory, founded in 1950, stated

that individual constituents once aggregated take on collective characteristics that are

usually not manifested in the properties of the individual components themselves

(Kempes, Koehl and West, 2019), in such a sense information is added which would not

emerge at the level of the individuum. This realisation already presages the tenets and in

a sense the importance of collective intelligence. Recent research around improving

decision-making suggested the promotion of active participation from groups of all sizes,

involving everyone and unlocking their potential (Steinhöfer, 2020).

Members of the Quirky network, a community-led invention platform, publish their

product notions with the help of the company’s internet presence. Subscribers vote on

how appealing each idea is and find ways to put them into practice (Crunchbase, 2015).

This is an example of collective intelligence. ‘Voting’ through a pricing mechanism, on

the analogy of buying and selling shares, carries the concept forward into a PM.

18

The collective decision-making ability of a group is at least as good as or better than that

of each individual member (Ho and Chen, 2007). The concept of collective intelligence

assumes teams to significantly outperform individuals in decision-making (Matzler,

Strobl and Bailom, 2016; Suran, Pattanaik and Draheim, 2020), and anonymity and

diversity further enhance the quality of results.

Anonymity prevents pressure from peers and bosses, and diversity helps to avoid

groupthink, for example. Studies had shown that groups of average people were able to

predict events with higher accuracy than a single expert (Malone and Bernstein, 2015;

McHugh et al., 2016; Peeters, 2018). The promise of more accurate forecasting is why so

many companies today use collective intelligence (Palak and Nguyen, 2017) where a PM

can host questions around strategic environment scanning (micro and macro) for instance

(Tiberius and Rasche, 2011). Environmental influences, such as competitors, politics and

economic factors, can be the subject of forecasts. However, sales forecasts are most

prevalent in the context of business (ibid.), additionally the prognosis of market share,

product growth rates, the potential of new products or the quantification of technology

trends (Clement and Schreiber, 2013).

In the literature, PMs are compared to and contrasted with other methods and models,

such as opinion polls (Acker, 2016) face-to-face meetings, nominal groups, and Delphi

(Graefe and Armstrong, 2011). Ford (2018) stated that artificial intelligence (AI) has

made its transition into our daily lives, outperforming people and also applications that

turn raw data into coherent and meaningful – albeit technology-based – statements about

the future. Because the approach of Big Data mimics and surpasses human decision-

making, even showing creativity in its forecasting capabilities (Ertel, 2016), it may be

ousting PMs. A cofounder of the MIT Media Lab projected that collective intelligence

technology, which includes PMs, will come not only from networked cerebra, but also

from vastly networked and intelligent devices (Brynjolfsson and McAfee, 2014, jacket

blurb).

However, there are still design flaws like a lack of traders (as can happen when the subject

is not sufficiently interesting) or contracts that do not clearly specify when they will and

will not pay off, that may cause PMs to fail (Snowberg, Wolfers and Zitzewitz, 2013).

19

Despite their power and versatility, Big Data and other AI incarnations contain many

biases and need human-based efforts to function. Humans can introduce flaws, for

example in strategic decision-making, which also have a direct negative influence on

strategic corporate planning (Dye, Sibony and Truong, 2009). Biases are more likely to

be eliminated or at least reduced with mechanisms like the trading of PMs (Peeters, 2018)

that average out distorting views.

2.1.2 Forecasting and PMs

A PM can be described as a stock market which pulls together information dispersed

amongst traders. Its aggregation mechanism – the trading – translates the information into

a price (Tsiaris and Tatsiopoulos, 2007). Concrete events occurring in the near and far

future are thus predicted for challenges of direct importance to an organisation (O’Leary,

2011). Businesses need to “forecast uncertain outcomes such as how a competitor will

respond to a new product-launch or how much revenue a promotion will generate”

(Shoemaker and Tetlock, 2016, p.75). Steve Ballmer’s assessment in 2007 that the iPhone

would not achieve a noticeable market share deprived Microsoft of the opportunity to

explore other scenarios (ibid.). Assumptions on a product’s potential success could,

however, be elicited through a PM. A market could, for instance, peg the probability of

first quarter sales of a new product being in the range of 9,000 and 11,000 units at 80%,

giving valuable information regarding production scheduling.

In PMs – still called an innovative forecasting method, only recently applied in businesses

according to Horn, Brem and Ivens (2014) – virtual shares, whose pay-outs are linked to

the result of uncertain future events, are sold if traders think they are overvalued, and

bought if deemed to be undervalued (Seemann, 2009). Traders form their own views on

the value of individual stocks. Their evaluation can change based on the appearance and

availability of new facts, thereby changing the price of a traded security (Coles, Lakhani

and McAfee, 2007).

Due to market dynamics, the trading price reflects buyers’ aggregated evaluation of the

consequences of a future matter, which are unknown during the trading period. Thus, the

price generated through buying and selling indicates and communicates a forecast

(Seemann, 2009). For example, if shares in ‘Team 1 wins against Team 2’ trade at 60

cents, this ‘market price’ can be taken as a 60% occurrence likelihood of the underlying

event (as shares can only be worth between 0 and 100 cents).

20

Even when a PM is carefully constructed, herding behaviour can lead to mispricing as

price movements stemming from such behaviour may just echo the psychological state

of the traders, not true information, and predictions based on this may be imprecise (Sung

et al, 2019). In such cases traders are not driving out collective biases (Angelini, De

Angelis and Singleton, 2019; Berg and Rietz, 2019), they ignore their original assessment

and follow the trend (cf. Spyrou, 2013; Hirshleifer, 2020).

The risks of bias and herding warrant giving attention to implementation as the ability to

forecast accurately is central to effective planning (Millett, 2011).

2.2 Why Forecast and Approaches to Forecasting Usually in a company the function of forecasting is to predict sales or production targets

linked to organising resources but also to forming strategy to cater for changing settings.

As Mintzberg (1994) observed, hard data can be shared through a system, soft data

cannot. PMs offer a way to share soft data, as private perceptions influence the price

deliberations of trading in a market.

2.2.1 Relevance of Forecasting and Relevance of PMs – A Decision-Making Keystone

Business environments change. The challenge is to predict what kinds of changes will

occur. Such predictions could be made by a PM, but only if the shares to be traded are

correctly set up to account for possible futures. As incertitude is the chink in the armour

of strategy design (Allaire and Firsirotu, 1989 in Mintzberg, 1994), there are only four

ways to deal with the future: ignore it, predict it, control it or respond to it (Dimma, 1985).

In the general area of strategy development, Lovallo and Sibony (2006) researched

distortions and deceptions in strategic decisions. They had observed that ‘the human

factor’ cannot be ignored in strategic decision-making. To take good decisions, awareness

of cognitive biases and how they mislead are prime factors, as is establishing a culture of

constructive debate.

Using models and other means, business executives examine their company’s competitive

edge for a fit between the capabilities of the firm and the market, and they predict how

these will evolve over time. They then create a plan to build and maintain advantageous

positions consolidated through methodical, consecutive planning rounds, using

quantitative forecasting to envision the future (Reeves, Love and Tillmanns, 2012).

21

In this sense, strategy is linked to planning and proper planning requires an assessment

of coming events. Nevertheless, it is important to carefully distinguish between

forecasting and planning (Kolassa, 2012; Ord, Fildes and Kourentzes, 2017).

Continually monitoring events as they progress for the purpose of strategic planning can

be difficult with conventional prognostications based on brainstorming, experts, Delphi

or scenario approaches. The outcomes of events need to be analysed, evaluated and

précised before they can be useful. PMs are established with the express purpose of

gathering real-time information. Carefully constructed PMs can immediately steer

decision-making (Luckner et al., 2012). Whether conceived as information markets,

virtual stock markets or idea markets some academics have lauded PMs for outperforming

forecasting instruments like questionnaires, surveys, polls, conjoint analysis, etc. (Graefe,

2009; Servan-Schreiber, 2012; Snowberg, Wolfers and Zitzewitz, 2013; Rothschild and

Wolfers, 2013), with quite encouraging results for the purpose of business forecasts

(Spann and Skiera, 2003).

Some widely discussed alternatives to PMs might be more fashionable than their return

on investment. As an example, trends like Big Data analytics and AI are technologies

that, according to McKinsey (Bughin and van Zeebroeck, 2018), are still in the early

stages of deployment. Here, as with PMs, the imperative is ‘caveat emptor’ to avoid

buyer’s remorse in one’s choice of a forecasting tool.

Prediction is at the heart of decision-making under uncertainty. Such decisions permeate

businesses and private life, and prediction tools increase prediction productivity and help

where uncertainty influences strategy. Such tools facilitate the development of new

business structures and competition strategies and are increasingly available. AI, for

example, will make forecasting more affordable, dependable and more easily accessible

(Agrawal, Gans and Goldfarb, 2018; Yeung et al., 2018). Agrawal, Gans and Goldfarb

(2018) recast the rise of AI due to a drop in the cost of prediction, reframing it as a cheap

forecasting commodity. With predictions at the heart of decision-making, AI’s

extraordinary potential becomes clear through the basic premise of their findings: with

the cost of prediction dropping decisions should improve overall.

Nevertheless, it is taking a final judgement about the prediction that assigns value to its

input (ibid.), and management, which is often sceptical about a prediction, does not

always consider it. Heskett (2012) expressed it quite strongly: even if predictions come

from legitimate origins they are only trusted if they fit the current agenda of a company.

22

Jobs or careers are often on the line in strategy processes (Bradley, Hirt and Smit, 2018),

so decision-makers are cautious about the predictions of PMs as insights from the markets

can evidence the over-promising or under-delivering of performance goals of managers.

Managers therefore shy away from “reference data with predictive power” (ibid., p.6).

An analysis of more than 1,000 decisions by Lovallo and Sibony (2010) revealed that

facts and analytics contribute only 50% to important strategic decisions; the other half of

decision-making results from dynamics between the decision-makers.

This underscored the point that managers do sometimes care more about maintaining

status than arriving at a meaningful strategy (Bradley, Hirt and Smit, 2018). Particularly,

when a situation is ambivalent one argues in a way that is selective enough to serve one’s

interests yet plausible enough to convince others (and oneself) that one is not intentionally

twisting the facts (Sibony, 2020a). Additionally, as biases enter decision-making, e.g. a

propensity to look for facts that support existing predilections, decision-makers are more

likely to overlook important information. Especially under pressure, information is often

not analysed efficiently enough to reach a conclusion (Zhang, 2015).

Such ‘shortcuts’ and ‘distortions’ appear to be the easy way out, because many of the

choices shaping the future of organisations need time-consuming thought, analysis, and

multiple considerations (Kahneman, Lovallo and Sibony, 2019). Complex decisions

cannot effortlessly be checked for quality, too. To improve them, work needs to focus on

the processes by which decisions are reached using fact-based and independently made

assessments ideally relying on evaluations supported by the expression of probabilities

(ibid.).

The above addressed needed diversity and independence would be provided by a

corporate PM, as well as likelihoods of occurrence. Another way to ameliorate the

pointed-out decision-making problems could be the use of data-driven instruments; an

increasing quantity of resolvable issues are now embedded in decision-supporting

software based on Big Data. However, executives often do not obey the dicta of those

tools, regardless if they are getting an explanation, and even if the forecasting instruments

are not meant to replace intuition, but to support and refine it (Gilboa, Rouziou and

Sibony, 2018). Furthermore, when taking strategy decisions, there often seems to be a

rush towards certainty – not odds (Bradley, Hirt and Smit, 2018). Probability-based

uncertainty and multiple realities cannot always be handled by executives (ibid.). These

executives’ limitations inhibit adoption of PMs for strategy planning as well as scenario

planning, as both are steeped in a probabilistic approach.

23

The judgement portion of a decision will become increasingly important and thus tools

supporting the judgement; in the past, the distinction between prediction and judgement

was rather academic, as humans always performed both together, whereas the two

functions can now be more meaningfully separated (Agrawal, Gans and Goldfarb, 2018).

2.2.2 ‘Machine Prediction’ Although a PM is used by humans, it is still, in a sense, a non-human prediction

mechanism, a ‘machine prediction’. The anatomy of decisions, from input to prediction

and judgement to action, has changed through the introduction of an automated prediction

element such as AI or machine learning. Still, involved forecasting methods confuse

random noise with information (Silver, 2012; Spielmann, 2016), through over-fitting

where the complexity of an algorithm makes it fit to the noise rather than the signal

(Spiegelhalter, 2019). This problem could increase as the Big Data era generates more

and more noise by producing more and more data (Heskett, 2012). Results from data

analytics have, therefore, still to be taken with a pinch of salt, and are not a universal cure

for difficulties in forecasting. Sometimes, when there are many variables or dimensions,

one dimension might simply get a lucky prediction result that appears to be statistically

significant (Stephens-Davidowitz, 2017).

Some experts in the finance industry, where many applications of the technology exist,

see machine learning as overrated and overhyped (Harding, 2019). Deploying the

technology at scale also poses a substantial challenge for companies; a majority only run

pilots. Companies find it hard to switch from leader-driven decision-making, and from

being rigid and risk averse to adopting the desired experimental and agile approaches

(Rigby, 2017; Fountaine, McCarthy and Saleh, 2019). Ceding control to an algorithm, a

non-management entity so to speak, whether based on predictive analytics or a PM

trading mechanism, is still extremely unusual (Benaich, 2019).

Research at MIT on collective intelligence nevertheless suggested sophisticated data

analysis to mine electronic communications such as email, telephone, chat, virtual

meetings, etc. This yields a set of structural time- and content-related social network

measures from which one can predict authentic insights (Gloor, 2017). When innovation

is in focus though – as increasingly happens with higher significance in companies

(Meissner, Polt and Vonortas, 2017; Agarwal, Oehler and Brem, 2021) – the focus should

be more on collaboration, knowledge sharing, and networking to arrive at innovation

productively (Gloor, 2006; Zwass, 2010; Bagherzadeh, Markovic and Bogers, 2021).

24

Methodologies to achieve competitive advantage in innovation rely on team-building and

other collaborative traits to be successful (Gloor, 2006). Challenges to Big Data and

machine learning implementations also tend to be organisational and cultural in nature

rather than technical. Fountaine, McCarthy and Saleh (2019) have found that culture is

key in adoption, and this goes for CPMs as well.

But, as non-human predictions become increasingly cheaper, reliance on them will grow;

prediction tools raise the value of data and complementary judgements and reduce the

worth of human forecasts (Agrawal, Gans and Goldfarb, 2018). Such an increase in use

carries the risk that an organisation will overlook the fact that humans remain better at

making predictions in unusual circumstances, in the absence of good data, and when

making analogies is needed (ibid.). With sparse data, the quality of predictions is at risk;

it is more likely that a prediction is provided with confidence by AI but is false. In sparse

data situations, AI is ineffective (ibid.). Also, the reliance on correlation rather than

causality leads AI into the same traps as anyone using simple data-based statistics (ibid.),

whereas missing data can often only be overcome by thoughtful human judgement.

Perhaps because established companies are sometimes wary of adopting new prediction

technologies, like Big Data and AI based machine learning (ibid.), commercial adoption

of these tools still lags behind expectations, according to a report by the McKinsey Global

Institute (Bughin, Hazan et al., 2017). Such prediction machines have limitations. They

are trained on specific data sets and thus can be susceptible to bias. AI and machine

learning depend on and reflect the ideas they are taught (Willis, 2018) and this could still

create a chance for PMs to prevail (Krug, 2019).

Statistical models can forecast sufficiently precise projections of events which occur

frequently, but not one-off situations without historical precedent. One way of estimating

the probability of such events is crowd-wisdom, where equities trading aggregates

opinions about future profits, betting markets reveal a consensus view about future news

and a fair representation of the overall crowd-wisdom (Brown and Reade, 2019; Clement,

2020; Hall, Jones and Klenow, 2020). Additional data-driven analytics can add valuable

information to such predictions (Brown et al., 2018).

25

In extrapolating from a particular situation or given sales figures to arrive at a forecast,

there is a tendency to over-interpret correlation, viewing it as a causal link and

interpreting the causal links as logical connections. Such logical pitfalls take place under

the guise of simplicity and efficiency. Humans’ inclination to trust our instincts and to

‘think fast’ leaves us vulnerable to pitfalls of this sort (Garrette, Phelps and Sibony, 2018),

but they can be avoided by using a PM. Because many people contribute, opinion outliers

and fallacious reasoning can be averaged out.

Working in teams with people using different mental models and frameworks can produce

innovation insights by sharing ideas and information (Gloor et al., 2020). Increasingly

though, the results of such deliberations are expressed in metrics, particularly if guided

by Big Data technologies (Gloor, 2017). Silver (2012) encouraged decision-makers to be

careful about forecasts that are expressed as percentages or probabilities. They clearly

represent the limits of our predictive capabilities. Probabilities are not easy to grasp

(Bingham, 2013) and biases can be introduced by misconceptions of probabilities

(Snowberg and Wolfers, 2010). In general, it is difficult for people to interpret statistical

values (Gigerenzer, 2015), which calls into question the usefulness of not only PMs, but

also Big Data approaches that provide probability results.

2.2.3 Managing for an ‘Unpredictable’ Future The IARPA tournament – a large study on the merits of different forecasting approaches

including PMs, conducted over four years by IARPA (the ‘Intelligence Advanced

Research Projects Activity’ within the US’ Office of the Director of National Intelligence)

with 25.000 forecasters and millions of predictions – reached the important insight that

general knowledge often outperforms specialist analysis, and that forecasting ingenuity

can be improved through carefully crafted training (Schoemaker and Tetlock, 2016).

These results suggested that well-selected participants and effort put into forecasting

could eliminate the need for a forecasting tool like group opinions, analogies or Delphi

(Ord, Fildes and Kourentzes, 2017) but still lean towards judgemental forecasting

methods, to which PMs belong.

The third suggestion from Schoemaker and Tetlock (2016) was to use teams to

outperform individuals. It is also important that forecasting teams are multifaceted (ibid.)

and that they allow for probability-focused methods of forecasting. This recommendation

could include the use of PMs, which aggregate general and specific knowledge, are

diverse when trading is not kept endemic and use a probabilistic mechanism in the form

of the price signal representing percentages.

26

One key component of an informative PM is that participants all invest equal or similar

amounts. Otherwise the volatility of the price makes it less likely that the market will

settle (Jansma, 2020) and therefore will not produce a useable forecasting result.

Nevertheless, PMs encapsulate data collection of future intentions well and incentivise to

dynamically elicit latent views. Complementary approaches such as Delphi, however,

produce results of similar quality (Ord, Fildes and Kourentzes, 2017).

The above qualitative forecasting methods, including PMs, exhibit clear advantages, but

researchers universally suggested contrasting them with qualitative approaches

(Robertson et al., 2017; Ansari, Holz and Tosun, 2018; Arvan et al., 2019; Sanzieri, Hall

and Anderson, 2019). To integrate human judgement into quantitative forecasting

methods, namely, to reconcile qualitative storylines and quantitative descriptions, both

approaches need to be considered, as seen in Table 2-1.

Table 2-1 Managing for an Unpredictable Future – Forecasting Approaches Compared (Developed for Research from Ord, Fildes and Kourentzes, 2017; Schoemaker and Tetlock, 2016; Armstrong, 2001b)

Forecasting Approach Pros Cons

Judgemental / qualitative

• Receptive to most recent environmental changes

• Incorporation of inside information • Compensate for one-time or unusual events

• Cognitive limitations via applied heuristics • Possibly lacking coherency • Biases towards salient information • Motivational biases • Subject to overconfidence • Perhaps time-consuming or costly

Quantitative

• Objective and consistent • Ability to process large data amounts • Variables and complex relationships accounted

for • Replicability

• Delayed reaction to changing environmental factors

• The model and the underlying data determine quality

• Soft information is costly to model if at all • Technical understanding by forecaster and user

required (‘black box’)

Combined

• “Forecasts derived from methods that differ substantially and draw from different sources of information” (Armstrong, 2001b, p.417) improves forecasting accuracy

• Especially useful when there exists uncertainty “about the situation … [or] which method is most accurate, and when … [wanting] to avoid large errors” (ibid.)

• Combining prognoses can yield a higher precision than their most accurate component and can be “more accurate than the typical component forecast” (ibid.)

• Data and methods need to potentially differ substantially for combining to be effective

• Armstrong (2001b, p.423) strongly advises to “use … forecasts from at least five methods when possible”

• The suggestion to use formal procedures for combining makes it complex

Bottom Line / Résumé The ‘sweet spot’, on which companies should focus, are forecasts where some data, logic and analysis can be utilised – but in which experienced and skilful judgement and careful questioning also play a key role (Schoemaker and Tetlock, 2016)

The weaknesses of judgemental forecasting can be captured in the phrase ‘heuristics and

biases’, defining the shortcomings and, to some extent, the advantages of judgemental

forecasting (Ord, Fildes and Kourentzes, 2017).

27

Forecasting of this type uses simple mental strategies, or heuristics, that can contain

failings like biases, as judgemental or qualitative forecasting integrates information based

on subjective beliefs. PMs can drive out the subjectivity and associated predispositions

through their structured trading and aggregating mechanism. Quantitative forecasting

applies a prescribed and explicit analysis of numerically coded data for extrapolation,

which is also supposed to drive out biases. Numerical coding cannot always incorporate

soft factors, however, which are not captured in the hard data. Both judgemental and

quantitative forecasts are valuable and should not be mutually exclusive. Schoemaker and

Tetlock (2016), for example, suggested applying careful human questioning to data and

logic, combining data analysis with human ingenuity.

Sharing the constraints under which forecasting recommendations apply makes them

more credible rather than less convincing (Garrette, Phelps and Sibony, 2018). As price

is the only information available in PMs, the boundaries or constraints of the forecast

usually cannot be conveyed. Limitations of the ‘advice’ from a forecast are nevertheless

important, as managerial overconfidence leads to inappropriate adjustments of forecasts

and to initial miscalibration. Where boundaries can be given, people make fewer

adjustments as a level of meta-knowledge, or the limits of the estimation, is conveyed

(ibid.). Humans make adjustments far too often, however, decreasing the quality of the

overall forecast, also when stemming from PMs (Eksoz et al., 2019; Van den Broeke et

al., 2019; Fildes and Goodwin, 2020; De Baets and Harvey, 2020).

2.2.4 Selection of a Forecasting Approach

When deciding on a new forecasting method, one could perhaps start with a blank sheet

of paper. If one wanted to incorporate new, modern methods into a forecasting program,

the focus would likely fall on Big Data – including predictive analytics, machine learning,

and AI – because there is no getting around this analysis and prediction method. One

reason for the omnipresence of these methods is that advanced analytics, as the

management consulting firm Bain & Company averred, can help firms to derive the kind

of proprietary insights that give an essential edge against rivals, and can complete these

efforts in a fraction of the time and at a fraction of the cost previously required

(MacArthur and Rainey, 2019).

In 2019, the International Journal of Forecasting published a special issue on Big Data.

Their editors expressed the opinion “that the forecasting community thus far has taken a

rather myopic view of big data” (Boone et al., 2019, p.125) and its capability of greatly

reducing forecasting errors.

28

Big Data can spot trends in vast datasets which are impossible for humans to detect and

can add to the information aggregation capabilities of PMs, almost as an antagonist

(LaRiviere et al., 2016). But despite this praise, the academic literature is not in sync with

experiences from practitioners the researcher approached to gauge the possibilities of Big

Data technologies. At a German manufacturer, production line forecasting via Big Data

only achieved 65% accuracy (Koß, 2020), underscoring this point. Furthermore, Big

Data’s usefulness as a forecasting tool, particularly when companies contemplate

broadening their forecasting arsenal, encounters similar hindrances as with CPMs.

Cultural and hierarchical issues in companies, fundamental barriers in enterprises,

remain, such as functional silos and the lack of commitment from leaders (Chui and

Malhotra, 2018).

The shift to digital technology in forecasting is nontrivial. The need to change business

processes to integrate cognition from data into the sequence of operations and to correctly

interpret data and get those insights in front of decision-makers (Manyika, 2017) can be

a hindrance to adoption. Mid-level executives and managers need to know not only how

they can use the data-driven insights – they must actually use them. As with PMs (cf.

Thompson, 2012), personal judgement still overrides AI-based decision-making

frequently (highlighted in a question rated by C-level executives at 3,073 companies

(Bughin, Hazan et al., 2017)). PMs, however, differ drastically when combining the

judgements of a group of individuals, according to Ord, Fildes and Kourentzes (2017),

and they are a ‘wallflower’. Compared to Big Data, PMs hardly ever appear in public

discourse, as can be seen in the graph below of searches for the two terms over the last

five years.

Figure 2.2 Google Trends – Results from Search Terms ‘Prediction Market’ (blue line) and ‘Big Data’ (in red) in January 2021 (Data Source: Google Trends – https://www.google.com/trends)

29

Any new forecasting method proposed to a company must stand out from Big Data. The

focus of this thesis is on PMs, but in order to be clear on the similarities and differences

between PMs and Big Data deliberations around data-based prediction are provided.

2.2.5 Oracle or Evidence? Predictive Planning as a Success Factor A research study conducted from May to June 2018 among employees with planning

backgrounds from 308 companies in Germany, Austria and Switzerland showed that

predictive planning and forecasting takes business-planning to the next level. The use of

statistical methods and machine learning facilitated faster simulations, more accurate

forecasts, and increased automation of corporate planning (Tischler, Fuchs and Engel,

2018). Thanks to the alleged maturity of the technology and easy deployment in the cloud,

statistical methods and machine learning are becoming affordable and relevant for an

increasing number of companies; indeed, their use is becoming one of the key trends for

optimising planning functions (ibid.). New tools and methodologies are used in data-

based analysis to provide the basis for simulations and anticipatory strategies. They

provide robust options for possible future events, ensuring greater planning certainty.

Applied Big Data analytics deploys and implements predictive capabilities in a business

environment (Jalonen and Lönnqvist, 2011) providing improved recognition of business

opportunities and threats, ‘eliminating oracles’ based on less suitable approaches. The

trend toward the adoption of Big Data analytics has been visible since 2008, when Bottou

and Bousquet (2008) published on the solution to the computational complexity of data

mining to support AI and machine learning, and to improve forecasting with these tools.

Data mining and predictive analytics to analyse future outcomes have been considered

for forecasting since 2007 (Hair, 2007). Sixteen years ago, Batchelor (2005) introduced

neural networks into the forecasting practice and demonstrated that they were able to

outperform more traditional forecasting approaches. He suggested neural networks, a

machine learning tool for large volumes of data, for data-rich environments. Neural

networks were becoming increasingly utilised ten years later when PMs made their

appearance (Horn, Ohneberg and Ivens, 2014).

Judgemental and data driven forecasting tools and approaches can therefore be seen as

competitors or can be considered to exist along a diverging path of tool choices, as

suggested by Armstrong (2001a).

30

Very few companies believe that if their industry continues to digitise at its current rate

and pace, their business models will remain economically viable. This acknowledgement

that the world has become data-driven might favour non-judgemental forecasting.

Widespread digitalisation causes disruption of and incompatibility with long-established

operating models used by many companies (Bughin et al., 2018). McKinsey sees

digitalisation as a near-instant, gratis, and frictionless way to join individuals, equipment,

and physical objects across all dimensions (ibid.). In their analysis, digitalisation led to

the automation of both processes and decisions, and to automated decisions based on data

mining which do not need PMs.

2.2.6 Prediction: Data or Market?

Furthermore, Bughin et al. (2018) stressed that strategic decisions need scenarios and not

point estimates, as provided by a PM. The same point forecasts also used in other

forecasting methods, like time series, emphasise minimising bias in the forecasts.

However, focusing on avoiding distortions may neglect the variability of the forecasts. In

this context, simple methods tend to have a large bias but a small variance, while

complexity reduces the bias at the expense of increasing variance. Several researchers

argued that simple methods are preferable to complex ones, even if the resulting forecasts

are more biased (Kolassa, 2016; Katsikopoulos and Syntetos, 2016; Spiegelhalter, 2019).

While complex approaches such as AI and data-driven systems enable and enhance our

comprehension of complex situations, they can also maintain prejudices or incorrect

assumptions, and can therefore produce unjust results (Napolitano, 2018; Solberg and

Maryilka, 2019). Prejudice in machine-based decisions can be very hard to detect because

the reasons for conclusions are hidden among many subtle trade-offs and thousands of

considered factors rather than articulated in clear rules (Brynjolfsson and McAfee, 2018).

Machine learning and AI are black box technologies and use mathematical-statistical

pattern processing principles of neural networks. Such methods do not come up with the

idea of looking for patterns on their own; these processes need to be set up by specialists

and then calibrated often at great expense (Wimmer, 2019). The researcher would argue

that AI only finds hidden patterns and behaviours repeating themselves that the ‘AI

system’ already knows about. The context of the data is often forgotten (Garrette, Phelps

and Sibony, 2018), and in this way an analysis of the underlying factors might go astray,

as context might change the outcome, or the wrong data is examined to begin with.

31

Many events that can be anticipated by human intuition and understanding of human

behaviour cannot be predicted by logic, and computers cannot duplicate such predictions.

Intuition is not data. Humans can incorporate perceived knowledge, a vague hunch and

suitable analogies (Meier, 2019) into a PM, which can thus beat computer-based

predictions, especially in the area of novel or unknown things, and will continue to do so

for the foreseeable future.

PMs are not necessarily as good as they appear, however. One yardstick used to measure

them in various academic treatises, namely election polls, is potentially flawed (Erikson

and Wlezien, 2008; 2012; Duquette et al., 2014; Lopez et al., 2017). It was claimed that

electoral predictions can be used for accuracy comparison, an example without losing

generality (Schoen et al., 2013) and with less statistical errors (Snowberg, Wolfers and

Zitzewitz, 2013). But, sixty-five weeks prior to the nominating convention for instance,

presidential primary polls were more wrong than right (Kennedy et al., 2018) and it was

easy to beat them with a PM. Duquette et al. (2014, p.358, researcher’s emphasis) found

that “election or voting market prices add nothing to the information that is offered by

scientific polling (i.e. candidate prices follow the polls)”; at best PMs are more accurate

than polls but still strongly influenced by them and on average closer to the polls than to

the actual election results (Calipha and Venezia, 2021). PMs might be able to outperform

polls, as these go spectacularly wrong on a regular basis, as shown by Hansen,

Klemmensen and Serritzlew (2019) for election periods between 2008 and 2016 or e.g.

in the recent US presidential elections in 2020 (The Economist, 2020). According to Horn

and Ivens (2015), a number of scholars demonstrated the predominance of political

markets, i.e. PMs, to predict political events over alternative prediction methods. They

put PMs ‘on top’ without critiquing them (Shen and Winstanley, 2019).

Casting a light at the inner workings and principles that underpin collective intelligence

next might help users to further understand whether to choose a PM.

For this type of cognisance the researcher relied on the research from Sibony (2020a;

2020b; 2021; cf. Kahneman, Lovallo and Sibony, 2019; Garrette, Phelps and Sibony,

2018; Gilboa, Rouziou and Sibony, 2018; Lovallo and Sibony, 2006; 2010) throughout

this thesis. Professor Sibony, whom the researcher also interviewed in 2013 and 2017,

centres his research on the effect of heuristics and biases in strategic decision-making and

procedures to improve the quality of decisions; Sibony’s deliberations helped the

researcher to gain valuable insights with a focus on intra-company stumbling blocks for

CPMs.

32

Another influence came from Gloor (2006; 2011; 2017; cf. Gloor et al., 2016; 2020), a

research scientist at the MIT Centre for Collective Intelligence who upholds the adage

‘Communicate, Collaborate, Innovate’ and conducts research on creativity from social

networks be it for forecasting or innovation. This heralded insights for the researcher on

collective intelligence, sophisticated data analysis, and innovation and on the importance

of these topics.

2.3 Principles of PMs

To overcome biases and group pressure, it is helpful to introduce anonymity, diversity,

and opportunity for dissent into forecasting processes as implemented in Delphi

forecasting. Because traders in a PM are independent of each other and do not know one

another’s input, PMs also allow for diverse voices.

Managers saw that hierarchy had a clear impact on what employees are willing to share

(Hosseini et al., 2015). Such employee reservations make a case for the anonymity that

PMs provide, including freedom from the influence of others or the corporate power

structure. When people feel free to say what they know, group decisions get better.

According to Sunstein and Hastie (2015), problems that afflict group decisions are largely

eliminated in structured approaches like PMs.

In PMs, participants buy and sell contracts anonymously, encouraging individuals to

reveal their personal viewpoints. The PM then assembles the previously dispersed

information, and the resulting share value represents traders’ collective views on the

outcome of a future event (Luckner et al., 2012). The true performance of PMs depends

on various factors; for the aggregation to really outperform experts, poor performers need

to be eliminated from the crowd (Budescu and Chen, 2015). Population size contributes

to collective accuracy and leads to a varying amount of bias (Economo, Hong and Page,

2016). Experts advised balancing wished-for forecasting performance with the allowance

for biases, like the favourite-longshot bias (Štrumbelj, 2016; Angelini, De Angelis and

Singleton, 2020), in a PM set-up.

2.3.1 Theory of PMs Based on Three Pillars Group deliberation, as Sunstein (2006) put it, often does not bear down to truth. Individual

errors can be magnified instead of being cleansed, leading to groupthink. If a group of

individuals wants to reach agreement on a controversial issue, pressure from equals and

managers leads individuals to change their dissenting opinions and, perhaps equally

importantly, not to share knowledge of conflicting information (Solomon, 2006).

33

In that way, groups often amplify the mistakes of their members rather than diminish

them, a feature of groupthink (Sunstein and Hastie, 2015) and research from HEC Paris

recognised that in group settings self-censure of the most creative ideas occurs and

divergent opinions, the emergence of differing options, are often suppressed (Sibony,

2021). The rejection of minority views can have a significant impact on the beliefs and

attitudes of the majority and thus on group decisions (De Dreu, 2007). Crowd-wisdom is

helped when judgements in groups systematically differ as much as possible. Even if

judgements of individuals are biased and correlated, the simple ‘averaging’ of the crowd

would be more accurate than one individual’s view (Davis-Stober et al., 2014; Dietrich

and Spiekermann, 2021). The average or majority opinion of a group emerges as

surprisingly accurate when everyone is included, and social influences are avoided.

Groups use information very efficiently, and if groups are dissimilar and their judgements

independent, there is a lot of it to be had (Sunstein and Hastie, 2015). The three main

pillars of PMs, namely independence, diversity, and anonymity, produce such positive

conditions embedded in the market.

To create momentum in the co-creation of new products, diversity in thinking and in the

backgrounds of participants are advantageous (Lewrick, Link and Leifer, 2017). Current

planning and management paradigms often do not react suitably to changes in the

environment. Other forms of collaboration and different mind-sets are needed to

creatively anticipate future events or new products and services (Rigby, Elk and Berez,

2020; Bambauer-Sachse and Helbling, 2021), like PMs which can be set up in a variety

of ways. For example, they can be in-house corporate prediction tools or publicly

available PMs, such as from PredictIt or CrowdWorx.

2.3.2 Pertinence of PMs in the Context of Strategic Planning

Biases are eliminated and diversity introduced plus views elicited ‘without fear’ due to

anonymity – hence constructively – via the use of PMs. Hall (2010) stated that PMs are

especially practical as forecasting tools that prevent traders from directly passing on the

information underlying their convictions which keeps out partialness.

It is important to introduce PMs to the decision-making process based on strategic

management as a discipline, according to Wooldridge (2011); management theories still

command a large audience among managers. As part of the management machinery,

project management and strategic planning represent the greatest ingenuity of human

beings according to Hamel (2007) and these two management tools could be improved

by PMs.

34

Bain & Company has been conducting regular surveys on leading management tools since

1993. The tools assayed are more high-level than forecasting but comprise topics which

either deal with predictions or have the remit to explore the future and enforce the

importance of predictions; the surveys do not mention PMs at all (Rigby, 2017; Rigby

and Bilodeau, 2018). A comprehensive study by Bloom et al. (2012) exhibited

management practices as a significant reason for the differences in productivity between

enterprises and countries. Management practice is closely linked to better business

performance in terms of productivity, return on investment, Tobin’s Q and revenue

growth (Bloom et al., 2005). The techniques of good management are widely known and

publicly available (Bloom et al., 2007), so improved performance could also stem from

the sort of management techniques taught in business schools (Wooldridge, 2011),

potentially including PMs, scenario planning, the Delphi method, and other tools included

in the curricula.

Projected futures need to be assigned a likelihood to be valuable (Borison and Hamm,

2010) and there would be a benefit from expressing the prospect of risk or probability of

the scenarios, which a PM would do through its price finding.

2.4 Traits and Attributes of PMs 2.4.1 Advantages and Disadvantages of PMs

Decision-making based on cooperative intelligence produces results that a person

working independently cannot. However, problems of coordination and motivation

hinder collaboration; groupthink might be unavoidable (Hackman, 2002), and companies

often lack effective problem-solving processes for group decisions (Lefort, McMurray

and Tesvic, 2015). PMs – which could be used in this context – do not always seem to

work as intended. In analysing the online PM Intrade, Woerle (2013) found no support

for strong efficiency and concluded that the market was not conclusively aggregating

private information. Furthermore, if the traded shares do not adequately represent the

questions asked, a PM can produce misleading results (Thicke, 2017).

Irrespective of their primary purpose, speculative market prices can nevertheless be

generally fairly accurate estimates of future events because they can aggregate large

quantities of obtainable information. For instance, ‘Orange juice futures’ forecast the

‘elements’ better than the US government, and betting markets give better predictions of

horse races than experts (Beneš, 2017).

35

PMs can be useful for capturing conditional estimates or considerations for a decision to

be made, and these evaluations can provide the basis for complex decisions in an

organisation. A PM provides information in the form of a stock price, which not only

represents an average assessment by market participants, but also their level of trust in

their forecasts. Also, because money is at stake (real, virtual or ‘reputational’), PMs

incentivise users to both gather relevant information and give truthful revelations. PMs

can thus be immensely powerful (ibid.) and their aggregation of information could be

called bias neutralisation. PMs lessen the frequency and severity of mistakes caused by

incomplete information by averaging it with other potentially patchy information.

They also reveal the beliefs of the participants without identifying the participants

themselves, and without identifying the information underlying those beliefs, which are

the keys of anonymisation – the ability to remove information from source-identifying

traits (Hall, 2010).

Thirty-six years ago, Gemünden and Hauschildt (1985) questioned approaches that would

negate or condense alternatives by compressing futures into a single price signal. They

conducted a longitudinal study analysing decision-making processes at the executive

board level and the importance of information searching that leads to decision results. In

this context Nutt (2007) saw that the most successful informational probes employed

intelligence gathering tactics akin to scenarios or group discussions with a focus on

alternatives, highlighting that in organisations the traditional forecasting and decision-

making processes are not only applied but even confirmed by research as superior.

Research indicated that the development of alternatives is an important aspect for intricate

decision-making problems (Pluchinotta et al., 2019). Implementing alternatives in a PM

would lead either to a complex set-up of stocks to be traded in a market or directly to the

use of scenario techniques because decisions with two alternatives, i.e. based on binary

options like ‘yes’ and ‘no’ or word pairs similarly expressing the affirmative and the

negative respectively, are simply too easy (Gemünden and Hauschildt, 1985).

The design and use of alternatives demand more creativity than the search for

information. Creativity is a decisive factor of well-rounded decisions, and creativity is

not adequately captured in a PM, in which participants decide for or against a traded

stock, setting a price, and nothing otherwise.

36

An objection to equating the PM price to a probability estimate is that it infers the

probability to be an impartial fact, not the subjective assessment that it is. PM prices

represent only estimates, opinions and nothing else (Tetlock and Gardner, 2015).

Particularly in complex decision-making processes – and strategic planning has complex

questions in abundance (Bingham, 2013) – the traded shares set up in a PM will fail to

fully encompass the inherent intricacy and lack the needed precision.

The inherent probability approach can be confusing for the users, and intricate decisions

might be better addressed through questionnaires or Delphi. PMs are recommended for

less complex and comparatively general issues (ibid.).

Dietl et al. (2004) posited that interviews/questioning are superior to PMs, in particular if

the information sought is specific, the questions are complex, or if the frequency of

needed information is high. Research concluded that in such circumstances experts are a

better alternative to PMs (Metcalf, Askay and Rosenberg, 2019).

2.4.2 The Three Pillars – Revisited

The alleged superior accuracy of PMs has become almost universally believed, at least in

(academic) publications, and so receives perhaps too little scrutiny, as does their overall

usefulness to companies. In research circles it was reported that CPM use, together with

social forecasting tools in general, are on the rise, “with 2010 being a breakthrough year

[for these tools]” (Ivanov, 2012, p.2) or expressed differently, rather colloquially as “the

methodology frog jumped to the fore [when mapping this toolset’s success]” (Phillips and

Linstone, 2016, p.161). However, argued in this thesis is that the academic literature

‘erred’, the contrary would be true as covered later in section 2.5 on the business use of

PMs. It seems to be fitting then, that the three PM pillars, anonymity, diversity, and

independence are not as unshakable as they are presented to be. Although PMs find

answers to a multitude of real-life problems, they need additional research to assure users

that they can fulfil their promise of diversity, independence, decentralisation, and

aggregation and can be proper determinants of collective wisdom (Nguyen et al., 2019).

Two essential factors influencing PMs acceptance have been neglected almost altogether

in academia and academic publications, giving a potentially wrong view on the overall

usefulness of PMs. These elements are management acceptance and company hierarchy,

both cultural aspects to be covered later in this thesis.

37

Researchers had recognised that when partly abandoning the anonymity principle of a

PM and allowing intermediate levels of informational signals between the participants,

but not maximal ones, produces higher levels of collective intelligence in an information

exchange (Kao and Couzin, 2014). Other research is conflicting. Some studies found that

information sharing diminishes PM accuracy (Lorenz et al., 2011; Graefe, 2011), while

others noted improvements (Farrell, 2011; Gürçay, Mellers and Baron, 2015; Vaughan

Williams and Reade, 2016).

The way that information is exchanged may be important. For example, intermittent

information exchange may improve market accuracy, according to a recent study that

found that (intermittent) collaborators arrived at a better average solution than groups

acting individually and independently (Bernstein, Shore and Lazer, 2018). Supported by

experiments regarding forecasting accuracy Becker, Brackbill and Centola (2017) argued

that ‘social influence’ in centralised or decentralised social networks would decrease

errors rather than forecasting quality, as claimed by Lorenz et al. (2011, p.9020) who

suggested that “social influence can undermine … [a PM’s wisdom-of-crowd] effect”.

How companies implement a PM impacts the quality of the results. Whether and how

employees are encouraged to interact alters the exactness and validity of the trading

results. Kao and Couzin (2014, abstract) highlighted in the title of their paper that

‘decision accuracy in complex environments is often maximized by small group sizes’: It

is not uncommon to find that limited, or small, groups that do not draw on crowd-wisdom

can increase the quality of a decision, as the wisdom-of-crowds phenomenon – where

collective accuracy increases in proportion to group size – tends to be rare. Large groups

are particularly vulnerable to the detrimental consequences of correlated data, even if only

a small part uses them. Furthermore, the way information is shared can influence the

ensuing accuracy of decisions and can often weaken the collective intelligence by

increasing confidence in collective decision-making without enhancing its exactness

(ibid.).

Each assessment in a decision-making process can be seen as having two parts: an

informational part and errors. Intuitively, if deviations from the basic truth are unbiased

and independent, averaging will largely counterbalance the errors. The diversity of

experience is assumed to prevent the phenomenon of ‘groupthink’ (Goldstein, McAfee

and Suri, 2014).

38

Research showed that it is better to follow the opinion of a highly qualified manager who

ranks among the top 30 of his colleagues than to aggregate information from a selection

of 30 randomly chosen individuals, although the mean of the top 100 executive scores is

inferior to that of the beliefs of 100 haphazardly selected individuals. Research showed

further that the wisdom of small, clever crowds can far exceed the quality of random

crowds. Smarter, smaller crowds can be found within a larger crowd by grouping together

only those with higher predicted abilities (ibid.). This research indicated that diversity

does not automatically lead to better market predictions, as is claimed so often (cf. Hartley

and Hassett, 2016).

The second pillar does not stand universally. According to Goldstein, McAfee and Suri

(2014), when expertise is not evenly distributed among a crowd, focusing on

concentrating expertise rather than adulterating it with lower quality experts tends to be

the better approach. Subsequently, experts contained in the crowd can demonstrate a

higher degree of wisdom than the crowd in its entirety – optimally one could disregard

information from less experienced managers or attribute a lesser value to it. As individual

human performance records become increasingly available, so too will opportunities to

enhance crowd prediction by detecting and utilising the wisdom of expert subsets of the

crowd. Research suggested that it may be prudent to limit participation in PMs to those

with demonstrated knowledge in the field to be predicted (ibid.). A lack of experts is

frequently identified as a challenge for a PM, however (Brigis, 2008; Graefe, 2008;

Bousoño-Calzón et al., 2018). Even the accuracy of PMs – their hallmark advantage –

has been called into question in experiments showing forecasts from PMs to be quite

deficient (Bousoño-Calzón et al., 2018).

The gathering of experts in meetings carries the risk that the validity of the consensus

opinion will be undermined by majority views and perceived pressure from peers and

superiors (Armstrong, 2006). PMs though create a (monetary) incentive for individuals

who have information to ‘report’ it and participate in the market. On the one hand this

selects unbiased predictors, but it also conveys new facts to the market instead of just

getting better predictions from existing information. New information tends to be

incorporated quickly and independently, too (Ozimek, 2014).

As the aim of a PM is to elicit the average of the crowd’s estimates, knowledge about that

target would drive – or ‘herd’ – trader behaviour towards the current, potentially

incorrect, value. Such convergence suppresses an individual’s real information.

39

In other words, herding occurs when traders choose to follow others’ judgements instead

of following their own convictions and observed facts. And people who are aware of the

viewpoints of others hesitate to look stupid if their view differs from that of the majority

(Galinsky and Schweitzer, 2015). Herding diminishes the wisdom-of-crowds effect in

cases where accuracy relies on the guesses being independent (Berg and Rietz, 2019).

Experiments showed that improvements of the collective error did not occur when social

influence impacted the crowd’s diversity (Lorenz et al., 2011; Mavrodiev and Schweitzer,

2021). The viewpoints of traders become much more similar due to direct links in social

networks (Takac, Hinz and Spann, 2011), which eliminates diversity and independency.

All three pillars show clear weaknesses that call into question the acceptance and even

usefulness of the results of PMs. The pillars are even more imperilled if a market is not

easy and appealing to use (Schrieber, 2004), this is mainly due to overly complex user

interfaces (ibid.). Taherdoost (2018) posited an acceptance model that beheld that fun and

convenience show direct and indirect influence on the adoption of a technology. Between

these intrinsic (‘fun’) and extrinsic (‘convenience’) motivators, convenience is more

important as it influences adoption to a higher degree than fun. The fact that mostly

dissatisfaction was noticed, even clear disinterest in deploying a PM (Bughin, Manyika

and Miller, 2008; Bughin, Chui and Harrysson, 2015), fits with Bandura’s Social

Cognitive Theory, which put forward that people are driven by external factors rather

than internal forces, with environmental factors representing external situational

influences (The World Bank, 2010), leading to seeing extraneous influences as more

important.

The research discussed in this section suggests that implementation plays a role in the

acceptance or rejection of a new tool. It also highlights the questionable assumptions

underlying PMs, which would require a culture of openness to be overcome.

Top executives recognise that company culture can be a challenge to the implementation

of PMs (Gässler, 2010). Companies with corporate cultures that focus on leadership,

employees, customers, and owners have increased their performance markedly compared

to peer companies with a hierarchical rather than collaborative culture which show low

to mediocre success (Flamholtz, 2001; Kotter and Heskett, 2011).

40

In their 2020 almanac on the future of IT, the International Data Group explicitly assumed

that companies will only be successful in the next five years if they follow the equation

‘implementation plus strategy plus culture equals success’ (Salzsieder, 2019). In this

equation, swarm (i.e. crowd) intelligence is considered crucial, as is leadership by

example, and meeting eye-to-eye with employees, which prohibits strong hierarchical

leadership styles (ibid.). An organisation’s culture is thus the key to its success but the

notion of leaving the locus of control with the team member, runs counter to many

organisations’ need for control. The desire to exercise control becomes a desire for

conformity, and such conformity threatens individuality (Buckingham and Goodall,

2019), potentially stifling one of the prerequisites for effective PMs. The top-down

approach does not give employees the freedom to contribute to the decision-making

process (Hlupic, 2018; Krekel, Ward and De Neve, 2019).

Due to demands on executive time and hierarchical structures, it is rare to find teams

contributing to collective intelligence in the upper management levels of companies

(Katzenbach and Smith, 2015). Such teams are also a success factor for the relatively new

phenomenon of agile development, a set of practices designed to condense development

life cycles and to constantly deliver high quality software. When responding to changes,

organisations face hurdles like constrained, regimented, and inflexible ways of working.

They need to adjust to the versatility, independence, and forthrightness of working in

teams (Higgins et al., 2019). This inevitable cultural change might clash with existing

company structures, as agile working requires development without hierarchical

boundaries (Vayghan et al., 2018), but would be beneficial for deploying CPMs as well.

Comparing the classical top-down management approach with a bottom-up consensus

model, Ciasullo et al. (2017) determined that, on average, teams in a bottom-up approach

performed better across a number of performance metrics than teams in a traditional top-

down management approach.

Performance-enhancing conduct by leaders fostered a sense of empowerment amongst

team members, not a hierarchical approach. Individual characteristics are usually stable

over time (The World Bank, 2010). It is therefore not easy to change or challenge the

power dynamics stemming from hierarchical structures because such dynamics become

embedded as personal traits in those who derive advantage from them.

41

This rigidity can be detrimental to leaders, as the chance of top executives becoming

inundated by even valuable insights grows with the prevalence of forecasts. Here, the

solution would be to democratise information by encouraging business units and

functions below the top echelons to make more and better decisions themselves

(Dewhurst and Willmott, 2014). Even though a transition away from such debilitating

structures could make the use of PMs more practical in business enterprises, attempts to

remove cultural barriers often fail. When cognition, social determinants, and personal

convictions affect an established behaviour, changes do not occur. Even when companies

have an intention to change, a small variance, a slight hesitance, may prevent the desired

change (The World Bank, 2010; Taherdoost, 2018).

Last but not least, following a model of technology adoption, the motivation of a user

stems from the two main beliefs of considered usefulness and ease-of-use which have a

significant impact on one’s inclination towards using a product (Taherdoost, 2018). PMs

are often not easy to use, do not have user-friendly interfaces, and are complex and thus

difficult to understand. Furthermore, hierarchy prevents a positive attitude toward the tool

leading to results not being fully accepted. Economists claim that big companies execute

many planning exercises, and that, from a planning perspective, firms are and will remain

highly hierarchical (Phillips and Rozworski, 2019), preserving this particular stumbling

block for PMs.

Rather than attempt to engineer the cultural shifts necessary for the acceptance and use

of PMs, one could abandon prediction altogether and work towards control instead. Taleb

(2012) had suggested to use a non-predictive approach in a way that is resistant to

interference – i.e., resilient to changes in future outcomes. The essential idea is that the

best manner to foresee the future is to devise it (Drucker, 2013) using a non-predictive

approach that is invulnerable to interferences and therefore robust to changes in future

outcomes. PMs are not robust in this way. They need to be refined on a regular basis.

Under most circumstances, PMs embedded in social networks had a better forecasting

performance than non-networked ones according to Qiu, Rui and Whinston (2013; 2014).

42

2.4.3 Alternative Approaches – Polls, Scenarios, and Non-Predictive Action

The inducements for exchanging information are considered better in prediction polls

than in PMs. Getting to all the information might pose a challenge in a PM as the market

is a zero-sum game: if one trader wins, another one loses (Atanasov et al., 2017). There

is no incentive to reveal beliefs and rationales to one another, and the market structure

precludes this, even if traders believe that sharing facts with someone else who knows

more would lead to an overall more informative picture. Sharing of information in

prediction polls works well in small settings. Polling a single person may be useful, while

a market with one participant is problematic.

The average of forecasts of even a few people provides reasonable results but PMs

experience liquidity problems (ibid.) when they have a limited number of participants.

Prediction polls also perform well in situations where managers need to gather forecasts

but have reasons to steer clear of giving consolidated predictions back to participants

(ibid.).

Still, PMs can provide accurate predictions of the likelihood of an event by summarising

traders’ input. Formal studies showed, however, that the market participants’ risk

attitudes can distort market equilibrium prices, thus making PM forecasting unreliable

(cf. Boulu-Reshef et al., 2016). Such unreliability can also result from revisions made

during trading, which tend towards significantly worse forecasting performance

(Singleton, Reade and Brown, 2019). Perhaps for that reason, businesspeople habitually

employ prediction- and control-based strategies to diminish incertitude and risk.

The former emphasise collecting intelligence to arrive at an estimation, i.e. prediction, of

prospective results while the latter focus on taking action to fashion circumstances which

are more beneficial for a business, i.e. controlling the outcome (Kuechle, Boulu-Reshef

and Carr, 2016). This approach is similar to approaches like Strategic Foresight where

one creates or shapes the future (Lewrick, Link and Leifer, 2017) rather than predicting

it. Being able to afford complex strategies worked in uncomplicated business landscapes

but simplification is now called for as we face more and more complex situations,

defining business direction with a few easy-to-understand rules without restricting it

(Eisenhardt and Sull, 2001). In such a way, business embraces control and action over

prediction.

43

But also, to accommodate forces that can have a considerable influence on one’s business

fortunes, i.e. in a precarious environment, the benefits of considering scenarios are clear-

cut (Hirt, Laczkowski and Mysore, 2019; Staples, 2020). When a single scenario is not

sufficient, it is necessary to develop plans positing several different tenable ways forward

to draw attention to the underlying uncertainty drivers (Dye, Sibony and Viguerie, 2009).

Whether based on scenarios or using collective intelligence, ‘control over prediction’ is

emerging as a theme in forecasting circles; instead of envisaging the future, using non-

predictive strategies or antifragile approaches (cf. Wiltbank et al., 2006; Taleb, 2012).

This could be in part because the future is not known but overall directions can be

conjectured and treated reasonably. Such ‘foresight’ moves beyond prediction and

includes aspects of networking and decision preparation regarding future facets, in this

way foresight (Cuhls, 2003) goes far beyond a mere prediction ansatz.

2.4.4 How Do Benefits and Drawbacks Balance?

The primary advantage of a PM is to induce a diverse community to disclose their private

information. The economic incentive present through the market mechanism drives

traders to look for the best facts available (Brown, Reade and Vaughan Williams, 2019;

Auld and Linton, 2019) and they alleviate many of the distortions that infect group

deliberations. Still, PMs can be affected by various trader biases (Restocchi, McGroarty

and Gerding, 2018; Berg and Rietz, 2019; Grant, Johnstone and Kwon, 2019). On balance

PM research sees both advantages and disadvantages as summarised in Table 2-2.

Table 2-2 PMs’ Strengths and Weaknesses (Developed for Research from Kloker et al., 2019)

PM Research – Selected Benefits and Limitations

Strengths Weaknesses

Fast reaction to and incorporation of new information with immediate feedback on one’s own estimation Complexity (in implementation and use)

Continuous forecasting possible Misses qualitative and background information

Robust against distortions Questions (i.e. traded stocks) cannot easily be changed when new insights arrive

Avoiding conformity in opinion forming Low to no visibility for alternative points

Gamification as motivating factor Long-time horizons not easily possible

Going one step further, Table 2-3 outlines the main favourable and unfavourable points

of PMs covered so far in the thesis, suggesting an unclear picture slightly skewed towards

the negative.

44

Table 2-3 Balancing Arguments Pro and Con of PMs (Developed for Research)

Comparing Positives and Negatives

Favourable Unfavourable

Accuracy Acceptance problems – trust issues in hierarchical organisations

Aggregation of varying beliefs and information Biases occur regularly

Anonymity Communication / information exchange deemed important, however harmful to accurate price forming

Bias reduction Crowd-wisdom not deployed on transformative or strategic topics

Conditional estimates possible Lacking a view of alternatives / scenarios – information too condensed

Effective information aggregation Liquidity level needed for successful trading doubtful

Encourages to seek new information, brings new information to the market Mispricing

Incentivises truthful revelations Not considered mainstream yet

Prices represent confidence levels, an additional informational feature

Other methods / alternatives seen as superior; PM shares cannot adequately reflect that

Preference towards non-predictive approaches – ‘future-forming’

Price / result adjustments and interpretation / improvements needed

Price as a probability hard to interpret

Suitability as a business tool called into question

Theoretical underpinnings unclear

Some unfavourable issues could be mitigated and there are enough favourable attributes,

so the evidence still supports the value of PMs for forecasting. Nevertheless, unfavourable

perceptions of PMs are obstacles to implementation. There are successful PM

deployments, like at Deutsche Telekom. Still, some wonder if the rewards are worth the

effort. Various PM providers (including Crowdworx, forming the basis for the market at

Deutsche Telekom) are moving away from the pure PM concept to a more inclusive and

manageable voting platform (Ivanov, 2013).

Also, an important tenet of a successful market, anonymity, runs contrary to the fact that

in decision-making a lot of reasoning is devoted to affirming the identity of the group one

belongs to and an individual’s position within it. This is not necessarily done to help

individuals to make better decisions but rather to help cooperation, according to Mercier

and Sperber (2017).

No clear picture has emerged yet as to the overall business suitability of PMs.

45

Alternatives like self-reported beliefs (the expression or explanation of an attitude or

belief on a numerical scale), seemed to be more useful than PMs. For many psychologists

it is a revelation how precisely market prices can estimate probabilities for a number of

occasions from different areas. Many economists may now be astounded by how

instructive it is to ‘simply ask’ (Dana et al., 2019).

Additional exploration was needed and is provided in this thesis.

2.4.5 Implementation at Scale and Proper Presentation to Management

Researchers looked at how, in the context of management decision-making, different PM

designs led to better predictions and hence decisions. A setup that incorporated advice

and interaction from management united the inducements for traders and decision-makers

in such a way that the market performed adequately and was useful for the company

(Dianat and Siemroth, 2020). In decision-making the way an idea is expressed is as

important as the insight that underpins it according to Dewey and Brown (2021). Thus,

adequate communication is required for judgemental forecasting to enhance confidence

in and reduce disconnectedness from the results amongst users (Hyndman and

Athanasopoulos, 2018; Berinato, 2019). However, a communication initiative detailing

contextual information like uncertainty and risk, as suggested by Finney (2019) or

Spiegelhalter (2019), would not be needed with a PM.

By its very nature, a PM delivers stock prices that directly represent probabilities (Wolfers

and Zitzewitz, 2006) and hence uncertainty and risk. Without clear communication of

constraints and levels of confidence present in a prediction, the field of forecasting will

lose tenability (Orrell and McSharry, 2009). To mitigate this, Teschner and Rothschild

(2013) suggested inputting confidence levels through a web-based interface, as

CrowdWorx, purveyors of a so-called Social Forecasting platform, have done, to improve

performance and acceptance of a PM (Ivanov, 2013). However, in deliberations from

Finney (2019) the forecasting method suggested was a scenario approach. Perhaps more

importantly, the manner in which a forecasting result is presented to management is

deemed to be significant and whether PMs or Big Data are seen as a valuable forecasting

tool, corporate culture plays a major role in the implementation success (Hölzle et al.,

2017). Only between 8% and 10% of predictive solutions unlocked any value for a

company (Fleming et al., 2018).

46

In the context of traits and attributes of PMs – this section’s heading – looking at seminal

papers on PMs by the editor of the Journal of Prediction Markets Vaughan Williams

(2011; cf. Vaughan Williams and Reade, 2016; Brown, Reade and Vaughan Williams,

2019; Reade and Vaughan Williams, 2019) and the associate editor for Foresight – The

International Journal of Applied Forecasting, Graefe (2008; 2009; 2010; 2011; 2019),

helped the researcher to understand the value of forecasting combinations but also to

detect a first inkling of the possible importance and future success of CPMs in the area of

innovation management (cf. Armstrong, Du, Green and Graefe, 2016; Marinovic,

Ottaviani and Sørensen, 2011, in a book on PMs which Vaughan Williams edited).

Nevertheless, observing publications from both these academics, who were three of the

five authors the researcher relied on most, also raised the researcher’s concerns that within

academia PMs in general were pretty much not looked on as a worthwhile tool for a firm’s

forecasting processes. For example, in the case of Graefe (2017; 2018) they were seen

more as a successful tool in election forecasting, an area the researcher established as a

weak benchmark for PMs (cf. section 2.2.6).

2.5 Business Use of PMs

The efficacy of PMs was a theme of a special section comprising eleven papers published

by the International Journal of Forecasting in 2019. Those papers lauded their precision,

accuracy, and information formation.

They proved to be outstanding prediction tools and outperformed experts and polls in

numerous instances (Brown, Reade and Vaughan Williams, 2019; Brown and Yang,

2019; Reade and Vaughan Williams, 2019; Strijbis and Arnesen, 2019). They were found

superior to traditional forecasting methods when information is scattered and on par with

them in settings where information is homogenous (Pennings, van Dalen and Rook,

2019).

PMs have the potential to place uncertain decisions on a better footing. In particular, their

theoretical basis, observational evidence and plausibility were considered being superior

to other methods (Borison and Hamm, 2010; Nayak, Misra and Behera, 2019). Despite

the admiration for PMs among economists, policymakers and enterprises, it has taken a

long time for them to be even minimally employed in decision-making (Cowgill and

Zitzewitz, 2015). In the decade leading up to 2006, interest in PMs increased in both the

private and public domain, driven by the hope that they would prove beneficial for

forecasting, decision-making, and risk management (Wolfers and Zitzewitz, 2006).

47

According to Horn and Ivens (2015) nothing had changed ten years later; PMs were not

yet generally known in 2015 despite their ability to organise scattered knowledge

efficiently and produce accurate forecasts. When integrating markets into existing

companies for forecasting purposes (ibid.), problems stemming from cultural issues and

motivation arise. After approaching several hundred companies, Rieg and Schoder (2011)

found very few successful corporate PMs and only a few companies interested in them.

In McKinsey’s annually conducted research rating Web 2.0 technologies, PMs

consistently ranked at the very bottom of overall adoption in the corporate world,

plateauing at 8%. And, importantly, usage is falling (McKinsey & Company, 2013).

Figure 2.3 based on the McKinsey data shows that, from 2011 to 2013, the use of CPMs

based on normalised actual numbers fell by 30% from its peak.

Figure 2.3 Usage of CPMs until 2013 (Developed by the Researcher from McKinsey & Company, 2013)

Based on his experience, a former director at McKinsey considered even these

percentages too high; he believed that some companies might have claimed PM adoption

when in reality they might have misunderstood the term (Sibony, 2013).

Such misunderstandings might be based on a lack of knowledge about PMs. In a 2013

study of 130 German companies (three quarters of which were large enterprises), this

crowdsourcing approach was unknown to 21.1% of questionnaire recipients and 45.1%

had heard the term PM but were not sure of its precise definition. These two categories

account for 66.2% of study respondents (Wagner, 2013), lending support to the

classification of CPMs as ‘fairly’ unknown.

100

71

191

250238

175

6%

5%

7%

7%

8% 8%

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

0

50

100

150

200

250

300

2008 2009 2010 2011 2012 2013

PM usage normalised, 2008=100

PM usage in percent

48

There is little reason to believe that there will be a surge in the use of corporate PMs.

Even in 2015, enterprise adoption of Web 2.0 tools seemed to be slowing down after a

decade of growth (Bughin, 2015). According to research from McKinsey on the overall

use of social technologies, not a single Web 2.0 technology had been adopted into

mainstream use (ibid.) but rather tapered off in actual numbers, as depicted below.

Figure 2.4 Corporate Use of Web 2.0 / Enterprise 2.0 Technologies, Percentages – Graph on Top – and Actual Numbers – Second Graph (Developed by the Researcher from McKinsey & Company, 2013)

Enterprise 2.0 tools require sustained funding and management discipline – which are

not always available – to drive integration forward (ibid.). The McKinsey investigations

also looked explicitly at the apparent impact of these aids on revenue and operating

expenses. The self-reported answers had been aggregated into a gauge of the company’s

perceived value creation. It was found that social networks, wikis, and blogs, the tools

with the highest usage rates, resulted in a self-reported value creation of a minimum of

5% apiece, while the impact of other social technologies was much lower (ibid.).

0%

10%

20%

30%

40%

50%

60%

70%

2007 2008 2009 2010 2011 2012 2013

Blogs Video Sharing Wikis

Podcasts Social Networking Prediction Markets

0

250

500

750

1.000

1.250

1.500

1.750

2.000

2007 2008 2009 2010 2011 2012 2013

Blogs Video Sharing Wikis

Podcasts Social Networking Prediction Markets

49

As PMs feature amongst those, the question arises why to use them with an average value

added of less than half a percentage point – even with 100% penetration within an

enterprise, the increase in value added is only 0.6% (see Table 2-4)!

Table 2-4 Diffusion of Social Technologies within Companies – Increase in Value-add in Percent per Level of Penetration (‘2007–15 McKinsey survey of 1,500 companies’; Bughin, 2015, researcher’s emphasis)

100% Penetration 75% Penetration 50% Penetration 25% Penetration

Social Networking 5.8 3.4 1.7 0.6

Wikis 5.7 3.1 1.5 0.5

Blogs 5.0 2.6 1.3 0.4

Video-Sharing 1.4 1.0 0.6 0.3

PMs 0.6 0.4 0.3 0.1

Podcasts 0.6 0.4 0.2 0.1

The low usage and the lack of benefits of PMs had already been reported in 2009. The

technology was adopted by only 9% of companies, and more than half of those, 5%, used

the tool and reported no measurable benefits (Bughin, Chui and Miller, 2009). Even in

situations where people are interested in the results from PMs, the most persistent obstacle

to their adoption was convincing managers that their own professional value in no way

depends on being the smartest person present (Thompson, 2012).

Ensuring that analytics tools can be used by as many different types of frontline staff as

possible will improve performance comprehensively (Bughin, 2016a). This observation

was supported by a survey conducted by Gartner of CIOs across 360 businesses.

Crowdsourcing – the umbrella term for methods including PMs – had the highest average

performance across ten methodologies, yet still showed the lowest usage-percentage of

all tools considered at just slightly above 5% (Weldon, 2016). Utilisation of Big Data,

according to Weldon’s study, also decreased over the last three years: from 50% to 41%

to 39% in 2014, 2015 and 2016, respectively, where the percentages represent the share

of CIOs naming each topic as one of the top three areas for new IT spending (ibid.).

In researching leadership and organisational studies, Tourish (2019) concluded that the

lack of inclusion of practical approaches like PMs stemmed from their remoteness from

practical problems faced by managers and employees.

50

Further support for this realisation is their omission from the 9th edition of a book

describing forward-looking instruments (Rustler, 2019); it covers in excess of 50 tools

without once referring to PMs. This omission is pretty surprising in light of the fact that

the handbook is based on more than 50 years of scientific research on creativity and

innovation, and one of the quintessential drivers of the handbook is the importance of

tapping into the wisdom and creativity of a company’s employees. Moreover, idea

filtering – an area where PMs would feature (Hamel, 2012; Buckley and McDonagh,

2014) – is one of the main subjects covered by Rustler (2019), who sees it as an important

instrument for experts in judging future ideas as opposed to mere intuition in strategic

thinking and strategy development.

The researcher had ‘combed through’ a multitude of management literature which

mentioned approaches that could benefit from or be amended by PMs (cf. e.g.

Eschenbach, Eschenbach and Kunesch, 2008; Simon and Gathen, 2010; Kerth, Asum and

Stich, 2011; Andler, 2016; Krogerus and Tschäppeler, 2017; van Aerssen and Buchholz,

2018). But of those even the recent publications on management tools and forecasting

instruments neglected to mention PMs although 51% of the most influential, most cited

papers on PMs were published prior to 2003 (Klingert, 2017). Tapping the collective

knowledge of groups was nevertheless recognised as important, for example in the

context of innovation management (cf. Puccio, Murdock and Mance, 2005; Puccio,

2020). Thus, the conundrum realised in this thesis is that PMs as a good predictive tool

have been neglected but might emerged again.

With “innovation … [being] the only sustainable strategy for creating long-term value”

(Hamel, 2012, p.6, original emphasis), and the advice that innovation management has to

be an indispensable part of a company’s strategy for value creation (Alfaro et al., 2019),

CPMs might successfully re-emerge through new approaches like idea markets. Here,

upper management looking for fresh ideas might accept what they did not for CPMs,

namely: collective wisdom from lower rungs of the corporate ladder and results that run

counter to these managers’ own perceived knowledge.

Catching the zeitgeist already in 1997, the concept of disruptive innovation has stood the

test of time so far (Lepore, 2019) and hence this could bode well for idea/prediction

markets. Research from Harvard strongly indicated that with the potential of using

technology to predict human behaviour, tools that encourage to innovate disruptively over

incremental product improvement will gain traction (Lepore, 2014; 2020).

51

As employees can provide a potentially strong and diverse set of knowledge that can be

drawn on, recognition of this opportunity can open the way for capturing their creative

ideas with the help of supportive structures like a PM (O’Leary, 2019). Research from

Gatu (2020) underscored the importance O’Leary (2019; 2020) gave to innovation

management and crowdsourcing approaches, a prospective way forward for PMs

encapsulated also in section 1.2.1 and Figure 1.3.

In this context and in the overall thesis the researcher also depended on Bughin (2013;

2015; 2016a; 2016b; 2017; 2018) who oversees the McKinsey Global Institute,

McKinsey’s business and economics research arm. Bughin’s research delivered many

insights chiefly around the usage of social (forecasting) technologies and Big Data within

companies or the lack thereof in the business world and the reasons behind that.

2.6 Impact of Company Hierarchy and Management Acceptance

One of the stumbling blocks of CPMs is corporate hierarchy, however, and research

indicated it is not going away, even in this time of digital upheaval (Carnabuci and

Diószegi, 2015; Carnabuci, Emery and Brinberg, 2018; Everett, 2019). “As they grow

companies tend to become more and more unidimensional, less flexible in their internal

structures [and] driven … by the inevitable ossification of the top-down … bureaucratic

needs perceived as necessary for operating a … company in the modern era” (West, 2018,

p.33, emphasis added; cf. Daepp et al., 2015). Particularly digital start-ups would be

expected to change to a firm culture with a flat ‘pecking order’ and collaborative team

arrangements which is not happening as expected (Baars and Spicer, 2017) and perhaps

explained by the research from West (2018). Structural and management changes that

social tools were expected to bring about could flatten or even erase the formal hierarchy

of an organisation (Bughin, Chui and Harrysson, 2016a). But hierarchy is still common

in all complex organisations (Besanko et al., 2017) continuously impacting the structure

of organisations (Pfeffer, 2013; 2015) and its usefulness increases as companies grow.

Managers prefer for a company’s ‘doers’ to stay separated from their ‘thinkers’, even

though this often leads to a strategy that lacks realism and is difficult to implement (Judah

et al., 2016). The loss of decision-making power is unpleasant, even though people are

pretty bad at making predictions. Predictions and decisions are almost inextricably linked

(McAfee and Brynjolfsson, 2017) and people like having the power to decide. They are

therefore not necessarily happy or willing to cede it to a ‘betting mechanism’, like a PM,

even in exchange for better results.

52

Managers seem willing to give up financial rewards to maintain the ability to make their

own choices, presumably for psychological gain (Bobadilla-Suarez, Sunstein and Sharot,

2017). They are also afraid that biased manager decisions would be exposed as such by

the crowd (McAfee and Brynjolfsson, 2017).

Group leaders still aim to maximise power differences, even sacrificing group goals for

the sake of self-interest (Maner and Mead, 2010; Maner and Case, 2013). Where signals

are reported to senior management by employees that are less influenced by the existing

complex and intricate structure of past decisions, they tend to be discarded or treated as

unimportant if they threaten established managerial wisdom, managers often ignore the

results (Cappelli, 2019). Authoritative leadership also often fosters environments where

people frequently disregard information that conflicts with their view of the world. Some

beliefs are a public part of one’s identity, treasured in their own right, and contrasting

views need to be managed as a challenge through what Bénabou and Tirole (2016) called

motivated reasoning, a believer avoids information offering conflicting evidence.

Self-managing groups are difficult to monitor and control (Besanko et al., 2017), so there

is a deep-seated concern about subordinates influencing management decisions.

Executives favour the viewpoints of their management consultants: at least you can talk

to the consultants, you can’t ‘talk to the market’. The CIO of Jaguar Land Rover ranked

his internal employees third behind his fellow CIOs and behind consultants in providing

trustworthy predictions (Vincent, 2014). A senior manager at Allianz, a global financial

services company, confirmed that executives often trust the view of management

consultants more than advice from their own employees (Häßner, 2014). This

subsequently is a demotivating factor, talented employees will leave if the company is

not committed to benefit from insights (cf. Wegener and Sinha, 2013) , PMs’ participants

need to see their advice is heeded (Dye, 2013) – as generally putting trust in employees’

personal responsibility and willingness to perform is a key motivational factor (Sprenger,

2014).

Chlupsa (2013) noted that commonly a decision-making procedure is very structured and

conditional on defined processes and monitoring by hierarchical levels, and one’s sway

is based on internal ranking and expert knowledge (ibid.). Management occurs from top

to bottom and people are dissuaded from risk-taking (Machnig, 2016).

53

The so-called Generation Y or millennials, digital citizens with a clear grasp of things

like social media (The Economist, 2013) who have been truly nurtured on social

technologies, have entered the workforce only a few years ago. One might expect that

management styles would change in the presence of these millennials who grew up in the

digital era that encourages (natural) collaboration. But, in a poll of 5,000 people, 41% of

millennials agreed that workers should do what their supervisors tell them to do, even if

they cannot see the reason (Deal and Levenson, 2016), suggesting that millennials are

individualists who partly prefer conventional attitudes regarding management forms,

rather than collaborators (Erickson, 2008; Studer 2021).

These management forms, though, often hinder employees to contribute creatively mostly

through a high level of bureaucracy layers according to research from Harvard and a

Gallup survey (Hamel, 2011; Harter, 2020). Managers must lessen their tendencies

towards centralisation and delegate power to individual business units, when trying to

tackle novel problems (Hamel and Zanini, 2020).

PMs overcome internal processes of ‘resistance’, particularly in innovation and idea

management. Market-based idea management is allowed to be objective, as suggestions

are not hindered, co-opted or misused, nor do they fail altogether through hierarchically

generated distrust (Barth, Keitel and Wille, 2002). Even though the market itself might

accept input from all levels of the hierarchy, it is not clear that the decision-makers will

actually heed its advice/suggestions if it runs counter to their power structures and such

organisational nimbleness is still elusive for most companies (Ahlbäck et al., 2017; Rigby

and Bilodeau, 2018). Especially digital transformations have first and foremost to do with

changes in company culture (Cole, 2015), in larger companies, structural issues are the

biggest obstacle to Web 2.0 initiatives (Gottlieb and Willmott, 2014). The issues of

hierarchy, trust, and availability of information thus call into question the very core of

PMs.

54

3. Literature Synthesis

The application of PMs has great potential for forecasting and strategic planning. But

particularly in corporations they have not received much interest and their applications

are few. CPMs have not been taken up for many reasons; e.g. lack of knowledge about

them, a perception of complicatedness, management’s concerns over the loss of power,

etc. However, with advances regarding CPMs’ implementation and their potential for

innovation and ideation creation, they might have a chance for resurgence. Such

(potential) progress is illustrated in Figure 3.1.

Figure 3.1 Possible Path for CPMs – Innovation Management Making Up for the Decline in Use and Interest (Developed for Research)

These arguments and counterarguments are taken forward by summarising the literature

on PMs, linking the literature to the research objectives, and identifying gaps in the

literature.

3.1 PMs and Their Potential By aggregating information from public and private sources, and from data – information

and knowledge which can be widely scattered at various levels of a company – CPMs can

provide superior forecasts of future events (Buckley, 2016). Frequently, upper echelons

are unable to tap into such vital intelligence due to obstacles stemming from the

hierarchical structures typical of complex organisations (Buckley and O’Brien, 2017),

and PMs in general present one of the few tools available to correct and alleviate biases

in available information and to bring hidden intelligence to the fore. A participatory, non-

hierarchical culture is rated as important for the success of a corporate PM (Buckley and

Doyle, 2017b), and when information is released and shared in an inclusive way, the

accuracy of a PM is often improved (Brown, Reade and Vaughan Williams, 2019).

Promising Fundamentals

“Phoenix Rising from the ashes”

Fortifying Innovation

Diminishing Usage Stumbling

Blocksin Corporations

55

More generally, “cultural diversity not only leads to friction or problem creation, but also

to enrichment and to generation of solutions” (Drogendijk and Zander, 2010, p.189).

Grant (2016) noted, however, that most changes in the business environment have been

in degree rather than kind, which might cause leaders to believe that established

paradigms can be maintained, and that new ones need not be promoted. But on occasion

change occurs rapidly and the scope for strategic action shifts. Consequently, there is a

need for tools that help to create options and for techniques that harness cross-functional

capabilities within companies (Webb, 2014) and include personnel beyond management.

A PM can aggregate tacit knowledge and disparate information from manifold

contributors, but an optimal decision-making process depends on choosing a system

appropriate for the given context (Buckley, 2016). Where large and dispersed pieces of

information need to be collected, PMs scale well, while methods like nominal group

technique and Delphi do not. Also, PMs operate in real time, and thus are suitable for

time-constrained decisions (Gaspoz, 2011), which is a clear advantage. In novel

situations, decisions often need to be made before complete analyses are available. With

PMs the needed information can be provided by market participants (Eagar et al., 2020),

a suitably easy-to-use interface furthers speedy collection of facts via the market. PMs’

quick reaction time can consequently help to incorporate new information efficiently and

quickly (Gao, Zhang and Chen, 2013) and corporate PMs would be an asset in times of

crises when quick decision-making is needed (Blackburn et al. 2020; Finn, Mysore and

Usher, 2020).

For input into value decisions and decisions involving judgements where additional

communication is sought, PMs are less useful, because they usually inhibit such

exchanges by dint of their set-up, the trading mechanism customarily features anonymity

(Buckley, 2016). But with careful implementation tweaks, PMs can help even in these

cases. Chen and Wortman Vaughan (2010) cited that PMs could cope with combinatorial

or infinite outcome spaces like ‘x will happen ahead of y’ or ‘either-or-situations’ or

‘revenue will be between x and y Euro’ by implementing different cost functions. For

these benefits to materialise, those who deploy a corporate PM must find ways to integrate

its output into the organisation’s decision-making processes. Then the results become

useful and utilised (Bughin, Byers and Chui, 2011; Buckley, 2016).

56

Also, to inspire trust and enhance a CPM’s acceptance, employees need to see that the

results are acted upon (McHugh and Jackson, 2012; Balboni et al., 2013), because if a

company does not respond to a forecast, it has no value beyond mere curiosity (Ord,

Fildes and Kourentzes, 2017).

The ‘Promising Fundamentals’ (Step 1 – Figure 3.1) do not translate into increased CPM

usage, however, as will be revealed in the following section.

3.2 CPM Adoption

While capturing the value of social technologies is becoming the norm, Web 2.0 tools,

including CPMs, have only just begun to realise their full potential (Bughin, Chui and

Manyika, 2012; Farmer et al., 2020). Companies are finding that ways of thinking are

hard to change, whether they are trying to convince employees to use social technologies

or to evolve to the point that information sharing is standard practice (Chui, Dewhurst

and Pollak, 2013). One of the ways that culture undermines the adoption of new

collaborative tools and processes is by focusing on the technology itself. Rather,

companies need to foster an atmosphere of empowerment to drive collaboration (ibid.).

In general, in the context of Web 2.0 tools the inclination to use new processes is much

greater in idea management (‘Finding new ideas’ in Figure 3.2) than in strategic planning,

which would actually be a promising result for CPMs in the guise of idea markets.

According to a survey of 4,261 companies conducted by McKinsey, in developing a

strategic plan 31% maintain the status quo of the approach.

Figure 3.2 A Mix of Old and New – Extent to Which Social Technologies Can Change Organisational Processes, Percentage of Respondents (Developed for Research from Bughin, Byers and Chui, 2011; Razmerita, Kirchner and Nabeth, 2014)

One could say that 81% of companies maintain a relatively traditional outlook on strategic

planning (31% report ‘no change’, 8% report ‘more traditional processes than new ones’,

and 42% claim an ‘equal mix’).

9

10

18

20

9

11

17

17

42

45

48

46

8

8

6

4

31

25

13

13

Developing4a4strategic4plan

Managing4projects

Finding4new4ideas

Scanning4external4environment

Entirely4new4process Mix4between4more4new4processes4and4fewer4traditional4processesEqual4mix4of4new4and4traditional4processes Mix4between4more4traditional4processes4and4fewer4new4processesNo4change4in4process

57

It was highlighted that new processes could emerge if obstacles to usage, such as cultural

barriers, were diminished (Bughin, Byers and Chui, 2011; Bughin, 2016b). The

uncertainty about whether this change will actually happen is underscored by the fact that

occasional use of social technologies does not offer their full benefits. Respondents from

various studies observed that techniques like social networking and blogs are better

adapted to certain types of business procedures such as finding new ideas, an area where

CPMs could perform well. However, CPMs and the technologies of tagging, rating and

mash-ups which are all below an implementation threshold of 10% were not included in

these study results (Bughin, Byers and Chui, 2011; Batrinca and Treleaven, 2015)! It is

not surprising then that in comparison to other social/Web 2.0 tools, CPMs continually

languished at the bottom of adoption in percentage terms as observed in

Figure 3.3.

Figure 3.3 Adoption of Web 2.0 Tools (Developed by the Researcher from McKinsey & Company, 2013; Jabeen et al., 2016; Pan et al., 2016; Bughin, Chui et al., 2017; Clement, 2019)

Such low usage points to an inconsistency between the conclusions of experts in academia

and those of corporate managers regarding CPMs (Azeem and Yasmin, 2016) where the

former would expect rising acceptance (Paul and Weinbach, 2015) and the latter did not

adopt the technology.

Overcoming such ‘Diminishing Usage’ (Step 2 – Figure 3.1) is possibly exacerbated by

rigid company structures.

29%

41%

35%

23%

58%

49%

13%16%

20%

13%

21%

28%

6%8%

5%

0%

10%

20%

30%

40%

50%

60%

2008 2013 2020 - ANTICIPATED

Overall Adoption of selected Web 2.0 Tools in Corporations

Blogs Social Networking Rating Tagging Prediction Markets

58

3.3 Obstacles and Hindrances to CPM Adoption

For corporate managers, the importance of status is clearly visible when leader-boards

disclosing relative positions of participants are used, as leader-boards are a reward

mechanism regularly applied in CPMs. Their use could jeopardise the information

aggregation, as knowledge about one’s own position in comparison to other traders led

CPM participants to prefer to improve on that position, even being willing to forego

financial gain; achieving kudos was more important (Woike and Hafenbrädl, 2020). The

‘knowledge as power’ phenomenon is more striking at the top of an organisation than at

the bottom (Simon, 2013; Barley, Treem and Kuhn, 2018) and will likely be difficult to

overcome.

Also, PMs had repeatedly been characterised as ‘new’, even recently (Freeman, Lahaie

and Pennock, 2016; Tieben, 2017; Zinger, 2018), which does not inspire trust for those

choosing a forecasting tool. Evidence that this is not necessarily the case, as a new tool

might actually raise interest by potential users, was given by Cummings, Pennock and

Wortman Vaughan (2016). However, they quoted some limitations, too. Apart from PMs

being a novelty, there were questions about their adoptability, accuracy and reliability.

In addition to the traits of the tool itself, institutional culture has also presented challenges

for CPMs. The normative conventions of a company can anchor decisions and their

controlling instruments so strongly that they are not called into question. They can even

cause managers to disavow disagreements to improve their alliances or uphold the

positive perception of others, and hold internal opponents at a distance (Baer, Heiligtag

and Samandari, 2017). When corporate culture does not foster questioning, a CPM alone

might not overcome this. As culture determines and limits strategy (Schein, 2010), the

importance of a company’s culture for its future was highlighted. Baer, Heiligtag and

Samandari (2017) saw that one way to remove bias and circumvent hierarchical

challenges is for decisions to be automated based on statistical algorithms like from Big

Data and machine learning.

In the end, the solution to eliminating potential barriers such as acceptance of prediction

results will be a readiness for cultural change (see Table 3-1). Managers need to

understand that they are biased and not omniscient, needing input from their staff on a

regular basis (Siegel, 2016).

59

Table 3-1 Obstacles to Using Social Technologies (Developed for Research)

Adoption of Social Technologies

Topic / Barrier Characteristic

Corporate culture • Undermines the adoption of new collaborative tools • Ways of thinking are hard to change

Emergence of new processes • Impediments to use, like cultural barriers, not yet diminished • Information sharing too sparse

Inclination to use new processes • Much less so in strategic planning • Maintaining the status quo

Readiness for cultural change • Orthodoxy in forecasting • Rare elimination of potential barriers such as acceptance of

prediction results

Enthusiasm about social technologies belies their actual measurable benefits. More data

do not necessarily automatically create added value. Just as the organisational structure

of a company is decisive in making the most of Big Data (Almquist, Senior and Springer,

2015), so it does for CPMs (Buckley and Doyle, 2017b). One major challenge are (data)

silos reflecting the internal boundaries of a company. Different divisions within an

enterprise that are afraid to give up their power loathe to share their data or their personal

viewpoints (Drezner and McKernan, 2020; Drezner et al., 2020). Data structures are often

only weakly disguised power edifices, and few companies have made organisational

changes to overcome this internal divide (Logan, 2020). Organisational culture, with its

ingrained silos, hinders agility, collaboration, and crowd-approaches (Blackburn et al.,

2020), and is a stumbling block for CPMs but also to attain the benefits of Big Data.

Bughin et al. (2017b) and Rigby and Bilodeau (2018) showed that in 2017 managers use

on average around half as many or even just a quarter of management tools as they had

in 2007, deducing that it is harder to justify the introduction of a new tool. With a clearly

suggested mandate to decrement the number of forecasting instruments in use (Rigby and

Bilodeau, 2018), another way to evaluate the success of an instrument is to examine the

relationship between its promoters and its detractors. On balance, dissatisfaction with

CPMs is higher among their detractors than is contentment among those who like the tool

(McKinsey & Company, 2013). Thus, culture is at least as important as strategy for

business success, which reiterates the point on the lack of general uptake of CPMs.

For the ‘Stumbling Blocks in Corporations’ (Step 3 – Figure 3.1) to be overcome, CPMs

need clear impulse and momentum from a meaningful putting-into-operation presented

at the end of section 3.4.2, a section which meanders from praise to criticism to an

important application possibility for CPMs; concluding to potentially deploy CPMs in

innovation management.

60

The section reveals the complexity around choosing a CPM but also the inherent promise

of their use for enterprises.

3.4 Summarising the Literature and Linking It to the Objectives

Rieg and Schoder (2011) pointed out that PMs in general are limited in their scope of

application, and they work best when people see themselves as independent individuals

who express their opinion anonymously and in a competitive environment under the

adage ‘may the best person win’. Certain environments within firms may therefore not be

conducive to a corporate PM.

If, for example, employees view a market with scepticism or react poorly to the

appearance of competition amongst them, CPMs might damage an organisational culture

of mutual trust, open discussion, and collective action (Luft, 2016). A sentiment of this

sort will not exist in every company; there are important differences in cultural attitudes

between countries and between companies (Hofstede, 1993). Such ‘destructive

competition’ and status considerations would be a problem for corporations that deploy

agile management approaches, in which employees from different sections are organised

in teams and encouraged to work together (Woike and Hafenbrädl, 2020), the competition

might hinder team cohesion.

Innovation as ‘Creative destruction’ (Schumpeter, 1942) can be supported by the arsenal

offered by PMs, though.

3.4.1 Research Objectives The approach to meet the research aim was laid out through the four research objectives.

The literature synthesis spanned the goal across them: This section touches on evidence

of corporate use, pertaining to objective 1;

Figure 3.3 highlights PM usage, too. Exploring challenges in using CPMs, as mentioned

above, is the essence of research objective 2, further exemplified in section 3.4.2.

How adoption could be facilitated – objective 4 – is summarised in the list of success

factors underneath Table 3-2. The last paragraph of section 3.4.2 recommends PMs in the

context of innovation buttressing objective 4. (Research objective 3 with its focus on

expert consultations is more generally informed by the literature which helped to identify

important questions to ask through the researcher’s appraisal.)

Following on from this résumé, Table 3-2 accounts for how the literature, synthesised in

this chapter, applies to the research questions.

61

Table 3-2 Literature Synthesis and Research Objectives (Developed for Research)

Linking Literature and Objectives

Abridged Research Objectives 1-4 Supportive Section in the Literature Synthesis

1. Evidence of corporate use CPM adoption – Section 3.2 Corporate Use – Section 3.4.2

2. Challenges in the use of CPMs Obstacle and Hindrances – Section 3.3 Low Corporate Use – Section 3.4.2

3. Forecasters’ and researchers’ knowledge of PMs Literature contributed to form interview questions (cf. Section 5.2.4)

4. Recommendations for companies on the use of PMs Success Factors – Bullet List below Innovation Management with CPMs – Section 3.4.2

Research suggested that PM forecasts need to be put into the centre of a forecasting and

decision-making context (Buckley, 2016; Sung et al., 2019). Success factors required to

promote corporate PMs would entail:

• Clear endorsement by management

• Acceptance of results from non-hierarchical approaches

• Acting on PM results

• Entrusting decisions to lower ranks in the company

• Implementation focusing on ease-of-use, optimal cost functions, and speedy

operation

• Integration of results into decision-making structures

The success criteria touched upon above would also support objective 4. The adoption

framework and recommendations sought in objective 4 need a basis, a backing for their

reasoning, provided by factors that promote CPMs as shown above.

3.4.2 PMs’ Low Use at Corporations – A Potential Misnomer CPMs consequently could be a mechanism that is easy to use and deployed expansively.

They would be able to fill gaps in a company’s knowledge about extraneous drivers like

trends derived from large-scale economic factors or steps competitors take or key

company performance indicators such as turnover or market share (Montgomery et al.,

2013; Qiu and Kumar, 2017).

But in CPMs’ traditional context, power structures, perceived complexity, and

implementation challenges such as requiring up to a full year to offer meaningful results

(Beckmann, 2010) speak loudly against CPMs. So, too, does the advice from Rigby and

Bilodeau (2018) to limit the introduction of new and unproven tools. Furthermore, Google

Trends from 2004 to 2020 displayed a downward trend of interest in PMs in general.

62

(In Figure 3.4 the four peaks of interest, in the months of October, coincide with the US

presidential elections in 2008, 2012, 2016, and 2018.)

Figure 3.4 Interest in ‘Prediction Markets’: Google Trends – Results from Search Term ‘Prediction Markets’ in December 2020 (Data Source: Google Trends – https://www.google.com/trends)

Figure 3.5 shows that publications about PMs taper off sharply after 2011, highlighted by

two literature reviews (Tziralis and Tatsiopoulos, 2007; Horn, Ohneberg and Ivens,

2014), as did corporate PM usage according to research conducted by McKinsey from

2007 to 2013.

Figure 3.5 Academic Interest in PMs and PM Usage by Corporations (Developed by the Researcher from Tziralis and Tatsiopoulos, 2007; Horn, Ohneberg and Ivens, 2014; McKinsey & Company, 2013)

The data show that by 2013, the adoption of CPMs had fallen 30% from its peak in 2011.

A drop of such proportions was termed a death knoll trajectory by Forrester analysts when

looking at the market share for Blackberry devices (Schadler, Bernoff and Ask, 2014).

0

50

100

150

200

250

300

350

0

10

20

30

40

50

60

70

1990 1991 1992 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Waning Academic Interest plus Decline in Usage

Yearly Publications PM (left axis) Usage in actual Numbers according to McKinsey (on the right)

63

The points mentioned above, and the low adoption shown in

Figure 3.3 considered CPMs from a point of traditional forecasting and how tools would

be embedded in that function; indicating that success is questionable. As mentioned

earlier in this thesis, use of CPMs in idea generation is gaining traction (Tomo et al.,

2019; Webster and Gardner, 2019). Innovation management consistently garners high

interest in the business world (Frank et al., 2019), making the case that an area of PM

application in corporations has been overlooked when gauging their general usage. The

importance of innovation for enterprises and the need for strategic orientation towards

collaborative tools (Adams, Freitas and Fontana, 2019; Solaimani, Talab and van der

Rhee, 2019) might invigorate the application of CPMs; in their original conception they

might have failed as forecasting methods, though.

‘Fortifying Innovation’ (Step 4 – Figure 3.1) is nevertheless a chance for CPMs.

3.5 Gaps in the Literature Regarding PMs in Corporate Forecasting

The sparse use of the tool in corporations indicates a clear gap in perceptions and practice

between academia and corporate planning: it is hard to portray CPMs as a success with

their clear decline in use (Horn and Ivens, 2015; Bughin, 2015; Weldon, 2016). CPMs

could improve a manager’s decisions (Dianat and Siemroth, 2019; 2020) but a lack of

their integration into (strategic) decision-making is also clearly noted (Vecchiato, 2012;

Buckley, 2016). Hierarchical considerations appear among the main reasons for this

(Hlupic, 2018; Krekel, Ward and De Neve, 2019) and might explain the scarce usage,

since without meaningful integration of a forecasting tool the results are likely to be

ignored or even rejected (Ferrier, 2018).

According to recent research (Verreynne, Ho and Linnenluecke, 2018; Pittaway et al.,

2019), the literature on PMs did not sufficiently address why this is happening and what

entrepreneurs need to underpin resilient forecasting processes. However, the overall

importance of good forecasting and to identify the best forecasting techniques was

repeatedly highlighted (Robson, 2020) as is managers’ and executives’ involvement.

The reasons behind the lack of use of corporate PMs and their stumbling blocks in

companies escaped the body of text on PMs, as did new avenues of use for corporate

PMs. The researcher considered it important to bridge or close this gap between the theory

and the practice of their application. To understand the apparent discrepancy would also

require information from both forecasters and users of forecasts (Canduela, 2009).

64

Thus, the rationale behind the conundrum of what appears to be a good forecasting tool

and its neglect in the business world is not apparent solely from the literature, the focus

in the literature is much more on the theory of PMs, e.g. how the information aggregation

works (Jolly et al., 2015; Choo and Kaplan, 2019) or only on the general success of the

principle, the PMs’ concept (McAfee and Brynjolfsson, 2017).

Therefore, further research is required to inform about reasons for corporate uptake or

lack thereof, which the researcher tried to provide with this thesis. The basis for such

research is now presented in the next chapter on theoretical conceptualisation.

65

4. Theoretical Conceptualisation

CPMs emerged as an interesting proposition at first but encountered many obstacles even

in situations where they were implemented, and they have fallen out of favour. The

challenges to acceptance by management deter companies from using CPMs to their full

potential.

Decisions that are the remit of the boardroom, the pinnacle of the company pyramid, are

usually not entrusted to the lower ranks (Thompson, 2012). Corporate superiors are

deeply suspicious of their employees, and therefore resistant to the type of ‘digital

change’ and corporate culture change which promotes inclusion brought on by a CPM.

CPMs require that employees who participate in them are empowered to influence

corporate decisions (Scheuss, 2012; Seemann, 2013).

A large sample of experimental PMs confirmed the importance of the relation between

self-reported comprehension of PMs and participation in the market. The need for

participants who have experience with PMs is an important issue for a successful PM

implementation (Boulu-Reshef et al., 2016). But it is hard to gain experience when the

tool is rarely in use. Adoption of a tool and its subsequent use depends on how well

information about the tool is communicated (Xiao, Witschey and Murphy-Hill, 2014).

An innovation needs diffusion (ibid.) which has not been achieved in the case of PMs,

the general ignorance about them prevents them from making it onto the list of forecasting

instruments in use by corporations (Hyndman, 2020). As PMs seldom appear in the

management literature (Andler, 2016; van Aerssen and Buchholz, 2018; Holt, 2020) it is

no surprise that most managers do not know about them or misunderstand them. Perhaps

businesses are also firmly set in their ways and resistant to opening up to new ideas.

Thompson (2012) saw a clear implementation challenge in the fact that businesses have

little understanding of PMs. About hierarchy and trust he said that managers using CPMs

do not care to admit that certain decisions might stem from lower rungs of the corporate

ladder (ibid.). The ensuing gap in the literature around implementation aspects and

company hierarchy was also highlighted by Dianat and Siemroth (2020) who saw

research on CPMs to be still relatively scarce, contributing to their obscurity in business

circles.

66

To consolidate the discoveries from the literature review, and to provide the theoretical

lens that will give direction to the empirical part of this thesis, the researcher developed

a research framework which draws on three key concepts which the literature review

brought forward as the main observations:

• Usage of CPMs is limited, and company structures and organisational set-up can

severely inhibit their acceptance and implementation

• PMs in general nevertheless have a strong theoretical underpinning and can be

effective when based on the three pillars of anonymity, diversity, and

independency

• Combining PMs with other methods and expanding their use cases could indicate

a route to successful adoption

Through the thesis’ framework these considerations steer the specific research questions,

which form the basis of this thesis.

4.1 Theoretical Framework and Research Questions

The relationships among the above findings are summarised in relevant conceptual

elements and depicted in the honeycomb structure in Figure 4.1 (the three different

colours, two greys and a dark blue, distinguish the three elements of the framework, with

two topics each).

Researching challenges in the use of PMs and how these might negate the benefits of the

method, but also looking at alternatives to or choosing different applications for PMs shed

light on the decline in the use of CPMs and would explain why and how the problem of

adoption exists.

67

Figure 4.1 Elements of the Theoretical Framework Supporting the Overarching Research Question (Developed for Research)

The discussion in this thesis on the fit of PMs with corporate culture and management

styles, and on the concepts described above, continue to make the research questions

posed at the beginning of this thesis important. The remainder of this thesis pursues their

overarching enquiry:

What are the barriers to adoption of PMs by corporations, and what schemes

would facilitate the decision for or against an adoption of PMs?

The researcher arrived at an overall conclusion to the summary research question by

answering the following questions in turn:

• What are the advantages and disadvantages of PMs?

(Framework element ‘Promoter / Benefit’)

• What are the barriers to adoption of PMs by corporations?

(Framework element ‘Barrier / Downside’)

• Are there ways to make PMs more adoptable for corporations?

(Framework element ‘Moderators’)

Promoter / Benefit

Barrier / Downside

What are the barriers to adoption

of PMs by corporations, and

what schemes would facilitate the decision

for or against an adoption of PMs?

Effectiveness of the method

Theoretical underpinning

Use of PMs in practice

Challenges through

company organisation

Combination of forecasting

methods

Alternative use cases and competition

Moderators

68

As can be seen, these questions are in line with the cadence of topics in the theoretical

framework and provide useful guidance. The main theoretical point of the thesis is about

adoption or otherwise of tools for strategic planning and forecasting and PMs as a

particular example of this process.

4.2 Framing a Concept for CPM Adoption Considering an assessment and embedding of a new forecasting process made it useful to

string out the topics and main elements of the theoretical framework in a sequence. The

following diagram shows how the different ideas (theories) expressed in the literature

come together and relate to each other.

Figure 4.2 Theoretical Framework: Inputs and Influences into PM Tool Adoption; Assess and Embed CPMs as a New Forecasting Process (Developed for Research)

Business and management theory did not span CPMs comprehensively enough to offer

explanations and the topic of this investigation came from several diverse contexts which

had rarely been systematically combined before. For this reason, the researcher used his

own and the ideas of others, both from academia and business, to draw conclusions

following the path laid out by the suggested framework.

CPMs as originally conceived have failed as a forecasting approach but the process of

rejection is of interest and by exploring this, a way to reinvent and reintroduce CPMs

might open up.

influence

impact

Effectiveness of the method

Theoretical underpinning

Use of PMs in practice

Challenges through

company organisation

Combination of forecasting

methodsUse cases

and usage of competitive

tools

moderate

CPM adoptionthrough

Alternative use cases,

e.g. innovation

management

Alternative use cases

69

Moving along the constructed framework as illustrated in Figure 4.2 presents a suitable

way to assess the incorporation of a CPM into a firm’s forecasting landscape. Taking the

inputs and influences into CPM adoption one by one can elucidate how a forecasting

approach that was hailed as a good idea was not accepted and if CPMs have a chance of

come-back, for instance in the area of ideation where they could support innovation

management processes.

4.2.1 Benefits Promoting CPM Usage – Method Effectiveness and Theoretical Underpinning

It has been shown that both traditional forecasting methods and modern AI techniques

perform poorly when modelling and aggregating expert knowledge (Tziralis and

Tatsiopoulos, 2006; Gröner and Heinecke, 2019). Research advocated the use of a PM

that incorporates expert forecasts and aggregates information into predictions for such

situations (ibid.). Compared to conventional prediction methods, PMs give more accurate

results and provide continual information updates (Wang, 2013). Several studies in the

PM literature have examined the information aggregation efficacy of PMs outside of

company environments, for example Page and Siemroth (2017, 2019), Corgnet et al.

(2018), Choo and Kaplan (2019) and Corgnet, DeSantis and Porter (2020). There is

evidence that PMs generate very precise forecasts and can ameliorate the capacity to

predict numerous phenomena, from geo-political events to the realisation of a new

technology to a firm’s revenue goals (Kingsley, 2015).

Implementation also plays a role in exploiting the advantages of PMs; companies need to

describe how the markets work and how they will be used for decision-making. With such

communication, PM prices become more meaningful (Dianat and Siemroth, 2019; 2020).

4.2.2 Barriers to Corporate Adoption – CPM Use and Organisational Aspects / Challenges

The literature review made it clear that PMs were introduced but not taken up, a

reasonable idea and notion to improve forecasting that was not accepted in the corporate

realm. Partially this was based on ignorance of the approach as highlighted in the

introduction to this chapter, and was not necessarily a direct rejection.

Even when a knowledge organisation is the central reality in a company and the centre of

gravity shifts to the knowledge worker (Drucker, 2017; Girdharwal, 2020), there is still a

focus on managing oneself for effectiveness on an individual basis (Drucker, 2017).

70

Such isolated conduct is not amenable to a crowd-aggregating mechanism. Furthermore,

an inflexible holding-on to know-how, role, or hierarchical structures and a focus on ego

are typical when we feel those things to come under threat (Berger and Achi, 2020). Such

mind-traps are unhelpful in addressing modern challenges, particularly as the best

solutions to those challenges require giving up the security a firm’s hierarchy can provide

and establishing a more inclusive corporate culture.

4.2.3 Moderation of the Challenges to Adoption – Alternative PM Use Cases

The process of strategic decision-making is about understanding where you are now,

where you are heading and how you will get there. When transforming the vision into

action, planning and forecasting should involve the managers at all levels of a company

(Kourdi, 2015). Viewing business strategy as an ongoing and evolving process, feedback

must be constantly requested, and can be elicited through a PM (Roughgarden, 2016;

2018). In the fields of accelerating innovation and understanding how digitalisation

changes the business logic of a company a PM can be used to produce informed prediction

in strategy development (Matzler et al., 2016; Rumelt, 2017).

Researchers suggested that management scholars should become involved in and

influence the process of managing innovation (Birkinshaw, Hamel and Mol, 2008; Mol

and Birkinshaw, 2009), and helping to choose practical new tools, like a PM, for gauging,

speeding up, and organising innovation, and the selection of risk mitigation strategies

when introducing new business models (Osterwalder et al., 2020). Managing innovation

portfolios is one possible path forward for CPMs (Schaer, 2019; Osterwalder et al., 2020).

Here, upper management seeking fresh ideas might accept both collective wisdom from

lower rungs of the corporate ladder and results that may run counter to their own

perceived knowledge. In that way, strategy-making focused on innovation embraces all

levels of a company, shifting and rejuvenating strategy by driving and refining the

available choices (Vanhaverbeke and Peeters, 2005).

4.2.4 CPM Adoption – Innovation Management as a Use Case but also Method Combination

A key to creativity and innovation in organisations is determining the ideas most likely

to succeed in the future. But predicting which new idea might win can be difficult. When

choosing the best new products, services, etc., people often overrate their individual ideas

(Berg, 2016). A CPM can mitigate this and facilitate the inclusion of sensitive data, which

is often not fully reported openly by participants (Krause and Caimo, 2019). The reporting

would happen under the guise of a market’s anonymity.

71

Furthermore, information can come from multiple sources, but the field of innovation

inevitably contains gaps in information or unavailable data from which the wrong

conclusions are easily drawn (Krause et al., 2020). The aggregation of facts via

establishing an idea market can mitigate this to a degree.

Nevertheless, alternatives to forecasting approaches exist. The so-called Blue Ocean

Strategy advocates to beat the competition by not competing, predict by not predicting

(Chan Kim and Mauborgne, 2015; Mebert and Lowe, 2017). The Blue Ocean Strategy is

an approach which constantly subjects the business strategy to an economic perspective,

creating the ability to adopt advantageous new developments and consistently adapt to

changes (Straubhaar, 2019). But, to achieve these results while extracting knowledge

from complex data and applying the findings, a strategic planner needs to select the most

relevant characteristics for the company to move forward strategically on the basis of the

possibilities presented (Chatfield et al., 2014; Schulz et al., 2020), and here judgemental

forecasting mechanisms, like PMs, can again play a valuable role.

When strategy is viewed as the process of weeding out scenarios, those best to eliminate

can be identified via a CPM. At the forefront of this choice process one needs to discover

all the critical factors of a situation that could be entrusted to a CPM, almost like using it

as an ideation valuation instrument. Something similar can be facilitated with data mining

techniques (Lu, Tsai, and Yen, 2010). The overlap illustrates that it could be worthwhile

to use a combination of methods, scenarios, market-based prediction and data analytics

in concert to evaluate the data sets to be used by a modelling system through a PM (Merrill

et al., 2019; Staples, 2020). In such a way, a PM acts as an enabler to machine learning,

in keeping with the advice that algorithms are created by humans and should do what

humans intend them to do (Dräger and Müller-Eiselt, 2019; Tai, 2020).

4.3 An additional Slant of the Theoretical Framework – a further Contribution

So far the literature review has elucidated two main points regarding PMs: their potential

failure as a general forecasting approach in corporations but also a way out of this.

Particularly the chosen theoretical framework shows how PMs could escape from the

‘Trough of Disillusionment’, as Gartner called the lack of maturity of a concept, moving

towards new use cases in the area of innovation management. However, the overall

process of rejection and the reasons behind it could contribute to and constitute a learning

in its own right, as the reflections and considerations from the literature also drove out

insights into the reasoning why a good idea floundered and did not get adopted.

72

For instance, company culture hindered collaborative approaches that might have helped

with the clear need to improve forecasting methods (May, 2012; Powell et al., 2013;

Lenart-Gansiniec, 2019), insights which also draw on technology acceptance models

(Taherdoost, 2018).

Strategic decision-making, too, is impacted by the type of hierarchical structures noted in

the review of PM literature (Accard, 2015). Companies often lack effective problem-

solving processes (Lefort, McMurray and Tesvic, 2015) and do not understand how they

could better underpin decision-making in uncertain times (Alexander, De Smet and

Weiss, 2020). Understanding the failure of PMs as an example could lead to promoting

different views and debates around general forecasting instruments and strategic

decision-making in general. The main issues of trust and hierarchy in the context of PM

rejection, which the conceptual framework shed light on, could also apply in missing out

on other conceptually good ideas for leadership (Foglesong, 2004).

In a sense, the same way PMs had been looked at ‘wrongly’ could possibly apply to other

new arenas of strategic decision-making (cf. Kandemir and Acur, 2012; Rosas, 2019);

and to make it clear and understand why that happened could help decision-makers to

learn from these non-fulfilments and initiate a change in behaviour accordingly.

The purpose of the thesis was nevertheless to evaluate CPMs and had less to do with the

forecasting function or its tool use in general or with universal technology acceptance

models.

73

5. Research Methodology

A former editor of the journal Academy of Management Review expressed that theory

development encourages new ideas from established concepts that derive empirical

questions not yet asked (Kilduff, 2006). In the same journal, it was averred that

contribution to theory development stems from and commonly involves perspectives

from other fields, encouraging a change to underlying rationales “altering … metaphors

and gestalts” (Whetten, 1989, p.493). Also, influential theories come from observing real

phenomena, not only from scientists essaying to find gaps in the literature to put it a little

extreme (Hambrick, 2005); still, a well-written article should also draw on extant

literature to emphasise what is appealing and dissimilar (cf. Davis, 1971).

Guided by this assertion, ontological and epistemological assumptions are now presented.

These shaped the overall research philosophy, design, and strategy of this doctoral thesis.

The approach to the research design is encapsulated in the building blocks shown in the

following graphical summary.

Figure 5.1 Research Method – Five Steps to Form the Findings (Developed for Research)

The figure shows data collection and interpretation tying in with the research paradigm.

5.1 Approach and Strategy – Philosophical Position and Research Design The choice of a theory in business research involves making assumptions about the nature

of reality, implicating the type and logic of inquiry a researcher undertakes. This view on

reality shapes the philosophy and the philosophical assumptions that inspire business

research methods. The way the data is collected and treated is a consequence of the chosen

methodology or research strategy.

Constructionism and Interpretivism

Cross-sectional qualitative interviewsand thematic analysis driving out actions,reasons, and social context according PMuse and challenges (cf. Andrews, 2012)underpinned by quantitative data on PMaccuracy to enhance data richness(cf. Creswell and Poth, 2016)

0510 interviews

Extended Interviews – Innovation

0135 interviews

Secondary Interview Data (Pre-Study)02 13 interviews

Pilot Study Interviews

037 interviews

Main Set Interviews04 data sets from a multi-year study

Statistical Analysis of PM Trading Data

74

Using a philosophy which studies human experiences in management settings rather than

a fact-based ontological approach, the challenges with successful CPM usage can be

explored (Jeddy, Klempner and Portner, 2018) as well as how CPMs work as a forecasting

method and how they can be adopted as a focus of innovation. Thus, the researcher

adopted primarily phenomenology or constructionism rather than positivism, recognising

reality as a social construct rather than a concrete structure or process (Morgan and

Smircich, 1980; Easterby-Smith, Thorpe and Jackson, 2008).

In the focus of this thesis were entities that are made real by the actions and

understandings of humans (Bell, Bryman and Harley, 2018), which is a hallmark of

constructionism. Its roots lie in phenomenology, but more recently it has also been

associated with postmodernism (Marks, 2019).

The researcher considered postmodernism as too divorced from reality as it views

research findings as versions of reality which are solely constructed (Alvesson, 2002;

Travers, 2006) and therefore did not choose postmodernism as a paradigm.

Constructionism (also called social constructionism (Marks, 2019)) supports research

which seeks to elicit more the ‘how’ than the ‘what’ (Bell, Bryman and Harley, 2018),

which was a goal of the research questions posed in this thesis. Fleetwood (2014) argued

that things that are called ‘socially real’ comprise market mechanisms, organisations but

also class or gender structures, norms, rules or conventions. This fit with the

organisational, managerial, and hierarchical aspects brought forward in the literature

review. It also supported making constructionism the ontology of choice for this thesis.

In constructionism, knowledge is gained by observing and interviewing actors to

understand how they comprehend the world. Seeking to comprehend the decline of PM

use and why and how the problem of adoption exists, fit with the theoretical framework

as depicted in section 4.1. This led to the choice of an interpretivist approach to data

collection and analysis adopted by utilising semi-structured interviews (Leitch, Hill and

Harrison, 2010).

The (evident) lack of management acceptance and the adherence to company hierarchy

is considered to deter companies from using PMs to their full potential. For research

dealing with such complex culture issues the researcher considered the duo of

constructionism and interpretivism fit to serve as an appropriate philosophy.

75

They were jointly used in this research as an evaluation of social interactions and

reasoning aligns with a philosophy of interpretivist research and synergies between

constructivism and contemporary interpretivist analysis exist, as argued by Walker and

Dewar (2000) and Hay (2015).

By choosing interviews as a data collection method, value was placed on personal

interaction, an axiology of research being value-bound. Furthermore, a researcher is a

central part of the research and should not be viewed as a separate entity. Therefore, such

research is subjective, which fits with the paradigm of constructionism and data collection

techniques using small sample sizes and being highly in-depth and qualitative (Marks,

2019). Questions around PMs and their challenges require a nuanced and complex

understanding of actions and reasons. The notion of social context and interactions

framing and constructing our realities is central to the constructionist perspective (ibid.).

As explained in section 4.3 on the theoretical framework (and in section 2.4.2),

considering the acceptance of PMs and their potential choice as a forecasting method shed

light on technological acceptance and decision-making approaches in companies with a

focus on PMs.

Facilitating organisational decisions considers phenomena within the social world, the

way organisations operate and the interactions between individuals. These points fit and

underscore the philosophy chosen for this research.

Also, assessing technology acceptance in organisational settings, the moderating effect

of leadership and culture, and perceived ease-of-use and usefulness related to the actual

application of a system complies with the findings from the literature review. Likewise,

this is consistent with conceptual comprehension of organisational issues and a

framework of interpretivism.

A research focus on experiences, events, and occurrences, to be understood through

interviews, would be supported pulling together and textually analysing (potential)

evidence from the literature – both past and current, as suggested by Christensen (2011).

This suited the approach of qualitative examinations in line with a philosophical position

of phenomenological aspects as forwarded by Bauer, Bicquelet and Suerdem (2014).

76

In line with the research questions and the theoretical framework, the researcher applied

a cross-sectional research design as advocated by Diekmann (2014) and Bell, Bryman

and Harley (2018) using interviews with forecasting experts or users drawn from relevant

companies, business consultancies, and academia. The use of cross-sectional design is

appropriate for the research questions to be answered employing semi-structured

interviewing with a number of people according to Scase and Goffee (2017) and allowed

evaluation of the research findings. Cross-sectional studies are common in business and

management research (Bell, Bryman and Harley, 2018).

Qualitative interviews are the fulcrum of the investigation of this thesis. However, there

exists a sizable repository of potentially useful PM trading data from a large multi-year

study run by IARPA focusing on prediction accuracy (cf. section 2.2.3). Exploring this

with the help of statistical analysis is appropriate to reduce subjectivity, results are often

underpinned by quantitative approaches to enhance data richness (Lambert and Loiselle,

2008). Such a process of using a triangulation methodology then calls for a mixed method

approach potentially anchoring the overall research in a postpositivist paradigm. In a

special issue on triangulation published in the Journal of Mixed Methods Research,

Mertens and Hesse-Biber (2012, p.75) suggested, however, that it is possible to use

“qualitatively framed mixed methods” and hence stay with qualitative, constructionist,

and interpretative pathways for the underlying research when triangulation with

qualitative data only plays a minor part.

Denzin (2012) criticised mixed methods approaches which he perceived as supporting a

methodological hierarchy in which quantitative methods prevail over qualitative

methods. Keeping a constructionist paradigm, as used in this thesis, can put the data into

a more comprehensive explanatory framework, according to Howe (2012), and Flick et

al. (2012) illustrated triangulation as a methodological framework within a study using a

constructionist approach.

Furthermore, Mackey and Gass (2015) and Rogers and Révész (2020) saw triangulation

more generally as a precautionary measure adopted by a researcher to ensure that their

collection, analysis, and interpretation of data in qualitative research is valid and

plausible. Put slightly differently, it is employed to make the qualitative phases of

research more objective and credible, not necessarily superseding the chosen main

research paradigm, as mixed method research was chosen in the past more by convention

and tradition (Gass, Loewen and Plonsky, 2020).

77

The rationale for this thesis’ design was that forecasts are established based on the

interplay between a prognosticator and the forecasting solution, which is often integrated

into a decision support system (Fildes and Goodwin, 2013; Kahn, 2014) and places

forecasting squarely into a dedicated forecasting department (Lapide, 2003) or the

strategy division of a company to concentrate expertise (Hyndman and Athanasopoulos,

2018). Here forecasting systems or tools are refined over time (ibid.) producing forecasts

that are drawn on elsewhere in the organisation (Ord, Fildes and Kourentzes, 2017). As

forecast experts in a company are developers and users of forecasting tools, they are

judged appropriate informants for this research, revealing if PMs are known and

employed, in a similar way as exemplified by Önkal et al. (2017). The researcher chose

such a qualitative research strategy as appropriate to address the aims of the thesis

allowing a full portrayal and analysis of a research topic with no restriction on the “scope

of the research and the nature of the participants’ responses” (Langos, 2014). To facilitate

credible semi-structured interviews of experts an interview guideline was developed, as

advocated by Misoch (2015). The interviews were then analysed using qualitative content

analysis in a thematic framework, in accordance with the advice of Saldaña (2015).

The implementation of this strategy is depicted in Figure 5.2 as a plan of the steps the

researcher followed. It starts with the research objectives leading into the literature

review, then building a theoretical framework supporting the research questions and

thereby guiding the interviewing process.

Figure 5.2 Progression of the Research – from Research Objectives to Conclusion (Developed for Research)

Define Research

Objectives

Pre-Study Interviews

(Secondary Data)

Pilot StudyInterviews

trigger

did in

form

Literature Review & Synthesis

guidesderiveTheoretical Framework

Main Study Interviews

data collection thematic analyses

thematic categorisation

main observations

are incorporated

build

advect into

areincorporated

is inc

orpora

ted

deduce

Formulation

Evaluation Distillation

validate

and guide

Discussion and

Conclusion

lead

Results and Findings

Activity

Investigation

Writings(Thesis

Chapters)

LEGEND

Statistical Analysis

IARPA Data*

triang

ulate

* IARPA forecasting tournament – four year study based on probabilistic predictions about geopolitical questions with a focus on the accuracy of the forecasts

78

The pilot tested the research design for the final stage and the pilot interviews validated

and guided the main study interview schedule. Three interview sessions carried out –

stemming from a pre-study (previous interviews used as a secondary source), the pilot,

and the main study – were investigated also regarding thematic themes from the literature.

Besides, statistical analysis of data from the IARPA forecasting tournament was used as

a triangulation and validation exercise. The collected and analysed data resulted in

findings to form and substantiate the conclusions.

The progression of the research in the flow-chart above illustrates the overall research

strategy. By using this strategy, the researcher tried to answer if and how PMs can be

made more adoptable for corporations to enable decision-making around their selection.

How this research aim is embedded in the methodological context is shown in Table 5-1.

Table 5-1 Adopted Research (Developed for Research)

Addressing Research Objectives

High-Level Research Objectives 1-4 Methodological Context and Data Collection

1. Evidence of corporate use, identifying advantages and disadvantages of PMs Constructionist approach to the literature review

2. Explore challenges in the use of CPMs and compare to alternatives Constructionist approach to the literature review

3. Gauge corporate forecasters’ and researchers’ knowledge of PMs, their choice of tools, and forecasting challenges Cross-sectional qualitative semi-structured interviews

4. Framework to guide PM adoption and recommendations for companies on the use of PMs

Interpretative paradigm applied via interviews and literature analysis

Guiding business research along those lines provided the required flexibility for this

research (Morrell and Learmonth, 2015).

The methods of data collection, the selection of the respondents, the research process, the

type of data analysis, the research limitations, and the ethical considerations of the thesis

are outlined in the next sections.

79

5.2 Sampling, Research Setting and Participants

To legitimise the choice of the expert interview as an independent research method, a

precise specification of the experts is necessary (Misoch, 2015). The concept of an expert

is competence-related and specified through capabilities (knowledge and skills) that are

acquired through a demanding, potentially long-term and extensive education and a

‘professionalisation’ through updating and expanding the existing specialist knowledge

(Hitzler, 1998). Wiig’s (1993) skill categorisations declared an expert to be highly

proficient in a particular area, and generally knowledgeable. Based on these explanations,

experts have knowledge that concerns a specific area and is not part of the general

knowledge (Misoch, 2015); they are representatives of a specific social group with a

specific knowledge, whose knowledge of a subject is of interest. The aspect of

professionalisation and the classification of expertise laid out here made it feasible to

identify experts. Using this meaning of expert the researcher sought professionals and

specialists as a source of information within a given organisational or social function with

at least five years of experience and complying with the criteria given in Table 5-2.

Table 5-2 Chosen Expertise Levels in Interview Partners for the Anticipated Interview Domains (Developed for Research)

Economic Professors Enterprise Interviewees Consulting Firms PM Software Platform Vendors

Professor or Assistant Professor

CEO or company director, managing director, general manager, Geschäftsführer

Manager or senior manager in a forecasting or strategy department (per their job title / functional specification)

Consultants that have published in the field and have a track record of advising practitioners

Business executive at the vendor

Regarding the number of interviews to be conducted, Romney, Weller and Batchelder

(1986) and Weller (2007) found that small samples, between nine and twenty-five

participants, are sufficient to provide comprehensive and credible information in a

specific cultural context, provided that participants have an appropriate level of expertise

in the field of investigation.

This is the case with experts in forecasting, from consulting, and academic research in

PMs. Önkal et al. (2017) and Bolger and Wright (2017) provided evidence that such

experts are needed as they have knowledge relating to anticipating the future and can

provide meaningful insights and feedback on assessing forecasting methods.

80

5.2.1 Inclusion of Secondary Data – Pre-Study Interviews

In line with the above, the researcher had already previously conducted 35 interviews and

captured their content in interview transcriptions between 2013 and 2015 which were

used as a secondary source. Even though, strictly speaking, secondary data would be data

that is collected and/or analysed by someone else, this does not automatically have to

apply (Bell, Bryman and Harley, 2018). It can also refer to a researcher’s original data

that is reworked at a later stage (ibid.). Qualitative data can be considered as secondary

data (Corti et al., 2019) and including additional sources of data, provided they are of

appropriate quality, can increase the value of an investigation (Alvarez, Canduela and

Raeside, 2012; Bell, Bryman and Harley, 2018).

When using secondary data, a sound methodological approach is essential. For qualitative

data collections, such as interviews, apart from the content that reflects the actual (data)

observations parlayed by the interview partners, information about the participants, such

as biographical and other contextual background, should be kept to increase the reuse

value (Haaker, Corti, and Van den Eynden, 2019), an approach the researcher applied.

The researcher employed high quality transcription, matching the analytic and

methodological aims of similar research and used an interview guide to ensure a level of

uniformity across data collection and assuring the quality of data as recommended by

Bucholtz (2000), Oliver, Serovich and Mason (2005), and Haaker, Summers and Corti

(2019). The researcher had availed himself of such advice regarding methodological

approaches to data collection prior to the first interviews conducted, and used an

exploratory-based, qualitative approach with semi-structured interviews to identify

causes and backgrounds of what was expected of CPMs.

The precautions exerted by the researcher (and set out above) justified the use and

inclusion of the previously conducted interviews as a pre-study (treated as secondary

data); the pre-study interview content was factored in during the data collection analysis

of the main study.

The pre-study also allowed the researcher to form a first high-level view on the usefulness

of the research objectives as the interview partners at that time clearly indicated perceived

challenges with management acceptance of PMs together with abandoning PMs and the

use of alternatives as the main points from these interviews, which helped to address

research objectives 2 and 3.

81

5.2.2 Research Participants

The researcher took care to ensure the representativeness of the sample of managers

interviewed in the course of the research study so that appropriate comparisons were

made and credible conclusions drawn.

As mentioned earlier, forecasters are usually the ones using the prediction tools in a

company, they then pass on their forecasts to recipients or consumers of their work who

basically have little or no influence on the methods used. Conducted interviews made this

clear as well as research on the organisational aspects of forecasting departments by

Lapide (2003) and Šindelář (2014).

For interviewing participants who are especially knowledgeable about the topic, the

researcher applied purposive sampling and contacted forecasting personnel on the level

of manager or senior manager within a company (see Table 5-2 above). These were

deemed to be suitable sources due to their knowledge, work experience, and daily

involvement in forecasting processes (Frey, 2018). Research indicated that to reach such

managerial levels within a company takes at least five to seven years and usually requires

at least a bachelor’s degree (Day et al., 2014; Veatch, 2016; Winkler, 2020).

As an example, the work experience of the pilot study participants, categorised by job

title, academic degree, involvement in forecasting, and the length of employment in the

respective position, is shown in Table 5-3. Some candidates from the pre-study were

business executives outside of the forecasting function, i.e. about 20% of all interviews,

adding a different flavour of expertise. Academics that research the field or specific

subjects also fulfilled the criteria of expertise and knowledge in the researcher’s view.

The pilot and main study interviewees could be badged as the main set of informants for

this research, rectifying a gap and expanding on information regarding the potential

applicability of CPMs in the field of innovation management – as highlighted in sections

4.2.3 and 4.2.4 for example. This led to including additional people to be interviewed

with knowledge in this specific area. The researcher made sure that they possessed a

similar breadth of experience as outlined above and illustrated for the pilot interviewees

in the table below.

82

Table 5-3 Pilot Interview Partner’s Experience and Qualifications (Developed for Research)

Interview Partner (Job Title or Similar) Qualification Forecasting Sphere

(Where Applicable) Nature of

Forecasting

Time Span in Forecasting Function (or Work Experience)

Principal Scientist PhD Running a PM for Sales Forecasting (Manufacturing) Short-term 20 years

Director of Research PhD New product development; Ideation Long-term 6.5 years

Professor PhD n/a n/a 18 years

Director of Business Development PhD Sales Forecasting new products

(Pharmaceuticals) Long-term 12 years

Professor PhD Preference Markets Short-term 14 years

Director of Business Development BSc Technology research Long-term 14 years

Professor PhD n/a n/a 15 years

Director Innovation DPhil Technology research Long-term 10 years

Head of Economic Research PhD Growth Rates, Foreign Exchange Rates, wage dynamics Long-term 13 years

Client Manager Underwriting MA Strategic Market Analysis Long-term 14 years

Senior Program Manager MSc Budget Planning and Forecasting (Manufacturing & IT) Short-term 7 years

President and CEO PhD Running a PM equivalent for Product Development Long-term 19 years

Senior Consultant Strategy & Processes PhD

Project Delivery, Assessment Business Environment Long-term 6 years

As with the chosen candidates for the pilot, all interview partners for the study came from

four areas: the researcher interviewed enterprises, consulting firms that have advised on

PM deployment, purveyors of CPM software, and economic professors that have

researched the field. Incorporated in the thesis is information from the pre-study, pilot

and main study, and additional interviews on the extension of PMs with respect to

innovation management as depicted in Table 5-4.

Table 5-4 Interview / Qualitative Data Collection Phases (Developed for Research)

Phase Activity Number of Interviews (65 Overall)

Phase 1 Collation of Secondary Data 35

Phase 2 Pilot 13

Phase 3 Main Set of Interviews 7

Phase 4 Extended Interviews 10

To achieve representativeness, the researcher asked the interview participants common

core questions based on an interview guide to conduct the interviews in as standardised a

way as possible and to derive comparable themes from them (Turner, 2010); data

saturation was assessed on the common core of questions. The interview schedule used

is displayed in Appendix D.

83

5.2.3 Sampling Technique / Approach

Choosing interviewees from companies regardless of the specific industry they are in was

feasible, as strategic planning develops in a similar way even in dissimilar companies

(Gluck, Kaufman and Walleck, 1980; Roley, 2006). Corporate forecasting also faces

similar problems in most cases (Daheim and Uerz, 2008), only specific details may vary

from company to company (Price, 2007); thus, interviewees from different companies

and industries can be compared on the main topics. Participants’ involvement was mainly

in long-term strategic forecasting such as in pharmaceutical or manufacturing sectors and

in environmental scanning or market intelligence, as shown in Table 5-3. For that reason,

the researcher did not seek to restrict interview partners from enterprises to a particular

branch of commercial activity and included forecasting managers or executives

regardless of industry.

The researcher recruited candidates by email, he was introduced to them by business

associates or through referrals from conducted interviews (cf. Heckathorn, 2015). Prior

to an interview an information letter was sent out by email (cf. Appendix C). The

researcher deemed knowledge about the interview topics to be necessary from the outset

and therefore used purposive, non-probabilistic sampling based on a candidate’s relevant

experience. When targeting such experts, potential candidates were accessed with the

help of outside introductions and through recommendations implying an element of

snowball sampling (cf. Daniel, 2012; Creswell and Creswell, 2018).

Even though the interviewee selection was based on accessibility (cf. Etikan, Alkassim

and Abubakar, 2016), the researcher had not drawn the interview subjects directly from

his personal or professional network, he never worked with them and they were suggested

by others; that made them, in a sense, contacts once removed. Homophily was thus

limited as the researcher came from a completely different background than the contacted

participants and hence appropriate candidate diversity was assured; the researcher did not

share professional characteristics or social ties with the participants, the traits of the

researcher and the recruited were independent, an important feature of an interview

process (Moemken, 2017; Crawford et al., 2018).

84

The analysis and results stage therefore encompassed and encapsulated information from

the pre-study, comprising 35 interviews as secondary data, the pilot which tested and

honed the interview process with 13 interviews, 7 interviews conducted for the main study

in 2020, and then a final and additional 10 interviews to address the expansion of PMs to

innovation management with additional questions on relevance and implementation of

PMs in this particular area (cf. Table 5-4).

5.2.4 Research Setting – Interview Guide

Supporting an interview process is done through an interview guide, which typically

consists of three to a maximum of eight topics comprising one to three main questions

each (Bogner, Littig and Menz, 2014). In this instance, the researcher chose three themes

as a preparation for content and method for the individual queries. Each topic contained

two questions to guide the dialogue with the interviewees:

Description of the company’s forecasting process

• How does the planning process work at the company? • What tools are used in forecasting (e.g. scenarios, Delphi, experts) and why?

Prediction markets in the context of forecasting

• Knowledge of prediction markets – Yes or No - If No, a short explanation: artificial stock market (example Iowa

election market) that can be used in corporations as well, exemplify with a market betting on the winner of ‘Italy versus France’ in soccer

• Can the interviewee see a use of prediction markets in the context of forecasting and in their own company?

- If they are already used, in what context? - What lessons are learnt from implementation?

Challenges of the existing forecasting approach

• Challenges in eliciting forecasts at the interviewee’s company • Does the interviewee see challenges stemming from the organisational

hierarchy in getting and aggregating information?

On a high level, the question topics concern challenges around eliciting and amassing

information, particularly related to company culture; the workings of a company’s

forecasting process including tool use; plus knowledge of PMs and gauging their

usefulness for company forecasting (this topic included a brief PM explanation). This

summary demonstrates how the questions the researcher developed map to the research

objectives. How individual questions associate with objectives is illustrated in Table 5-5.

85

Table 5-5 Interview Questions’ Fit to the Research Objectives (Developed for Research)

Research Objective (Abbreviated) Interview Question

Objective 2 – Challenges in the use of CPMs

• Challenges in eliciting forecasts at the interviewee's company

• Does the interviewee see challenges stemming from the organisational hierarchy in getting and aggregating information?

Objective 3 – Knowledge about PMs in forecasting circles Objective 3 – Understand challenges in a forecasting process

• Knowledge of prediction markets - Yes or No • Can the interviewee see a use of prediction markets in the

context of forecasting and in their own company

• How does the planning process work at the company? • What tools are used in forecasting (e.g. scenarios, Delphi,

experts) and why?

Objective 4 – Guiding Adoption of PMs • What lessons are learnt from implementation?

Thus, the questions asked gave support to the overall research aim by underpinning the

research objectives.

5.3 Analysis Plan

The researcher deployed qualitative content analysis treating the texts or interviews to be

analysed as material containing data. In content analysis data is taken from the texts or

interview transcripts, i.e. raw data is extracted, processed and evaluated. With the

extraction, information is taken from the text and evaluated, the corresponding text is

read, and it is decided which information is relevant for the investigation. To derive the

content to be analysed it was decided not to use text analysis software, qualitative analysis

packages such as NVivo, which also made the use of transcript coding unnecessary (cf.

Bell, Bryman and Harley, 2018).

The researcher preferred to be closer to the data and angling for deep saturation, making

sense of the interviews and identifying themes himself, thereby, with his background,

understanding the content better than letting a ‘machine’ doing the examination. The data

for this thesis was considered being fairly small-scale and hence manual content

processing by the researcher was expected to suffice. McLellan, MacQueen and Neidig

(2003) also suggested that structured interview notes are adequate as a basis for analysis,

the same as a taped and transcribed conversation.

By adopting a structured text analysis, the researcher also created an open category

system which reduced the amount of information, and structured it according to the

objective of the investigation. The openness of a category system means that the

characteristic values are described freely (Gläser and Laudel, 2010).

86

The scale or list of values (keywords or key aspects) were created by the researcher during

the extraction process, which ensured that unanticipated features were adequately

incorporated. Characteristics did not need to be fitted into an ex ante predetermined

system (Kromrey, 2006). The existing literature can also be the population, the ‘textual’

data where a researcher needs to go for data collection and analysis as opposed to people

being the object of a study (Katz, 2002; Bell, Bryman and Harley, 2018). Therefore, it is

meaningful to categorise appropriate literature sources into major themes and subtopics

(Friese, 2019), allowing a purposeful comparison of literature and interview results,

which the researcher adhered to.

A certain quantification of results from qualitative research tends to support revealing the

universality of the outlined phenomena (Silverman, 2013; 2017). Miles and Huberman

(1994) stressed that using an interview script a researcher can categorise interview

responses by theme and thus highlight the main concepts, themes, and issues and

furthermore provide their frequency of occurrence. The researcher used a count of

individual key phrases (i.e. derived conceptual themes which comprise keywords and

their sentiment expressed) to gauge their comparative importance.

Re-reading the interview scripts, the researcher labelled certain passages or paragraphs

as to their higher-level content and cross-compared these chosen ‘codes’ within an

interview and between different interviews as prescribed by Silverman (2016; 2019). For

instance, “knows about PM operation at other companies” was given the label ‘Business

example’. Furthermore, the researcher refined – expanded or collapsed – the labelling as

the analysis progressed, at the end ‘Business example’ was honed to ‘Market in Use’ as

a conceptual theme through such thematic analysis (cf. Bell, Bryman and Harley, 2018).

Such coding helps to make sense of derived data and to interpret it (ibid.). Following this

approach, the researcher arrived at the possibility to link the data to the research question

and theoretical concepts.

For the triangulation purposes alluded to in section 5.1 data from a forecasting tournament

run by IARPA as the lead sponsoring agency was used, such prize competitions are

regularly initiated by US federal agencies. “Forecasting tournaments are level-playing-

field competitions that reveal which individuals, teams, or algorithms generate more

accurate probability estimates on which topics” (Tetlock et al., 2014, p.290), the term

‘algorithm’ would cover the subject of PMs.

87

As a challenge to develop innovative solutions “for integrating crowdsourced forecasts

and other data into accurate, timely forecasts” (Office of Science & Technology Policy,

2019, p.B-90), IARPA created a tournament from 2011 through 2015 called ‘Aggregative

Contingent Estimation’ focusing on optimal performance by harnessing the wisdom-of-

crowds. To improve the accuracy and timeliness of forecasts, the tournament compared

five competing research programs around “closed-ended forecasting questions that

concerned specific, objectively verifiable geopolitical events” (ibid., p.B-92). All

forecasting questions were open for estimation on a platform for a specified period of

time. Until a question was closed, tournament participants were able to update their

forecasts.

The researcher analysed these data descriptively with a focus on PM accuracy leveraging

box plot diagrams; the data are publicly available at a data repository from Harvard

University.

5.4 Pilot Study – Interviews In any research, it is helpful to pilot the interviews with a couple of participants before

the actual data collection (Gläser and Laudel, 2010; Bogner, Littig and Menz, 2014) to

test if an interview schedule is clear, understandable and the interviewee capable of

answering the research questions, or if any changes to the interview schedule are required.

Also testing if the administrative arrangement for the interviews works and if the

responses can be analysed to give useful information. This section reports how the pilot

for this research was designed, run and analysed.

5.4.1 Interview Approach Following the interview guide from section 5.2.4, the researcher conducted thirteen

interviews between 2018 and 2020. Forecasting and planning managers from different

business sectors as well as academics and consultants were interviewed on their views of

the value of forecasting tools. The thirteen pilot study participants (see Table 5-3 above)

were contacted either by mail or telephone prior to the interview. Four interviews took

place face-to-face in the office of the participant, three via Skype, and six per telephone.

The individual interviews lasted between 45 minutes (the two shortest) and three hours

(the longest interview) – for the remainder, four conversations went on for an hour, four

proceeded for two hours, and two interviews took 90 minutes and an hour and a quarter,

respectively. The interviews were conducted in English or German.

88

To conduct the pilot, the interview guide was first tested with six candidates. The

researcher interviewed candidates to check the comprehensibility of the questions and

their logical structure and to ensure that the interviews took a course which did not appear

‘halting’ or unstructured. This preparatory study yielded both results and feedback from

the interviewees that the questions regarding the subject under investigation were

practical, meaningful, and understandable, and allowed useful data to be acquired. Some

concerns were expressed regarding a question about the knowledge of PMs where the

researcher used an example to illustrate the workings of a PM. Accordingly, a more

comprehensive exemplification was used to increase the probability of a response and

reduce misunderstanding.

As in the first set of interviews the researcher could clarify the confusion around what a

PM would be during the interview by expounding on the topic, their substance could

therefore be kept as well. A further seven interviews were carried out, arriving at thirteen

pilot interviews in total as shown in Table 5-6. Feedback from the additional interviews

did not necessitate any further changes to the interview structure or approach.

Table 5-6 Interview Partners while Piloting (Developed for Research)

Economic Professors Enterprise Interviewees Industry Sectors Consulting Firms PM Software Platform

Academics (3)

Pharmaceuticals (1) Manufacturing (2) Services (1) Telecommunication (2)

Gartner (1) Inovex (1)

McKinsey (1) CrowdWorx (1)

Stier (1999) and de Casterlé et al. (2012) supported such an approach, as they saw no

guarantee for optimal interview guidelines, rather semi-structured interviews would be

especially appropriate to identify causes and backgrounds with a scheme of questions and

an interview guide where the sequence and wording of questions is not fully

predetermined.

5.4.2 Analysis Approach to Pilot Interviews

For the analysis of qualitative data one approach is to code interview transcripts to help

uncover or infer patterns and ideas classifying content into a pre-described category

(Woolf and Danahy, 2017), an advice that resembles the researcher’s analysis plan in

section 5.3. Such an approach also allows for content clustering with the help of keywords

or word clusters (ibid.) supporting the researcher’s choice of using conceptual themes in

the analysis of the interviews conducted.

89

All interviews were evaluated and analysed individually using thematic analysis; the

researcher coded and closely examined the data to identify broad themes and patterns

useful to the topic and research question explored, as suggested by Braun and Clarke

(2012). The themes generated by the researcher established the key thematic aspects

pertaining to the utilisation and acceptance of PMs.

It is noted from Verschuren (2001) that each formalisation based on a higher-level

description and exploration of phenomena needed for comparison across interviews is

reductionist in nature. However, extracting contextual meaning from unstructured

(interview) texts can be done by focusing on core themes and relevant messages (Miner

et al., 2012) giving credence to the analysis method chosen by the researcher.

Amounts of such evidence – quantities of thematic aspects – can be viewed as useful data

according to Kvernbekk (2013), and context themes are suitable in studying experience

and meaning, “the participant’s life-world” (Daher et al., 2017, p.2). Coding of the typical

features of the transcripts is a prerequisite (Nassehi, 2019).

These arguments raised the question if following this approach, using frequencies and the

conceptual themes themselves, would remove the context and sentiment too much.

Decomposing a research object into its key or basic elements and treating those

analytically and even numerically could perhaps leave aspects of social reality contained

in the interviews underexposed and not fully incorporated into derived results. A

reductionist approach could potentially fail to fully grasp the whole of an object under

investigation, isolate it from its context.

The researcher was convinced that this had not happened, only negligible loss would have

occurred, as demonstrated in Table 5-7, Table 5-8, and Table 5-9 based on partial

transcripts of three interviews. The researcher employed verbatim quotes from these

interviews to expose the chosen conceptual themes which later support the derived

inferences. Some ‘phenomena’ expressed in the dialogues might be too ambiguous to

allow objectification, though (Little, 2016). When an unambiguous key phrase could not

be assigned, such an expression was included as a core aspect text of the interview to

avoid losing sentiment from such paragraphs or sentences (exemplified beneath Table

5-7); but generally, there were only very few of those expressions that could not be

assigned a specific thematic aspect.

90

Table 5-7 Example 1: Interview Content and Derived Conceptual Themes – Coded Transcript Sample from Enterprise Interviewee (Developed for Research)

Conceptual Theme Interview Transcript Part Juxtaposed with Its Specific Topic

Part of Work Processes “Existing planning and forecasts are based on controlling tools and empirical values, the company wanted to include ‘outsiders’ more strongly, e.g. field staff, as equipment planning is very volatile”

Implementation Issue

“With 800 employees at one hundred locations, the flow of information is very low. But field sales representatives have a feeling for the market” “Decided for PM-tool, because of optically very good user interface and playfully implemented (CrowdWorx)” “25 to 30 questions are in one stock exchange, these are perhaps almost too many questions; it must go fast and simply, one has no time to occupy oneself with it for a long time” “Length at the beginning 2 weeks was too short, then 1 month - must be adjusted individually” “put a lot of commitment into it to promote it every time anew; Participation not as high as hoped for; No self-runner, a permanent resource required; Promotion but also evaluation was necessary - extremely important”

Fosters Employee Motivation “Incentive: you can influence what happens in the company Forecasts were accurate, but not accurate enough, but the comments (over 1,000) were very valuable - information about the market, fed back to management, was seen positive”

Market in Use “Forecast exchange: 3 to 4 times a year for approx. 1 month”

This interview also contained “Make it clear and plausible to the employees that the data

will not be used against them, e.g. if one was wrong with the forecast” which was kept

separately in accordance with the remark above.

Table 5-8 Example 2: Interview Content and Derived Conceptual Themes – Coded Transcript Sample from Interview with Economic Professor (Developed for Research)

Conceptual Theme Interview Transcript Part Juxtaposed with Its Specific Topic

Weak Benchmark

“Election study comparisons always ignore the topic, e.g. are political markets really superior to polls as election predictors? Prediction markets were not compared with strong, established methods/tools.”

Not a Topic of Interest “There is still very little knowledge about the forecasting quality of PMs, papers still read like 10 years ago.”

Market Principle Questioned “PM vs. polls: PMs only work because there are polls. Little is known about the conditions under which PMs achieve good results.”

Management Acceptance

“To set up/position certain markets, decision-makers must disclose information which the company’s ‘man in the street’ should not know. Preserving the asymmetry of information. Markets function very well when the asymmetry is reversed, the participant knows it, the decision maker does not know it and cannot enquire about it (project management as an example).”

Market Abandoned “The PM at the university in Karlsruhe ran out of interest and expired. Chris Masse, a ‘PM-hero’, running the Toronto PM blog: Hype did not fit with what the research results say”

91

Table 5-9 Example 3: Interview Content and Derived Conceptual Themes – Coded Transcript Sample from Enterprise Interviewee (Developed for Research)

Conceptual Theme Interview Transcript Part Juxtaposed with Its Specific Topic

Implementation Issue “the 2nd or 3rd market has to be waited for to achieve improvements in quality (1st market is a learning effect). Usability significantly improved in the last 7 to 8 years, users more internet-savvy as well as more open to playful, intuitive aspects”

Management Acceptance

“When the market closes, the ideas have to be filtered/clustered even further to be usable, which is also an important process in line with the strategic roadmap.” “A market can be contrary to the organizational culture, philosophy, hierarchy/power is levered out (everyone has the say) - creates barriers”

Market in Use “With very fast changes of the topic much faster information today about PMs – Big Data is passive - PM is active; our PM is much more dynamic”

Use of Alternatives “80% of prognosis tools work by time series analyses” “It's always just another additional source of information.”

Wimsatt (2006) and Little (2016) averred that generalisations or contextual

simplifications can be based on reductionist approaches describing higher-level systemic

phenomena appropriately and allowing to infer properties of the whole in a pragmatic

holistic way. In such a fashion, non-eliminative reductionism would exist which

recognises the reality of complex human-level structures of meaning as derived from

interview interpretation (Slingerland, 2018). Here human emotion and dispositions can

undergo a successful formalised representation as in the use of key phrases (Schuller,

Ganascia and Devillers, 2016). To ‘translate’ textual regularities into a conceptual theme

makes keywords and their expressed sentiment manageable; quantification of themes can

help discover further relatively invisible regularities (Nassehi, 2019). This seemingly

radical simplification, a reduction to numbers, to the ‘digital’, makes it possible to

compare things that are all too different in their ‘analogue’ forms, context, and sentiments

(ibid.). The researcher subsequently used such an account of individual key phrases to

assess their relative importance. A demonstration from the three transcripts in this section

is given in Table 5-10 and the approach applied to achieve the first results from the overall

pilot interviews in section 5.4.3.

Table 5-10 Summary of Thematic Aspects from Three Interview Transcripts (Developed for Research)

Conceptual Themes Stemming from Example Interview Content

Fosters Employee Motivation from Example 1

Implementation Issue from Example 1, Example 3

Management Acceptance from Example 2, Example 3

Market Abandoned from Example 2

Market in Use from Example 1, Example 3

Market Principle Questioned from Example 2

Not a Topic of Interest from Example 2

92

Conceptual Themes Stemming from Example Interview Content

Part of Work Processes from Example 1

Use of Alternatives from Example 3

Weak Benchmark from Example 2

The researcher’s main objective to arrive at indicators of content comparison was thus

achievable by distilling all or most parts of the interviews into conceptual themes but also

subsequently into counts of those phrases without losing reference to the underlying

content. This also confirmed that the content of all thirteen pilot interviews could be used

in the main results as originally hoped for. Since the searcher had access to a reasonably

large amount of interview partners, the above results substantiated the researcher’s

decision to aim for this more expansive preliminary / pilot analysis approach (see also

next section).

5.4.3 First Results Based on Pilot Interviews

The researcher examined all interviews with a focus on major issues revealed, thereby

identifying key aspects related to the use of PMs as viewed from corporations and found

in the academic literature. Categorising and clustering the respective content from an

interview analysis based on section 5.3, and the theoretical aspects given in section 5.4.2,

allowed twelve key thematic aspects depicted in Table 5-11, with frequencies shown in

brackets.

Table 5-11 Key Thematic Aspects from Pilot Interviews (Developed for Research)

Conceptual Themes and Frequency of Occurrence

Accuracy (3) Fosters Employee Motivation (1) Hierarchy (8)

Implementation Issue (6) Management Acceptance (4) Market Abandoned (7)

Market in Use (4) Market Principle Questioned (5) Not a Topic of Interest (3)

Part of Work Processes (1) Use of Alternatives (7) Weak Benchmark (1)

In the first instance, the researcher decided to keep all derived key aspects to be used for

further analysis. Three aspects only occurred once, but these subjects of ‘Fostering

employee motivation’, ‘Part of Work Processes’, and ‘Weak Benchmark’ were all

substantiated in the literature to play a role in the context of PMs by e.g. Gerhart and Fang

(2015), Horn and Ivens (2015); Vandenberg, Richardson and Eastman (1999), Lee et al.

(2019); and Duquette et al. (2014), Santillán-Salgado, Ulin-Lastra and Escobar-Saldivar

(2018), respectively.

93

As the literature furnished evidence of their noteworthiness, the researcher intended to

use all key phrases in subsequent content analysis to be conducted in line with the analysis

plan given in section 5.3. During thematic analysis the researcher took care to ‘monitor’

text recurrences and refined conceptual themes to arrive at the final core themes or

categories to concentrate on as advised by Silverman (2013; 2017) and Bell, Bryman and

Harley (2018).

A full report on the pilot interviews is incorporated in Chapter 6 ‘Results / Findings’ of

this thesis with the next section covering the pilot’s results in the context of addressing

the research questions and objectives.

5.4.4 Pilot Study Reflection The pilot scheme carried out yielded feedback regarding the administration of the

interviews, and the need to change an ambiguous question from the interview schedule

was identified. Thereby, the researcher ascertained that his study approach was viable.

Credibility of the analysis was achieved by ‘measuring’ the extracted key thematic

aspects against prevalence in publications.

A first comparison to generalised themes based in the literature showed that the derived

topics would make sense. Scouring the literature review found roughly 30 areas that

belong to the aspects of management acceptance (including questioning the market

principle) and a firm’s hierarchy, which have the two highest appearances in the

dissection of the pilot interviews, thus numbering nine and eight, respectively (see Table

5-11 above). Company hierarchy and the acceptance of a CPM’s results tie in with

research objective 2 of investigating the challenges of CPM’s use.

Furthermore, the interview results also contributed to research objectives 3 and 4.

Questions from the second and third interview topic – about knowledge of PMs and trials

regarding information capture plus company culture, respectively – pertained to gauging

PM knowledge and learning about perceived challenges (objective 3), which led to

adoption guidance (objective 4). Also resultant key concepts filled the quest for

understanding tool choice, reasons and challenges, from objective 3, brought up through

aspects like ‘Accuracy’, ‘Market Abandoned’, ‘Market in Use’, ‘Not a Topic of Interest’

or ‘Use of Alternatives’; their underlying content also helped to derive recommendations

in line with objective 4.

94

The learning from the pilot showed the applicability of the chosen research design and

approach and underlined the feasibility to obtain meaningful results for the main study as

well (cf. Chowdhury, Ahmmed and Hossain, 2020). The main study therefore continued

with a qualitative research strategy based on semi-structured interviews after

incorporating the small adaptions indicated.

5.5 Main Study – Interviews

Based on the successful pilot the process and logistics of the main study continued with

the research strategy as outlined in section 5.1. The specifics of the sample frame of

reference, sample type, where the respondents will have been drawn from, interview type,

and detailed proposed analysis of the results did not change for the main study.

The main study data collection, continuing the interview process and containing a

triangulation exercise, was embedded in the overall approach whose sequence was

presented in Figure 5.2. The flow of interviews was maintained but expanded beyond the

scope originally anticipated. The motive for this was based on the cadence of reasoning

leading to a suggested adoption of CPMs, a future role, in the area of idea management,

presented and argued for from the Introduction (Figure 1.3), to the Literature Synthesis

(Figure 3.1), and finally continued with the Theoretical Framework (Figure 4.2).

The qualitative information collected for the analysis and results stage was investigated

by means of a thematic analysis as in the pilot (Fereday and Muir-Cochrane, 2006; Yin,

2017). When determining the thematic topics in the analysis of the transcribed data,

comparison continued to be a dominant principle. Comparison is a targeted method to

classify interview data, enabling inductive theory development by linking categorisations

and brackets of data together (Skinnarland, 2013; Creswell and Creswell, 2018). The

responses from all interviews were compared via their ‘conversion’ into key aspects based

on the thematic analysis method from sections 5.3 and 5.4.2 which resulted in the key

thematic aspects in Table 5-11 derived from the pilot.

Leading on from the original investigation and adding the interviews on innovation

management was intended to determine the potential for future success and use of CPMs.

The groundwork to arrive at insights cumulating in the main study was put in motion with

an appropriate research design. Following sequential approaches laid out as a first concept

in the theoretical framework and subsequently in Figure 5.1 and Figure 5.2 the research

design comprised the logical steps visible in the flowchart of Figure 5.3.

95

The process abstract below contains dotted lines indicating reflective iterations which

were not necessarily performed.

Figure 5.3 Flowchart of the Research Stages (Developed for Research)

This schematic summary addresses the components of the research design with its

sequential and iterative steps.

Initial Research Questions

Literature Review and Conceptual

Model

Final Definition of Research Questions

Formulation of Research Strategy

Pilot study (13 interviews)

Main Interviews (n =7)

Extended Interviews (n =10)

Analysis of Secondary Data

(35 Interviews)

Validation(incl. qualitative data)

Finalise Thesis

Analysis

Analysis

Assessment of Pilot

& Secondary Analysis

96

5.6 Potential Methodological Constraints

Awareness of the bounds of one’s research puts the relevance of the findings into a

context of possible limitations. An adequate delineation of how constraints could affect

internal and external validity is not just about being self-critical: A researcher who

identifies limitations and clarifies how they might impact the study results shows rigour

(Price and Murnan, 2004; Greener, 2018). When appraising conceivable constraints,

internal validity relates to potential limitations of the study design and its internal

integrity, and external validity to the outward generalisability of the reported results

(Greener, 2018).

5.6.1 Validity

From a validity point of view, it is imperative to account for how the results came about,

which is explained by the detailing of the research strategy and design. Recipients ought

to be able to understand the analysis and the conclusions drawn from it being provided

with a description of the transcription and analysis process (Elo et al., 2014), which the

researcher undertook.

The findings were pertinent to the research questions, i.e. the researcher derived relevant

answers that were attributable to the mechanism of the framework used, explained and

demonstrated in section 5.4.4: Research questions asked were relevant in that they were

derived from the literature and operated within the framework forwarded. Further validity

was provided as participants were selected in an unbiased way and a recognised interview

procedure following a question guide was used.

The procedure of interviewing was unbiased and dispassionate, not guiding participants

to answer in a particular way and their participation was voluntary and informed, i.e.

ethical. To check validity of interpretation and analysis feedback was given to participants

to arrive at a consensus on the connotation of what they had said.

Enhancing the qualitative aspect would happen through a triangulation with quantitative

sources of information. The researcher planned to undertake some statistical analysis of

data from the IARPA forecasting tournament based on probabilistic predictions about

geopolitical questions with a focus on the accuracy of the forecasts. Thus, quantitative

results triangulate the qualitative analysis from the interviews, and the qualitative and

quantitative data gathering aspects lend credence to each other. Such a slight expansion

of the overall research strategy did not alter the methodology approach in a significant

way, the overall research was still primarily qualitative.

97

5.6.2 Bias in Qualitative Research

When planning the research, selecting interviewees, and conducting interviews, biases

can occur. Phenomenological or similar studies should always employ appropriate

precautions which will reduce researcher bias (Allan, 2003). An awareness of bias is

vitally important in all research, but whereas in quantitative research bias should be

eliminated altogether, in qualitative research, the major concern is acknowledging its

inevitable occurrence (Shuttleworth, 2009) and trying to mitigate it. Distortions can come

about in all phases of research, especially in an ‘interview context’. As interviews are a

major focus of data collection, the researcher mitigated this through use of a structured

guide.

The researcher was conscious that as phenomena are explored through relaying personal

experiences, participants and the researcher might influence these discussions. Awareness

of personal bias, and role and position concerning the topic studied is required (Howard

et al., 2016; Kross and Giust, 2019).

But rather than trying to eliminate the biases, monitoring and understanding if they shape

the collection and interpretation of data is sufficient according to Merriam and Grenier

(2019). The researcher achieved this by informing the interviewees that chose to

participate about the way the content would be used in the thesis, collecting their

feedback. This happened after the interviews took place to avoid response bias, and it

confirmed an unprejudiced representation in the thesis.

5.6.3 Generalisability

Often researchers aim to apply particular research findings to settings beyond those in

which they were originally situated to draw conclusions that are generalisable. Given the

specificity of forecasting within individual companies contacted by the researcher for

interviewing, the attempt to generalise and draw conclusions from the interview content

may not have completely contributed to closing the gap between the theory and practice

of PMs as much as hoped for, as each time there was a dependence on the situation and

the organisational details.

Conditional upon a sample’s appropriateness and representative or knowledgeable

contributors regarding the research subject, a sampling strategy is typically chosen for its

methodology and topic when pertaining to qualitative research and not because

generalisability of the results is required (Cruz and Higginbottom, 2013; Treves, 2017).

98

The researcher took care to include interview participants into his research commensurate

with such advice arriving at a general applicability of the results, since section 5.2.3

already explained that forecasting processes show sufficient similarities. At least a degree

of generalisability was achieved as broad principles were formed. The wider context this

creates is often generalisable, as generalisability is not a binary concept (Futoma et al.,

2020).

5.6.4 Reliability

Scholars should think about how they can confirm the credibility and concordance when

organising the analysis. Used expressions drawn from an analysed text need to accurately

reflect the information provided by the participants, achieving consistency of results, and

they should not be ‘invented’ or changed by a researcher in a way that they would not fit

the underlying data anymore (Polit and Beck, 2012). The researcher achieved this by

presenting his findings back to a majority of the interview partners who confirmed the

validity of the chosen terms in each case.

5.6.5 Reflections Regarding the Validity and Reliability of the Pilot

Reflecting on the pilot an acceptable level of validity and reliability was assured. It arose

from the choice of participants, method of gathering data, the analysis of that data and the

conclusions drawn. This was further supported by feedback and agreement on

conclusions with those interviewed. The concept applied to the pilot gave reassurance to

continue with the planned approach to be implemented in this thesis.

5.7 Ethics As a doctoral candidate of Heriot-Watt University, the researcher abided by University

regulations and standards, compliance with the University code of ethics was assured,

always acting honestly and professionally.

For the intended contribution of this work, outcomes were not exaggerated; this was

avoided by not embellishing, modifying or fabricating achieved results. Attention was

paid to process only data that had been validly received and contradictory facts or

countervailing factors were not suppressed, maintaining impartiality. Study participants

were informed about the purpose and process of the research before having been asked to

consent to participate and only if they voluntarily gave consent were they approached for

an interview. They were also informed about the results and were able to withdraw their

data and responses if they so desired.

99

As interviews were the main vehicle of data capture, prior to an interview’s

commencement the study details together with ethical principles (e.g. anonymity,

confidentiality, the option to withdraw) were relayed to the participant (cf. Legard,

Keegan and Ward, 2003; Gill et al., 2008). This is a central aspect of an informed consent

process (Kadam, 2017), i.e. a researcher explains the research to the potential respondent

who then chooses to volunteer or not, without any coercion. Statements made previously

by one person or persons in an interview were not relayed to others, unless previously

agreed upon; direct quotes attributed to the originator would only be used if explicit

consent had been given, otherwise ensuring all data and results remain anonymous and

cannot be traced to a specific contributor. Persons interviewed would not have been put

under pressure or answers demanded, immediately accepting a situation where people

would refuse to answer a question to maintain an atmosphere of fairness and

responsibility.

100

6. Results / Findings

The literature review successfully established PM’s theoretical underpinnings and

challenges in using them; the latter had been reconfirmed during data collection depicted

in this chapter which points out possible use cases for corporate PMs, too, from interviews

and literature. PMs have a potentially tarnished record, scant use, complex

implementation, and diminished usability and methods like Big Data emerged to

challenge traditional forecasting approaches. PMs as they are theorised in the literature

abstracts too much from ‘facts on the ground’, the literature saw PMs almost as an

anachronism. PMs re-emergence in idea management could be the anachronism, though,

pointing to a useful (future) area of application for corporate PMs.

Data collection based on interviews (particularly around questions of corporate usage)

was intended to unpack the reasons and rationale jointly with additional literature analysis

to tie together hindrances but also use cases for CPMs. Innovation appeared as a theme

to make CPMs successful (again) which led the researcher to a stance on carrying out

additional interviews (past the data collection exercise in section 6.1) with a specific focus

on innovation management, described in section 6.6.

6.1 Data Collection via Interviews For data collection the researcher continued the interview approach outlined in section

5.4 adding to the pool of dialogues from both the pilot and the pre-study. This yielded a

further seven interviews in 2020 resulting in 55 interviews overall to be analysed

(represented in Table 6-1). For research projects which are about understanding

experiences “among a group of relatively homogeneous individuals, twelve interviews

should suffice” (Guest, Bunce and Johnson, 2006, p.79) and a number as low as six would

be consistent for qualitative research according to Guest, Namey and Chen (2020).

Therefore, adding seven new interviews to the thirteen from the pilot study in addition to

the data from the pre-study qualified as a comprehensive and saturated data pool to base

a thematic analysis on (Braun and Clarke, 2019; Guest, Namey and Chen, 2020).

Table 6-1 Interviews from Data Collection, Pre-Study, and Pilot (Developed for Research)

Economic Professors (12) Enterprise interviewees (30) Industry Sectors Consulting Firms (7) PM Software Platform (6)

Academics (12)

Chemicals/Pharmaceuticals (2) Financial Services (4) Manufacturing (10) Media (1) Professional Services (7) Services (3) Telecommunication (3)

Gartner (1) Hochschule St. Gallen (1) Ignite Consulting (1) Inovex (1) McKinsey (3)

CrowdWorx (5) Predicat (1)

101

The pilot scheme had served to test the approach, the pre-study had already garnered

useful material for inspection, and the additional data collection rounded out that picture,

on balance creating comprehensive responses from the interview partners. The researcher

thus acquired sufficient information to form an informed view on important aspects of

CPM usage and the still existing challenges.

6.2 Interview Content, Illustration, Analysis, and Results

All interviews were reviewed and analysed the same way, following the outline from

section 5.3 and the analysis rationale in section 5.4.2. In section 5.4.3 concerning the pilot

study a partial analysis had been carried out which will be expanded upon. The 35

interviews from the pre-study could be slotted into the same analytical process as the

research methods used in this thesis had not changed significantly compared to the chosen

concepts on which theses interviews were based on (cf. section 5.2.1).

The pilot study content is presented more comprehensively below, which serves as a

demonstration of the way the overall review was performed (similar to the examples from

section 5.4.2) and highlights first results in a more detailed fashion.

6.2.1 Transcription, Analysis and Content Portrayal – Thematic Aspects

The researcher conducted the interviews per telephone, in person or via video conference

and subsequently transcribed them into separate Word documents.

All interviews were then reviewed individually and analysed with a focus on key points,

consequently establishing key thematic aspects pertaining to the utilisation of PMs, by

which the respective content would be categorised and clustered. The researcher then

created a spreadsheet to present an overview of the contained key points and cross-

checked if they fit the key aspects of the individual interviews applying a thematic

analysis (Silverman, 2016; 2019).

Lastly, a core aspect was added to emphasise the respective approach to the utilisation of

PMs, and to highlight the discrepancy between the known benefits of PMs and the reasons

for their lack of use when apparent.

Twelve key aspects had emerged from the pilot study, analysing the pre-study data and

the main study interviews did not change those themes, as can be seen in Table 6-2.

102

Table 6-2 Key Thematic Aspects from Interviews (Developed for Research)

Conceptual Themes

Accuracy Fosters Employee Motivation Hierarchy

Implementation Issue Management Acceptance Market Abandoned

Market in Use Market Principle Questioned Not a Topic of Interest

Part of Work Processes Use of Alternatives Weak Benchmark

As no change occurred, the interpretation of the interview results was subsequently

undertaken in an overall approach pulling all content together. To show how the content

was derived into the thematic key aspects, the pilot interviews are used as a

demonstration: their key interview points are tabulated together with the associated

conceptual theme in section 6.2.2 below.

Among the entire 176 occurrences of the key aspects, ‘works-council’ was also a topic

but was only mentioned in three interviews with companies from Europe where it might

be a more important issue. At the time of writing, of 29 European countries in the EU

plus the UK and Norway, 18 had works councils established in companies but the

researcher did not consider the subject further in the overall analysis due to its small

prevalence and potential lack of generalisability.

6.2.2 Representation of Interview Content – Pilot Study – Thirteen Interviews All interviews depicted in this section have the same structure and headings:

Interview Category, e.g. Academia Interviewee organisation or industry sector, e.g. Telecommunication

Tabulation (sample depiction):

Conceptual Theme Key Interview Points per occurring conceptual Theme

Hierarchy Some managers can’t handle the truth – not everybody wants the truth to come out, crowd wisdom influencing final decisions is troubling, … …

Core Aspect: i.e. “an appropriate, salient quote from the interview”

The core aspect emphasises a core statement of or a major conclusion from the interview

or represents a special point that the interviewee wanted to make which was not

adequately covered by a conceptual theme.

Interviews are now depicted per the four interview domains of academia, enterprises,

consulting firms and software vendors.

103

Academia

Anderson School at UCLA

Conceptual Theme Key Interview Points

Management Acceptance

By its very nature the information contained in a market is ‘public’ – every trader becomes aware of the stock, lack of privacy in the data is troubling for lots of people, they don’t want contradicting evidence that is publicly known Why does not every company in the U.S. run a permanent PM?

Hierarchy Some managers can’t handle the truth – not everybody does want the truth to come out, crowd wisdom influencing final decision is troubling, managers want to retain more control

Market Principle Questioned People do not believe the market, not everybody grasps the idea

Implementation issue Practical issue/logistical challenges – have to have software, how to recruit, keep interest/motivation

Market Abandoned For a PM at the Pentagon the reason for failure was not that it did not work but politics (National Security Advisor John Pointdexter wanted it). Microsoft: people running it did not like it

Use of Alternatives You can capture most of what you can learn from a market from private opinions: solves the privacy problem because you do not reveal the stock value – protects information, resolves CEO-misgivings, no public humiliation

Core Aspect: “I can either do a traditional research study or a PM, but then the PM loses

pretty quickly, as even people in the midst of the game sometimes struggle with the

concept.”

Polytechnic Burgenland

Conceptual Theme Key Interview Points

Not a Topic of Interest Has never used the method himself The topic did not occur at the university in the context of strategic planning, i.e. it was never discussed.

Market Principle Questioned Don’t see the connection between typical questions of management strategy and bets placed in a market People cannot estimate how they will behave in the future when it comes to changing trends

Core Aspect: “PMs are an interesting tool, but not a promising concept. Not more suitable than other survey tools for strategic topics.”

Ludwig-Maximilians-University Munich

Conceptual Theme Key Interview Points

Weak Benchmark

Election study comparisons always ignore the topic, e.g. whether political markets in forecasting elections are really nonpareil to polls PMs were not compared with strong, established methods/tools.

Not a Topic of Interest There is still very little knowledge about the forecasting quality of PMs, papers still read like 10 years ago.

Market Principle Questioned PM vs. polls: PMs only work because there are polls. Little is known about the conditions under which PMs achieve good results.

Implementation Issue

To set up/position certain markets, decision-makers must disclose information which the company’s ‘man in the street’ should not know, it is hard to form appropriate stocks to be traded.

Market Abandoned The PM at the university in Karlsruhe run out of interest and expired. Chris Masse, a ‘PM-hero’, running the Toronto PM blog: Hype did not fit with what the research results say

Hierarchy Preserving the asymmetry of information. Markets function very well when the asymmetry is reversed, the participant knows it, the decision maker does not know it and cannot enquire about it – but this does not go down well in traditionally organised companies

104

Core Aspect: “PMs have no future as prediction tools.”

Business – Enterprises

Manufacturing Industry – two interview partners

Conceptual Theme Key Interview Points

Implementation issue

Benefit does not seem to justify the cost Motivation is a permanent issue Too many conditions must be right for a PM to be successful PMs are quite a labour-intensive endeavour to undertake If you don’t have a large number of people, it is hard to run a PM successfully albeit the results are still accurate

Market Principle Questioned The company has hundreds of figures to be forecasted – impossible that you would run PMs on all of them – hence who decides which ones are the most important – how would a PM encompass this

Market Abandoned Newsfutures (2000 – 2010, now Lumenogic) used to have a business model to do PMs for companies – they changed their business model away from PMs even though they were the major player, most visible at the time

Management Acceptance

PM was always one of many tools used. Other tools have been deployed more as a standard, e.g. statistical forecast. An internal tool (like a PM) does get rid of PM issues by being as simple as possible

Use of Alternatives Planning tied to assignment of scores or probabilities to time-series data, and doing an environmental scan to prepare for the planning process identifying 3 to 5 trends / opportunities

Hierarchy “I do not want the CEO to know the sales forecast before I do!”

Core Aspect: “Too many conditions have to be right for a PM to be successful.”

Pharmaceuticals

Conceptual Theme Key Interview Points

Implementation issue

Example of predicting mobile device usage: it was too much to ask of the employees to do additional work beyond their daily tasks (lowered participation) Implementation hurdles, PM not working when it appears to be boring – if it is only of interest to senior management, if it ‘insults’ the participants (at Eli Lilly) – they did not believe in the intention of the question

Use of Alternatives McKinsey pitched the idea of a PM at Charles River Laboratories (drug discovery and development) the company still did not believe that the idea had reached its full potential and leaned towards alternatives

Market Abandoned Eli Lilly: PM does not exist anymore – lots of technical obstacles

Fosters Employee Motivation PM value is in community participation, not in accuracy

Core Aspect: “A prediction is only worth it if you can take action, a prediction market is

not designed to facilitate that.”

Telecommunications – two interview partners

Conceptual Theme Key Interview Points

Management Acceptance A prototype PM was built, executive decision-makers opted not to pursue this

Market Abandoned The market was abandoned because no one carried on the work and participant interest had waned

105

Core Aspect: “The PM never started in earnest as it did not raise the interest of

executives.”

Services - Open Innovation

Conceptual Theme Key Interview Points

Market Principle Questioned PMs only answer ‘what’ and not ‘why’ questions which are more important The probabilistic expression of their results is too hard to grasp

Hierarchy Inherent skills play a major role, these change only marginally - i.e. a hierarchical approach essentially remains the same

Core Aspect: “Quoting Nate Silver (2012, p.403): ‘What makes a forecast scientific, is

that it is concerned with the objective world. What makes forecasts fail is when our

concern only extends as far as the method, the maxim of the model’ which happens with

PMs, i.e. they fail.”

Consulting Firms

McKinsey

Conceptual Theme Key Interview Points

Management Acceptance Managerial discomfort continues with these tools with Fortune 500 companies

Not a Topic of Interest It is not obvious in the literature what is required to keep a PM up and running – or participants must be extremely driven to participate. Battled mightily to introduce PM to the McKinsey client base

Accuracy

A bunch of random users (e.g. movie-goers) do not help with proper forecasting – clear underperformance French movie-chain: only two groups outperformed other forecasts, theatre manager, HQ-staff – knowledgeable and passionate people – you need to find them

Implementation Issue

Remembers quite some excitement at the beginning, applied PMs with client during the time as a consultant, Best Buy discovered that these tools require a lot of maintenance and marketing – more costly, more labour-intensive than the outcome justifies If the end date is not soon enough employee enthusiasm tapers off. How much bandwidth should it consume against the rest of the job? Study with a French movie-chain: a real PM needs a certain expertise by the users

Market in Use

McKinsey tried it internally (used the term collective intelligence – the gamut from surveys to future markets) on the performance of the firm – management later declined, even though risk free because McKinsey is a private company that never publishes results

Core Aspect: “PM rejection by managers may be overrated, yet they still feel very

uncomfortable because a PM can bring out things they don’t want to hear.”

106

Gartner

Conceptual Theme Key Interview Points

Implementation Issue Some companies frown upon a gaming environment (betting markets are considered as such)

Part of Work Processes

It is a challenge to set-up and to make people come back, appropriate rewards in a workplace are a challenge to set-up particularly where prospective traders are not attracted to markets where time scales extend years into the future Is the topic interesting enough (‘do I care’) to maintain interest as part of normal work procedures, scepticism towards diverse opinion versus views expressed by experts

Market Abandoned PMs had interest only periodically but never been adopted wholesale Companies set up pilots but stopped using PMs Gartner’s own market was not successful, lacking participants

Core Aspect: “The question of the real advantage of PMs has still not been answered –

benefit of setting it up vs. systems that are in existence.”

Inovex

Conceptual Theme Key Interview Points

Accuracy Used as a tool at the 2006 World Cup, predicted results showed remarkable correctness

Hierarchy

Sales forecasting is an emotional thing, often gut decisions related to (individual) people. Looking at prediction via a tool, as an organisation this has to be learned – ‘gut persons’ (type) have a problem with analysts and trading. Scepticism towards (purely) number-driven or number-accented results, particularly when from lower hierarchical ranks

Market in Use

Everything that cannot be measured with data cannot be measured in an analytical way: e.g. which properties of a product make sense for a person/human being. Product success can be ascertained with a PM run in companies

Use of Alternatives

Prediction mechanism Big Data: Patterns can be separated from randomness - clear strength of BD; e.g. detect seasonal phenomena (no longer manually detectable due to the size of the data volume) - but as to pattern recognition the limit of BD is also visible To make better decisions faster, success comes more likely from data-driven analysis (an example in marketing see Beaudin, Brinda and Ding, 2016)

Core Aspect: “Customers tend to go even for bad predictive models if they give/deliver

a reason! People want to understand, e.g. the investigation of a supply chain predicts a

problem but not why.”

Software Vendors

CrowdWorx

Conceptual Theme Key Interview Points

Management Acceptance Management doesn’t want to let PM in, even though that’s the level where PM software companies want to be / pitch their services

Not a Topic of Interest You always start at zero with every customer and only gradually create the trust which is necessary

Core Aspect: “Apple became a very successful company through simple design and

usability, these points are also very important for PMs. CrowdWorx brings the PM idea

into a new era by changing their concept: only when easy to use do they work.”

107

6.2.3 Summary of Pilot Interviews: Key Interview Points and Core Aspects

Interview topics based on key aspects extracted from all interviews are summarised and

put into context in section 6.3.1 below. Recapping noteworthy points from the pilot in

this section gives a first glance of what the overall results will bring.

Retaining control was the most important aspect that emerged; it was the overarching

motif stemming from the topics of ‘Management Acceptance’, ‘Hierarchy’, and ‘Market

Principle Questioned’. Management apparently needs to avoid having “contradicting

evidence … [to be] publicly known”, requires “preserving the asymmetry of information”

as “forecasting is an emotional thing … [raising] scepticism towards … number-accented

results”. A PM could contravene the aforementioned needs, creating “managerial

discomfort” when the truth, true forecasting results not in line with management’s

expectations (and perhaps previously communicated ‘facts’), would come out – “some

managers can’t handle the truth”, “do not believe the market”, “management doesn’t want

to let a PM in” – and last but not least, a PM is “not designed to facilitate [action-taking]”,

which was seen as the prerogative of managers (cf. Badrinarayanan, Ramachandran and

Madhavaram, 2019). Possibly because “people [also] want to understand … [the] why”,

retaining control over a decision-making process is hard to come by with a PM, as it does

not convey reasoning.

The second major theme appeared to be that PMs are “not a promising concept” – “people

struggle with the concept”. A PM’s “benefit does not seem to justify the cost” of

implementing “a labour-intensive endeavour” with “lots of technical obstacles” where “it

is hard to form appropriate stocks to be traded”, particularly versus “systems that are in

existence”. “Participant interest … waned” when effort in its upkeep ceases – “a lot of

maintenance and marketing” is required – and suitably “knowledgeable and passionate

people” need to be enticed and incited against internal resistance to a “… [frowned-upon]

gaming environment” where its concept is not that easy to grasp; sometimes “participants

have to be extremely driven to participate”. Even with “results [that] showed remarkable

correctness” it is still essential to “gradually create the trust which is necessary” for its

acceptance which indicated long-term effort and engagement needed for a CPM’s

success. This statement about trust-building loops back to the managerial and cultural

obstacles to be overcome “you always start with zero with every … [implementation]”.

Nevertheless, the general PM idea can move into a new era when appropriate uses can be

found and they “work when easy to use”.

108

Only the interviews from the pilot study have been depicted in the structure explained

and exemplified above in section 6.2.2. The ‘summary and interpretation’ section below

contains content from all interviews – including pre-study, pilot, and data collection –

and their content had been incorporated into the final analysis of key aspects and was

used throughout the thesis as appropriate.

6.3 Overall Interview Summary and Interpretation

6.3.1 Interview Topics – Key Thematic Aspects Apart from the above-mentioned works-council, twelve key aspects had emerged from

the interview analysis, with their frequency counted. Being able to draw meaningful

interpretations and conclusions from ‘just’ those thematic aspects and their occurrence

has been established as a sound methodological concept in section 5.4.2: namely that

patterns can be derived from qualitative data in an informative way.

A focus on the detection of regularities, with the help of such patterns, is of interest in a

formation of order, i.e. a re-constructible form of the exclusion of other possibilities and

interpretations (Nassehi and Saake, 2002; Daher et al., 2017). Hereby, a distinction

between facts and interpretations can be represented (Nassehi, 2018), and regularities are

exactly the material for which it is worth to use discrete forms and an objectification of

observations (Nassehi, 2019). Based on such a syllogistic approach deduced aspects can

support the interpretation and generalisation of qualitative content (Baur and Knoblauch,

2018).

Eight aspects have a negative connotation, depicted in the table below as D, while positive

ones are shown as C. The term ‘Management Acceptance’ was classed as negative

because the researcher elected for it to denote situations where, for instance, critical and

constructive information ‘voiced’ via a PM is not taken into account by the decision

makers (Welsch, 2010), and discarded or treated as unimportant if they threaten

established managerial wisdom (ibid.).

Overall, the key aspects mainly gelled with the high-level subjects arrived at during the

literature review, which is shown with a cross (‘X’) in the second column in Table 6-3;

for instance, hierarchical power continues to impact the structure of organisations and

therefore position affects the information managers and employees share and receive

(Pfeffer, 2013; 2015), an important factor in a decision to pass on information or not is

the motivation to avoid a mistake vis-a-vis superiors (Nasher, 2018; 2019).

109

Per key aspect the overall occurrence across the interviews (column ‘Sum’ in Table 6-3)

is broken up to show their prevalence in the respective areas (domains) the interview

partners hailed from. The researcher exacted the classification of the individual key

aspects from the interview content and sought and reached consensus on his chosen

scheme with a majority of interviewees as indicated in section 5.6.1 to achieve validity.

Table 6-3 Summary of Key Aspect Occurrence from Interviews (Developed for Research)

Key Aspect * ** Academia Enterprises Consulting Firms Software Vendors Sum Implementation Issue X D 7 12 6 1 26

Market Abandoned X D 7 12 4 2 25

Management Acceptance X D 4 13 3 2 22

Market Principle Questioned X D 7 7 1 5 20

Hierarchy X D 5 9 2 0 16

Use of Alternatives D 3 10 2 0 15

Market in Use C 0 12 0 0 12

Part of Work Processes C 2 7 1 0 10

Not a Topic of Interest D 4 2 2 2 10

Accuracy X C 3 3 2 0 8

Fosters Employee Motivation C 1 4 2 0 7

Weak Benchmark X D 2 0 0 0 2

SUM 45 91 25 12 173

* Topic also exists in the literature review

** Positive or negative connotation/view

From the 55 interviews – split across 12 economic professors, 30 enterprises, 7 consulting

firms, and 6 PM software platform vendors – not all showed each of the 12 key aspects

but the individual interviews mostly had both positive and negative views.

As an example, in academia the 45 overall aspects stemming from the 12 interviews had

three instances of dialogues with only adverse viewpoints and the rest showed 26%

positively slanted ones on average. The percentage-wise distribution across all

conversations can be seen in Table 6-4, showing negative aspects first and positive views

second; within these categories they are sorted by domain total in descending order.

110

Table 6-4 Negative and Positive Key Aspect Occurrence in Percentages across Interview Domains and in Total (Developed for Research)

View Key Aspect Academia Enterprises Consulting Firms

Software Vendors

Total of all Domains

Cumulative Total

Negative

Implementation Issue 15,6% 13,2% 24,0% 8,3% 15,0% 15,0%

Market Abandoned 15,6% 13,2% 16,0% 16,7% 14,5% 29,5% Management Acceptance 8,9% 14,3% 12,0% 16,7% 12,7% 42,2%

Market Principle Questioned 15,6% 7,7% 4,0% 41,7% 11,6% 53,8%

Hierarchy 11,1% 9,9% 8,0% 0,0% 9,2% 63,0% Use of Alternatives 6,7% 11,0% 8,0% 0,0% 8,7% 71,7% Not a Topic of Interest 8,9% 2,2% 8,0% 16,7% 5,8% 77,5%

Weak Benchmark 4,4% 0,0% 0,0% 0,0% 1,2% 78,6%

Positive

Market in Use 0,0% 13,2% 0,0% 0,0% 6,9% 85,5% Part of Work Processes 4,4% 7,7% 4,0% 0,0% 5,8% 91,3%

Accuracy 6,7% 3,3% 8,0% 0,0% 4,6% 96,0% Fosters Employee Motivation 2,2% 4,4% 8,0% 0,0% 4,0% 100,0%

As the two summarising tables above show, academics discerned mostly negative topics

like ‘Implementation Issue’, ‘Market Abandoned’, and ‘Market Principle Questioned’,

with almost 50% of answers being negatively slanted – 21 occurrences in relation to 45

key aspects derived from the interviews with academics overall; even more so when

adding lack of ‘Management Acceptance’ and ‘Hierarchy’.

This was mirrored in enterprises which put ‘Management Acceptance’, ‘Market

Abandoned’, and ‘Implementation Issue’ in focus; ‘Hierarchy’ – company structure – as

a topic was ranked low (sixth place), denoted by 9 instances out of the total of 91.

Purveyors of PM software primarily questioned the market principle.

Academia also gave hierarchy a low importance which is an interesting notion as the

literature clearly saw this as a highly influencing factor (see Figure 6.2 and Table 6-9

further below). The interviewees did not notice it as much in their own companies. For

consulting firms a focus on implementation would foster PM success followed by

ingraining PMs more with company leaders (‘Management Acceptance’) to affect

success.

Almost 79% of answers mentioning the topics derived from the interviews had a negative

connotation (cf. Table 6-4), only 21% of utterances showed a positive slant (key aspects:

‘Accuracy’, ‘Fosters Employee Motivation’, ‘Market in Use’, ‘Part of Work Processes’),

and the actual key aspects themselves had nearly a two thirds to one third propagation in

that respect. The individual (overall) spread in percent is shown in the diagram below.

111

Figure 6.1 Types of Topics Derived from Interviews, Not Showing Topics Less Than Four Percent (Developed for Research)

The total picture showed a clear priority on implementation as a critical factor, directly

followed by items related to non-acceptance – putting together management acceptance,

market abandonment, company hierarchy (which does not foster acceptance of markets’

prognoses), and whether the market principle works at all – as the major foci influencing

a CPM’s success, its lack of use.

This matched the view from Powell et al. (2013) that even with copious efforts poured

into the development of forecasting methods, it has proven difficult to obtain better

forecasts and PMs have not brought a noticeable improvement to the table. For this

reason, PMs’ uptake would not be encouraged beyond academia (where the concept is

lauded), which was clearly expressed in the interviews.

Research also showed that contemporary company culture hinders collective intelligence

as opposed to advancing its use and in addition, many organisations are characterised by

a high level of inertia and a low level of risk tolerance which both hinder the necessary

cultural change and improvements in and enticements for working collaboratively (May,

2012; Lenart-Gansiniec, 2019). Moreover, acceptance by managers of collaborative

approaches is not easy as they can be out of touch with the reality of the limitations of

their leadership style and goings-on outside the boardroom (Boaz and Fox, 2014).

Not a Topic of Interest5,8%

Market in Use6,9%

Hierarchy9,2%

Use of Alternatives8,7%

Part of Work

Processes5,8%

Market Principle Questioned

11,6%

Management Acceptance

12,7%

Market Abandoned14,5%

Implementation Issue15,0%

Accuracy4,6%

Fosters Employee Motivation

4,0%

112

6.3.2 Topics Related to the Literature Review

The results from the interviews and what is expressed in the literature review come

together on the level of derived thematic topics (literature sources were also categorised

into high-level topics following the approach set forth in section 5.3 and the topics’

incidences counted) but showed notable differences as depicted below in Figure 6.2.

Furthermore, the derived data from the interviews provided categories that are within the

topics of past research and in this way both contributing to but also transforming extant

views on PMs. The subjects that were not overly present in the literature all stemmed

from aspects germane to those of a firm’s organisation and processes around forecasting,

i.e. the ‘inner workings’ of a company, viz. ‘Part of Work Processes’, ‘Market in Use’,

‘Use of Alternatives’, ‘Not a Topic of Interest’, ‘Fosters Employee Motivation’.

The topic of ‘Weak Benchmark’ appeared almost exclusively in the literature, not

surprisingly as companies would not be concerned with esoteric arcana around a little-

known forecasting mechanism (Atkinson-Grosjean, 2000) – leaving six topics to be

compared across interviews and literature review in Figure 6.2. The figure’s horizontal

axis is based on the frequencies of occurrence stemming from the thematic analyses of

both interviews and literature.

Figure 6.2 Key Topics Stemming from Interviews versus the Prevalence of Major Topics in the Literature, in Ascending Order by Occurrence in Interviews (Developed for Research)

Interviews

Management Acceptance

Market abandoned

Accuracy

Implementation Issue

Literature Review

Hierarchy

Market Principle Questioned

0 5 10 15 20 25 30 35 40 45 50

113

As can be seen above, the subject of hierarchy had a much higher importance in the

literature opposed to what was expressed during the interviews. Accuracy was only a

minor item in the interviews but of considerably higher importance in the literature on

PMs – how well markets performed was, however, often expressed under the rubric of

implementation issues in the interview process, which was the most important item there.

Market Abandonment only played a negligible role in the literature whereas interviewees

saw it as important. It occurred with a higher prevalence possibly because the

interviewees at companies were directly involved in the trials of PMs.

As abandoning a PM could also be expressed as not accepting the market or various

aspects of it, this goes hand in hand with questioning the principle and overall

management acceptance. The interviews bore this out.

Thus, between interviews and literature analysis three topics came into focus as the most

important ones to be addressed, namely accuracy, (company) hierarchy, and management

acceptance (which would include the two ‘qualms’ about PMs mentioned above).

6.3.3 Topic ‘Accuracy’ – Analysis and Findings from the IARPA Forecasting Competition

As accuracy emerged as a significant topic and is also rated among the most important

forecasting criteria (Gerhart and Fang, 2015; Hyndman and Athanasopoulos, 2018;

Kurvers et al., 2019), the researcher decided to expand his examination purview beyond

the thematic analysis of interviews and literature. Forestal, Zhang and Pi (2020) recently

established PMs to be 79% more accurate than alternative forecasting methods in a meta-

analysis of PM literature scanning 12,866 papers and PMs were claimed to “generate

large improvements in accuracy when forecasting geopolitical events” (Horowitz, 2021);

this reinforced to take an approach of examining PM precision. As a validation activity

(cf. section 5.6.1), undertaking a statistical analysis of datasets from the IARPA program

on aggregate contingent estimation (Joseph, 2014) added an additional and less subjective

appraisal of what PMs can achieve in the realm of forecasting in the opinion of the

researcher. However, the examination of this large study on the merits of different

forecasting approaches including PMs was constricted to the aspect of just assessing PM

accuracy, and how fast they actually achieve meaningful results.

114

Adding this quantitative element technically constitutes multiple methods but the

majority of the field work was and remained qualitative interviews and analysis based on

a qualitative research paradigm specified in section 5.1. As laid out in the middle of that

section such an approach can be classed to stay on course of a qualitative paradigm.

Research literature and academic writings indicated that the detailed exposition of a

study’s specific methods may be contained in the relevant chapter and not in the research

methodology chapter, as applied here by the researcher (Rudestam and Newton, 2014;

Dunleavy, 2014; Weinstein, 2020).

The IARPA forecasting tournament was conducted over four years covering thousands

of forecasts on various topics. Five renowned research teams competed to produce

accurate forecasts running from September 2011 to June 2015 and IARPA thus evaluated

different forecasting approaches (Tetlock and Gardner, 2015). The tournament covered

“questions on topics ranging from Brent crude oil prices to Sino-Japanese clashes in the

East China Sea to leadership turnover in Russia and Zimbabwe” (Tetlock et al., 2014,

p.291). For PMs these questions were expressed as stocks to be traded.

Using R, a programming language for statistical computing and graphics, the researcher

analysed 329,280 trades (based on 498 traded stocks) from the IARPA data repository

which in its totality is publicly available at Harvard University (‘Good Judgment Project

Dataverse’), further details are included in ‘Appendix A: IARPA Data and Analysis’. The

existing stocks equated to questions that were categorised by the researcher in two ways,

one per region the question pertained to and one per type of question, visible in Table 6-5,

categorisation examples are given in Table 6-6.

Table 6-5 IARPA Data – Categorisation for Analysis (Developed for Research)

All Stocks (‘Traded Questions’) Were Classed via the Two Categories of Region and Question Type

Region (8) Question Type (6)

AFRICA E, Economics

ASIA G, Geography or Climate

CARIBBEAN H, Health

EUROPE I, International Affairs

MIDDLE_EAST P, Politics (Domestic)

NORTH_AMERICA S, Sport or Entertainment

SOUTH_AMERICA

WORLD

115

Table 6-6 IARPA Data – Categorisation Examples (Developed for Research from Moore et al., 2017)

Question (‘Traded Stock’) – Region and Question Type

“Will Standard & Poor's improve Tunisia’s sovereign credit rating or outlook before 10 April 2013?” – AFRICA and E

“Will the Six-Party talks (among the US, North Korea, South Korea, Russia, China, and Japan) formally resume in 2011?” – ASIA and I “Between 21 February 2012 and 1 April 2012, will the UN Security Council announce any reduction of its peacekeeping force in Haiti?” – CARIBBEAN and I

“Who will be inaugurated as President of Russia in 2012?” – EUROPE and P

“Will Libyan government forces regain control of the city of Bani Walid before 6 February 2012?” – MIDDLE_EAST and P

“Will US nonfarm payroll employment increase by more than 175,000 workers in July 2013 compared to June 2013?” – NORTH_AMERICA and E “How many countries in Central or South America will decriminalize possession of marijuana between 7 August 2013 and 1 May 2014?” – SOUTH_AMERICA and I “As of 31 March 2014, what will be the last total value of cumulative pledges to the Least Developed Countries Fund (LDCF) reported by the Global Environmental Facility (GEF)?” – WORLD and E

The IARPA data contained all trades in an R-readable format spanning three years plus

an Excel file with all stocks, i.e. ‘the questions’, a short title, a description, and their

‘answers’. The answers gave the correct value for pay-out of a stock, i.e. the actual result

of the event forecasted. Mostly these were binary questions with a yes or no answer, like

forecasting “Will China officially announce a peak year for its carbon emissions before

1 June 2015?” – the correct answer would have been ‘yes’ (cf. Moore et al., 2017).

As to the set-up of the market, two stocks would usually be available for such an example

and trading in the ‘yes’ and ‘no’ shares would drive their individual price (in probability

percentages). One would expect the ‘yes’-share to converge on a price of 100 and the

‘no’-share towards zero over time, or vice versa, depending on the anticipated outcome.

How closely and quickly this boundary value is reached is a measure of the performance

of the underlying PM.

The stocks are distributed regarding the researcher’s categorisation as visible in Table

6-7. In the ensuing analysis the stocks for the question types Geography (G), Health (H),

and Sport (S) were not included by the researcher because of their very low numbers. The

roughly 330,000 trades in the remaining categories split into 16.3% for question type

Economics (E), 53.4% for International Affairs (I), and 30.3% for Domestic Politics (P).

Subsequent results depict only these categories when detailed accordingly.

116

Table 6-7 IARPA Traded Stocks per Question Type and Region (Developed for Research from Good Judgment Project, 2016)

Number of Stocks per Region and Question Type

E G H I P S

AFRICA 2 0 2 16 26 0

ASIA 23 1 1 70 40 0

CARIBBEAN 0 0 0 1 2 0

EUROPE 30 0 1 36 39 1

MIDDLE_EAST 3 0 1 87 45 0

NORTH_AMERICA 3 0 0 1 3 3

SOUTH_AMERICA 3 0 0 6 17 0

WORLD 15 2 2 16 0 0

AFRICA 2 0 2 16 26 0

Total 79 3 7 233 172 4

The researcher analysed all available markets as well as the shares and prices they

contained together (eleven PMs in total, five run on the trading platform from Lumenogic,

six implemented on the platform from Inkling). This ensued as the attention was focused

on the accuracy of PMs in general, not on individual stocks – which can always be an

outlier – or individual markets of a single year. This meant pulling all trades, regardless

of the year they stemmed from, into a single analysis file normalising the different trading

dates contained in the file regarding their distance from the date the market closed.

For instance for the share ‘Will the Nikkei 225 index finish trading at or above 9,500 on

30 September 2011?’ where trading expired on September 28th and for ‘Who will win the

January 2012 Taiwan Presidential election?’ closing on January 12th these dates would

both be classed as day 0 and the respective dates seven days prior – trading on and prices

from September 21st and January 5th – would be normalised to occur on day -7 to render

the trading outcomes comparable.

Furthermore, as 83% of all trades were on binary shares and as in the vast majority of

these cases just one ‘answer’ was traded, only this ‘winning share’ is shown in the graphs

(meaning that usually a no-share did not capture any trades, just the yes-answers, which

anticipated the event to happen, were traded – or e.g. when asking who would win the

presidential primary for the opposition in 2012 in Venezuela, only the winning

contestant’s share was included, Henrique Capriles Radonski, as Pablo Perez, Diego

Arria, and Maria Corrina Machado were not traded or hardly at all).

117

This also made it easier to compare accuracy behaviour over time as the trades thus

analysed and summarised were from the same ilk, i.e. answers were expected to move

(closely) towards a price of 100.

In the two box plot-graphs further below (cf. Glossary of Terms), the line in the box itself

is the median per day shown, the asterisk depicts the mean. The median and mean values

both show an upward slope when they near the end of the trading, which is depicted by

day 0 to the right of the x-axis in Figure 6.3 as explained above (negative numbers relate

to the respective number of days prior to the event occurring, prior to the end of trading).

This behaviour denotes improved accuracy, as the ‘best value’ in this way of analysis

depiction would be a price of 100, denoting 100% expected probability of the event

occurring.

This ‘improvement’ or ‘correctness’ already started 70 days before the end of the market,

the median price of the according PMs correctly expected the event to happen with a

probability clearly above 80% from that point in time onward. Also, the box itself

continued to become noticeably smaller, i.e. shorter, all the prices of the traded shares at

the individual points in time were close together predicting the ‘same’ high probability.

The median had moved upwards to a more desirable level, and importantly, variability

had decreased; however, more outliers emerged. The very positive performance across

all markets from Figure 6.3 can be seen in more detail in Figure A.1 in Appendix A which

shows the spreading of all prices per time slot in addition to the box plot.

Three weeks (21 days) before the predicted events were supposed to happen, three-

quarters of the trading the prices (the lower end of the box corresponds to the third quartile

of data dispersion) sat above a 75% probability (price 75) of the event happening. The

median – hence half of all trades – was in fact higher than 90% at that time. The same

positive performance occurred even when accounting for three quite distinct areas of

prediction, namely economics, international affairs and local politics in Figure 6.4.

Just to reiterate, PMs fulfilled their claim of reaching an accurate prediction and doing

this quickly. Forecasting an event which will actually come to pass as predicted with 90%

probability three weeks out and better than 80% more than two months before could

justifiably be called very good.

118

Figure 6.3 IARPA Trades from All Markets (Developed by the Researcher from Good Judgment Project, 2016)

Based on the graph below in Figure 6.4 the researcher concluded that PMs repeatedly

show remarkable accuracy, noting that the picture encompasses well over 300,000 trades

split into three categories.

Figure 6.4 Trades from All Markets per Category ‘Type’ (Developed by the Researcher from Good Judgment Project, 2016)

119

However, as with all aggregation, it can make sense to break it up into smaller

components, less clustered ones. Shown in Figure 6.5 are what the researcher classed as

long, medium, and short trades, where trading started only 50 days (‘short’) before market

closure (day 0), 100 days (‘medium’) or beyond 100 days (‘long’), respectively. This

examination summarised the traded stocks regarding their classifications. Here, in

contrast to the box plots, the average overall stock prices for each individual trading day

were calculated.

Showing 21 iterations, seven regions per three question types, the lines in the nine

quadrants below correspond to the smoothed average (arithmetic mean). In comparison

to the box plots these graphs therefore indicate the averaged change over time (the grey

areas around the lines show the 95% confidence level), the box plots show the distribution

of the individual prices exactly ‘x’ days before the end of the trading period.

Figure 6.5 IARPA Market Behaviour per the Three Categories and Long, Medium, and Short Markets (Developed by the Researcher from Good Judgment Project, 2016)

The results may seem erratic at first, but looking at the graphs, prices converged above

80% estimated probability regularly, and noticeably earlier than the predicted events were

supposed to happen, a performance better than a coin-toss, showing a satisfying accuracy.

Some question/region combinations might also have been hard to predict, which would

impact the precision to be achieved.

120

However, to paraphrase Shakespeare, something may still be rotten in the state of affairs

for PMs: rolling individual surveys based on expert-based responses, i.e. individual, not

market-based estimates which were also captured over the four years of the IARPA

project showed a similar performance to PMs visible in Figure 6.6 (‘PM boxes’ shown to

the left side of each day depicted). The IARPA contest called the underlying data files

that contain these estimates ‘survey forecasts’, a term the researcher kept in the graphs

below, even though these appraisals were not surveys in the strict sense but rather from

participants who delivered their individual assessments unprompted. Based on this, one

could consider the perceived accuracy of PMs of lower quality as an individual just made

three ‘guesses’ on average per question over the time period the survey was open.

Conversely, an individual gave estimates to about 47 questions on average, some to even

more than 100, which might be somewhat artificial compared to a real-life setting.

The researcher analysed both measures in the same way to allow a comparison of the

784,494 estimations made to the trades from the PMs, the contrast visible via the box plot

graph in Figure 6.6.

Figure 6.6 Comparison between PMs and ‘Surveys’ Run During the IARPA Project (Developed by the Researcher from Good Judgment Project, 2016)

IARPA ‘surveys’ did not perform worse than PMs, their spread or dispersion is somewhat

greater than with PMs, though. In the smoothed average comparison in Figure 6.7

questioning individuals always performed better than PMs over time, apart from the only

exception of short-term economics questions.

121

Figure 6.7 Comparison between PMs and ‘Surveys’ per Question Type; Long, Medium, and Short Trading or ‘Survey’ Periods (Developed by the Researcher from Good Judgment Project, 2016)

But broken down by category of question in Figure 6.8 (PMs again on the left) the lead

of ‘surveys’ did not quite hold anymore. The spreading of data from ‘survey’ answers is

noticeably larger than with PMs two-thirds of the time, indicating a lower overall

accuracy. With the ‘surveys’’ greater interquartile range (representing the 50% of the data

closest to the median), a greater spread in this section of the plot suggests data that is less

reliable as an indicator of highly probable values, impacting the quality claim of the

‘surveys’, albeit their median value is higher than the PMs’.

Figure 6.8 Comparison between PMs and ‘Surveys’ Box Plot per Question Type (Developed by the Researcher from Good Judgment Project, 2016)

122

Overall PMs were therefore not a clear winner. Nevertheless, PMs evidently reached a

high level of accuracy and may therefore warrant an inclusion in the canon of forecasting

instruments, particularly if anonymity, diversity, and independence would be deemed

important by a company to weed out biases, groupthink, or too little information inclusion

in their decision-making or judgemental forecasting. It could furthermore be problematic

to run public or publicly available opinion polls or surveys on strategy questions or about

new products, such disclosure of proprietary information is often avoided by companies

to maintain competitive advantages (Arya, Frimor and Mittendorf, 2010), favouring an

internal forecasting process like with a corporate PM.

Amid comparable performance, another aspect to consider is the cost of implementing a

forecasting tool or prediction approach. With PMs an accurate prognosis can be achieved

with participant numbers as low as 12 to 20 (Bingham and Nagar, 2013, Jester-Pfad,

2015) and quite good results can be reached with active participation of 30 to 50 traders

(Riekhof, Riekhof and Brinkhoff, 2012; Riekhof and Brinkhoff, 2014). Surveys must

reach at least 100 people for reasonable results (Karlan and Appel, 2016); for the

achievement of robust and reliable scores participant numbers have to be well above

1,000 representative participants (Ortmann, 2019; Schnell, 2019; Schumann, 2019). This

can carry considerable costs, four to five-digit amounts may be needed for a single survey

of high quality (Appinio GmbH, 2020; Seraei, 2020; Syperek, 2020), large polls can run

into millions of dollars (Calipha and Venezia, 2021).

Internet-based surveys might be cheaper but it is still questionable if representative

population surveys can be conducted via the internet in a meaningful way. Those who

participate in online surveys are often considerably different from partakers in other

surveys and results need to be cross-checked or validated against other approaches –

adding costs or, if not carried out, diluting results – plus representativeness is often

questionable when having an insufficient concordance between target and survey

population (Bandilla, 2016; Bytzek and Bieber, 2016).

Therefore, these considerations did not tilt the scales away from PMs. Eventually, how

well markets can perform is also subject to the existing company culture covered in the

following section.

123

6.3.4 Topics ‘Hierarchy’ and ‘Management Acceptance’ and Merging Literature and Interview Results

The two tables below give an overview of where the major viewpoints in the literature

review stemmed from and what they encompassed, apart from accuracy handled in the

section above. Even outside of direct PM literature there were 28 areas in the literature

review which belong to the criteria of cultural acceptance and the impact of company

hierarchy. ‘Cultural Acceptance’ is used as a term in the next two sections in lieu of

‘Management Acceptance’ (which as a thematic topic stemmed from the interview

analysis) as in the literature this particular topic is slightly broader, covering change and

decision-making aspects that relate to cultural aspects within a firm (Mokyr, 2016;

Spiegler, Heinecke and Wagner, 2019; Mikalsen et al., 2019).

Rereading the literature (review) with a focus on these two subjects the researcher

categorised the respective sources into major themes and subtopics in Table 6-8 and Table

6-9.

On the subject of how management would or would not accept the views stemming from

a PM, 13 references exist in Table 6-8 that were drawn from perceptions garnered by

authors outside of PM publications.

Table 6-8 Perspectives from Researching the Literature – Cultural Acceptance in the Literature Review (Developed for Research)

Cultural Acceptance –Management Literature Outside of Prediction Market Literature

Theme – Culture Subcategory Article Reference

Cultural Change • Threaten established managerial wisdom • Corporate adaptation • Culture and strategy

Cappelli, 2019; Lewrick, Link and Leifer, 2017; Schein, 2010; Welsch, 2010

Decision making • Complexity of past decisions • Avoiding conflicting evidence • Management resistance

Bénabou and Tirole, 2016; Dye, 2013; Sibony, 2013*

Decision making flaws • Groupthink • Biases Baer, Heiligtag and Samandari, 2017; Brem, 2017

Big Data in the context of management tools

• Risk • Scaling, Agility • Machine Learning

Babel et al., 2019; Fountaine, McCarthy and Saleh, 2019; Rigby, 2017; Wegener and Sinha, 2013

* the references to Dye and Sibony are interviews

Similarly, 15 references were found which address various aspects of the impact of

company hierarchy, illustrated below in Table 6-9.

124

Table 6-9 Perspectives from Researching the Literature – Company Hierarchy in the Literature Review (Developed for Research)

Company Hierarchy – Management Literature Outside of Prediction Market Literature

Theme – Hierarchy Subcategory Article Reference

Expression of Power • Power and influence positively correlated

with hierarchical rank • Maximise power structures

Chlupsa, 2013; Maner and Mead, 2010; Maner and Case, 2013; Pfeffer, 2015

Hindering Success

• Culture has a clear bearing on success • Hierarchies hinder employee engagement • Organizational nimbleness is still elusive

for most companies • Hierarchies are evidently here to stay

Agrawal and Harter, 2010; Ahlbäck et al., 2017; Cole, 2015; Gottlieb and Willmott, 2014; Hosseini et al., 2015; Pfeffer, 2013; Zinger, 2018

Organisational Structure

• Hierarchical structures are still the norm in complex organisations

• Decisions enforced via instructions • Planning is a defining feature • Digitally aware companies move away

from hierarchical structures

Barth, Kiefel and Wille, 2002; Besanko et al., 2017; Bughin, Manyika and Miller, 2008; Rigby and Bilodeau, 2018

Some authors that were classed to cover ‘Cultural Acceptance’ also expanded on the last

two themes of the challenges stemming from company hierarchies (Baer, Heiligtag and

Samandari, 2017; Bénabou and Tirole, 2016; Cappelli, 2019; Lewrick, Link and Leifer,

2017; Welsch, 2010), making company hierarchy the most important factor influencing

the uptake of CPMs in the literature review.

As hierarchy also has a clear bearing on company culture (Corley, 2004; Spiegler,

Heinecke and Wagner, 2019) this overall subject would play a key role for the success of

implementing and integrating a CPM into corporate forecasting processes and thus could

explain the failure of CPMs. This conclusion was borne out by the interviews as well, as

36% of all topics connoted nonacceptance of CPMs based on hierarchical and managerial

aspects, and this would rise to almost 50% if doubts from users on a PM’s principles –

which also does not foster or encourage corporate PM use – are included.

Such doubts are perhaps understandable when PMs’ accuracy is quite high but the ability

to stand out from the field of prediction approaches is possibly only mediocre in

comparison to ‘just asking’ or individual repeated ‘guesses’ as shown in section 6.3.3.

Thus both the literature and the interviews did not instil corporate PMs as a clear

forerunner in a company.

125

6.4 Analysis of CPM Usage

In 2018 three researchers lauded PMs yet again to be applied in corporate settings,

regarding the increasingly important phenomenon of innovation management which

fundamentally changes how companies create and capture value. A PM was suggested to

be used in business practices as a mechanism to tap into a crowd in the early stages of an

innovation process (Horn, Bogers and Brem, 2018). Evaluating this application of PMs

to represent new product ideas – idea markets – the researchers gave twelve examples of

PMs used in business. Of these, at least three (at Siemens; Hewlett Packard; and Ford) do

not exist anymore, and research on a further six is more than ten years old (at General

Electric; at a technology company; a finance company; a communications company; a

basic materials company; in the movie industry), this questions the validity of suggesting

corporate PMs as a way forward.

Between February and July 2020, a research study conducted by Leadership Choices, an

international professional services firm specialising in executive development and

coaching, looked at what type of changes would be needed most in companies in the next

few years and what kind of company culture would be needed in the future for a company

to stay successful. The study revealed that the second most important item in these two

categories is a less hierarchical set-up and cross-functional collaboration respectively. To

fulfil this, a company needs to increase flexibility and adaptability and leaders would need

to be more trusting and empowering (Lewe, 2020). Such advice was also reflected when

asking what organisations would look like in 2025, the emphasis was on “enhancement

[of] cross-functional … collaboration with new technology” (Mitchell et al., 2019, p.4).

CPMs foster the suggested company collaboration, particularly answering important

strategic questions or helping with innovation processes but are possibly hindered by the

still existing ‘importance’ of hierarchies as strongly highlighted in the first and second

paragraph of the preceding section 6.3.4. Understanding the (puzzling) present from the

past sheds light on these matters.

6.4.1 Modelling of CPM Adoption

McKinsey & Company conducted a large cross-industry survey on social technology

usage every year between 2006 and 2016 which pointed to a steep decline in CPM usage.

Until 2013, the results were published in the public domain; the researcher was able to

obtain the unpublished ratings for 2014 to 2016. McKinsey retired the survey on social

technologies from their annual rotation in 2016 (Seiler, 2018), so there was no further

updated data from 2017 and beyond.

126

Tracking the adoption and spreading of social technologies through a sophisticated survey

programme (Bughin, 2015; Bughin, Chui and Harrysson, 2015; Bughin, Chui and

Harrysson, 2016b; Bughin, Chiu et al., 2017), McKinsey used a database of 11,000

executives among which they organised regular examinations by drawing random

samples of up to 4,500 survey participants. This research comprised topics such as blogs,

PMs, podcasts, video sharing, social networking, and wikis.

A model was constructed from the results when those technologies would reach critical

mass – i.e. the peak of a noncumulative adoption curve in Figure 6.9 – following the

approach from Wong et al. (2011) which styled diffusion patterns of new innovations like

email, blogs, discussion forums, etc. based on an appropriate bellwether; e.g. for Twitter,

introduced in September 2006, critical mass was foreseen at the beginning of 2012.

Figure 6.9 Model of Corporate Adoption of Enterprise 2.0 Technologies in Percentage Terms (Bughin, 2015)

So, for CPMs (depicted as ‘Prediction markets’ above) a growth phase as rapid as in other

technologies was never anticipated. Still, there was great anticipation in 2015 regarding

the future of CPMs per the modelling (Paul and Weinbach, 2015). Constant growth was

predicted regarding their adoption. This contrasts with the actual surveys, also conducted

by McKinsey, where reported usage never exceeded 12% at its pinnacle in 2014 (see next

section 6.4.2) and which even illustrated an overall decline looking at the actual numbers!

The estimations in the model, however, reached that usage mark already at about 2009

and expected almost 25% penetration in 2014.

127

Based on actual observations rather than expectations from a model, a ‘gloomy’ picture

of CPMs’ success already emerged at the beginning of McKinsey’s data gathering

exercise. Seemann (2009) advised that PMs in general lack both a foundation in

forecasting and overall adoption and this gelled with the very low figures on uptake in

2008 of less than 1% quoted by Gartner (Cain and Drakos, 2008) in contrast to the model.

6.4.2 Analysis of Actual CPM Applications

In Figure 6.10, based on the yearly research by McKinsey, the data show that the adoption

of CPMs fell by 63% from its peak in 2011 to 2016, as far as absolute numbers, and not

percentage figures are concerned. The research contained the number of total respondents

per year and the percentage of companies which used PMs in a given year, actual absolute

usage was calculated by the researcher. (The actual usage number is derived from

applying the usage percentage numbers contained in the surveys to the total number of

participants, e.g. with 1,988 survey participants in 2008 and six percent CPM usage at

that time, this results in an absolute usage figure of 119; this value from the data series

‘Actual Usage’ is visible to the left in the graph below.)

In the literature review where only officially published figures were beheld until 2013,

this fall was shown to be only 30% from 2011 to 2013, hence a further drastic decline

since then (cf. section 2.5).

Figure 6.10 Decrease in CPM Usage, Actual Usage Calculated from Usage in Percent and Total Number of Participants (Developed by the Researcher from McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui and Harrysson, 2016b; Bughin, Chui et al., 2017)

4%5%6%7%8%9%10%11%12%13%14%

0

50

100

150

200

250

300

350

2008 2009 2010 2011 2012 2013 2014 2015 2016

Actual Usage Usage in Percent

128

A breakdown into a specific portion of employees at each organisation that were using

CPMs is only available until 2015; information about companies’ overall usage exists up

to and including 2016. In companies exploiting CPMs to begin with, the portion of

employees using them fell by 44% from 2011 to 2015.

The strata of individual employee usage in the surveys were 1 to 10 percent, 11 to 30

percent, 31 to 50 percent, and 51 percent or more (and ‘Don’t Know’ – which was

excluded in this analysis); answering the question “Prediction markets: Approximately

what share of your employees is using each of the following technologies or tools?”

(Bughin, 2013, p.196). In the bracket where CPMs were used by more than 50% of

employees, between the pinnacle in 2011 and 2015, there was a regular yearly decline of

on average 4% translating into an overall diminution in the complete period of 15%

though (McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui

and Harrysson, 2016b).

Casting the view back to 2008, the beginning of the McKinsey investigations, PMs in

general were down the ‘Peak of Inflated Expectations’ in Gartner’s famous hype cycle

(Cain and Drakos, 2008), then in 2009 they were sliding into the ‘Trough of

Disillusionment’, and in 2011 they were taken off the cycle (Mann, 2011).

In the whole of 2012, as revealed in an interview, Gartner received a maximum of two

enquiries about the topic. The hype did not fit the research results (Rozwell, 2013), rather

it was suggested that there is a total lack of interest. One could thus argue that PMs ceased

to exist as early as 2012 as a valid forecasting instrument to be considered for

corporations, which is borne out by the above graph on their usage. Sibony (2013; 2017)

considered the percentage numbers as always way too high from his experience when he

published on PMs. With innovation and innovative concepts slow to trickle down in an

organisation generally this would be true for a new tool like corporate PMs.

So, CPMs had high expectations which have not yet been fulfilled.

Reflecting and adapting the concept of ‘net-promoter score (NPS)’ (cf. Reichheld,

2003), a ratio of promoters to detractors where NPS = promoters (in % of all respondents)

minus detractors (in % of all respondents), for CPM usage led to the following result: the

researcher classed employee usage greater than 50% as promotion (high) and below 30%

as detraction (medium to low) which showed a constantly negative delta between ‘high’

and ‘medium to low’ usage.

129

The negative gap fluctuates between at best (roughly) minus 35, and a worst point of

almost minus 60 in Figure 6.11.

Figure 6.11 Net Promoter Score – Constant Negative Delta between ‘High’ and ‘Medium to Low’ Usage (Developed by the Researcher from Reichheld, 2003; McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui and Harrysson, 2016b)

This means many more actual and potential users would be ‘detracted’ from the concept

rather than ‘promote’ it and it appeared that users are highly dissatisfied with CPMs.

Particularly so-called power users who typically get the most value from social

technologies (Bughin, 2015) were anticipated to exploit them, which was not happening

as elaborated below.

Companies are classed as ‘power users’ when 51% of staff or more use a certain tool in

the realm of social forecasting. CPMs being cast away by particularly the very companies

that would be expected to use them most and their leaning towards other tools can clearly

be seen in Figure 6.12 in the numbers that are the lowest in all years bar one in the stratum

of power users (‘high usage’) as opposed to usage amongst the classifications of very

low, low, and medium.

-60%

-55%

-50%

-45%

-40%

-35%

-30%2008 2009 2010 2011 2012 2013 2014 2015

NPS Linear Trend

130

Figure 6.12 Portion of Employees Using Prediction Markets by Usage Stratum, Excluding ‘Don’t Know’ Answers (Developed by the Researcher from McKinsey & Company, 2013; Bughin, Chui and Harrysson, 2015; Bughin, Chui and Harrysson, 2016b)

All this would be one explanation – a number-based one – for the very low usage numbers

overall and the fact that companies are dispensing with CPMs, which was also parlayed

in the interviews the researcher conducted, particularly from enterprises.

6.5 Significance of CPMs and a Possible New Field of Application Not only are companies rarely using CPMs, manufacturers of PM software, used by

companies to establish and implement their CPMs, have moved away from this arena of

product offerings. As many of these software fabricators would need to be considered

small boutiques rather than fully fledged purveyors of PM software (platforms), this

indicated an overall lack of usage.

To strengthen such an inference, Sibony (2017) suggested looking at the revenues of PM

software vendors. If these are flat or miniscule, he would strongly submit this to be

another symbol of the insignificance of PMs. In this vein, PM software providers are

mainly minnows (see Table 6-10 further below). One such provider advocated himself

that the market for PM software had always been very small and is now infinitesimally

small (Ivanov, 2017). The company, CrowdWorx, is only sustainable because they have

switched to innovation management, where collaborative forecasts are just an aspect

(called Social Forecasting, one out of six offered modules).

0%

10%

20%

30%

40%

50%

60%

70%

2008 2009 2010 2011 2012 2013 2014 2015

very low (1-10%) low (11-30%) medium (31-50%) high usage (>50%)

131

Cursorily looking at the market for PMs revealed about 37 providers (cf. Struhl, 2017;

Pauw, 2018; Shchory, 2020). There would be some Open Source PMs as well (like for

instance Serotonin Prediction Markets, Idea Futures Prediction Market or MarMix, etc.),

but as these are more like toolkits to build a PM and would need extensive programming,

they were not considered here. Fourteen market providers no longer exist, amongst them

well-known contenders like Intrade which suspended all trading in March 2013 or

TradeSports which closed for business in November 2015. Five vendors provide PM

platforms where ‘pre-packaged’ prediction items can be traded but one cannot influence

what forecasts are sought or would be of specific interest to one’s firm. Large contenders

here would be the Hollywood Stock Exchange (movies) and the Iowa Electronic

Markets (mostly politics), who set up their own markets for interested parties to trade in.

The focus on providing software for firms to run a PM was always a specialised segment

of the overall market (Barbu and Lay, 2012; Ruberry, 2013) and comprises 18 vendors

depicted in the table below.

Table 6-10 Gauging the Size of Selected Prediction Market Vendors (Developed for Research)

Purveyors of Prediction Market Software Annual Revenue and Number of Employees*

$25M revenue $9M revenue $5.5M revenue < $1M revenue $2.2M revenue 90 employees 63 employees 36 employees 30 employees 34 employees

Private company, small Private company, small $10M revenue $1.4M revenue $0.6M revenue 10 employees 126 employees 4 employees

Austrian GmbH, small $28M revenue $1.3M revenue

German GmbH, small Privately held

125 employees 20 employees < 50 employees

$4.9M revenue $0.3M revenue $4.4M revenue 39 employees 4 employees 20 employees

* based on company information and on business profiles gleaned from the Internet platform Owler which provides competitive insights on firms

As shown, companies in the realm of PM software provision are all miniscule aspirants,

and many are still where they had been ten years ago from a revenue perspective, trapped

in an irrelevant market (Ivanov, 2020).

To put the above numbers into perspective and show their true tininess, the contrast to

Betfair with $530M annual revenue and 1,739 employees is illuminating.

132

Containing political betting markets whose predictions are often used in the public

domain (Pierce and Thomas, 2007; Abramowicz, 2008; The Economist, 2019), UK-based

Betfair is more than 20 times the size of the largest company in the table above and

provides first and foremost an internet-based betting exchange for sports gamblers. PMs

are just a sideshow under politics betting and the company does not provide platforms for

enterprises to use.

Interestingly, purveyors of PM software have partially grown in revenue after the focus

shifted away from merely providing a PM – which Ivanov (2019) referred to as an

absolute niche – to offering market research services instead (System1, 2020). Qmarkets

had taken the path towards idea management, Spigit bought Crowdcast in 2012 and

switched to innovation and idea management, too.

ConsensusPoint moved towards market research, and Augure offered a PM based on

blockchain technology which in itself is still an evolving technology (Siba and Prakash,

2017; Hin, 2019). Looking at the top ten rivals for both Spigit and Qmarkets underlines

this view as well. They are all providing innovation management software, viz. IdeaScale,

Brightidea, SoapBox, BrainBank, Wazoku, Mindjet, Officevibe, Imaginatik, Aikon Labs,

and Idearium. Other pioneers in the field of PMs had also switched focus, Intengo

(belonging to Infosurv Research) switching to ideation processes for concept testing and

screening. This harnesses the wisdom-of-crowds to uncover new product and service

ideas for their customers. PredictWise called their new product offering Audience

Technology for the 21st Century – a machine learning approach for customer

segmentation and advertising campaigns – but still runs political PMs on their website.

Another example of a software vendor moving completely away from PMs is predictX.

In 2010 the co-founder Sebastian Diemer still praised the concept of PMs and the

company even developed a new market-maker for their trading platform (Bruns, 2010).

But now the focus is on predictive analytics for various industry verticals, preferring data

over human ingenuity (“The Predict.X platform is a cluster of tools and technologies

which leverage the capabilities of artificial intelligence and machine learning” – from the

company website at https://www.predictx.com/our-platform/predictx-platform/).

133

As Big Data analytics extract game-changing insights for companies to use (Olanrewaju,

Smaje and Willmott, 2014) and providers of PM software have moved away from their

original endeavours as shown above, all of this might sound like the ‘death’ of corporate

PMs. But it also hinted at a hope for CPMs, as the focus of these firms now seems to be

on innovation management. New approaches to innovation to deal with shortened product

life cycles have included idea markets. An extension of CPMs that can foster innovation

across disciplines (Hamel, 2012), both rely on the ‘wisdom-of-crowds’.

While traditional PMs facilitate acting on the outcome of fluid future events, idea markets

provide a platform for generating and evaluating ideas by trading virtual stocks that

represent products and concepts (Buckley and McDonagh, 2014). A 2012 study of idea

markets reported that with the stock prices indicating their probable success this helps to

focus on those ideas worth pursuing further, in such a way reducing the number of

proposals management has to pay attention to (Soukhoroukova, Spann and Skiera, 2012).

Gauging the possible success of a new product is still a difficult task. Conventional

market research is costly, time-consuming, and prone to errors. PMs have been instituted

as a practicable substitute (Matzler, et al. 2013). Deutsche Telekom, for example, has

been using a corporate PM to forecast the potential of new products on a weekly basis

since 2011 (Meyer-Berhorn, 2013; Ivanov, 2020). The fact that a CPM has been in

operation for such a long time, when applied to new product development, seems to

confirm its continuous usefulness to the company in such an area. Here, perhaps, lies the

future of CPMs.

The next section thus reflects upon the extension of PMs into innovation management.

6.6 Additional Interviews – Focus on Innovation Management

Engaging the crowd has become an increasingly popular way for entrepreneurs to get

access to alternative funding as well as to connect with future customers. Catering to their

need with appropriate products is paramount, one way to achieve this is to actively seek

out ideas from outsiders, which helps to gauge potential market response, too (Stanko and

Henard, 2016; 2017).

As a CPM could assist with such endeavours, the researcher saw the area of crowd-based

innovation management for companies worthwhile further exploring, following Bar Am

(2020) who advised to prioritise innovation in products and services to unlock growth,

particularly in crises.

134

Furthermore, as with CPMs, the challenge from ease-of-use and having an appealing

interface to make people trust and want to use an innovation platform is an aspect to

explore. Consideration of emotional aspects in the way humans interact with user

interfaces (Raeside, Peisl and Canduela, 2019) influences the acceptance of a platform.

Gathering and synthesising market intelligence ought not be an isolated task and

organising such material to enable a quick conversion into new merchandise, services or

business concepts is paramount (Bar Am, 2020). To understand how such a focus could

be achieved with a supportive tool the researcher carried out ten additional interviews in

2020 with experts in the commercial field, consulting, and from academia (illustrated in

Table 6-11) beyond the ones already presented in section 6.1 on data collection.

Table 6-11 Interview Partners ‘Focus on Innovation Management’ (Developed for Research)

Interviewee Abbreviation (To Ascertain Anonymity) Area Job Title or

Equivalent Research Focus

(Where Appropriate) Work

Experience

IM_Int_1 Academia Professor Innovation Management and Prediction Markets 18 years

IM_Int_2 Consulting Founder/Managing Director Digitalisation of Processes 15 years

IM_Int_3 Consulting Managing Director Innovation Strategy, Innovation Management & Digital Transformation 15 years

IM_Int_4 Enterprise General Manager Germany and Austria 25 years

IM_Int_5 Enterprise Head of Team Business Intelligence 36 years

IM_Int_6 Academia Professor Futurist 17 years

IM_Int_7 Enterprise Head of Corporate Strategy 24 years

IM_Int_8 Academia Professor of Economics Decision-making in organisational Settings and Prediction Markets 16 years

IM_Int_9 Enterprise Head of IoT Program 16 years

IM_Int_10 Academia Research Fellow Applied Machine Learning 8 years

The researcher used his original approach of conducting semi-structured interviews

amending the interview guide by one additional question asking interview partners about

innovation management and knowledge and suitability of idea markets, which are an

application of PMs for ideation processes (Siemon, 2019).

6.6.1 Content Elicitation The type of topics that originated from the conversations suggested for the responses to

be clustered into four areas around implementation, their applicability for innovation, PM

challenges, and more general remarks.

135

The researcher chose to present the interview content not always directly assigning

content to individual persons (hence with only sparse direct quotations; this also accounts

for the occasional desire for anonymity; when a citation is made, only the anonymised

‘names’ from Table 6-11 are used and the year is not added as they are all from 2020, and

not all interview participants are directly quoted). The ‘reporting’ from the interviews’

findings happens in continuous text; small but inconsequential editing (without altering

content or meaning) took place to conserve the flow. Nevertheless, this section

exclusively represents the insights and views of the interviewees. The structure, however,

was chosen by the researcher and occasional references to other authors underscoring a

point made by an interviewee are also his insertions.

An interview partner that has researched but also implemented PMs still believed firmly

in the method of PMs – however “the success of the market stands and falls with the

expertise of the participants” (IM_Int_1). To attract knowledgeable participants, plug-

and-play implementation needs to be fostered, otherwise there would be a barrier for

PMs’ diffusion, especially if the method is not known and no internal expertise is

available. Simple, i.e. simple-to-use platforms can be an intermediate step on the way to

casting a wide net of interested parties, but one “needs at least a minimum knowledge

about the content / question – potentially leaving certain people out of the market” (ibid.).

Technical implementation through good visual design was already important in the past;

“today this is still a challenge” (IM_Int_10). But if the design heeds certain prerequisites

one could unleash a significant benefit: “by combining emotion and data, human and

machine, analog and digital” (Tobaccowala, 2020, p.ii) into interfaces for example.

In the car manufacturing industry as a whole, as expressed by an interview candidate,

“sales forecasts have long been poor and could improve significantly” (IM_Int_7). When

the use of a competition-based tool – and a PM would be viewed as such – was looked at

by a car manufacturer, it was considered too political and rejected by the works council.

“But on an external platform the internal hurdles for a PM, like the political element,

would fall away; company specific questions could then be accepted there” (ibid.).

Attracting participants with the necessary expertise with a ‘pleasurable’ and well-

designed user interface could make CPMs work for innovation.

136

Constant innovation is an important topic to pursue for a company to stay successful,

openness in this context is of paramount importance. Developing portfolios that are then

evaluated should combine internal idea cultivation programmes with external sources of

innovation, relying on a venture capitalist-style approach rather than on rigid budgeting

rounds (Bland and Osterwalder, 2020) – a process to tie internal and external innovation

together would be beneficial.

PMs could support Open Innovation especially for strategic (environmental) screening –

early recognition (“Strategische Früherkennung” (IM_Int_6)) – as it is important to

identify relevant future fields. To gather and evaluate those fields with a PM can make

sense. In the area of qualitative evaluation for new markets or new products a company

could simply ‘make’ the market (like e.g. the introduction of the iPhone).

But one still needs number ranges for production based on experience, the numbers could

be determined by a PM. Relative benefit arises when no historical data exists and one is

dependent on the assessment of participants – for new products, there is little to learn

from the past (there is no 10-year product history).

A start-up culture was seen as amenable to idea markets and perhaps even corporate PMs

in general, as for example also expressed by Ries (2017): “it could be important to seeking

[sic] to create, nurture, and sustain entrepreneurial thinking in companies at any size and

scale”. This was also brought forward by Bocken et al. (2019), averring that implements

that emphasise ideation, for instance back-casting or games, are an important facet

highlighted in fields like design science and practice.

One interviewee also recognised innovation as an important aspect within the logistics

industry as a matter of high priority (the imparted knowledge stems from conferences and

trade forums) – for instance “the current Corona crisis very much diminishes revenue at

both UPS and FedEx from bulk cargo deliveries and Corona-related costs have risen in

the established processes” (IM_Int_4). The necessity arises to come up with new markets

to expand into, e.g. home deliveries. To assess these new markets but also the importance

of a new competitor like Amazon or even “gauging the success of [future] automated

deliveries [via unmanned vehicles, cars, trucks or even drones] is high on the agenda”

(ibid.).

Companies viewed innovation as a distinguishing factor to stay competitive and a

forecasting mechanism which sorts the “wheat from the chaff” when considering new

products is needed – CPMs could step into that breach as an appropriate tool.

137

When forecasting “one’s human side is always shown” (IM_Int_2) and for that reason

one comes to a consensus on the forecast figures through negotiation, which leaves human

ego intact to a degree; it might be taken away by an automated mechanism in the view of

an interviewee. Consensus is achieved through hierarchy; management heeds a strong

works-council which sees the playful element of a betting-based market as critical.

As revealed by an interview partner, a Sales Manager Europe wanted to decide for

himself, not wishing to hand over decision-making authority to a ‘democracy’; he would

see his authority, his competence questioned – it emerged that in principle he thinks he

knows better. “Culture and the sensitivities of bosses are basically a challenge as the boss

feels having less influence on the ranking of ideas when facilitated by a market”

(IM_Int_7). Therefore, the right management culture is very important, “at a minimum it

needs to be hierarchical participative (like in design thinking, Scrum, agile)” (IM_Int_6)

– but not everybody likes not being able to rule.

Hierarchy is a clear challenge – and would be a real issue if a PM would be deployed.

“People in the organisation are jockeying for advance and do not want bad news, that

could emerge if other people have input via a market, revealed” (IM_Int_7). As the term

PM had negative connotations, “corporate crowdfunding” (ibid.) was chosen as this was

seen as a catchier and less controversial concept.

Apart from company culture the feeling amongst many users is still that PMs are more

suitable to short term predictions – somewhat questioning the applicability in innovation

where the outcome is rather unknown. As with other arenas of forecasting, combining

methods might be a good way forward, pursuing a variety of approaches to innovation.

When crafting strategy focus is called for, but innovation benefits from making multiple

bets. In this context, internal programs rely more on hackathons or idea challenges and

dedicated time for experimentation to generate fresh ideas.

As an alternative to PMs one could also see pair comparisons – these calculate an

attractiveness score (an algorithm builds a kind of marketplace). “At the German car

manufacturer Daimler AG 150,000 pair comparisons were carried out” (IM_Int_1) – the

algorithm works very well with such data volume; here it might be difficult for a PM to

succeed and even handle such an amount of data.

If hierarchical aspects can be overcome, CPMs could foster innovation management; at

the very least in combination with other tools.

138

Effective transformations in organisations depend on a balance of intuitive power of

human beings and data-supported insight (this speaks for combining methods and could

include CPMs).

There are warning signs of data-blinded companies (the focus these days is perhaps too

much on Big Data) which lead to cultures with little human interaction and ‘starved’

innovation due to employees being suffocated and no longer encouraged to come up with

ideas and insights, as the current focus is on data analytics and automated processes

around that (cf. Tobaccowala, 2020).

Good innovation needs a ‘human element’ which would be provided by a CPM as it

amalgamates views from human participants.

6.6.2 ‘Innovation Interviews’ – Summary Integrating ideas using appropriate tools bringing together thought leaders helps to

improve the process of innovation and overcoming traditional, hierarchical ways of

operating. These are crucial conditions for success according to latest research on

innovation (Caimi, 2019; Courtright, 2020). The interviewees seconded these findings

and they saw PMs as beneficial for innovation processes. The negative impact of

hierarchy was frequently stressed as well. The number of occurrences of the derived key

aspects from the interviews is depicted in Table 6-12. These are the same thematic key

aspects that came to the fore and were used during data collection analysis with one new

addition, namely ‘Advantageous for Innovation Processes’ as the focus of the additional

interviews was on innovation management and idea markets.

Innovation management processes comprise idea generation and scenario management

followed by idea evaluation and portfolio management – or, expressed differently,

‘scanning and screening’ plus ‘breeding and incubation’ (Huesig and Endres, 2019; Ini-

Novation, 2020). An idea market typically supports the capture, selection, and filtering

parts of the process, helping to evaluate the ideas (Soukhoroukova, Spann and Skiera,

2012) – the above speaks for the choice and naming of the additional key aspect.

As a result of the interviews, an attribute of CPMs’ future successfulness could very well

be its application as a backing element to business innovation – innovation process

support appearing at a quarter of all key aspects (17 manifestations from a total of 67).

Hindrances (terms with a negative meaning in the table below) could be ‘engineered

away’ as useful improvements had been suggested regularly to make PMs in general more

beneficial (Buckley, 2016; Sung et al., 2019; Stershic and Gujral, 2020).

139

The overall outcome from the new interviews grouped around the same subjects as the

interview results from the main study when leaving out the newly introduced key aspect

relating to the advantage of idea markets. Amongst thematic aspects, implementation was

considered as vital, trailed by items related to an unfavourable reception: hierarchy,

needed alternatives, management acceptance, market abandonment, etc.

In this context, however, implementation could be seen as a hurdle to be overcome and

bridgeable (Gerlach and Brem, 2017), viewing the term as neutral, rather than negative.

The negatively slanted key aspects then start at a distant third with hierarchy having six

occurrences in Table 6-12.

Table 6-12 Major Points from Interviews around PMs and Innovation Management (Developed for Research)

Results from Interviews with Economic Professors (4), Enterprises (4), Innovation Consulting Firms (2)

Key Thematic Aspect Connotation Tally of Occurrences

Implementation Issue Negative – reclassed as Neutral 19

Advantageous for Innovation Processes* Positive 17

Hierarchy Negative 6

Use of Alternatives Negative 6

Market in Use Positive 5

Management Acceptance Negative 4

Part of Work Processes Positive 3

Market Abandoned Negative 2

Accuracy Positive 2

Market Principle Questioned Negative 1

Not a Topic of Interest Negative 1

Fosters Employee Motivation Positive 1

SUM 67

* New Topic derived from ‘innovation interviews’

For a high-level comparison with the interview interpretations from the main study the

researcher clustered the key thematic aspects in Table 6-13 in a similar way as explained

at the end of section 6.3.1. Connotations were kept as from above, e.g. implementation as

neutral. Two items were given a cluster in their own right – ‘Implementation’ and

‘Exactitude’ – although the ensuing agglomeration only comprises one term; so decided

because ‘Implementation issue’ was re-classed and reached quite high importance and

‘Accuracy’ was a major topic in this thesis across interviews and literature review.

140

Table 6-13 Clustering of Key Thematic Aspects (Developed for Research)

Grouping Including Their Characteristics Leaving Out the Thematic Aspect ‘Advantageous for Innovation Processes’*

Cluster Trait Key Thematic Aspects

Non-Acceptance Negative Management Acceptance, Market Abandoned, Hierarchy, Market Principle Questioned

Use of Alternatives Negative Use of Alternatives, Not a Topic of Interest

Implementation Neutral Implementation Issue

Company Acceptance Positive Market in Use, Part of Work Processes, Fosters Employee Motivation

Exactitude Positive Accuracy

* as it only occurred in the ‘innovation interviews’ and leaving out ‘weak benchmark’ from the main study with its very low occurrence

Based on the introduced clustering, Table 6-14 shows a similar picture across the two

different types of interviews the researcher had conducted (10 around innovation

management, 55 around general usage of CPMs), the prevalence of thematic subjects in

percent is about the same in three clusters with two notable exceptions in

‘Implementation’ and ‘Non-Acceptance’.

Table 6-14 Comparing Interviews Focusing on Innovation Management with Interviews from the Main Study (Developed for Research)

Contrasting Results per Cluster in Percent

Clustered Aspect Occurrence Innovation Management Occurrence Main Study

Non-Acceptance 19% 49%

Use of Alternatives 10% 15%

Implementation 28% 15%

Company Acceptance 13% 17%

Exactitude 3% 5%

CPMs did not seem to need to burnish their reputation in the arena of innovation as much

as in general business use, non-acceptance was two and a half times as much an issue

there (almost 50% in the main study interviews versus roughly 20% in the ‘innovation

interviews’). As already mentioned before (cf. section 2.6), a company’s management

accepts ‘downsides’ easier, they play a lesser role, when trying to further new ideas.

When it comes to innovation firm culture is not the most important variable impacting it

in an organisation according to Bahcall (2019), as can also be seen in Table 6-14.

141

The importance of developing and promoting new technologies and products is put forth

regularly (Sopra Steria, 2020) but the cornerstone of success is proper implementation as

can be seen in the table above; it was the most important issue mentioned in the

‘innovation interviews’ at 28% of all answers and with 15% of answers in the main study,

also not negligible. Overcoming this could lead corporate PMs to excel in innovation

management according to the above expounded interview schedule around this aspect.

New applications for PMs do appear from time to time also in other areas than innovation,

even though more in academic research (cf. Deck and Porter, 2013; Huang, Fildes and

Soopramanien, 2019; Abolghasemi and Dimitrov, 2020) and hence not underpinning

corporate usage in general, bar perhaps in the special field of innovation as idea markets.

6.7 CPMs’ Chances in a Nutshell

Summarising the results from the researcher’s analyses and arguments brought forth

regarding CPMs painted a picture of an artefact with remarkable forecasting accuracy but

low usage in corporations. This was mainly due to concerns stemming from company

culture and subsequent acceptance challenges of CPM outcomes by management, with

company hierarchy being the most important factor accounting for the poor uptake of

CPMs.

Innovation management, however, emerged as a topic enhancing the likelihood that

CPMs will be perceived a success because companies see innovation as a differentiator

to remaining competitive. If hierarchical aspects can be overcome, CPMs could enhance

innovation and idea management – possibly in combination with other forecasting

instruments – due to the belief that successful innovation needs a ‘human element’, which

a CPM offers through its aggregation of assessments from human participants.

142

7. Discussion

A consideration of PMs starts with the assertion that based on the representation of a

future event as a stock to be bought and sold, the properties of anonymity, diversity, and

independency brought forward by a PM allow for judgemental forecasting which is

considered equal to or better than other methods such as polls, surveys, Delphi, etc. These

three premises result in an alleged superior forecasting ability compared to tools which

suffer from deficiencies like a lack of diversity or groupthink. Aggregating diverse views

and information can tease out ‘overlooked’ but important information when making

forecasts. As PMs can facilitate such aggregation, this is a clear advantage especially as

visibility of information plays an important role in risk assessment during decision-

making about future events, according to Blastland and Spiegelhalter (2013).

The mentioned arguments failed to translate into a clear CPM success, though. The thesis’

contribution is working out reasons for this. Namely corporate culture, especially in

hierarchical configurations, is not amenable to accepting viewpoints stemming from

outside (top) management circles, particularly when presented by a lesser known or even

unknown mechanism like a PM.

Corporate culture might need to change, however, as in a rapidly changing economy and

society, spurred by advancing digitisation, there is a need for action; every company has

to decide how it wants to position itself. In a digital age, companies should actively

harness the power of innovation as a competitive factor. This pressure is especially

prominent for those whose focus is on innovation when their choosing of the timing and

route to innovation is critical in the face of competition from other companies. Two

strategy professors at INSEAD suggested using creative power and technology to assess

new opportunities in an easy way and go beyond conventional means and methods (Chan

Kim and Mauborgne, 2019). To build collaborative processes and nurture collaboration

potentially opens the way for methods like CPMs as they foster both opportunity

assessment and collaboration, setting a company towards a successful way the

“innovation game is played” (Rigby, Gruver and Allen, 2013, p.71). A corporate PM

could facilitate both the information aggregation and a price-based check on the most

promising opportunity when venturing along an innovation path.

143

The need for innovation management rises with the increasing volatility and uncertainty

of business environments. Promoting openness in company structures is necessary to

establish receptiveness and comprehension to deal with the ambiguity of digitalisation, a

plea for an innovation management which uses appropriate tools to sustain and facilitate

inclusion (Frishammar et al., 2019; Goffin et al., 2019; Wilken and Kirfel, 2020), as

facilitated with a CPM. For a forecasting mechanism to succeed in this arena Smith and

Saritas (2011) and Baczyk (2019) focused on appropriate prerequisites for success:

Collaboration – i.e. synergies from shared insights –, variety – rather than reductionist

analytics –, and diversity – including iconoclastic perspectives –, highlighting once again

the tenets of a PM.

7.1 Literature Reappraisal

Tying in with the above, the literature review in this thesis concluded three important

points regarding the usefulness of CPMs: a strong theoretical underpinning, limited use

in business, and the facilitation of innovation management. The literature suggested that

challenges to the use of PMs outweigh the benefits; as a consequence of the findings in

this thesis the reviewed literature needs to be reappraised. The reappraisal focuses on the

breadth of research offered by the Journal of Prediction Markets and recent publications

from 2020 and 2021.

7.1.1 The Journal of Prediction Markets (JPM)

It is notable that the first review of PM literature (Tziralis and Tatsiopoulos, 2007)

published in the first issue of the JPM still ranks among the top 25 of all publications on

PMs; other influential papers were also published in the JPM (Klingert, 2017).

The JPM, established in 2007, demonstrated PMs’ concepts, advantages, and challenges

in its first years of publication, with a special feature on corporate applications in the year

2009. Since then, however, it seems to have completely moved on to other subjects,

indicating a lack of business use for PMs. A journal explicitly dedicated to PMs – “The

journal will be … publishing articles on every aspect of the study of prediction markets”

(The Journal of Prediction Markets, 2020, researcher’s emphasis) – failed to focus on

actual applications in business, often the content of articles published were almost

exclusively about sport betting.

144

This lack of business leaning prevailed. Amongst other articles relating to PMs, just one

article in 2013 on ‘Managing Risk Using Prediction Markets’ showed a business leaning

with a novel idea on supporting risk management with PMs (Varma, 2013). Between

2014 and 2019 the JPM published a further 55 articles of which only four (two in 2014

and two in 2017) are notable exceptions: the literature review on innovation markets

(Buckley and McDonagh, 2014), the literature reviews on PMs in general (Horn,

Ohneberg, and Ivens, 2014; Klingert, 2017), and two professors assessing PM

performance in a corporate context (Buckley and Doyle, 2017a). Generally, however,

business themes were only marginally referred to.

As the JPM has been acquired by another publishing house, it has now been relaunched

and its 19 new issues, having arrived in March, September, and December 2020, barely

cover PMs at all anymore let alone PM topics with a business leaning. Particularly the

focus on idea or innovation management supported by PMs – which the researcher

forwarded in this thesis – has not been covered in the recent past at all; articles around

these subjects only stem from 2009, with one exceptional case from 2014 as visible in

Table 7-1.

Table 7-1 The JPM and Its Coverage of PMs in the Context of Innovation (Developed for Research)

Only Six Articles Around Innovation or Idea Markets Exist – References per Year, Alphabetical Order

Kamp, G. and Koen, P.A. (2009) ‘Improving the idea screening process within organizations using prediction markets: a theoretical perspective’, Journal of Prediction Markets, 3(2), pp.39-64.

Lavoie, J. (2009) ‘The innovation engine at rite-solutions: Lessons from the CEO’, Journal of Prediction Markets, 3(1), pp.1-11.

Litan, B. (2009) ‘Comment on Bell Article’, Journal of Prediction Markets, 3(1), pp.111-112.

Ottaviani, M. (2009) ‘The Design of Idea Markets: An Economist’s Perspective’, Journal of Prediction Markets, 3(1), pp.41-44.

Spears, B. and LaComb, C. (2009) ‘Examining trader behavior in idea markets: an implementation of GE’s imagination markets’, Journal of Prediction Markets, 3(1), pp.17-39.

Buckley, P. and McDonagh, E. (2014) ‘Idea Markets: a literature review and classification scheme’, Journal of Prediction Markets, 8(2), pp.76-88.

Even though research also aimed at advancing practical applications, PMs seem to be an

arcane tool generating mostly theoretical interest in academic articles focusing on specific

attributes of PMs rather than useful applications to this day (Klingert, 2017), even beyond

the JPM: typified through e.g. reporting on stylised facts on price changes, conceptual

information acquisition in a PM, their market mechanisms, and exploring PMs solely

through modelling (cf. Cummings, Pennock and Wortman Vaughan, 2016; Page and

Siemroth, 2017; Bottazzi and Giachini, 2019; Restocchi, McGroarty and Gerding, 2019).

145

A brief examination of PM publications from 2020 to February 2021 – based on a search

using Google Scholar – revealed 281 articles which lacked any serious business-related

aspects. Topics covered were PMs’ methodological basis with 53% of all papers, topics

related to (mostly theoretical) economic questions and political events or elections,

accounting for 7% and 12% of published items. Furthermore, how blockchain technology

could work in the context of PMs was looked at with 16% of occurrences which could

also be classed as covering PM methodology. Within academia, interest was mostly

around assessing the replicability of research or the furtherance of science (10%).

Just four papers (2%) had the goal of advancing the area of business and management:

around combining and supporting the scenario technique with PMs (Tiberius, Siglow and

Sendra-García, 2020), customer foresight methods (Eller, Hofman and Schwarz, 2020),

ethical risk identification (Floridi and Strait, 2020), and developing business strategies

(Malone, 2020). The details regarding the number of published articles per year are given

in Figure 7.1, categorised by the researcher in a classification scheme which differentiates

between descriptive works regarding PM methodology, articles around gauging

economic impact, election-oriented studies, articles dealing with blockchain PMs,

publications pursuing internal academic topics, and the business accentuated papers from

2020.

Figure 7.1 Classification of Published PM Articles Based on a Google Scholar Search with the Term ‘Prediction Markets’ and ‘Sort by Relevance’ (Developed for Research)

Academia & Science

25Blockchain

38

Economic18

Methodology128

Political Events -Elections

24

Acade-mia

4

Block-chain

6

Methodology21

Political Events10

2020

Business4

Jan-Feb 2021

Economic | 3

146

All these examples encapsulate two important points: the theoretical underpinnings are

still debated and even advanced by research, but the debate is about discussing insights

and findings from its own field (Klingert, 2017). Thus, as even a dedicated journal did

not cover practical applications, then a lack of promotion in the business world is no

wonder. The CPM’s promise in innovation could also be questioned given the scant

coverage in the literature.

Support for innovation management would be important, though, as raising (as far as

possible) the amount and range of ideas offered can be associated with success. For Syed

(2019), the disposition towards knowledge sharing would be valuable in a world of

complexity as employees are frequently too bound by social conventions to openly

express their opinions which leads to management pushing extensive projects in the face

of unseen misgivings. This was borne out by Billinger and Rosenbaum (2019), whose

study showed that organisational hierarchy matters in fostering cooperation and openness

in expressing an opinion. The more rigid the hierarchy the more openness is inhibited.

Companies should see an advantage to actively promote a diversity of opinions according

to Syed (2019), and as already put by the German poet, naturalist, and statesman Johann

Wolfgang von Goethe in a verselet in 1814: no living thing is a one, it is always a

multitude (von Goethe, 2013).

7.1.2 Literature on Alternatives to PMs In recent years the focus in the strategy and planning literature has moved away from a

sole and single future to gearing more towards scenarios intervening in the area of

strategic management, providing multiple credible states of the future (MacKay and

McKiernan, 2010; Goodwin, 2019). Here Big Data analytics could evolve into an ersatz

PM or even a full successor; with the advantage of such tools stemming from their alleged

ability to anticipate disruptive new technologies or identifying and tracking possible

warning signs (MacArthur and Rainey, 2019). However, Big Data acts passively,

conclusions are extracted from data in an automated fashion, whereas PMs act actively:

trading is active, and findings and conclusions are put to the test of other traders’ views

immediately through the underlying pricing mechanism, a possible advantage from the

use of a CPM over autonomous systems.

147

But recent publications to enhance PMs also focused more on the ‘enhancement’ itself

than on PMs, like surveys, Delphi or nominal groups (Dai, Jia and Kou, 2020; Bolger,

Nyberg et al, 2020; Bolger, Rowe et al, 2020) and companies need to look at several

probable futures and no longer plan based on solitary projections alone (Wulf et al.,

2013). For instance, private equity firms realised that more than 30 percent of decisions

reach a different conclusion when pitting a variety of opportunities against each other

(Bradley, Hirt and Smit, 2018).

Scenario planning was quoted as being widely used within multinational firms

(Vecchiato, 2019; Boyonas, Olavarria and Saenz, 2020). This is questionable, as Bain

and Company pointed out that its utilisation has fallen from 69% at the pinnacle in 2007

to a mere 19% lately (Rigby and Bilodeau, 2018). In creating a high-performing digital

enterprise moving from good to great would require, amongst others, performing granular

diagnostics (‘Big Data’) but also carrying out scenario tests to check various business

model assumptions and industry trends (Desmet et al., 2015). This combines Big Data

and not CPMs with scenario techniques.

Although the literature seems to be downplaying PMs, another important contribution of

this thesis is to show clear pathways to their acceptance in companies.

7.2 Result Appraisal – Findings in the Context of the Research Questions

When adopting a new technology which looks proficient but has no proven track record

like with a PM, one way forward could be to choose simple, proven technologies familiar

to the company, highlighted by recent research (Blumberg, Delaet and Kartikeya Swami,

2020). Innovation management, its importance stressed several times in this thesis, could

be the exception. With the high level of uncertainty inherent in the process (Berker, 2010)

the inclusion of more unknown processes and tools is accepted to a higher degree (Engels,

2020). The conceptual framework in this thesis (cf. section 4.2) has mapped out a path to

answer its central question regarding the relevance and usefulness of PMs for companies.

From the central findings of literature review and data collection, the framework had

made the option of supporting innovation management with corporate PMs already

apparent. Using the elements of the framework, the overall high-level thesis results are

summarised in tabular form below, comprising findings from 65 interviews and a

thematic analysis of the literature.

148

In Table 7-2 each framework element is depicted together with major aspects pertaining

to it, expressed with findings and/or (direct) quotes pre-eminently from Starbuck (1993;

2006a; 2006b; 2009; 2013; 2014a; 2014b; 2015; 2017). As the sources drawn on from

Starbuck’s work illustrate the breadth of the subject investigated in this thesis and support

the researcher’s own findings and conclusions, they relate them to existing research.

Hence the researcher saw this particular artifice of presentation as a useful summary

encapsulation. An explanation from the researcher of the points mentioned is added,

highlighting the researcher’s findings (emphasised in italics).

Table 7-2 Adoption of PMs in Business – Aspects from Data Collection and Analysis in Line with the Theoretical Framework (Developed for Research)

Inputs and Influences into PM Tool Adoption; Assess and Embed CPMs as a New Forecasting Process

Framework Element Major Aspects Path to Facilitate / Mitigate Adoption

• People have “unrealistic assumptions about … [their] knowledge and about their abilities to forecast accurately” (Starbuck, 2006a, p.474), averaging out prediction errors (in a market) leads to precise results

• “Managers’ perceptions of their [company’s] environment do [often] not correlate with ‘objective’ measures of those environments” (Starbuck and Mezias, 1996, p.113), eliminating errors and biases in such a context would lead to higher accuracy in deliberating futures

• “High control over the predicted variable tend [sic] to increase the accuracy of forecasts” (Barnett, Starbuck and Pant, 2003, p.653), facilitated by encapsulating the information sought into a traded stock appropriately

Problems which make group decisions difficult are mostly discarded in structured approaches such as PMs, which show clear evidence of high accuracy

• Change strategies in response to new in-formation (Starbuck, 2006a), PMs moti-vate frequent information updates and thus incorporating hidden information as well – “learning from rare events is erratic, but potentially very profitable” (Starbuck, 2009, p.933)

• Weak signals are often overlooked in forecasting and relying on supposedly proven mechanisms often create inaccurate predictions and over-emphasise unlikely events (Desai et al., 2015), challenges which would be overcome by PMs

Efficient information aggregation is one of the hallmarks of PMs

• Consensus can be dangerous and unlikely to produce wanted outcomes and it “may … be a liability, insofar as it induces managers to focus too narrowly and to underestimate the actual uncertainty of future events” (Starbuck, 1993, p.357)

• Research about decision-making usually assumes managers are aware of and understand organisations’ environments (Mezias and Starbuck, 2009), they often lack insights that do not stem from their sphere of influence

PMs effect proper probability estimates and data gathering with their independent trading mechanism

• Managers need to be informed about organisational and environmental properties full scale, to do that they have to exploit improved technology (Mezias and Starbuck, 2003)

• “Learning from rare events is statistically unusual and rare, … both individuals and organisations can benefit significantly from active efforts to learn from rare events” (Starbuck, 2009, p.925), better to design “robust organisations that can tolerate misperceptions” (Mezias and Starbuck, 2003)

• “A … desirable tactic is to increase the diversity of advice and information” (Starbuck, 2017, p.37) to improve forecast results

Anonymity, diversity, and independence contribute to effective forecasting, PMs grant that by being very comprehensive as far as information inclusion is concerned

Method Effectiveness

Theoretical Underpinning

149

Framework Element Major Aspects Path to Facilitate / Mitigate Adoption

• Firms “devote most of their information-gathering efforts to sources that have proven to be useful” (Starbuck, 2017, p.36)

Companies are not readily using the unproven, a challenge for PMs

• Inject common sense into a forecasting process, make sure existing resources / information are not suppressed (Starbuck, 2017)

• A collective mind and mindfulness may be helpful for organisations (Starbuck, 2015), a view that fosters collective intelligence

Including diverse voices helps, companies often lack effective problem-solving processes for group inclusion and decisions

• Formalisation, particularly stemming from hierarchy, undercuts strategising’s potential value (Starbuck, 2006a) especially in serious crises

• Learning is unlikely to occur much in a large, divisionalised firm (Baumard and Starbuck, 2005)

• Hierarchical environments strongly influence what is learned (Starbuck and Hedberg, 2001)

• Senior executives dominate daily operations in large global corporations (Starbuck, 2014a)

• Certain “knowledge creation blocks discovery, because when people decide that an explanation or observation is true, they create barriers to the acceptance of alternative truths” (Calhoun and Starbuck, 2003, p.471)

Keeping the status quo relying on top-down ‘truths’ is the norm in hierarchical firms

• “Managers often perceive [new information] inaccurately … assuming that they have precisely correct expectations about future events” (Salgado, Starbuck and Mezias, 2002, p.168)

• Changing an organisational design can improve organisations’ efficacy (Dunbar and Starbuck, 2006)

• Inject realism into managers’ perceptions (Starbuck, 2006a) to foster trust and good feelings for employees to enhance the ‘width’ of managers’ viewpoints

PMs – ‘if allowed to’ – can overcome informational hurdles and silos in a firm

• “Complex interactions among processes make designers’ forecasts unreliable, serious future problems can be avoided by keeping processes dynamically balanced” (Hedberg, Nystrom and Starbuck, 1976, p.41), forecasting processes are in need of suitable supportive tools

• There is immense power in conceptualisation and creative logic (Starbuck, 2013)

Pointing out the need for the support of creative deliberations in companies

• To advance a company is to understand “how an organisation can be designed to meet social and technological changes and to reap advantage from them” (Hedberg, Nystrom and Starbuck, 1976, p.41)

• “With appropriate cultures and leaders, small and egalitarian discussion groups that we call ‘collegial nests’ can become powerful generators of innovative ideas” (Schwab and Starbuck, 2016, p.168), particularly if supported by appropriate modern communication technology to foster group collaboration

Future orientation and forward-looking approaches are seen as advantageous

• Avoiding complication usually performs better than intricate approaches and “complex forecasting methods [can] mistake random noise for information” (Pant and Starbuck, 1990, p.433)

• Baseline modelling and iterative approaches, and a comparison across methods is advised (Schwab and Starbuck, 2013)

Simple methods are preferable to complex ones, PMs would be considered as potentially too multifaceted

• Deploy more than one level of analysis, like in triangulation, look at the same object from two or more perspectives (Starbuck, 2006b)

• Hedging of bets (Baumard and Starbuck, 2005) and scenario thinking (Clegg and Starbuck, 2009) create improved future states for a company

Scenarios but also combining forecast methods improve forecasting quality, PMs could be included

• Procedures are needed that yield more robust knowledge (Starbuck, 2006b)

• Strategists can emphasise informal communication (Starbuck, 2017) to advance knowledge generation and sharing

Innovation and product creation occur based on knowledge gathering (Starbuck, 2014b)

• “Surprises occur when organisations try to exceed the limits of their capabilities. The surprises include both serious accidents and remarkable discoveries” (Farjoun and Starbuck, 2007, p.541)

Discovering new ideas and learning from the past is an important aspect of a company’s success and path to the future

PM Deployment

Impact Company Culture

CPM Use cases

Forecast Combination

and Competition

Alternative Use Cases –Innovation

Management

150

As the elements of the theoretical framework were developed by the researcher to support

the central summary research question, the researcher’s findings associated with the

individual elements also steered answers to the specific research questions posed which

are now addressed.

Research Question 1 – What are the advantages and disadvantages of PMs?

Problems such as groupthink, which complicate (group) decision-making, are usually

obviated in structured approaches such as PMs, which also show clear evidence of high

accuracy. Efficient and comprehensive information aggregation is one of the hallmarks

of PMs plus the provided anonymity, diversity, and independence which contribute to

effective predictions. However, implementation and usability of PMs is sometimes seen

as complicated. The overall thesis results in this context are encapsulated in one negative

and three positive high-level points:

• High degree of accuracy (+)

• Efficient aggregation of dispersed information (+)

• Bias elimination (+)

• Perceived complexity (-)

Research Question 2 – What are the barriers to adoption of PMs by corporations?

Inclusion of diverse voices would help in decision-making, but companies often lack

effective problem-solving processes for group involvement and group decision-making.

Maintaining the status quo by relying on top-down ‘truths’ is often the norm in

hierarchical companies and people create barriers to accepting alternative truths. A

summary is embodied in these statements:

• There is often managerial resistance to ‘unwanted’ and/or ‘not curated’

information

• Managerial influence on information dispersal within the organisation is lacking

• Hierarchical settings tend not fully accept ‘wisdom-from-below’

151

Research Question 3 – Are there ways to make PMs more adoptable for corporations?

Discovering new ideas and learning from the past is an important aspect of a company’s

success and path to the future. Forward-looking approaches such as PMs are seen as

beneficial to supporting efficient forecasting mechanisms which overcome hierarchical

challenges, here usability plays an important role. Thesis results are anchored in two

overarching aspects:

• Seek out areas of application where pressing business needs overcome hindrances

from company culture, for instance in innovation management

• Focus on ease-of-use in implementation

In summary, the answer to the overall enquiry of which barriers to the adoption of CPMs

exist, and what schemes would facilitate the decision for or against their adoption is firstly

to accept the underlying difficulties in many companies from hierarchical challenges.

Secondly, to find an application for CPMs in an area where robust and innovative insights

are needed to be gathered from all parts of an enterprise.

It would also be prudent to make CPMs (and PMs) more known to further their use, “top

managers … [need] to loosen the bonds of obsolete learning [and obsolete tools],

including their own commitments to current beliefs and strategies” (Starbuck, 2017, p.37)

and focus on the inclusion of new techniques that further imagination to support

potentially unusual business strategies (ibid.), and to support innovation.

Discovery and validation of ideas were described as highly important and the validation

can benefit from a screening mechanism (Bland and Osterwalder, 2020). The application

of corporate PMs in innovation is thus a possible way out – the additional interviews

beyond the original data collection showed innovation management as a possibility for

CPMs (cf. section 6.6.2) and research from Stanford University underscored this

(Doorley et al., 2018). Particularly creating an environment for the encouragement of

diverse perspectives and lending authority to all participants to avoid exclusion of

important viewpoints were considered being important cornerstones of the advancement

of innovation (Turner et al., 2017), facilitated by a CPM via its underpinnings (Pathak,

Rothchild and Dudik, 2015).

152

Research from Harvard, McKinsey, and the University of Sydney (Beer, Finnström and

Schrader, 2016; Rinaudo, 2020; Koller, Lovallo and Brown, 2020) demonstrated

practices that managers can use to break with cognitive and organisational preconceptions

that hinder knowledge creation through managers’ predispositions towards hierarchical

approaches (cf. Ferreira, Fernandes and Ratten, 2017). The major insight was that one

cannot change individuals easily, the rules around how organisations lead and make

decisions must be changed looking at debiasing instruments and structured approaches

which result in ‘neutral’ and aligned results.

Dialogues about results worked better when management recognised the value of irksome

information rather than ‘running away from it’ or figuratively putting their heads in the

sand and avoiding the information.

Such discernments constitute both a chance and once again a stumbling block for CPMs.

The needed elimination of biases and overcoming “employees’ fears of telling the senior

team about obstacles” (Beer, Finnström and Schrader, 2016, p.50) can be facilitated with

the help of a CPM as it provides the necessary anonymity and diversity that would make

decision results much more resilient, particularly in innovation (Lisak et al., 2016;

Duchek, Raetze and Scheuch, 2020). On the other hand, acceptance of such ‘market-

driven’ results by management, and taking on unpleasant or ambiguous results in general,

are still lacking (Gray and Ulbrich, 2017).

As pointed out above, CPMs actively integrate users into innovation activities and tap the

productive potential of a crowd to accomplish business goals like product innovation

(Dowling, Noll and Zisler, 2019), which presents a clear opportunity for a CPM.

7.3 Limitations of the Research

As the obstacles to CPM acceptance could also have a root in resistance to change,

technology acceptance, or culture and decision-making theories, a more comprehensive

theoretical underpinning of these areas might be called for. However, the researcher’s

findings highlighted CPM reception based on the effects of leadership and culture in

practice and as this thesis focused more on a perspective of knowledge generation about

PM usefulness rather than on explicit theoretical considerations about them, the exclusion

of a deeper consideration of the above-mentioned theories seemed to be justified.

153

The theoretical framework (cf. Figure 4.2) focused on PM use cases, alternatives to PMs,

and forecasting method/tool combinations. How entrenched the use of PM alternatives

and more traditional tools in company settings really is and whether their deficiencies

were apparent could have been fleshed out more if the researcher had had a chance to

increase or expand the pool of interviewees. In addition, a more comprehensive interview

sample could have helped to draw more expansive conclusions, as only 13 out of the

overall 65 interviews conducted by the researcher resulted in showing PM use at the

companies or organisations the interviewees worked for1.

As management acceptance together with a firm’s hierarchy – or more generally company

culture – was amongst the main stumbling blocks for CPMs, higher-level executives as

consumers of forecasts made by others and relegated to them (including via a CPM) could

have made additional worthwhile contributions. It would have been beneficial if these

people had been included in greater numbers (just 10 out of the total of 34 ‘business’

interviewees) yet the researcher did not have more access to these particular company

personnel.

The researcher did not believe that this ‘reduced’ interview sample limited the scope of

his analyses, though and was convinced for it to have had only minor impact and not to

introduce constraints. The examined results were comprehensive and clear and

unambiguous conclusions could be drawn from them. It emerged that applying CPMs in

innovation management would prove to be advantageous.

The researcher did not discuss alternative communities such as the military or government

services, where PMs had also been debated in the past (Hanson, 2007; Borison and

Hamm, 2010; Bell, 2011), for two reasons. PMs often received unfavourable publicity;

one government-sponsored market was called the ‘Terrorism Futures Market’ in the press

(Weigle, 2007). Exposure to gambling laws and PMs’ legality in that context were viewed

quite critically so that public sector PMs have rarely been continued (Ozimek, 2014;

Weijers, 2021). These points were aptly summarised as “bureaucrats recoil from legal

[and reputational] uncertainty” (Bell, 2011, p.434). Furthermore, with a focus of this

thesis on applications for enterprises, the transferability was seen as limited by the

researcher, especially as the public sector does not contain profit-making entities.

1 it is coincidental that the number of 13 interviews here is the same as in the pilot. Only three of the pilot interviews were with people who had experienced PM deployment

154

Furthermore, as this work depends a lot on the researcher’s interpretations, his own biases

might have exerted some limitations. However, strategies were consistently applied to

mitigate this, for example by running analysis results past interview partners to seek their

feedback and substantiating findings with references from the literature as much as

possible. The researcher also pointed out potential shortcomings and addressed those

critically throughout this thesis.

7.4 Research Impact

Noting the limitations explained above this thesis has made contributions to the academic

canon and has helped to aid decisions about the deployment of PMs in companies.

The researcher accredited and compiled the importance of certain topics, issues related to

corporate culture and subsequent management acceptance, the prominence of which

showed a clear discrepancy to their theoretical acclaim (cf. Figure 6.2). The literature on

PMs addressed these themes only sparsely, the thesis amended and developed such core

PM concepts further with perspectives from literature in other fields. By bringing this to

light the researcher closed a clear gap in academic publications, as the subjects of

management acceptance and company hierarchy have proven to be essential factors

influencing PM acceptance in corporations, as determined in this thesis.

The second contributory aspect follows from the researcher having presented CPMs as

an approach to future-oriented information, and as a method to produce strategic insights,

especially to create long term views on strategic forecasting. This is particularly so when

choosing approaches to enable innovation management. Here CPMs, by enhancing a

company’s innovation capability, can be a factor contributing to stronger business

performance and sustainability. Thus, it is recommended that CPMs be deployed in the

process of innovation management.

155

8. Conclusion

8.1 Epilogue The overall perspectives arrived at in this thesis put CPMs in the context of improving a

company’s decision-making and planning and made important additions to the literature,

as academic publications on PMs did not sufficiently address how PMs would be

worthwhile in a company setting and what managers should look for to underpin robust

forecasting processes in this particular context.

A renewed study by the researcher of the topics and topoi offered by a search of PM

literature resulted in 1,792 ‘hits’ originating from the period from 2019 to February 2021.

From those, 630 literature sources remained which actually had to do with the subject of

PMs of which in turn only 13 covered themes related to business management, company

culture or hierarchies in the context of PMs – the latter two themes being key results from

this thesis. Considering more recent PM literature, a very similar ‘desultory’ outcome had

surfaced in section 7.1.1, here just 4 out of 281 matching scholarly articles or papers

remained when the results were narrowed down to ‘business articles’ in a stricter sense.

Table 8-1 indicates the subjects that emerged and how they fit into the business-oriented

PM literature, for 2021 no such articles have appeared yet.

Table 8-1 PMs Associated with ‘Business Management’ in a Forecasting Milieu – Literature Search 2019-2021 (Developed for Research)

Business Plus Forecasting Related Topics from PM Literature – Thirteen Results

Topic Brief High-Level Content Summary Source

Hierarchy Command-and-control leadership has not been recommended for some time, but no fully-fledged alternative has emerged. This is partly because senior leaders are ambivalent about changing their own behaviour

Ancona, Backman and Isaacs (2019)

Business Management The crowd prevails over the individual in forecasting performance Fladerer and Kurzmann (2019)

Hierarchy An investigation if the information that employees have is reflected and factored in the expectations and decisions of top managers

Huang, Li and Markov (2019)

Company Culture

Business performance is in need of expertise that cannot be distilled by the strictly technical aspect of mechanical data analysis. Blending in firm culture and overall capabilities is needed to increase the company’s performance

Mu, Zhang and Gilliland (2019)

Company Culture Encourage stakeholders to communicate useful information and not to withhold it

Remidez, Stodnick and Beldona (2019)

Company Culture Integrated idea management needs to solve the problem of information integration, PMs can do this to some extent Thom and Brem (2019)

Business Management Combinations of methods are suitable to conduct customer foresight studies

Eller, Hofman and Schwarz (2020)

Business Management There is a need to anticipate or predict the ethical issues that newly observed technologies or systems supplying new needs may present

Floridi and Strait (2020)

Business Management Developing business strategies by accepting that humans and system-based approaches need to work together Malone (2020)

Company Culture The part that culture plays in knowledge sharing and innovation is paramount

Passerini, Osatuyi and Stipe (2020)

156

Topic Brief High-Level Content Summary Source

Business Management

When forecasting, human judgement is often secondary to algorithmic methods. However, when it comes to including sociological phenomena, human judgement is frequently favoured over algorithms and is necessary to discern and sustain judgemental reasoning

Rona-Tas (2020)

Business Management Faced with the uncertainty of daily decisions, managers rely on their common sense, experience and judgement. But such management approaches are the source of countless errors in team decision-making

Sibony (2020b)

Business Management Enhancing forecasting mechanisms by combining methods Tiberius, Siglow and Sendra-García (2020)

This review of what the academic PM literature is concentrating on continues to illustrate

the contrast to the outcomes from this thesis. The few search results with a business

context point to aspects which the researcher had also recognised. However, the

researcher’s findings were much broader and more comprehensive and showed the issues

of management acceptance (or company culture) and hierarchy problems to be of major

importance, topics appearing in only half of the thirteen cases from recent PM-focused

literature, as can be seen in Table 8-1. The researcher had established that company

culture and its influence on the uptake of CPMs had not been covered sufficiently. For

example, the researcher’s literature review had identified 28 pertinent articles pertaining

to this area from outside of PM literature alone, plus according findings from the

researcher’s thematic interview analyses, all of which exposed the important issue

explaining the challenges in using CPMs to stem from a firm’s hierarchy. These findings

constitute an essential contribution to the body of academic knowledge about CPMs and

PMs.

The discoveries in this thesis established that for companies to improve forecasting

processes, and to become more innovative for instance, managers need to be willing to

push power and decision-making downwards in the organisation. This would also form

the basis for capturing knowledge with collaborative tools, like a CPM, as these rely on

combining but also accepting contributions from various hierarchical layers in a

company. So despite the fact that from a literature perspective, CPMs’ chances might be

construed as falling short of the mark, this thesis, however, demonstrated that CPMs can

become successful when deployed in the right application areas.

In this thesis, it is shown that there are chances and opportunities to surmount obstacles

to the success of CPMs as especially when faced with a crisis, leaders ought to abandon

their conviction that top-down approaches provide stability, and move away from

command-and-control structures.

157

Organising rapid problem-solving and knowledge collection via teams has clear

advantages and employees need to bring critical challenges or opportunities to a manager

(D’Auria and De Smet, 2020; Fisher, Wisneski and Bakker, 2020), fitting with the

cornerstone of the information gathering of a PM, where insights are aggregated and

automatically evaluated based on a PM’s price mechanism and the anonymous trading

defeats inherent biases in decision-making processes.

Management decisions create payoffs linked to future choices, so they entail options or

scenarios that need evaluation. PMs possess the necessary requisites and fundamentals to

support that but need to overcome stumbling blocks to arrive at an accepted and

successful use in companies, which is explained in Figure 8.1 showing a ‘pathway’ of

influences.

Figure 8.1 Outline to Arrive at a Recommendation for Corporate Prediction Markets, a Path to and Factors for Success of Prediction Markets in Corporations: Support of Innovation Management (Developed by the Researcher)

Apart from overcoming hierarchical aspects, however, PMs need to be made more known

to gain use and acceptance. When chief forecasting technologies were summarised in a

recent major handbook of innovation management, only Delphi and expert judgements

were mentioned under collective wisdom; neither PMs nor idea markets were named (Yu,

2019). This goes to the heart of the lack of knowledge about PMs. For enhanced decision-

making, forecasting should encompass more stakeholders to arrive at more robust

forecasts (Petrakis and Konstantakopoulou, 2016; Gurkov, 2020), like pulling opinions

together in a PM which can only happen if this forecasting approach is known and used.

Gladwell’s (2000) book ‘The Tipping Point’ might give a hint on how to prevail over

such lack of recognition, as it illustrated putting levers in place to spark ideas about

implementing a new product or tool in an engaging way within a company.

PM Success in Corporations taking into account the influencing factors can lead to a successful route from PMs’ basis to CPMs’ acceptance

CPM Challenges

CPM usage is limited, organisational set-up

limits acceptance

CPMs’ Tipping Point

Combining PMs with other methods and

expanding their use cases

Fortifying Innovation

CPM adoption via innovation management

in companies

PM – Promising Fundamentals

PMs have a strong theoretical underpinning

and can be effective

158

This would be added to by systematically and transparently combining innovation finding

mechanisms to achieve a company’s modernisation, more mature companies usually

deploy several digital transformation practices and tools according to recent research

(Boehm and Smith, 2021) which the researcher confirmed with both literature

examinations and dialogues in the space of PMs.

As particularly the digital world presents companies with many opportunities to improve

but also to innovate, a hybrid structure with both centralised decision-making and

allowance for contributions from lower rungs in the hierarchy offers definite advantages,

and these contributions are encouraged by a CPM. Senior management culture and rigid

hierarchies need to be challenged to arrive at a more effective and comprehensive

decision-making and also to contain the emotional and political influences that can work

against the establishment of good and effective tools. As hierarchy has a clear impact on

corporate culture, this overarching theme from this thesis would play a key role in the

success of implementing and integrating a CPM into a company’s forecasting processes.

Strategic decision-making in the main is also affected by the type of hierarchical

structures identified in the review of PM literature. Companies often lack effective

problem-solving processes and do not understand how they could better underpin

decision-making especially in uncertain times. It has been established in this thesis that

for such precarious situations CPMs can make the needed innovation process particularly

robust and reliable.

Innovation opportunities could be evaluated with a CPM in an effective way obtaining an

appraisal of high-quality ideas for new products and even novel business strategies from

participants. Discussions the researcher had about innovation management bore this out.

Humans are needed to imagine novel solutions and to judge and evaluate them – possibly

with CPMs or idea markets, which, again, was highlighted by the interviews conducted.

A transformation to a new prediction approach, like a CPM in the shape of an idea market,

needs to work across functions and business units, to empower workers, and to change

the company culture to a collaborative one. Often the most remarkable innovations occur

when people explore the limits of their imagination together, collaboration leads to an

ensemble, a big picture (Dräther, Koschek and Sahling, 2019), and to express a result

coherently, as by the price function in a CPM. The trading aspect provides the necessary

rhythm to advance speedily when new insights arrive – a potential benefit of a CPM.

159

In addition, the academic literature supported the fact that increasing and improving

innovation leads to improved business results, an analysis of 70 research papers found a

strong relationship to that effect (Rigby, Elk and Berez, 2020). Recording customer and

market wishes and corresponding signals and having them evaluated with the help of

experts is essential both for the process of ideation and for early management warnings.

To make efficient decisions, the prioritisation of emerging options for action or new

product ideas is the sine qua non for decision-making and a market mechanism – a

corporate prediction market – is a way to achieve this, articulated through this work.

To foster the necessary participation in a CPM for such endeavours companies can use

elements of gamification to engage participants from all areas in a firm, gamification

would improve extrinsic motivation to participate and collaborate (Jenney et al., 2020;

Sykes, 2021). A company can achieve increases in collaboration through a well-setup

user-interface and maintain participation through continued use of easy-to-understand

approaches (Palmer and Hugo, 2013), an idea forwarded for CPMs in this thesis. Such

approaches and implementations could overcome what the researcher had identified as a

hindrance for CPMs, decision-makers’ reluctance of tapping collective knowledge as the

ultimate recipients of the prediction results. But the future of CPMs stands out from its

past in crucial ways as using a CPM in the context of innovation would be like a weather

report for an organisation, reporting on and forecasting future events and product

successes reasonably accurately on a continuous, potentially real-time basis, as

encapsulated in this thesis.

Overall, the researcher recognised limited use of PMs in the corporate world, mostly

attributable to company structures. On the other hand the researcher has also verified the

high degree of accuracy a PM can provide and therefore an application of such a wisdom-

of-crowd approach in a company setting could be deemed to achieve success. Innovation

management would be an efficacious application – using a CPM in the form of an idea

market – and has been established as a route to possible success in this thesis.

Prophecy regarding future developments often entails the consideration of making

something new – innovare in Latin. Innovation’s ‘magic ingredients’ are openness,

diffusion of decisions, and free speech (Danielsson, 2016). Looking at forecasting to

sustain innovation, particularly via technological support, can mimic or perhaps

paraphrase these three components into the three ‘magic’ pillars, the tenets, of a PM:

anonymity, diversity, and independence (Magnusson, Wästlund and Netz, 2016),

showing a PM’s match to support innovation and hence an idea market’s as well.

160

The current state of CPMs and their opportunities have been systematically exposed in

this thesis, these matters and issues had previously been obscured. Encapsulated again,

CPMs have a future, especially in new product development and in dealing with novel

situations. It was shown in this thesis that for this to happen, senior management and

decision-makers need to be more receptive to wider consultation and accept challenges

to their thinking.

As a final contribution in light of a valuable method not easily finding acceptance within

companies, as exemplified in the paragraphs above, it has been indirectly shown how

other concepts that had been shelved would be worth further consideration. Learning

around CPMs’ ‘failings’ but also their chances for re-emergence could generally provide

insight on why a good idea faltered or was considered ‘wrong’ and how to potentially

rectify this in other areas of decision-making as well. The methodology and approach

demonstrated in this thesis would also help researchers to get more insight into decision-

making ideas.

Last but not least, the researcher’s analysis of PM accuracy (cf. section 6.3.3) had been

received well by two specialists in the field of PMs (Koch, 2020; Ivanov, 2021); their

viewpoints on this researcher’s results (cf. Appendix B) underscored the utility of using

a CPM.

8.2 Recommendations for further Research Collective judgement, teaming, and an aggregation mechanism are three components for

successful forecasts in innovation (Good Judgment Inc., 2020). These ‘ingredients’ are

provided by a CPM. The preceding chapters in this thesis demonstrated that the

challenges for CPMs might be overcome when deployed in ideation processes. The

researcher would therefore recommend identifying and gauging useful application areas

for CPMs and to provide guidance on how to implement them successfully, all of which

the researcher saw as a further need. Furthermore, the researcher has demonstrated a

methodology and methods which could be applied to research decision-making and

management strategies.

For academia, by filling gaps in the literature, this research showed how forecasting and

decision-making could be improved as well as a methodology and a way to allow

innovation management to be more successful was demonstrated.

161

For practical recommendations this thesis revealed the same topics for CPMs to have

merit especially for innovation management. However, to embrace CPMs the issue of

hierarchies and silo thinking needs to be overcome. Senior management needs to change

its own culture and an organisation needs to come to terms with digitisation to elicit ideas

and operationalise CPMs.

The suggestions the researcher made for further research considerations are geared

towards researching areas this thesis brought to the fore but did not investigate in detail

as its focus was to seek more high-level explanations and not to investigate concrete

aspects of changes in PM set-up, implementation or other technical aspects.

8.2.1 Recommendations to Academia Academic consideration and research should focus on understanding how a market

concept could further an innovation practice within a firm. Innovation needs valid

evaluation mechanisms where contributors voice their opinions and concerns freely and

form a cogent result via aggregation.

To bolster usage of CPMs and enhance their chances of acceptance they need to be put

on more solid theoretical ground for instance by apprehending how different trading

interfaces would influence acceptance of and positive outcomes in a market, as PMs in

general have been criticised for their user-unfriendly operability and further research had

already been called for in this area (cf. Scheiner et al., 2013; Ivanov, 2013; 2019).

Thus research comparing different trading interfaces would be useful, the trial of user-

friendliness and an attractive interface to gain people’s trust and enticing them to want to

use an innovation platform is a facet to be further investigated. For instance, what types

of information a PM should present to its users to be informative to their decision-making.

The accuracy of market prediction with different user interfaces should be compared

further to other mechanisms, such as estimated probabilities of an event occurring, as

advocated by Berg and Proebsting (2012) and Hanson et al. (2013).

The researcher therefore suggests initiating a PM platform which can present differing

interfaces to the users and recruit participants that are then (randomly) allocated to the

different incarnations of the pilot market, all participants trading in the same

stocks/questions. The questions themselves should be pre-tested to ascertain if they are

comprehensible and can be answered in a reasonable way.

162

This is required because the focus of this proposed research is on the

workings/performance of PM user-interfaces and their influence on prediction accuracy

and should not be compromised by possible challenges stemming from the questions put

to the markets. As a possible extension, participants could be asked to also trade in the

platforms from a variety of PM platform providers and chronicle their impression of the

ease-of-use of these platforms.

Different types of questions could also be tested in such research to determine if there are

particular question domains which would markedly alter the performance of a PM; that

heterogeneity in trading behaviour could exist in such a context was established by Waitz

and Mild (2020) and Tindale, Winget, and Hinsz (2020). Testing would happen by

clustering participants to question types accordingly, such as attributing them to ‘yes/no’

stocks (e.g. anticipated success of a certain product, deadline of a project to be met), or

to gauging number ranges (classic in sales forecasts).

Publication of results from these two submissions could give guidance and assurance to

future users of CPMs on choosing successful implementation options for their corporate

markets.

On a different vein, as this thesis brought out management acceptance of a CPM’s results

to be one of the major obstacles hampering successful deployment, exploring hierarchical

structures in companies and their impact on technology acceptance could open an avenue

of research to further CPM understanding. This could even have an explicit focus on the

integration of a technology into corporate forecasting processes. Past research already

yielded meaningful insights regarding the importance of different acceptance levels from

higher-level managers compared to end users and managers in lower-level business

settings which was also reflected in the findings of this thesis. Expanding research in this

area would further insights in decision-making and management strategies more in

general.

With the particular focus of this thesis, such research would be worthwhile to be further

extended for CPMs, too. The researcher suggests following an approach advocated by

Lai (2017) and Gómez-Ramirez, Valencia-Arias and Duque (2019): Surveys or

questionnaires could be deployed to gauge if acceptance of a new tool is based more on

the characteristics of the technology itself, or if other influences, such as interaction

within one’s (social) network, exist.

163

Other determinants such as cognitive and situational aspects or even the implementation

context could also have an impact on effective utilisation; the magnitude of the effect of

all of these factors would be of interest. Questions to be asked in such an enquiry would

revolve around personal innovativeness, one’s image, subjective norms, perceived

behavioural control, perceived usefulness and perceived ease of use of the tool, as

recommended by Yi et al. (2006).

To understand causes of user intention and factors shaping technology acceptance, the

researcher proposes to obtain ratings from lower and higher-level employees, perhaps on

a Likert scale, to evidence the most significant effects on usage intention and to what

degree differences between hierarchy levels might exist. From these results measures and

interventions could be derived for implementation strategies better illustrating the

potential advantages associated with PM technology. A second (later) round of surveys

could determine if changes in behaviour towards and/or perception of CMPs would have

been achieved and hence if their advantages could successfully be demonstrated within a

modified implementation approach.

As findings from such research are not uncommon to be disseminated in more general

management publications rather than in more expressly PM-related journals this might

contribute to make CPMs more known in management circles.

8.2.2 Recommendations to Practitioners Following from the submissions of this thesis more research would be helpful to

understand how idea markets could be embedded into innovation management processes

contributing positively to the success of the process. The researcher also identified a

lacuna in the literature around properly gauging the success of idea markets, particularly

around successful implementation procedures.

Practitioners could therefore advance the field by setting up an idea market pilot and

documenting and subsequently publishing the results of such a test. The aim would be to

evaluate employees’ product choices/ideas with the help of an idea market focusing on a

limited and clearly delineated set of product ideas within the framework of a pilot project.

164

To achieve this, the researcher recommends instituting ideas as virtual shares to be traded

as an exercise to gauge a product’s anticipated success. For instance, a share could reflect

the expected amount of units sold of a future product or asking staff to pinpoint emerging

technologies for the company the next 10 years hence. Ideally one would compare this

exercise against the efficiency of a ‘competitor’ approach, perhaps an ‘idea filtering

mechanism’ that is already in existence in the company.

To make such a pilot successful, it is important to define transparent quality criteria and

(accepted) evaluation categories amongst others. In implementing the results and

practical recommendations from a project/pilot the researcher discerned that CPMs

require commitment from the company and clarity of purpose and need a designated

champion to pronounce relevance and readiness to accept the results; hence the researcher

recommends having these structures in place. Additionally, such a structure and approach

would help to gather the pilot results in a comprehensive way.

Following one part of the suggestions for further academic research, laid out towards the

end of section 8.2.1, what needs to be done to gain management acceptance regarding

forecasting results from CPMs should also be investigated by practitioners. The approach

could be to deploy different tools competitively for the same forecasting challenge of

relevance to a company – for instance market research, an expert survey, agent-based

models, plus a PM – and presenting the individual prognoses and information about which

tool was used to achieve which forecasting result to decision-makers.

An appraisal of such an investigation would then focus on how participants would value

different attributes defining the specific tools applied (e.g. features, function, benefits).

Employing conjoint analysis, an implicit valuation of the individual elements making up

a product or service can be successfully determined, identifying the most influential

factors regarding respondent choice or acceptance. Other similarly applicable tools might

be used to assess feature comparisons. The researcher did not undertake to come up with

a firm recommendation here as he felt that instruments a future researcher is most

comfortable or familiar with should be used. The outcome would show whether CPMs

can stand out from other forecasting candidates and where potential acceptance problems

could exist and help to set CPMs on a path for success.

165

8.2.3 Integrating Academic and Practice Research Recommendations

As pointed out in section 8.2.1 the results of further academic research could provide

guidance and assurance to future (‘practical’) users of CPMs. Vice versa, existing studies

recommended to provide a practitioner’s perspective on how academic insights can be

best applied and delivered in practice (Glover and Reidenbach, 2012; del Amo and

Sanesi, 2017).

Along those lines, academia could demonstrate to practitioners the benefits of CPMs and

how to overcome obstacles to adoption from a theoretical background and based on this

both parties could investigate how to efficiently implement CPMs. Such a co-production

of research could be of mutual benefit, resembling an approach of academics as advisors

providing critical questioning with the general research impetus for practical applications

of CPMs stemming from industry. Particularly for a forecasting product which is not that

well known yet eliciting diverse research approaches could lead to joint synergy between

academia and practitioners – one area offering rigour in methodological approaches and

perspectives, the other providing worthwhile research subjects to look at for companies

wanting to deploy CPMs.

Another point to be made is that a gap between industry and academia persists, being of

concern particularly as technology and organisation complexity grow. Thus, closer

collaboration between industry and academia would be desirable according to Melese et

al. (2009) and Allen et al. (2021) and CPMs might be a vehicle to facilitate this.

For instance, open innovation networks are recommended and, particularly for industry,

to seek collaborations with academia to stimulate innovation. Adopting a new mindset on

information exchange is needed as current set-ups hinder innovation and collaboration

(Melese et al., 2009) – a path for PMs to be further investigated both from academia and

practitioners.

The same stance on collaboration is highlighted by Allen et al. (2021) citing that “big

data analytics and machine learning [are] ranked as high-benefit but also high-complexity

tools to implement” where only agile and collaborative responses would safeguard the

best fusion of these technologies, and narrowing the gap which currently exists between

what is theoretically actionable from new or novel research and what is used in the daily

lives of the people handling data in industry (ibid.); these arguments strengthen the

researcher’s recommendations to incorporate both practical and theoretical aspects in

future research undertaken for PMs as laid out in the first two paragraphs of this section.

166

In a different vein, one of this thesis’ results was that PMs are not just useful for

(supporting) innovation – and research recommendations have been made to that effect –

corporate PMs can also be a useful tool in any planning/decision-making situation and

should not have been dismissed so readily. Therefore, academia could conduct research

to overcome cultural barriers, especially the perceived loss of power, and also carry out

examinations on the (technical) implementation side to allow easier application of PMs

and presentation of their results. Similarly, recommendations for practitioners would be

to reconsider corporate PMs for strategy formation and decision-making. This would help

to get organisational involvement at all levels and could lead to co-production (McNae,

2010) positively challenging existing views of relevant planning and forecasting

processes. Such involvement at all levels ought to extend to (PM) research, too, which

should be carried out jointly by academics and practitioners according to Shams and

Kaufmann (2016) as it would foster, including both strategic and operational perspectives

more comprehensive and meaningful results.

167

9. References

Abeliuk, A., Benjamin, D.M., Morstatter, F. and Galstyan, A. (2020) ‘Quantifying machine influence over human forecasters’, Scientific Reports, 10(1), pp.1-14. Abeliuk, A., Huang, Z., Ferrara, E. and Lerman, K. (2020) ‘Predictability limit of partially observed systems’. To be published in ACM SIGCAS Computers and Society [Preprint]. arXiv:2001.06547. Abolghasemi, Y. and Dimitrov, S. (2020) ‘Determining the causality between US presidential prediction markets and global financial markets’, International Journal of Finance & Economics, 25(4), DOI: 10.1002/ijfe.2029. Abramowicz, M. (2008) Predictocracy: market mechanisms for public and private decision making. New Haven: Yale University Press. Accard, P. (2015) ‘Complex hierarchy: the strategic advantages of a trade-off between hierarchical supervision and self-organizing’, European Management Journal, 33(2), pp.89-103. Acker, D. (2016) Prediction markets: reality and theory. School of Economics, Finance and Management, Accounting and Finance Discussion Paper 16/5. Bristol: Bristol University. Adams, P., Freitas, I.M.B. and Fontana, R. (2019) ‘Strategic orientation, innovation performance and the moderating influence of marketing management’, Journal of Business Research, 97, pp.129-140. Aeppel, T. (2017) ‘AI & machine learning disruption timeline conference’, AI & Machine, 2017(3), pp.1-4. van Aerssen, B. and Buchholz, C. (eds.) (2018) Das große Handbuch Innovation: 555 Methoden und Instrumente für mehr Kreativität und Innovation im Unternehmen. München: Verlag Franz Vahlen. Agarwal, N., Oehler, J. and Brem, A. (2021) ‘Constraint-based thinking: a structured approach for developing frugal innovations’. To be published in IEEE Transactions on Engineering Management [Preprint]. DOI: 10.1109/TEM.2020.3042929. Agrawal, A., Gans, J. and Goldfarb, A. (2018) Prediction machines: the simple economics of artificial intelligence. Boston, MA: Harvard Business Review Press. Agrawal, S. and Harter, J. (2010) The cascade effect of employee engagement: a longitudinal study. Omaha, NE: The Gallup Organization. Ahlbäck, K., Fahrbach, C., Murarka, M. and Salo, O. (2017) How to create an agile organization. McKinsey Global Survey Results. Available at: https://cdn.ksrinc.com/mckinseysurvey/article29.pdf (Accessed: 11 September 2017). Alexander, A., De Smet, A. and Weiss, L. (2020) Decision making in uncertain times. Available at: https://www.mckinsey.com/business-functions/organization/our-insights/decision-making-in-uncertain-times (Accessed: 04 April 2020). Alfaro, E., Yu, F., Rehman, N. U., Hysa, E. and Kabeya, K. (2019) ‘Strategic management of innovation’, in Chen, J., Brem, A., Viardot, E. and Wong, P.K. (eds.) The Routledge companion to innovation management. Milton Park: Routledge, pp.107-168. Allan, G. (2003) ‘A critique of using grounded theory as a research method’, Electronic Journal of Business Research Methods, 2(1), pp.1-10.

168

Allen, L., Atkinson, J., Jayasundara, D., Cordiner, J. and Moghadam, P.Z. (2021) ‘Data visualization for Industry 4.0: A stepping-stone toward a digital future, bridging the gap between academia and industry’, Patterns, 2(5), DOI: 10.1016/j.patter.2021.100266.

Allmendinger, P. (2017) Planning theory. 3rd edn. London: Palgrave. Almquist, E., Senior, J. and Springer, T. (2015) Three promises and perils of Big Data. Boston: Bain & Company. Altrock, U. (2008) ‘Strategieorientierte Planung in Zeiten des Attraktivitätsparadigmas’, in Hamedinger, A., Frey, O., Dangschat, J.S. and Breitfuss, A. (eds.) Strategieorientierte Planung im kooperativen Staat. Wiesbaden: VS Verlag für Sozialwissenschaften, pp.61-86. Alvarez, J., Canduela, J. and Raeside, R. (2012) ‘Knowledge creation and the use of secondary data’, Journal of Clinical Nursing, 21(19), pp.2699-2710. Alvesson, M. (2002) Postmodernism and social research. Buckingham: Open University Press. del Amo, R.A. and Sanesi, G. (2017) ‘Tying it all together’ in Pearlmutter D. Calfapietra, C., Samson, R., O'Brien, L., Krajter Ostoić, S., Sanesi, G. and del Amo, R.A (eds) The urban forest. Cham: Springer, pp.303-304. Ancona, D., Backman, E. and Isaacs, K. (2019) ‘Nimble leadership’, Harvard Business Review, 97(4), pp.74-83. Andler, N. (2016) Tools for project management, workshops and consulting: a must-have compendium of essential tools and techniques. 3rd edn. Erlangen: Publicis Publishing. Andrews, T. (2012) ‘What is social constructionism?’, Grounded Theory Review, 11(1), pp.39-46. Angelini, G., De Angelis, L. and Singleton, C. (2019) Informational efficiency and price reaction within in-play prediction markets. Economics & Management Discussion Papers em-dp2019-20. Henley-on-Thames: Henley Business School, Reading University. Angelini, G., De Angelis, L. and Singleton, C. (2020) Informational efficiency and price reaction within in-play prediction markets. Economics & Management Discussion Papers em-dp2020-20. Henley-on-Thames: Henley Business School, Reading University. Ansari, D., Holz, F. and Tosun, H.B. (2018) ‘Fossil fuels, global energy, and international policy: qualitative and quantitative scenario forecasting towards 2055’, Transforming Energy Markets, 41st IAEE International Conference. Groningen, Netherlands, 10-13 June. Cleveland, Ohio: International Association for Energy Economics, pp.77-79. Appinio GmbH (2020) Verstehen Sie die Meinung Ihrer Zielgruppe. https://www.appinio.com/de/pricing (Accessed: 28 September 2020). Archibugi, D. (2017) ‘Blade Runner economics: will innovation lead the economic recovery?’, Research Policy, 46(3), pp.535-543. Armstrong, J.S. (2001a) ‘Selecting forecasting methods’, in Armstrong, J.S. (ed.) Principles of forecasting: a handbook for researchers and practitioners. New York: Springer Science+Business Media, pp.365-386.

169

Armstrong, J.S. (2001b) ‘Combining forecasts’, in Armstrong, J.S. (ed.) Principles of forecasting: a handbook for researchers and practitioners. Dordrecht, Netherlands: Kluwer Academic Publishing, pp.417-439. Armstrong, J.S. (2006) ‘How to make better forecasts and decisions: avoid face-to-face meetings’, Foresight: The International Journal of Applied Forecasting, 5, pp.3-8. Armstrong, J.S., Du, R., Green, K.C. and Graefe, A. (2016) ‘Predictive validity of evidence-based persuasion principles: an application of the index method’, European Journal of Marketing, 50(1/2), pp.276-293. Arvan, M., Fahimnia, B., Reisi, M. and Siemsen, E. (2019) ‘Integrating human judgement into quantitative forecasting methods: a review’, Omega, 86, pp.237-252. Arya, A., Frimor, H. and Mittendorf, B. (2010) ‘Discretionary disclosure of proprietary information in a multisegment firm’, Management Science, 56(4), pp.645-658. Atanasov, P., Rescober, P., Stone, E., Swift, S.A., Servan-Schreiber, E., Tetlock, P., Ungar, L. and Mellers, B. (2017) ‘Distilling the wisdom of crowds: prediction markets vs. prediction polls’, Management Science, 63(3), pp.691-706. Atanasov, P., Witkowski, J., Ungar, L., Mellers, B. and Tetlock, P. (2020) ‘Small steps to accuracy: incremental belief updaters are better forecasters’, Organizational Behavior and Human Decision Processes, 160, pp.19-35. Atkinson-Grosjean, J. (2000) ‘Who knows what tomorrow brings?’, Science and Public Policy, 27(6), pp.463-464. Auld, T. and Linton, O. (2019) ‘The behaviour of betting and currency markets on the night of the EU referendum’, International Journal of Forecasting, 35(1), pp.371-389. Azeem, M.F. and Yasmin, R. (2016) ‘HR 2.0: linking Web 2.0 and HRM functions’, Journal of Organizational Change Management, 29(5), pp.686-712. Baars, G. and Spicer, A. (eds.). (2017) The corporation: a critical, multi-disciplinary handbook. Cambridge: Cambridge University Press. Babel, B., Buehler, K., Pivonka, A., Richardson, B. and Waldron, D. (2019) ‘Derisking machine learning and artificial intelligence’, McKinsey Insights. Available at: https://www.mckinsey.com/business-functions/risk/our-insights/derisking-machine-learning-and-artificial-intelligence (Accessed: 02 April 2019). Baczyk, N. (2019) Strategic analysis in a global, dynamic business environment. Master of Science. Johannes Kepler Universität Linz. Badrinarayanan, V., Ramachandran, I. and Madhavaram, S. (2019) ‘Resource orchestration and dynamic managerial capabilities: focusing on sales managers as effective resource orchestrators’, Journal of Personal Selling & Sales Management, 39(1), pp.23-41. Baer, T., Heiligtag, S. and Samandari, H. (2017) ‘The business logic in debiasing’, McKinsey Insights, May 2017. Available at: https://www.mckinsey.com/business-functions/risk/our-insights/the-business-logic-in-debiasing (Accessed: 21 March 2017). Bagherzadeh, M., Markovic, S. and Bogers, M. (2021) ‘Managing open innovation: a project-level perspective’, IEEE Transactions on Engineering Management, 68(1), pp.301-316. Bahcall, S. (2019) ‘The Innovation Equation’, Harvard Business Review, 97(2), pp.74-81.

170

Balboni, F., Finch, G., Rodenbeck Reese, C. and Shockley, R. (2013) Analytics: a blueprint for value – converting big data and analytics insights into results. Armonk, NY: IBM Institute for Business Value. Bambauer-Sachse, S. and Helbling, T. (2021) ‘Customer satisfaction with business services: is agile better?’. To be published in Journal of Business & Industrial Marketing [Preprint]. DOI: 10.1108/JBIM-04-2020-0221. Bandilla, W. (2016) Web surveys. GESIS survey guidelines. Mannheim: GESIS – Leibniz Institute for the Social Sciences. Bannister, F. and Connolly, R. (2020) ‘The future ain't what it used to be: forecasting the impact of ICT on the public sphere’, Government Information Quarterly, 37(1), pp.1-14. Bar Am, J. (2020) Reviving the center aisle: an interview with Kellogg’s chief growth officer. Available at: https://www.mckinsey.com/industries/consumer-packaged-goods/our-insights/reviving-the-center-aisle-an-interview-with-kelloggs-chief-growth-officer (Accessed: 15 July 2020). Barbu, A. and Lay, N. (2012) ‘An introduction to artificial prediction markets for classification’, Journal of Machine Learning Research, 13(71), pp.2177-2204. Barley, W.C., Treem, J.W. and Kuhn, T. (2018) ‘Valuing multiple trajectories of knowledge: a critical review and agenda for knowledge management research’, Academy of Management Annals, 12(1), pp.278-317. Barnett, M.L., Starbuck, W.H. and Pant, P.N. (2003) ‘Which dreams come true? Endogeneity, industry structure and forecasting accuracy’, Industrial and Corporate Change, 12(4), pp.653-672. Barth, K., Kiefel, J. and Wille, K. (2002) Gefilterte Märkte: ein Anreizsystem zur Förderung von firmeninternen Wissensmärkten. Wiesbaden: Deutscher Universitäts-Verlag. Batchelor, R. (2005) ‘A primer on forecasting with neural networks’, Foresight: The International Journal of Applied Forecasting, 2, pp.37-43. Batrinca, B. and Treleaven, P.C. (2015) ‘Social media analytics: a survey of techniques, tools and platforms’, AI & Society, 30(1), pp.89-116. Bauer, M.W., Bicquelet, A. and Suerdem, A.K. (2014) ‘Text analysis: an introductory manifesto’, in Bauer, M.W., Bicquelet, A. and Suerdem, A.K. (eds.) Textual analysis. London: SAGE Benchmarks in Social Research Methods, pp.xxi-xvii. Baumard, P. and Starbuck, W.H. (2005) ‘Learning from failures: why it may not happen’, Long Range Planning, 38(3), pp.281-298. Baur, N. and Knoblauch, H. (2018) ‘Die Interpretativität des Quantitativen’, Soziologie – Forum der Deutschen Gesellschaft für Soziologie, 47(4), pp.439-461. Beaudin, L., Brinda, M. and Ding, J. (2016) ‘The math, the magic and the customer’, Bain & Company Publication. Available at: http://www.bain.com/publications/articles/the-math-the-magic-and-the-customer.aspx (Accessed: 28 April 2016). Becker, J., Brackbill, D. and Centola, D. (2017) ‘Network dynamics of social influence in the wisdom of crowds’, Proceedings of the National Academy of Sciences, 114(26), pp.E5070-E5076.

171

Beckmann, F. (2010) Prognosemärkte als Instrument des Wissensmanagements im Unternehmen. Eine Analyse der Potenziale am Beispiel der Deutschen Telekom AG. Bachelor of Arts in Business Administration. FHDW Bielefeld: GRIN Verlag. Beer, M., Finnström, M. and Schrader, D. (2016) ‘Why leadership training fails – and what to do about it’, Harvard Business Review, 94(10), pp.50-57. Bell, E., Bryman, A. and Harley, B. (2018) Business research methods. 5th edn. Oxford: Oxford University Press. Bell, T.W. (2011) ‘Government prediction markets: why, who, and how’, Penn State Law Review, 116, pp.403-436. Bénabou, R. and Tirole, J. (2016) ‘Mindful economics: the production, consumption, and value of beliefs’, The Journal of Economic Perspectives, 30(3), pp.141-164. Benaich, N. (2019) RAAIS 2019 – State of AI by Nathan Benaich (Air Street Capital). Available at: https://www.youtube.com/watch?v=0hzrB49e-SE (Accessed: 5 October 2019). Beneš, N. (2017) ‘The power of prediction markets’, Gnosis Blog, 25 September. Available at: https://blog.gnosis.pm/the-power-of-prediction-markets-fedea0b71244 (Accessed: 18 September 2019). Berg, H., 2007. Prediction markets at Microsoft. Available at: http://people.ku.edu/~cigar/PMConf_2007/HenryBerg(PredictionPoint%20KC%20071101).pdf (Accessed: 02 April 2014). Berg, H. and Proebsting, T.A. (2012) Combined estimate contest and prediction market. U.S. Patent 8,229,824. Available at: https://patents.google.com/patent/US8229824B2/en (Accessed: 12 December 2020). Berg, J.E. and Rietz, T.A. (2019) ‘Longshots, overconfidence and efficiency on the Iowa Electronic Market’, International Journal of Forecasting, 35(1), pp.271-287. Berg, J.M. (2016) ‘Balancing on the creative highwire: forecasting the success of novel ideas in organizations’, Administrative Science Quarterly, 61(3), pp.433-468. Berger, J.G. and Achi, Z.G. (2020) ‘Understanding the leader’s ‘identity mindtrap’: personal growth for the C-suite’, McKinsey Quarterly, 56(1). Available at: https://www.mckinsey.com/business-functions/organization/our-insights/understanding-the-leaders-identity-mindtrap-personal-growth-for-the-c-suite (Accessed: 30 March 2020). Bergstrom, C.T. and West, J.D. (2020) Calling bullshit – the art of scepticism in a data-driven world. New York: Random House. Berinato, S. (2019) ‘Data science & the art of persuasion’, Harvard Business Review, 97(1), pp.126-137. Berker, T. (2010) ‘Dealing with uncertainty in sustainable innovation: mainstreaming and substitution’, International Journal of Innovation and Sustainable Development, 5(1), pp.65-79. Bernstein, E., Shore, J. and Lazer, D. (2018) ‘How intermittent breaks in interaction improve collective intelligence’, Proceedings of the National Academy of Sciences, 115(35), pp.8734-8739. Besanko, D., Dranove, D., Shanley, M. and Schaefer, S. (2017) Economics of strategy. 7th edn. Hoboken, NJ: John Wiley & Sons.

172

Billinger, S. and Rosenbaum, S.M. (2019) ‘Discretionary mechanisms and cooperation in hierarchies: an experimental study’, Journal of Economic Psychology, 74(2019), Article 102193.

Bingham, A. (2013) Telephone interview with Thomas Wolfram, 28 February. Bingham, A. and Nagar, Y. (2013) ‘Prediction markets – a practitioner’s guide’, Expert Series YourEncore. Available at: https://www.consensuspoint.com/wp-content/uploads/2017/01/Practitioners_Guide-1.pdf (Accessed: 03 May 2019). Birkinshaw, J., Hamel, G. and Mol, M.J. (2008) ‘Management innovation’, Academy of Management Review, 33(4), pp.825-845. Blackburn, S., LaBerge, L., O’Toole, C. and Schneider, J. (2020) Digital strategy in a time of crisis. Available at: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/digital-strategy-in-a-time-of-crisis (Accessed: 23 April 2020). Bland, A. and Osterwalder, A. (2020) Testing business ideas. Hoboken, NJ: Wiley & Sons, Inc. Blastland, M. and Spiegelhalter, D. (2013) The norm chronicles. London: Profile Books. Bloom, N., Dorgan, S., Dowdy, J., Van Reenen, J. and Rippin, T. (2005) Management practices across firms and nations. Centre for Economic Performance, The London School of Economics and McKinsey. Available at: http://eprints.lse.ac.uk/4669/1/Management_Practices_Across_Firms_and_Nations.pdf (Accessed: 03 September 2015). Bloom, N., Dorgan, S., Dowdy, J. and Van Reenen, J. (2007) Management practice and productivity: why they matter. Centre for Economic Performance, The London School of Economics and McKinsey. Available at: http://www.growingjobs.org/downloads/management_practice.pdf (Accessed: 03 September 2015). Bloom, N., Genakos, C., Sadun, R. and Van Reenen, J. (2012) ‘Management practices across firms and countries’, The Academy of Management Perspectives, 26(1), pp.12-33. Blumberg, S., Delaet, T. and Kartikeya Swami, K. (2020) Ten ‘antipatterns’ that are derailing technology transformations. Available at: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/ten-antipatterns-that-are-derailing-technology-transformations (Accessed: 07 August 2020). Boaz, N. and Fox, E.A. (2014) ‘Change leader, change thyself’, McKinsey Quarterly. Available at: http://www.mckinsey.com/insights/leading_in_the_21st_century/change_leader_change_thyself (Accessed: 02 April 2014). Bobadilla-Suarez, S., Sunstein, C. and Sharot, T. (2017) ‘The intrinsic value of choice: the propensity to under-delegate in the face of potential gains and losses’, Journal of Risk and Uncertainty, 54(1), pp.187-202. Bocken, N., Strupeit, L., Whalen, K. and Nußholz, J. (2019) ‘A review and evaluation of circular business model innovation tools’, Sustainability, 11(8), p.2210. Boehm, J. and Smith, J. (2021) Derisking digital and analytics transformations. Available at: https://www.mckinsey.com/business-functions/risk/our-insights/derisking-digital-and-analytics-transformations (Accessed: 14 January 2021).

173

Bogner, A., Littig, B. and Menz, W. (2014) Interviews mit Experten: eine praxisorientierte Einführung. Wiesbaden: Springer Fachmedien. Bolger, F., Nyberg, E.P., Belton, I., Crawford, M.M., Hamlin, I., Alvandi, A.O., Pearson, R., Sissons, A., Lūka, C.T.B., Thakur, S. and Vasilichi, A. (2020) ‘Improving the production and evaluation of structural models using a Delphi process’. To be published in EFSA Supporting Publications [Preprint]. DOI: 10.31219/osf.io/4ufzg. Bolger, F., Rowe, G., Belton, I., Crawford, M.M., Hamlin, I., Sissons, A., Lūka, C.T.B., Vasilichi, A. and Wright, G. (2020) ‘The simulated group response paradigm: a new approach to the study of opinion change in Delphi and other structured-group techniques’. To be published in EFSA Supporting Publications [Preprint]. DOI: 10.31219/osf.io/4ufzg. Bolger, F. and Wright, G. (2017) ‘Use of expert knowledge to anticipate the future: issues, analysis and directions’, International Journal of Forecasting, 33(1), pp.230-243. Bonabeau, E. (2009) ‘Decisions 2.0: the power of collective intelligence’, MIT Sloan management review, 50(2), pp.45-52. Boone, T., Boylan, J.E., Fildes, R., Ganeshan, R. and Sanders, N. (2019) ‘Perspectives on supply chain forecasting: special section: supply chain forecasting’, International Journal of Forecasting, 35(1), pp.121-127. Borghesi, R. (2013) ‘Can prediction markets mitigate price biases?’, Journal of Prediction Markets, 7(1), pp.1-11. Borison, A. and Hamm, G. (2010) ‘Prediction markets: a new tool for strategic decision making’, California Management Review, 52(4), pp.125-141. Bothos, E., Apostolou, D. and Mentzas, G. (2008) ‘IDEM: a prediction market for idea management’, in Weinhardt, C., Luckner, S. and Stößer, J. (eds.) Designing e-business systems. Markets, services, and networks. Heidelberg: Springer, pp.1-13. Bottazzi, G. and Giachini, D. (2019) ‘Far from the madding crowd: collective wisdom in prediction markets’, Quantitative Finance, 19(9), pp.1461-1471. Bottou, L. and Bousquet, O. (2008) ‘The tradeoffs of large scale learning’, Advances in Neural Information Processing Systems 22 (NIPS 2008). Vancouver and Whistler, Canada, 8-11 December. San Diego, CA: Neural Information Processing Systems, pp.161-168. Boulu-Reshef, B., Comeig, I., Donze, R. and Weiss, G.D. (2016) ‘Risk aversion in prediction markets: a framed-field experiment’, Journal of Business Research, 69(11), pp.5071-5075.

Bourgeois, D.T. (2018) Information systems for business and beyond. Toronto: Dspace. Bousoño-Calzón, C., Molina-Bulla, H., Escudero-Garzás, J.J. and Herrera-Gálvez, F.J. (2018) ‘Expert selection in prediction markets with homological invariants’, IEEE Access, 6, pp.32226-32239. ‘Box plot’ (2020) Wikipedia. Available at: https://en.wikipedia.org/wiki/Box_plot (Accessed: 12 October 2020). Boyonas, M.I., Olavarria, L. and Saenz, M.J. (2020) ‘Scenario planning: preparing the future of global supply chains’, in Information Resources Management Association (ed.) Supply chain and logistics management: concepts, methodologies, tools, and applications. Hershey, PA: IGI Global, pp.2019-2060.

174

Bradley, C., Hirt, M. and Smit, S. (2018) Strategy beyond the hockey stick. Hoboken, NJ: Wiley & Sons, Inc. Braun, V. and Clarke, V. (2012) ‘Thematic analysis’, in Cooper, H. (ed.) APA handbook of research methods in psychology. Washington, D.C.: American Psychological Association, pp.57-71 Braun, V. and Clarke, V. (2019) ‘To saturate or not to saturate? Questioning data saturation as a useful concept for thematic analysis and sample-size rationales’, Qualitative Research in Sport, Exercise and Health, 12(2), pp.175-191. Brem, A. (2017) ‘Creativity and routine: conceptual considerations on managing organisational ambidexterity in entrepreneurial ventures’, International Journal of Entrepreneurship and Innovation Management, 21(3), pp.261-273. Bresnahan, T.F., Brynjolfsson, E. and Hitt, L.M. (2002) ‘Information technology, workplace organization, and the demand for skilled labor: firm-level evidence’, The Quarterly Journal of Economics, 117(1), pp.339-376. Brigis, A. (2008) ‘What’s holding back enterprise prediction markets?’, Future Blog, 17 April. Available at: http://futureblogger.net/futureblogger/show/884-what-s-holding-back-enterprise-prediction-markets- (Accessed: 13 August 2014). Brown, A., Rambaccussing, D., Reade, J.J. and Rossi, G. (2018) ‘Forecasting with social media: evidence from tweets on soccer matches’, Economic Inquiry, 56(3), pp.1748-1763. Brown, A. and Reade, J.J. (2019) ‘The wisdom of amateur crowds: evidence from an online community of sports tipsters’, European Journal of Operational Research, 272(3), pp.1073-1081. Brown, A., Reade, J.J. and Vaughan Williams, L. (2019) ‘When are prediction market prices most informative?’, International Journal of Forecasting, 35(1), pp.420-428. Brown, A. and Yang, F. (2019) ‘The wisdom of large and small crowds: evidence from repeated natural experiments in sports betting’, International Journal of Forecasting, 35(1), pp.288-296. Bruns, M. (2010) förderland Business Magazin für Entscheider. Available at: https://www.foerderland.de/gruendung/news-gruenderszene/die-aufgaben-sind-bei-uns-klar-verteilt-alexander-baut-das-auto-ich-verkaufe-es/ (Accessed: 20 September 2019). Brynjolfsson, E., Hitt, L.M. and Yang, S. (2002) ‘Intangible assets: computers and organizational capital’, Brookings Papers on Economic Activity, 2002(1), pp.137-198. Brynjolfsson, E. and McAfee, A. (2014) The second machine age. New York: W.W. Norton & Company. Brynjolfsson, E. and McAfee, A. (2018) ‘Von Managern und Maschinen’, Harvard Business Manager, Edition 3/2018, pp.32-39. Bucholtz, M. (2000) ‘The politics of transcription’, Journal of Pragmatics, 32(10), pp.1439-1465. Buckingham, M. and Goodall, A. (2019) Nine lies about work: a freethinking leader’s guide to the real world. Boston: Harvard Business Press. Buckley, P. (2016) ‘Harnessing the wisdom of crowds: decision spaces for prediction markets’, Business Horizons, 59(1), pp.85-94.

175

Buckley, P. and Doyle, E. (2017a) ‘Evidencing the forecasting performance of predication markets: an empirical comparative study’, Journal of Prediction Markets, 11(2), pp.60-76. Buckley, P. and Doyle, E. (2017b) ‘Individualising gamification: an investigation of the impact of learning styles and personality traits on the efficacy of gamification using a prediction market’, Computers & Education, 106, pp.43-55. Buckley, P. and McDonagh, E. (2014) ‘Idea markets: a literature review and classification scheme’, The Journal of Prediction Markets, 8(2), pp.76-88. Buckley, P. and O’Brien, F. (2017) ‘The effect of malicious manipulations on prediction market accuracy’, Information Systems Frontiers, 19(3), pp.611-623. Budescu, D.V. and Chen, E. (2015) ‘Identifying expertise to extract the wisdom of crowds’, Management Science, 61(2), pp.267-280. Bughin, J. (2013) ‘Enterprise 2.0 impact on company performance in developing countries’, in Azab, N.A. (ed.) Cases on Web 2.0 in developing countries: studies on implementation, application, and use. Hershey, PA: IGI Global, pp.172-197. Bughin, J. (2015) ‘Taking the measure of the networked enterprise’, McKinsey Quarterly, 51(3), pp.19-22. Bughin, J. (2016a) ‘Big data: getting a better read on performance’, McKinsey Quarterly, 52(1), pp.8-11

Bughin, J. (2016b) ‘Big data, big bang?’, Journal of Big Data, 3(1), pp.2-17. Bughin J., Byers, A.H. and Chui, M. (2011) ‘How social technologies are extending the organization’, McKinsey Quarterly, 47(2), pp.1-10. Bughin, J., Catlin, T., Hirt, M. and Willmott, P. (2018) ‘Why digital strategies fail’, McKinsey Quarterly, 54(1), pp.61-75. Bughin, J. and Chui, M. (2010) ‘The rise of the networked enterprise: Web 2.0 finds its payday’, McKinsey Quarterly, 46(4), pp.2-8. Bughin, J., Chui, M. and Harrysson, M. (2015) McKinsey global survey on social tools and technologies in the business, 2014. [Data file]. McKinsey & Company. Unpublished dataset, cited with permission. Bughin, J., Chui, M. and Harrysson, M. (2016a) ‘How social tools can reshape the organization’, McKinsey Global Survey Results. Available at: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/how-social-tools-can-reshape-the-organization (Accessed: 09 June 2016). Bughin, J., Chui, M. and Harrysson, M. (2016b) McKinsey global survey on social tools and technologies in the business, 2015. [Data file]. McKinsey & Company. Unpublished dataset, cited with permission. Bughin, J., Chui, M., Harrysson, M. and Lijek, S. (2017) McKinsey global survey on social tools and technologies in the business, 2016. [Data file]. McKinsey & Company. Unpublished dataset, cited with permission. Bughin, J., Chui, M. and Manyika, J. (2012) ‘Capturing business value with social technologies’, McKinsey Quarterly, 48(1), pp.72-80. Bughin, J., Chui, M. and Miller, A. (2009) How companies are benefiting from Web 2.0 – McKinsey global survey results. New York: McKinsey & Company.

176

Bughin, J., Hazan, E., Ramaswamy, S., Chui, M., Allas, T., Dahlström, P., Henke, N. and Trench, M. (2017) Artificial intelligence – the next digital frontier? Discussion paper June 2017. New York: McKinsey Global Institute. Bughin, J., Manyika, J. and Miller, A. (2008) ‘McKinsey global survey results: building the Web 2.0 enterprise’, McKinsey on Business Technology Fall 2008, pp.38-47. Bughin, J. and van Zeebroeck, N. (2018) ‘Artificial intelligence: why a digital base is critical’, McKinsey Quarterly, 54(3), pp.13-16. Burgman, M.A., McBride, M., Ashton, R., Speirs-Bridge, A., Flander, L., Wintle, B., Fidler, F., Rumpff, L. and Twardy, C. (2011) ‘Expert status and performance’, PLoS One, 6(7), doi.org/10.1371/journal.pone.0022998. Bytzek, E. and Bieber, I.E. (2016) ‘Does survey mode matter for studying electoral behaviour? Evidence from the 2009 German Longitudinal Election Study’, Electoral Studies, 43, pp.41-51. Caimi, G. (2019) ‘Navigating the route to innovation’, Bain Brief. Available at: https://www.bain.com/insights/navigating-the-route-to-innovation/ (Accessed: 12 August 2020). Cain, M. and Drakos, N. (2008) Hype cycle for social software 2008. Stamford, CT: Gartner Research Inc. Calhoun, M. and Starbuck, W.H. (2003) ‘Barriers to creating knowledge’, in Easterby-Smith, M. and Lyles, M.A. (eds.) The Blackwell handbook of organizational learning and knowledge management. Chichester: Blackwell Publishing, pp.471-490. Calipha, R. and Venezia, I. (2021) ‘Prediction markets: do they predict the polls or the election results? The case of the Israeli elections in April 2019’, in Venezia, I. (ed.) Behavioral finance – a novel approach. Singapore: World Scientific Publishing, pp.3-17. Camerer, C.F., Dreber, A., Forsell, E., Ho, T.H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Gideon Nave, G., Pfeiffer, T., Razen, M. and Wu. H. (2016) ‘Evaluating replicability of laboratory experiments in economics’, Science, 351(6280), pp.1433-1436. Canduela, J. (2009) Forecasting in fast moving consumer goods organisations – a case study. Beau-Bassin, Mauritius: Lambert Academic Publishing. Canonico, L.B., Flathmann, C. and McNeese, N. (2019) ‘The wisdom of the market: using human factors to design prediction markets for collective intelligence’, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 63(1), pp.1471-1475. Cappelli, P. (2019) ‘Your approach to hiring is all wrong – outsourcing and algorithms won’t get you the people you need’, Harvard Business Review, 97(3), pp.47-57. Carnabuci, G. and Diószegi, B. (2015) ‘Social networks, cognitive style, and innovative performance: a contingency perspective’, Academy of Management Journal, 58(3), pp.881–905. Carnabuci, G., Emery, C. and Brinberg, D. (2018) ‘Emergent leadership structures in informal groups: a dynamic, cognitively informed network model’, Organization Science, 29(1), pp.118-133.

177

Chan Kim, W. and Mauborgne, R. (2015) Blue ocean strategy: how to create uncontested market space and make the competition irrelevant. Rev. edn. Boston: Harvard Business School Publishing. Chan Kim, W. and Mauborgne, R. (2019) ‘Nondisruptive creation: rethinking innovation and growth’, MIT Sloan Management Review, 60(3), pp.46-55. Chatfield, A. T., Shlemoon, V.N., Redublado, W. and Rahman, F. (2014) ’Data scientists as game changers in big data environments’, Proceedings of the 25th Australasian Conference in Information Systems. Auckland, New Zealand , 8-10 December. Available at: https://ro.uow.edu.au/cgi/viewcontent.cgi?article=6674&context=eispapers (Accessed: 20 December 2019). Chen, Y. and Wortman Vaughan, J. (2010) ‘A new understanding of prediction markets via no-regret learning’, Proceedings of the 11th ACM conference on Electronic commerce. Harvard University, Cambridge, MA, 7-11 June. New York: Association for Computing Machinery, pp.189-198. Chlupsa, C., 2013. The impact of implicit motives on the business to business decision making process. Doctor of Philosophy. Plymouth: Plymouth Business School. Choo, L. and Kaplan, T.R. (2019) ‘Information aggregation in Arrow-Debreu markets: an experiment’, Experimental Economics, 22(3), pp.625-652. Chowdhury, M.S., Ahmmed, F. and Hossain, M. (2020) ‘Methodological dilemma in microfinance research: applicability of a qualitative case study design’, Qualitative Report, 25(2), pp.271-290. Christensen, C.M. (2011) The innovator’s dilemma: when new technologies cause great firms to fail. New York: Harper Business. Chui, M., Dewhurst, M. and Pollak, L. (2013) ‘Building the social enterprise’, McKinsey Quarterly, 49(4), pp.8-11. Chui, M. and Malhotra, S. (2018) ‘Notes from the AI frontier: AI adoption advances, but foundational barriers remain’, McKinsey Survey results. Available at: https://www.mckinsey.com/featured-insights/artificial-intelligence/ai-adoption-advances-but-foundational-barriers-remain (Accessed: 21 November 2019). Chui, M., Manyika, J., Bughin, J., Dobbs, R., Roxburgh, C., Sarrazin, H., Sands, G. and Westergren, M. (2012) ‘The social economy: unlocking value and productivity through social technologies’, McKinsey Global Institute, pp.1-170. Ciasullo, M.V., Cosimato, S., Gaeta, M. and Palumbo, R. (2017) ‘Comparing two approaches to team building: a performance measurement evaluation’, Team Performance Management: An International Journal, 23(7/8), pp.333-351. Clegg, S.R. and Starbuck, W.H. (2009) ‘Can we still fix M@n@gement? The narrow path towards a brighter future in organizing practices’, M@n@gement, 12(5), pp.332-358. Clement, D. (2020) ‘Superstars in the age of information’, For All – Magazine of the Opportunity & Inclusive Growth Institute, Spring 2020, pp.6-14. Clement, J. (2019) ‘Number of global social network users 2010-2021’, Statistica, 4(7), p.2020. Clement, R. and Schreiber, D. (2013) ‘Kollektive Intelligenz und Crowdsourcing’, in Clement, R. and Schreiber, D. (eds.) Internet-Ökonomie: Grundlagen und Fallbeispiele der vernetzten Wirtschaft. Berlin: Springer, pp.463-490.

178

Cole, T. (2015) Digitale Transformation. München: Verlag Franz Vahlen. Coles, P.A., Lakhani, K.R. and McAfee, A (2007) ‘Prediction markets at Google’, Harvard Business School Case, 9-607-088. Collins, P.H. (2019) Intersectionality as critical social theory. Durham: Duke University Press. Corgnet, B., Deck, C., DeSantis, M. and Porter, D. (2018) ‘Information (non) aggregation in markets with costly signal acquisition’, Journal of Economic Behavior & Organization, 154, pp.286-320. Corgnet, B., DeSantis, M. and Porter, D. (2020) ‘The distribution of information and the price efficiency of markets’, Journal of Economic Dynamics and Control, 110, DOI: 10.1016/j.jedc.2019.02.006. Corley, K.G. (2004) ‘Defined by our strategy or our culture? Hierarchical differences in perceptions of organizational identity and change’, Human Relations, 57(9), pp.1145-1177. Corti, L., Van den Eynden, V., Bishop, L. and Woollard, M. (eds.) (2019) Managing and sharing research data: a guide to good practice. 2nd edn. London: SAGE Publications Ltd. Costa Sperb, L.F., Sung, M.C., Ma, T. and Johnson, J. (2019) ‘Keeping a weather eye on prediction markets: improving forecasts by accounting for environmental conditions’, International Journal of Forecasting, 35(1), pp.321-335. Courtright, C. (2020) Faster, simpler, smarter: the future of insights. Plano, TX: dynata. Cowgill, B. and Zitzewitz, E. (2015) ‘Corporate prediction markets: evidence from Google, Ford, and Firm X’, Review of Economic Studies, 82(4), pp.1309-1341. Crawford, F.W., Aronow, P.M., Zeng, L. and Li, J. (2018) ‘Identification of homophily and preferential recruitment in respondent-driven sampling’, American Journal of Epidemiology, 187(1), pp.153-160. Creswell, J.W. and Creswell, J.D. (2018) Research design: qualitative, quantitative, and mixed methods approaches. 5th edn. London: Sage Publications. Creswell, J.W. and Poth, C.N. (2016) Qualitative inquiry and research design: Choosing among five approaches. London: Sage Publications. Crunchbase (2015) The on-demand economy. Available at: https://www.crunchbase.com/organization/the-on-demand-economy (Accessed: 04 June 2015). Cruz, E.V. and Higginbottom, G. (2013) ‘The use of focused ethnography in nursing research’, Nurse Researcher, 20(4), pp.36-43. Cuhls, K. (2003) ‘From forecasting to foresight processes – new participative foresight activities in Germany’, Journal of Forecasting, 22(2-3), pp.93-111. Cummings, R., Pennock, D.M. and Wortman Vaughan, J. (2016) ‘The possibilities and limitations of private prediction markets’, Proceedings of the 2016 ACM Conference on Economics and Computation. Maastricht, The Netherlands, 24-28 July. New York: Association for Computing Machinery, pp.143-160. Czwajda, L., Kosacka-Olejnik, M., Kudelska, I., Kostrzewski, M., Sethanan, K. and Pitakaso, R. (2019) ‘Application of prediction markets phenomenon as decision support instrument in vehicle recycling sector’, LogForum, 15(2), pp.265-278.

179

Daepp, M.I., Hamilton, M.J., West, G.B. and Bettencourt, L.M. (2015) ‘The mortality of companies’, Journal of The Royal Society Interface, 12(106), DOI: 10.1098/rsif.2015.0120. Daheim, C. and Uerz, G. (2008) ‘Corporate foresight in Europe: from trend based logics to open foresight’, Technology Analysis & Strategic Management, 20(3), pp.321-336. Daher, M., Carré, D., Jaramillo, A., Olivares, H. and Tomicic, A. (2017) ‘Experience and meaning in qualitative research: a conceptual review and a methodological device proposal’, Forum Qualitative Sozialforschung, 18(3), pp.1-25. Dai, M., Jia, Y. and Kou, S. (2020) ‘The wisdom of the crowd and prediction markets’. To be published in Journal of Econometrics [Preprint]. DOI: 10.1016/j.jeconom.2020.07.016. Dana, J., Atanasov, P., Tetlock, P. and Mellers, B. (2019) ‘Are markets more accurate than polls? The surprising informational value of “just asking”’, Judgment and Decision Making, 14(2), pp.135-147. Daniel, J. (2012) Sampling essentials: practical guidelines for making sampling choices. Los Angeles, CA: SAGE. Danielsson, K. (2016) Three is a magic number: an exploratory investigation of the mechanisms behind the practical triple helix. MSc in Management of Innovation and Business Development. Copenhagen Business School. D’Auria, G. and De Smet, A. (2020) Leadership in a crisis: responding to the coronavirus outbreak and future challenges. Available at: https://www.mckinsey.com/business-functions/organization/our-insights/leadership-in-a-crisis-responding-to-the-coronavirus-outbreak-and-future-challenges (Accessed: 29 December 2020). Davis, M.S. (1971) ‘That’s interesting! Towards a phenomenology of sociology and a sociology of phenomenology’, Philosophy of the Social Sciences, 1(2), pp.309-344. Davis-Stober, C.P., Budescu, D.V., Dana, J. and Broomell, S.B. (2014) ‘When is a crowd wise?’, Decision, 1(2), pp.79-82. Day, D., Fleenor, J., Atwater, L., Sturm, R. and McKee, R. (2014) ‘Advances in leader and leadership development: a review of 25 years of research and theory’, The Leadership Quarterly, 25(1), pp.63-82. Deal, J. and Levenson, A. (2016) What millennials want from work: how to maximize engagement in today’s workforce. New York: McGraw-Hill Education. De Baets, S. and Harvey, N. (2020) ‘Using judgment to select and adjust forecasts from statistical models’. To be published in European Journal of Operational Research [Preprint]. Doi: 10.1016/j.ejor.2020.01.028. de Casterlé, B.D., Gastmans, C., Bryon, E. and Denier, Y. (2012) ‘QUAGOL: A guide for qualitative data analysis’, International Journal of Nursing Studies, 49(3), pp.360-371. Deck, C. and Porter, D. (2013) ‘Prediction markets in the laboratory’, Journal of Economic Surveys, 27(3), pp.589-603. De Dreu, C.K. (2007) ‘Minority dissent, attitude change, and group performance’, in Pratkanis, A.R. (ed.) The science of social influence: advances and future progress. Frontiers of social psychology. New York: Psychology Press, pp.247-270.

180

‘Delphi method’, 2019. Wikipedia. Available at: https://en.wikipedia.org/wiki/Delphi_method (Accessed: 12 September 2019). Denzin, N.K. (2012) ‘Triangulation 2.0’, Journal of Mixed Methods Research, 6(2), pp.80-88. Desai, V., Liu, C., Madsen, P.M., Maslach, D., Carroll, J.S., Farjoun, M., Roberts, K.H., Starbuck, W.H. and Tamuz, M. (2015) ‘Oversight or oversight?: lessons learnt about managing and governing organizations facing disaster’, Academy of Management Annual Meeting Proceedings. Available at: https://doi.org/10.5465/ambpp.2015.16617symposium (Accessed: 19 September 2020). Desmet, D., Duncan, E., Scanlan, J. and Singer, M. (2015) ‘Six building blocks for creating a high-performing digital enterprise’, McKinsey Digital September 2015. Available at: http://www.mckinsey.com/insights/organization/six_building_blocks_for_creating_a_high_performing_digital_enterprise (Accessed: 31 December 2015). Desouza, K.C. and Jacob, B. (2017) ‘Big data in the public sector: lessons for practitioners and scholars’, Administration & Society, 49(7), pp.1043-1064. Dewey, R. and Brown, A. (2021) ‘Toil and trouble, don't get burned shorting bubbles’, SSRN Electronic Journal, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3782759. Dewhurst, M. and Willmott, P. (2014) ‘Manager and machine: the new leadership equation’, McKinsey Quarterly, 50(3), pp.76-83. Dianat, A. and Siemroth, C. (2019) ‘How can corporate prediction markets improve manager decisions? An experimental study’, SSRN Electronic Journal, https://doi-org.eres.qnl.qa/10.2139/ssrn.3345569. Dianat, A. and Siemroth, C. (2020) ‘Improving decisions with market information: an experiment on corporate prediction markets’. To be published in Experimental Economics [Preprint]. DOI: 10.2139/ssrn.3345569. Diekmann, A. (2014). Empirische Sozialforschung - Grundlagen, Methoden, Anwendungen. 9th edn. Hamburg: Rowohlt Verlag GmbH. Dietl, H., Rese, M., Krebs, A. and Franke, B. (2004) Virtuelle Informationsbörsen zur Prognose und Investitionsabsicherung. Köln: Josef Eul Verlag. Dietrich, F. and Spiekermann, K. (2021) ‘Social epistemology’ in Knauff, M. and Spohn, W. (eds.) The handbook of rationality. Cambridge, MA: MIT press, pp.27-46. Dimma, W.A. (1985) ‘Competitive strategic planning’, Business Quarterly, 50(1), pp.22-26. Dobbs, R., Manyika, J. and Woetzel, J. (2015) No ordinary disruption: the four global forces breaking all the trends. New York: PublicAffairs. Doorley, S., Holcomb, S., Klebahn, P., Segovia, K. and Utley, J. (2018) Design thinking bootleg. Available at: https://dschool.stanford.edu/resources/design-thinking-bootleg (Accessed: 16 August 2020). Dowling, M., Noll, E. and Zisler, K. (2019) ‘Innovation in the digital age’, in Chen, J., Brem, A., Viardot, E. and Wong, P.K. (eds.) The Routledge Companion to Innovation Management. Milton Park: Routledge, pp.278-296.

181

Dräger, J. and Müller-Eiselt, R. (2019) Wir und die intelligenten Maschinen: Wie Algorithmen unser Leben bestimmen und wir sie für uns nutzen können. München: Deutsche Verlags-Anstalt. Dräther, R., Koschek, H. and Sahling, C. (2019) Scrum kurz & gut. 2nd edn. Heidelberg: dpunkt.verlag GmbH. Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B.A. and Johannesson, M. (2015) ‘Using prediction markets to estimate the reproducibility of scientific research’, Proceedings of the National Academy of Sciences, 112(50), pp.15343-15347. Drezner, J.A. and McKernan, M. (2020) ‘Considerations for managing program data in the emerging acquisition environment: summary’, RAND Corporation Research Report. Available at: https://www.rand.org/pubs/research_reports/RR3130z1.html (Accessed: 04 June 2020). Drezner, J.A., McKernan, M., Sollinger, J.M. and Newberry, S. (2020) ‘Issues with access to acquisition information in the Department of Defence: a series on considerations for managing program data in the emerging acquisition environment’, RAND Corporation Research Report. Available at: https://www.rand.org/pubs/research_reports/RR3130.html (Accessed: 04 June 2020). Drogendijk, R. and Zander, L. (2010) ‘Walking the cultural distance: in search of direction beyond friction’, in Devinney, T., Pedersen, T. and Tihanyi, L. (eds.) The past, present and future of international business and management. Bingley: Emerald Publishing Limited, pp.189-212. Drucker, P. (2013) ‘The best way to predict the future is to create it’, Philosiblog, 17 May. Available at: https://philosiblog.com/2013/05/17/the-best-way-to-predict-the-future-is-to-create-it/ (Accessed: 02 August 2019). Drucker, P. (2017) The effective executive: the definitive guide to getting the right things done. 50th anniversary edn. New York: HarperCollins. Duchek, S., Raetze, S. and Scheuch, I. (2020) ‘The role of diversity in organizational resilience: a theoretical framework’, Business Research, 13(2), pp.387-423. Dunbar, R.L. and Starbuck, W.H. (2006) ‘Learning to design organizations and learning from designing them’, Organization Science, 17(2), pp.171-178. Dunleavy, P. (2014) ‘In a PhD or academic book, do you really need a methodology chapter early on?’, Medium Blog, 06 Feb. Available at: https://medium.com/advice-and-help-in-authoring-a-phd-or-non-fiction/in-a-phd-or-academic-book-do-you-really-need-a-methodology-chapter-early-on-e596ac518f75 (Accessed: 20 November 2020). Duquette, C., Mixon Jr, F.G., Cebula, R.J. and Upadhyaya, K.P. (2014) ‘Prediction markets and election polling: granger causality tests using InTrade and RealClearPolitics data’, Atlantic Economic Journal, 42(4), pp.357-366. Dye, R. (2013) Telephone interview with Thomas Wolfram, 19 March. Dye, R., Sibony, O. and Truong, V. (2009) ‘Flaws in strategic decision making’, McKinsey Quarterly, 45(1), pp.25-31. Dye, R., Sibony, O. and Viguerie, S.P. (2009) ‘Strategic planning: three tips for 2009’, McKinsey Quarterly, 45(1), pp.32-35. Eagar, R., Teixeira, T., Taga, K. and Caldani, S. (2020) Leading businesses through the COVID-19 crisis. Available at: https://www.adlittle.de/en/COVID19_CEOFirstLearnings (Accessed: 23 April 2020).

182

Earle, W.J. (2018) ‘Recent work on rationality’, The Philosophical Forum, 49(1), pp.105-129. Easley, D. and Kleinberg, J. (2019) ‘Google’s use of an internal prediction market’, Course blog for INFO 2040/CS 2850/Econ 2040/SOC 2090. Available at: https://blogs.cornell.edu/info2040/2019/11/24/googles-use-of-an-internal-prediction-market/ (Accessed: 18 August 2020). Easterby-Smith, M., Thorpe, R. and Jackson, P. (2008) Management research. 3rd edn. London: SAGE Publications Ltd. Economo, E., Hong, L. and Page, S.E. (2016) ‘Social structure, endogenous diversity, and collective accuracy’, Journal of Economic Behavior & Organization, 125, pp.212-231. Edmondson, A.C. (2018) The fearless organization: creating psychological safety in the workplace for learning, innovation, and growth. Hoboken, NJ: John Wiley & Sons. Eisenhardt, K.M. and Sull, D.N. (2001) ‘Strategy as simple rules’, Harvard Business Review, 79(1), pp.106-119. Eksoz, C., Mansouri, S.A., Bourlakis, M. and Önkal, D. (2019) ‘Judgmental adjustments through supply integration for strategic partnerships in food chains’, Omega, 87, pp.20-33. Eller, E., Hofman, R. and Schwarz, J.O. (2020) ‘The customer foresight territory’, Marketing Review St Gallen, 3, pp.888-895. Elo, S., Kääriäinen, M., Kanste, O., Pölkki, T., Utriainen, K. and Kyngäs, H. (2014) ‘Qualitative content analysis: a focus on trustworthiness’, SAGE Open, 4(1), pp.1-10. Elwyn, G., Stiel, M., Durand, M.A. and Boivin, J. (2011) ‘The design of patient decision support interventions: addressing the theory-practice gap’, Journal of Evaluation in Clinical Practice, 17(4), pp.565-574. Engels, F. (2020) Testing technologies – re-configuring societies. Doktor der Philosophie. Technische Universität München. Erickson, T. (2008) Plugged in: the Generation Y guide to thriving at work. Boston: Harvard Business Press. Erikson, R.S. and Wlezien, C. (2008) ‘Are political markets really superior to polls as election predictors?’, Public Opinion Quarterly, 72(2), pp.190-215. Erikson, R.S. and Wlezien, C. (2012) ‘Markets vs. polls as election predictors: an historical assessment’, Electoral Studies, 31(3), pp.532-539. Ertel, W. (2016) Grundkurs Künstliche Intelligenz: Eine praxisorientierte Einführung. 4th ed. Wiesbaden: Springer Vieweg. Eschenbach, R., Eschenbach, E. and Kunesch, H. (2008) Strategische Konzepte. Stuttgart: Schäffer-Poeschel Verlag. Etikan, I., Alkassim, R. and Abubakar, S. (2016) ‘Comparison of snowball sampling and sequential sampling technique’, Biometrics and Biostatistics International Journal, 3(1), pp.55-56. Everett, C. (2019) ‘Flat hierarchies - progressive organizational thinking or digital's 'flat earth' theory?’, Diginomica Blog, 21 February. Available at: https://diginomica.com/flat-hierarchies-progressive-organizational-thinking-or-digitals-flat-earth-theory (Accessed: 18 June 2020).

183

Farjoun, M. and Starbuck, W.H. (2007) ‘Organizing at and beyond the limits’, Organization Studies, 28(4), pp.541-566. Farmer, D., Markopoulou, F., Beinhocker, E. and Rasmussen, S. (2020) Collaborators in creation. Available at: https://aeon.co/essays/how-social-and-physical-technologies-collaborate-to-create (Accessed: 4 March 2020). Farrell, S. (2011) ‘Social influence benefits the wisdom of individuals in the crowd’, Proceedings of the National Academy of Sciences, 108(36), p.E625. Fenn, J. and Linden, A. (2005) Gartner’s Hype Cycle special report for 2005. Stamford, CT: Gartner Research Inc. Fereday, J. and Muir-Cochrane, E. (2006) ‘Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development’, International Journal of Qualitative Methods, 5(1), pp.80-92. Ferreira, J.J., Fernandes, C.I. and Ratten, V. (2017) ‘Entrepreneurship, innovation and competitiveness: what is the connection?’, International Journal of Business and Globalisation, 18(1), pp.73-95. Ferrier, M. (2018) ‘Just-in-time project task scheduling’, Proceedings of the Future Business Technology Conference (FTC) 2018. Vancouver, Canada, 13-14 November. Basel: Springer Nature Switzerland AG, pp.959-964. Ferris, S.P., Javakhadze, D. and Rajkovic, T. (2019) ‘An international analysis of CEO social capital and corporate risk‐taking’, European Financial Management, 25(1), pp.3-37. Fildes, R. and Goodwin, P. (2013) ‘Forecasting support systems: what we know, what we need to know’, International Journal of Forecasting, 2(29), pp.290-294. Fildes, R. and Goodwin, P. (2020) Stability and innovation in the use of forecasting systems: a case study in a supply-chain company. Department of Management Science Working Paper 2020:1. Lancaster: Lancaster University. Finn, P., Mysore, M. and Usher, O. (2020) When nothing is normal: managing in extreme uncertainty. Available at: https://www.mckinsey.com/business-functions/risk/our-insights/when-nothing-is-normal-managing-in-extreme-uncertainty (Accessed: 03 November 2020). Finney, A. (2019) ‘Communicating the forecast: providing decision makers with insights’, Foresight: The International Journal of Applied Forecasting, 54, pp.44-48. Fisher, G., Wisneski, J. and Bakker, R. (2020) Strategy in 3D: essential tools to diagnose, decide, and deliver. New York: Oxford University Press. Fladerer, J.P. and Kurzmann, E. (2019) The wisdom of the many: how to create self-organisation and how to use collective intelligence in companies and in society from management to managemANT. Norderstedt: BoD–Books on Demand. Flamholtz, E. (2001) ‘Corporate culture and the bottom line’, European Management Journal, 19(3), pp.268-275. Fleetwood, S. (2014) ‘Bhaskar and critical realism’, in Adler, P., Du Gay, P., Morgan, G. and Reed, M. (eds.) Oxford handbook of sociology, social theory and organisation studies: con-temporary currents. Oxford: Oxford University Press, pp.182–219.

184

Fleming, O., Fountaine, T., Henke, N. and Saleh, T. (2018) ‘Ten red flags signaling your analytics program will fail’, McKinsey Analytics. Available at: https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/ten-red-flags-signaling-your-analytics-program-will-fail (Accessed: 01 August 2018). Fletcher-Hill, P. (2019) Create your own prediction market. Available at: https://medium.com/veil-blog/create-your-own-prediction-market-d41ec7c19675 (Accessed: 20 August 2020). Flick, U., Garms-Homolová, V., Herrmann, W.J., Kuck, J. and Röhnsch (2012). ‘“I Can’t Prescribe Something Just Because Someone Asks for It...” using mixed methods in the framework of triangulation’, Journal of Mixed Methods Research, 6(2), pp.97-110. Floridi, L. and Strait, A. (2020) ‘Ethical foresight analysis: what it is and why it is needed?’, Minds and Machines, 30, pp.70-97. Flostrand, A. (2017) ‘Finding the future: crowdsourcing versus the Delphi technique’, Business Horizons, 60(2), pp.229-236. Foglesong, R.H. (2004) ‘Leadership from flight level 390’, Air & Space Power Journal, 18(3), pp.5-11.

Ford, M. (2018) Architects of intelligence. Birmingham, UK: Packt Publishing. Forestal, R.L., Zhang, C.P. and Pi, S.M. (2020) ‘Prediction markets: a systematic review and meta-analysis’, Proceedings of the 20th International Conference on Electronic Business. Hong Kong, 5-8 December. Atlanta: ICEB, pp.250-264. Fountaine, T., McCarthy, B. and Saleh, T. (2019) ‘What it really takes to scale artificial intelligence’, Mckinsey Digital Blog, 18 June. Available at: https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/digital-blog/what-it-really-takes-to-scale-artificial-intelligence. (Accessed: 24 June 2019). Frank, H., Kessler, A., Bachner, C., Fuetsch, E. and Suess-Reyes, J. (2019) ‘Principles for innovation management in family firms: an analysis of long-term successful good practices with a practitioner validation of the principles’, Journal of Family Business Management, 9(3), pp.319-348 Freeman, R., Lahaie, S. and Pennock, D.M. (2016) ‘Crowdsourced outcome determination in prediction markets’. To be published in Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence [Preprint]. Available at: arXiv preprint arXiv:1612.04885 (Accessed: 02 April 2019). Frey, B.B. (ed.) (2018) The SAGE encyclopedia of educational research, measurement, and evaluation. London: Sage Publications. Friese, S. (2019) Qualitative data analysis with ATLAS. ti. London: SAGE Publications Limited. Frishammar, J., Richtnér, A., Brattström, A., Magnusson, M. and Björk, J. (2019) ‘Opportunities and challenges in the new innovation landscape: implications for innovation auditing and innovation management’, European Management Journal, 37(2), pp.151-164. Futoma, J., Simons, M., Panch, T., Doshi-Velez, F. and Celi, L.A. (2020) ‘The myth of generalisability in clinical research and machine learning in health care’, The Lancet Digital Health, 2(9), pp.e489-e492.

185

Gässler, A. (2010) ‘Auf dem Weg zum erfolgreichen Spitzenteam’, in Kreuser, K. and Robrecht, T. (eds.) Führung und Erfolg – Eigene Potenziale entfalten, Mitarbeiter erfolgreich machen. Wiesbaden: Gabler Verlag, pp.59-77. Galinsky A. and Schweitzer, M. (2015) Friend and foe: when to cooperate, when to compete, and how to succeed at both. New York: Crown Business. Gandomi, A. and Haider, M. (2015) ‘Beyond the hype: big data concepts, methods, and analytics’, International Journal of Information Management, 35(2), pp.137-144. Gao, X.A., Zhang, J. and Chen, Y. (2013) ‘What you jointly know determines how you act: strategic interactions in prediction markets’, Proceedings of the fourteenth ACM conference on Electronic commerce. University of Philadelphia. PA, 16-20 June. New York: Association for Computing Machinery, pp.489-506. Garrette, B., Phelps, C. and Sibony, O. (2018) Cracked it! How to solve big problems and sell solutions like top strategy consultants. Cham: Palgrave Macmillan. Gaspoz, C. (2011) ‘Prediction markets as Web 2.0 tools for enterprise 2.0’, Proceedings of the Seventeenth Americas Conference on Information Systems. Detroit, Michigan, 4-8 August. Atlanta, GA: Curran Associates, Inc., pp.3808-3817. Gass, S., Loewen, S. and Plonsky, L. (2020) ‘Coming of age: The past, present, and future of quantitative SLA research’, in Porte, G. (ed.) Language teaching – survey and studies. Cambridge: Cambridge University Press, pp.1-14. Gatu, G. (2020) ‘Are crowdsourced analysts MICE or RASCLS?’. Assignment for UNDA23 20201, BSc Law and Political Science, Lund University. Unpublished. Geissdoerfer, M., Bocken, N. and Hultink, E.J. (2016) ‘Design thinking to enhance the sustainable business modelling process – a workshop based on a value mapping process’, Journal of Cleaner Production, 135, pp.1218-1232. Gemünden, H.G. and Hauschildt, J. (1985) ‘Number of alternatives and efficiency in different types of top-management decisions’, European Journal of Operational Research, 22(2), pp.178-190. George, G., Howard-Grenville, J., Joshi, A. and Tihanyi, L. (2016) ‘Understanding and tackling societal grand challenges through management research’, Academy of Management Journal, 59(6), pp.1880-1895. Gerhart, B. and Fang, M. (2015) ‘Pay, intrinsic motivation, extrinsic motivation, performance, and creativity in the workplace: revisiting long-held beliefs’, Annual Review of Organizational Psychology and Organizational Behavior, 2(1), pp.489-521. Gerlach, S. and Brem, A. (2017) ‘Idea management revisited: a review of the literature and guide for implementation’, International Journal of Innovation Studies, 1(2), pp.144-161. Geurts, Y.E. (2020) Improving the forecast accuracy by analyzing the behavior of planners in a time series forecasting model. Master Thesis. Eindhoven University of Technology. Gigerenzer, G. (2015) Das Einmaleins der Skepsis: über den richtigen Umgang mit Zahlen und Risiken. 3rd edn. München/Berlin: Piper Verlag. Gilboa, I., Rouziou, M. and Sibony, O. (2018) ‘Decision theory made relevant: between the software and the shrink’, Research in Economics, 72(2), pp.240-250.

186

Gill, P., Stewart, K., Treasure, E. and Chadwick, B. (2008) ‘Methods of data collection in qualitative research: interviews and focus groups’, British Dental Journal, 204(6), pp.291-295. Girdharwal, N. (2020) ‘The effective executive: the definitive guide to getting the right things done’, Indian Journal of Marketing, 50(1), pp.65-69. Gladwell, M. (2000) The tipping point: how little things can make a big difference. New York: Back Bay Books. Gläser, J. and Laudel, G. (2010) Experteninterviews und qualitative Inhaltsanalyse. 4th edn. Wiesbaden: VS Verlag für Sozialwissenschaften. Gloor, P. (2006) Swarm creativity: competitive advantage through collaborative innovation networks. Oxford: Oxford University Press. Gloor, P. (2011) Coolfarming: turn your great idea into the next big thing. New York: AMACOM. Gloor, P. (2017) Sociometrics and human relationships: analyzing social networks to manage brands, predict trends, and improve organizational performance. Bingley: Emerald Publishing Limited. Gloor, P., Paasivaara, M., Miller, C.Z. and Lassenius, C. (2016) ‘Lessons from the collaborative innovation networks seminar’, International Journal of Organisational Design and Engineering, 4(1-2), pp.3-25. Gloor, P.A., Colladon, A.F., Grippa, F., Hadley, B.M. and Woerner, S. (2020) ‘The impact of social media presence and board member composition on new venture success: evidence from VC-backed US startups’, Technological Forecasting and Social Change, 157, https://doi.org/10.1016/j.techfore.2020.120098. Glover, H.D. and Reidenbach, M. (2012) ‘Auditor reporting model modifications: Practical insights from the academic community’, Current Issues in Auditing, 6(1), pp.C7-C14. Gluck, F.W., Kaufman, S.P. and Walleck, A.S. (1980) ‘Strategic management for competitive advantage’, Harvard Business Review, 58(4), pp.154-161. von Goethe, J.W. (2013) Gedichte: Ausgabe letzter Hand 1827. Scotts Valley, CA: CreateSpace Independent Publishing Platform. Goffin, K., Åhlström, P., Bianchi, M. and Richtnér, A. (2019) ‘Perspective: state‐of‐the‐art: the quality of case study research in innovation management’, Journal of Product Innovation Management, 36(5), pp.586-615. Goldstein, D.G., McAfee, R.P. and Suri, S. (2014) ‘The wisdom of smaller, smarter crowds’, Proceedings of the fifteenth ACM conference on Economics and Computation. Palo Alto, CA, 8-12 June. New York: Association for Computing Machinery, pp.471-488. Gómez-Ramirez, I., Valencia-Arias, A. and Duque, L. (2019) ‘Approach to M-learning acceptance among university students: An integrated model of TPB and TAM’, International Review of Research in Open and Distributed Learning, 20(3), pp.141-163. Good Judgment Inc. (2020) See the future sooner – actionable, early insight and better decisions start with the science of superforecasting. Available at: https://goodjudgment.com/ (Accessed: 09 June 2020). Good Judgment Project (2016) ‘GJP data’. Doi:10.7910/DVN/BPCDH5 (Accessed: 09 September 2020)

187

Goodwin, P. (2019) ‘Scenarios and forecasts: complementary ways of anticipating the future?’, Foresight: The International Journal of Applied Forecasting, 52, pp.7-10. Gottlieb, J. and Willmott, P. (2014) McKinsey global survey results: the digital tipping point. Available at: http://www.mckinsey.com/business-functions/business-technology/our-insights/the-digital-tipping-point-mckinsey-global-survey-results (Accessed: 03 November 2015). Graefe, A. (2008) ‘Prediction markets – defining events and motivating participation’, Foresight – The International Journal of Applied Forecasting, 9, pp.30-32. Graefe, A. (2009) Prediction markets versus alternative methods: empirical tests of accuracy and acceptability. Doktor der Wirtschaftswissenschaften. University of Karlsruhe. Graefe, A. (2010) ‘Are prediction markets more accurate than simple surveys?’, Foresight – The International Journal of Applied Forecasting, 19, pp.39-43. Graefe, A. (2011) ‘Prediction market accuracy for business forecasting’, in Vaughan Williams, L. (ed.) Prediction markets – theory and applications. New York: Routledge, pp.87-95. Graefe, A. (2017) ‘The PollyVote’s long-term forecast for the 2017 German federal election’, PS: Political Science & Politics, 50(3), pp.693-696. Graefe, A. (2018) ‘Predicting elections: Experts, polls, and fundamentals’, Judgment & Decision Making, 13(4), pp.334-344. Graefe, A. (2019) ‘Accuracy of German federal election forecasts, 2013 & 2017’, International Journal of Forecasting, 35(3), pp.868-877. Graefe, A. and Armstrong, J.S. (2011) ‘Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task’, International Journal of Forecasting, 27(1), pp.183-195. Grainger, D. and Stoeckl, N. (2019) ‘The importance of social learning for non-market valuation’, Ecological Economics, 164, pp.1-9. Grant, A., Johnstone, D. and Kwon, O.K. (2019) ‘The cost of capital in a prediction market’, International Journal of Forecasting, 35(1), pp.313-320. Grant, R.M. (2016) Contemporary strategy analysis: text and cases edition. Hoboken, NJ: John Wiley & Sons. Gray, K. and Ulbrich, F. (2017) ‘Ambiguity acceptance and translation skills in the project management literature’, International Journal of Managing Projects in Business, 10(2), pp.423-450. Greener, S. (2018) ‘Research limitations: the need for honesty and common sense’, Interactive Learning Environments, 26(5), pp.567-568. Griffin, P.A. and Wright, A.M. (2015) ‘Commentaries on big data's importance for accounting and auditing’, Accounting Horizons, 29(2), pp.377-379. Gröner, S. and Heinecke, S. (2019) Kollege KI: Künstliche Intelligenz verstehen und sinnvoll im Unternehmen einsetzen. München: Redline Verlag. Gruca, T.S. and Berg, J.E. (2007) ‘Public information bias and prediction market accuracy’, The Journal of Prediction Markets, 1(3), pp.219-231. Gürçay, B., Mellers, B.A. and Baron, J. (2015) ‘The power of social influence on estimation accuracy’, Journal of Behavioral Decision Making, 28(3), pp.250-261.

188

Guest, G., Bunce, A. and Johnson, L. (2006) ‘How many interviews are enough? An experiment with data saturation and variability’, Field Methods, 18(1), pp.59-82. Guest, G., Namey, E. and Chen, M. (2020) ‘A simple method to assess and report thematic saturation in qualitative research’, PLoS One, 15(5), DOI: 10.1371/journal.pone.0232076. Gurkov, I. (2020) ‘Strategy techniques for the times of high uncertainty’, Journal for East European Management Studies, 15(2), pp.177-186. Haaker, M., Corti, L. and Van den Eynden, V. (2019) ‘Documenting and providing context for data’, in Corti, L., Van den Eynden, V., Bishop, L. and Woollard, M. (eds.) Managing and sharing research data: a guide to good practice. 2nd edn. London: SAGE Publications Ltd, pp.70-101. Haaker, M., Summers, S. and Corti, L. (2019) ‘Formatting, organising and transforming data’, in Corti, L., Van den Eynden, V., Bishop, L. and Woollard, M. (eds.) Managing and sharing research data: a guide to good practice. 2nd edn. London: SAGE Publications Ltd, pp.102-134. Hackman, J.R. (2002) Leading teams: setting the stage for great performances. Boston: Harvard Business Press.

Häßner, R. (2014) Personal interview with Thomas Wolfram, 30 July. Hair Jr, J.F. (2007) ‘Knowledge creation in marketing: the role of predictive analytics’, European Business Review, 19(4), pp.303-315. Hall, C. (2010) ‘Prediction markets: issues and applications’, Journal of Prediction Markets, 4(1), pp.27-58. Hall, R.E., Jones, C.I. and Klenow, P.J. (2020) ‘Trading off consumption and COVID-19 deaths’, Federal Reserve Bank of Minneapolis Quarterly Review, 42(1), pp.1-13. Hambrick, D.C. (2005) ‘Upper echelons theory: origins, twists and turns, and lessons learned’, in Smith, K.G. and Hitt, M.A. (eds.) Great minds in management: the process of theory development. New York: Oxford University Press, pp.109-127.

Hamel, G. (2007) The future of management. Boston: Harvard Business Press. Hamel, G. (2011) ‘First, let’s fire all the managers’, Harvard Business Review, 89(12), pp.48-60. Hamel, G. (2012) What matters now: how to win in a world of relentless change, ferocious competition, and unstoppable innovation. San Francisco: Jossey-Bass. Hamel, G. and Zanini, M. (2020) Humanocracy: creating organizations as amazing as the people inside them. Boston, MA: Harvard Business Review Press. Hansen, S.W., Klemmensen, R. and Serritzlew, S. (2019) ‘Losers lose more than winners win: asymmetrical effects of winning and losing in elections’, European Journal of Political Research, 58(4), pp.1172-1190. Hanson, R. (2007) ‘The policy analysis market (a thwarted experiment in the use of prediction markets for public policy)’, Innovations: Technology, Governance, Globalization, 2(3), pp.73-88. Hanson, R., Wilson, B., Parod, A. and Savage, D. (2013) Prediction market system and methods. U.S. Patent 8,612,331. Available at: https://patents.google.com/patent/US8612331B2/en (Accessed: 12 December 2020).

189

Harding, D. (2019) ‘8th FERI hedge funds investment day’, The Hedge Fund Journal, 5 September. Available at: https://thehedgefundjournal.com/in-conversation-with-martin-lueck-aspect-capital/ (Accessed: 04 October 2019). Harter, J. (2020) ‘4 factors driving record-high employee engagement in U.S.’, Gallup Survey. Available at: https://www.gallup.com/workplace/284180/factors-driving-record-high-employee-engagement.aspx (Accessed: 6 November 2020). Hartley, J.S. and Hassett, K.A. (2016) Statistically measuring 2016 presidential candidate electability: evidence from prediction markets. AEI Economics Working Paper 2016-04. Washington, DC: American Enterprise Institute. Hay, C. (2015) ‘Social constructivism’, in Bevir, M. and Rhodes , R.A.W. (eds.) The handbook of interpretive political science. London: Routledge, pp.99-112. Heckathorn, D.D. (2015) ‘Snowball versus respondent-driven sampling’, Sociological Methodology, 41(1), pp.355-366 Hedberg, B.L., Nystrom, P.C. and Starbuck, W.H. (1976) ‘Camping on seesaws: prescriptions for a self-designing organization’, Administrative Science Quarterly, 21(1), pp.41-65. Herzberg, G., Panikkar, R., Whiteman, R. and Sahu, A. (2020) The imperatives for automation success. New York: McKinsey & Company. Heskett, J. (2012) ‘Should managers bother listening to predictions?’, Harvard Business School Working Knowledge. Available at: https://hbswk.hbs.edu/item/should-managers-bother-listening-to-predictions (Accessed: 05 July 2018). Higgins, J.P., Larrea, E., Prabha, S., Singla, A. and Sood. R. (2019) ‘How to extract maximum value from a zero-based design approach to customer journeys’, McKinsey Digital – Our Insights. Available at: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/how-to-extract-maximum-value-from-a-zero-based-design-approach-to-customer-journeys (Accessed: 02 January 2020). Hin, L.H. (2019) ‘Critical review: future of block chain economy in financial services’, Global Journal of Management and Business Research, XIX(VII), pp.10-18. Hirshleifer, D. (2020) ‘Presidential address: Social transmission bias in economics and finance’, The Journal of Finance, 75(4), pp.1779-1831. Hirt, M., Laczkowski, K. and Mysore, M. (2019) ‘Bubbles pop, downturns stop’, McKinsey Quarterly, 55(2), pp.10-18. Hitzler, R. (1998) ‘Wissen und Wesen des Experten’, in Schulz, W.K. (ed.) Expertenwissen: Soziologische, psychologische und pädagogische Perspektiven. Wiesbaden: Vieweg+Teubner Verlag, pp.13-30. Hlupic, V. (2018) Humane capital: how to create a management shift to transform performance and profit. London: Bloomsbury Academic. Ho, T.H. and Chen, K.Y. (2007) ‘New product blockbusters: the magic and science of prediction markets’, California Management Review, 50(1), pp.144-158. Hölzle, K., Schoder, T., Spiri, N. and Götz, V. (2017) ‘Big Data und technologiegetriebene Geschäftsmodellinnovation’, in Schallmo, D., Rusnjak, A., Anzengruber, J., Werani, T. and Jünger, M. (eds.) Digitale Transformation von Geschäftsmodellen. Wiesbaden: Springer Fachmedien, pp.355-374. Hofstede, G. (1993) ‘Cultural constraints in management theories’, Academy of Management Perspectives, 7(1), pp.81-94.

190

Holt, C.A. (2020) E-mail to Thomas Wolfram, 10 March. Horn, C., Bogers, M. and Brem, A. (2018) ‘Prediction markets for crowdsourcing’, in Tucci, C.L., Afuah, A. and Viscusi, G. (eds.) Creating and capturing value through crowdsourcing. Oxford: Oxford University Press, pp.292-309. Horn, C., Brem, A. and Ivens, B. (2014) ‘Damaging brands through market research: can customer involvement in prediction markets damage brands? ’, Marketing Intelligence & Planning, 32(2), pp.232-248. Horn, C. and Ivens, B. (2015) ‘Corporate prediction markets for innovation management’, in Brem, A. and Viardot, E. (eds.) Adoption of innovation - balancing internal and external stakeholders in the marketing of innovation. Cham, Switzerland: Springer International Publishing, pp.11-23. Horn, C., Ohneberg, M. and Ivens, B.S. (2014) ‘Prediction markets – a literature review following Tziralis and Tatsiopoulos’, Journal of Prediction Markets, 8(2), pp.89-126. Horowitz, M.C. (2021) ‘Forecasting political events’, in Mintz, A. and Terris, L. (eds.) The Oxford handbook of behavioral political science. DOI: 10.1093/oxfordhb/9780190634131.013.16. Horx, M. (2011) Das Megatrend-Prinzip: wie die Welt von morgen entsteht. München: Deutsche Verlags-Anstalt. Hosseini, M., Moore, J., Almaliki, M., Shahri, A., Phalp, K. and Ali, R. (2015) ‘Wisdom of the crowd within enterprises: practices and challenges’, Computer Networks, 90, pp.121-132. Howard, J., Thompson, C., Nash, K. and Rodriguez, S. (2016) ‘Missing stories: the messy processes, multifaceted risks, & multiple roles of critical ethnographers’, Critical Questions in Education, 7(3), pp.318-338. Howe, J. (2009) Crowdsourcing: how the power of the crowd is driving the future of business. New York: Random House. Howe, K.R. (2012) ‘Mixed methods, triangulation, and causal explanation’, Journal of Mixed Methods Research, 6(2), pp.89-96. Huang, K., Li, M. and Markov, S. (2019) ‘The information asymmetry between management and rank-and-file employees: Determinants and Consequences’, SSRN Electronic Journal, DOI: 10.2139/ssrn.3233855. Huang, T., Fildes, R. and Soopramanien, D. (2019) ‘Forecasting retailer product sales in the presence of structural change’, European Journal of Operational Research, 279(2), pp.459-470. Huesig, S. and Endres, H. (2019) ‘Exploring the digital innovation process’, European Journal of Innovation Management, 22(2), pp.302-314. Hyndman, R.J. (2020) ‘A brief history of forecasting competitions’, International Journal of Forecasting, 36(1), pp.7-14. Hyndman, R.J. and Athanasopoulos, G. (2018) Forecasting: principles and practice. 2nd edn. Melbourne: OTexts. Ini-Novation (2020) Innovation management process in INI-Novation GmbH. Available at: http://www.ini-novation.com/wp-content/uploads/2018/07/Innovation-Management-Process-in-INI-Novation-GmbH.jpg (Accessed: 12 August 2020). Inkling (2015) Make better decisions. Available at: https://inklingmarkets.com/ (Accessed: 26 July 2020).

191

Ivanov, A. (2012) ‘The rise of social forecasting - how the crowd learned to be smart’, CrowdWorx Whitepaper. Available at: http://dailycrowdsource.com/crowdsourcing-research/whitepapers/viewdownload/2-whitepapers/51-the-rise-of-social-forecasting-how-the-crowd-learned-to-be-smart (Accessed: 12 January 2014). Ivanov, A. (2013) Telephone interview with Thomas Wolfram, 11 June.

Ivanov, A. (2017) Telephone interview with Thomas Wolfram, 31 March. Ivanov, A. (2019) Telephone interview with Thomas Wolfram, 31 December.

Ivanov, A. (2020) E-mail to Thomas Wolfram, 02 April. Ivanov, A.B. (2021) Skype interview with Thomas Wolfram, 05 February. Jabeen, F., Khusro, S., Majid, A. and Rauf, A. (2016) ‘Semantics discovery in social tagging systems: a review’, Multimedia Tools and Applications, 75(1), pp.573-605. Jalonen, H. and Lönnqvist, A. (2011) ‘Exploring the critical success factors for developing and implementing a predictive capability in business’, Knowledge and Process Management, 18(4), pp.207-219. Jansma, M. (2020) Belief elicitation in prediction markets. Bachelor Thesis, Artificial Intelligence. Rijksuniversiteit Groningen. Jayles, B., Kim, H.R., Escobedo, R., Cezera, S., Blanchet, A., Kameda, T., Sire, C. and Theraulaz, G. (2017) ‘How social information can improve estimation accuracy in human groups’, Proceedings of the National Academy of Sciences, 114(47), pp.12620-12625. Jeddy, A., Klempner, B. and Portner, M. (2018) How private equity fared in 2017, and what’s ahead. Available at: https://www.mckinsey.com/industries/private-equity-and-principal-investors/our-insights/how-private-equity-fared-in-2017-and-whats-ahead (Accessed: 23 June 2020). Jenney, S.L., Jung, H., Seif, H. and Petzold, F. (2020) ‘Gamification as a means to improve stakeholder management in urban planning’, in Koehler, T., Schoop, E. and Kahnwald, N. (eds.) Communities in new media – from hybrid realities to hybrid communities. Dresden: TUDPress, pp.90-98. Jester-Pfad, T.O. (2015) Smart Money? Der Erfolg von Prognosemärkten. Bachelor Thesis. Universität Trier. Jolly, R.D., Zwick, M., Wakeland, W. and Woods, J. (2015) ‘The mechanisms of information integration in experimental prediction markets’, International Journal of Economics and Business Research, 9(1), pp.100-129. Joseph, R. (2014) ‘Keeping score: better policy through improved predictive analysis’, 5th International Conference on Future-Oriented Technology Analysis (FTA). Brussels, 27-28 November. Brussels: Joint Research Centre of the European Commission, pp.27-28. Judah, M., O’Keeffe, D., Zehner D. and Cummings, L. (2016) ‘Strategic planning that produces real strategy’, Bain & Company Publication. Available at: http://www.bain.com/Images/BAIN_BRIEF_Strategic_planning_that_produces_real_strategy.pdf (Accessed: 28 April 2016). Junglas, I., Goel, L., Abraham, C. and Ives, B. (2013) ‘The social component of information systems – how sociability contributes to technology acceptance’, Journal of the Association for Information Systems, 14(10), pp.585-616.

192

Kadam, R.A. (2017) ‘Informed consent process: a step further towards making it meaningful!’, Perspectives in Clinical Research, 8(3), pp.107-112. Kahn, K.B. (2014) New‐product forecasting: an applied approach. Wiley international encyclopedia of marketing. London: Routledge. Kahneman, D., Lovallo, D. and Sibony, O. (2019) ‘A structured approach to strategic decisions’, MIT Sloan Management Review, 60(3), pp.67-73. Kandemir, D. and Acur, N. (2012) ‘Examining proactive strategic decision‐making flexibility in new product development’, Journal of Product Innovation Management, 29(4), pp.608-622. Kao, A.B. and Couzin, I.D. (2014) ‘Decision accuracy in complex environments is often maximized by small group sizes’, Proceedings of the Royal Society B: Biological Sciences, 281(1784), Article ID: 20133305. doi: 10.1098/rspb.2013.3305. Karlan, D. and Appel, J. (2016) Failing in the field: what we can learn when field research goes wrong. Reprint. Princeton: Princeton University Press, 2018. Kasparov, G. (2017) Deep thinking: where machine intelligence ends and human creativity begins. New York: Hachette Book Group. Katsagounos, I., Thomakos, D.D., Litsiou, K. and Nikolopoulos, K. (2020) ‘Superforecasting reality check: evidence from a small pool of experts and expedited identification’, European Journal of Operational Research, 288(3), pp.1-38. Katsikopoulos, K. and Syntetos, A. (2016) ‘Bias-variance trade-offs in demand forecasting’, Foresight: The International Journal of Applied Forecasting, 40, pp.12-19. Katz, W.A. (2002) Introduction to reference work: reference services and reference processes, Vol. 2. New York: McGraw-Hill. Katzenbach, J.R. and Smith, D.K. (2015) The wisdom of teams: creating the high-performance organization. Boston: Harvard Business Review Press. Kauko, K. and Palmroos, P. (2014) ‘The Delphi method in forecasting financial markets – an experimental study’, International Journal of Forecasting, 30(2), pp.313-327. Kempes, C.P., Koehl, M.A.R. and West, G.B. (2019) ‘The scales that limit: the physical boundaries of evolution’, Frontiers in Ecology and Evolution, 7(242), DOI: 10.3389/fevo.2019.00242. Kennedy, C., Blumenthal, M., Clement, S., Clinton, J.D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D. and Saad, L. (2018) ‘An evaluation of the 2016 election polls in the United States’, Public Opinion Quarterly, 82(1), pp.1-33. Kerth, K., Asum, H. and Stich, V. (2011) Die besten Strategietools in der Praxis. 5th edn. München: Hanser Verlag. Kilduff, M. (2006) ‘Editor’s comment: publishing theory’, Academy of Management Review 2006, 31(2), pp.252–255. Kim, H.C. (2015) ‘Acceptability engineering: the study of user acceptance of innovative technologies’, Journal of Applied Research and Technology, 13(2), pp.230-237. Kingsley, J. (2015) Prediction as sport: the promise of prediction polls and markets. Nesta Working Paper No. 15/02. London: Nesta.

193

Klingert, F. (2017) ‘The structure of prediction market research: important publications and research clusters’, Journal of Prediction Markets, 11(1), pp.51-65. Klingert, F. and Meyer, M. (2018) ‘Comparing prediction market mechanisms: An experiment-based and micro validated multi-agent simulation’, Journal of Artificial Societies and Social Simulation, 21(1), pp.7-32. Kloker, S., Klatt, F., Höffer, J. and Weinhardt, C. (2018) ‘Analyzing prediction market trading behaviour to select Delphi-experts’, Foresight, 20(4), pp.364-374. Kloker, S., Straub, T., Kranz, T. and Weinhardt, C. (2019) ‘Delphi-Märkte’, in Niederberger, M. and Renn, O, (eds.) Delphi-Verfahren in den Sozial- und Gesundheitswissenschaften: Konzept, Varianten und Anwendungsbeispiele. Wiesbaden: Springer, pp.125-148. Kloker, S., Straub, T. and Weinhardt, C. (2017) ‘Designing a crowd forecasting tool to combine prediction markets and real-time Delphi’, in Maedche, A., vom Brocke, J. and Hevner, A. (eds.) International conference on design science research in information systems. Cham: Springer, pp.468-473.

Koch, G. (2020) E-Mail to Thomas Wolfram, 02 May. Kokotajlo, D. and Grace, K. (2019a) ‘Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post’, AI Impacts Blog, 7 February. Available at: https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/ (Accessed: 22 December 2019). Kokotajlo, D. and Grace, K. (2019b) ‘Evidence on good forecasting practices from the Good Judgment Project’, AI Impacts Blog, 7 February. Available at: https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/ (Accessed: 22 December 2019). Kolassa, S. (2012) ‘Two notable new forecasting texts: Principles of business forecasting by Keith Ord & Robert Fildes – Forecasting: principles and practice by Rob Hyndman & George Athanasopoulos’, Foresight: The International Journal of Applied Forecasting, 27, pp.40-45. Kolassa, S. (2016) ‘Sometimes it's better to be simple than correct’, Foresight: The International Journal of Applied Forecasting, 40, pp.20-26. Koller, T., Lovallo, D. and Brown, S. (2020) ‘Four ways to assess projects and keep them on track’, McKinsey Insights. Available at: https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/four-ways-to-assess-projects-and-keep-them-on-track (Accessed: 26 October 2020). Koß, A. (2020) Personal interview with Thomas Wolfram, 22 January. Kotter, J.P. and Heskett, J.L. (2011) Corporate culture and performance. New York: Simon and Schuster. Kourdi, J. (2015) Business strategy – a guide to effective decision-making. 3rd edn. London: Profile Books Ltd. Krause, R.W. and Caimo, A. (2019) ‘Missing data augmentation for Bayesian exponential random multi-graph models’, International Workshop on Complex Networks. Tarragona, Spain, 18-21 March. Cham: Springer, pp.63-72. Krause, R.W., Huisman, M., Steglich, C. and Snijders, T. (2020) ‘Missing data in cross-sectional networks – an extensive comparison of missing data treatment methods’, Social Networks, 62, pp.99-112.

194

Krekel, C., Ward, G. and De Neve, J.E. (2019) Employee well-being, productivity, and firm performance: evidence and case studies – global happiness and well-being policy report. Boston: Harvard Business School. Krogerus, M. and Tschäppeler, R. (2017) 50 Erfolgsmodelle - Kleines Handbuch für strategische Entscheidungen. 3rd edn. Zürich: Kein & Aber.

Kromrey, H. (2006) Empirische Sozialforschung. Stuttgart: Lucius & Lucius. Kross, J. and Giust, A. (2019) ‘Elements of research questions in relation to qualitative inquiry’, The Qualitative Report, 24(1), pp.24-30. Krug, J. (2019) ‘Joey Krug | What is Augur? Why do prediction markets matter for real estate?’, Propy Blog, 23 June. Available at: https://propy.com/blog/joey-krug-what-is-augur-why-do-prediction-markets-matter-for-real-estate/ (Accessed: 28 September 2019). Kuechle, G., Boulu-Reshef, B. and Carr, S.D. (2016) ‘Prediction- and control-based strategies in entrepreneurship: the role of information’, Strategic Entrepreneurship Journal, 2016(10), pp.43-64. Kuhn, T.S. (2012) The structure of scientific revolutions. 4th edn. Chicago: University of Chicago Press. Kurvers, R., Herzog, S., Hertwig, R., Krause, J., Moussaid, M., Argenziano, G., Zalaudek, I., Carney, P. and Wolf, M. (2019) How to detect high-performing individuals and groups: decision similarity predicts accuracy. Available at: https://doi.org/10.31234/osf.io/ryvnw (Accessed: 09 January 2020). Kvernbekk, T. (2013) ‘Evidence-based practice: on the function of evidence in practical reasoning’, Studier i Pædagogisk Filosofi, 2(2), pp.19-33. Lai, P.C. (2017) ‘The literature review of technology adoption models and theories for the novelty technology’, JISTEM-Journal of Information Systems and Technology Management, 14(1), pp.21-38. Lambert, S.D. and Loiselle, C.G. (2008) ‘Combining individual interviews and focus groups to enhance data richness’, Journal of Advanced Nursing, 62(2), pp.228-237. Landry, S. (2010) The Gartner Hype Cycle for social software, 2010 – Gartner Report G205425. Stamford, CT: Gartner Research Inc. Langos, S. (2014) Athens as an international tourism destination: an empirical investigation to the city’s imagery and the role of local DMO’s. Master of Science. University of Derby. Lapide, L. (2003) ‘Organizing the forecasting department’, The Journal of Business Forecasting, 22(2), pp.20-21. LaRiviere, J., McAfee, P., Rao, J., Narayanan, V.K. and Sun, W. (2016) ‘Where predictive analytics is having the biggest impact’, Harvard Business Review. Available at: https://hbr.org/2016/05/where-predictive-analytics-is-having-the-biggest-impact (Accessed: 03 April 2019). Larkins, A.G. and McKinney, C.W. (2012) ‘Four types of theory: implications for research in social education’, Theory & Research in Social Education, 8(1), pp.9-17. Lavoie, J. (2009) ‘The innovation engine at Rite-solutions: lessons from the CEO’, Journal of Prediction Markets, 3(1), pp.1-11.

195

Lawrence, M., Goodwin, P., O’Connor, M. and Önkal, D. (2006) ‘Judgmental forecasting: A review of progress over the last 25 years’, International Journal of Forecasting, 22(3), pp.493-518. Lee, H.W., Pak, J., Kim, S. and Li, L.Z. (2019) ‘Effects of human resource management systems on employee proactivity and group innovation’, Journal of Management, 45(2), pp.819-846. Lefort, F., McMurray, D. and Tesvic, J. (2015) Secrets to implementation success. Available at: https://www.mckinsey.com/industries/consumer-packaged-goods/our-insights/secrets-to-implementation-success (Accessed: 18 September 2019). Legard, R., Keegan, J. and Ward, K. (2003) ‘In-depth interviews’, in Ritchie, J. and Lewis, J. (eds.) Qualitative research practice: a guide for social science students and researchers. London: Sage Publications, pp.139-169. Leitch, C.M., Hill, F.M. and Harrison, R.T. (2010) ‘The philosophy and practice of interpretivist research in entrepreneurship: quality, validation, and trust’, Organizational Research Methods, 13(1), pp.67-84. Lenart-Gansiniec, R. (2019) ‘Innovation management within the organisation’, in Chen, J., Brem, A., Viardot, E. and Wong, P.K. (eds.) The Routledge companion to innovation management. Milton Park: Routledge, pp.363-376. Lepore, J. (2014) ‘The disruption machine’, The New Yorker, 23, pp.30-36. Lepore, J. (2019) ‘The robot caravan: automation, AI, and the coming invasion’, The New Yorker, 95(2), pp.20-24. Lepore, J. (2020) If then: how the simulmatics corporation invented the future. New York: Liveright Publishing. Lewe, P. (2020) Resilienz ist die Kernkompetenz für jetzt und die Zeit nach der Krise. Available at: https://www.leadership-choices.com/de/thinkabout/article/resilienz-ist-die-kernkompetenz-fuer-jetzt-und-die-zeit-nach-der-krise.html (Accessed: 30 August 2020). Lewrick, M., Link, P. and Leifer, L. (2017) Das Design Thinking Playbook. München: Verlag Franz Vahlen. Lisak, A., Erez, M., Sui, Y. and Lee, C. (2016) ‘The positive role of global leaders in enhancing multicultural team innovation’, Journal of International Business Studies, 47(6), pp.655-673. Little, D. (2016) New directions in the philosophy of social science. London: Rowman & Littlefield. Logan, D. (2020) ‘Workshop: the art of culture Hhcking: how to create a data-driven culture’, Gartner Data & Analytics Summit. Starling Hotel & Conference Center, Geneva, Switzerland, 15-16 June. Stamford, CT: Gartner Research Inc. Lopez, J.C.A.D., Collignon-Delmar, S., Benoit, K. and Matsuo, A. (2017) ‘Predicting the Brexit vote by tracking and classifying public opinion using twitter data’, Statistics, Politics and Policy, 8(1), pp.85-104. Lorenz, J., Rauhut, H., Schweitzer, F. and Helbing, D. (2011) ‘How social influence can undermine the wisdom of crowd effect’, Proceedings of the National Academy of Sciences, 108(22), pp.9020-9025. Lovallo, D. and Sibony, O. (2006) ‘Distortions and deceptions in strategic decisions’, McKinsey Quarterly, 42(1), pp.18-29.

196

Lovallo, D. and Sibony, O. (2010) ‘The case for behavioral strategy’, McKinsey Quarterly, 46(1), pp.30-43. Lu, Y.H., Tsai, C.F. and Yen, D.C. (2010) ‘Discovering important factors of intangible firm value by association rules’, International Journal of Digital Accounting Research, 10(1), pp.55-85. Luckner, S., Schröder, J., Slamka, C., Franke, M., Geyer-Schulz, A., Skiera, B., Spann, M. and Weinhardt, C. (2012) Prediction markets – fundamentals, designs and applications. Wiesbaden: Gabler Verlag. Luft, J. (2016) ‘Cooperation and competition among employees: experimental evidence on the role of management control systems’, Management Accounting Research, 31, pp.75-85. Lundblad, J.P. (2003) ‘A review and critique of Rogers' diffusion of innovation theory as it applies to organizations’, Organization Development Journal, 21(4), pp.50-64. MacArthur, H. and Rainey, B. (2019) Global private equity report 2019. Boston: Bain & Company. Machnig, M. (2016) ‘Industriepolitik im Strukturwandel’, in Bäcker, G., Lehndorff, S. and Weinkopf, C. (eds.) Den Arbeitsmarkt verstehen, um ihn zu gestalten: Festschrift für Gerhard Bosch. Wiesbaden: Springer VS, pp.301-313. MacKay, B. and McKiernan, P. (2010) ‘Creativity and dysfunction in strategic processes: the case of scenario planning’, Futures, 42(4), pp.271-281. Mackey, A. and Gass, S.M. (2015) Second language research: Methodology and design. 2nd edn. New York: Routledge. Maertens, A. and Barrett, C.B. (2013) ‘Measuring social networks' effects on agricultural technology adoption’, American Journal of Agricultural Economics, 95(2), pp.353-359. Magnusson, P.R., Wästlund, E. and Netz, J. (2016) ‘Exploring users' appropriateness as a proxy for experts when screening new product/service ideas’, Journal of Product Innovation Management, 33(1), pp.4-18. Makridakis, S., Hyndman, R.J. and Petropoulos, F. (2020) ‘Forecasting in social settings: The state of the art’, International Journal of Forecasting, 36(1), pp.15-28. Malone, T.W. (2020) ‘How can human-computer “superminds” develop business strategies?’ in Canals, J. and Heukamp, F. (eds.) The future of management in an AI world. Cham: Palgrave Macmillan, pp.165-183. Malone, T.W. and Bernstein, M.S. (eds.) (2015) Handbook of collective intelligence. Cambridge, MA: MIT Press. Maner, J.K. and Case, C.R. (2013) ‘The essential tension between leadership and power’, American Psychological Association – Science Brief. Available at: https://www.apa.org/science/about/psa/2013/10/leadership-power. (Accessed: 18 September 2015). Maner, J.K. and Mead, N.L. (2010) ‘The essential tension between leadership and power: when leaders sacrifice group goals for the sake of self-interest’, Journal of Personality and Social Psychology, 99(3), pp.482-497. Mann, J. (2011) Hype Cycle for Social Software, 2010 – Gartner Report G00215875. Stamford, CT: Gartner Research Inc.

197

Manyika, J. (2017) ‘What’s now and next in analytics, AI, and automation – briefing note 2017’, McKinsey Global Institute. Available at: https://cdn.ksrinc.com/mckinseysurvey/whats_now_and_next_in_analytics_ai_and_automation.pdf (Accessed: 05 February 2018). Marinovic, I., Ottaviani, M. and Sørensen, P.N. (2011) ‘Modeling idea markets: between beauty contests and prediction markets’, in Vaughan Williams, L. (ed.) Prediction markets – theory and applications. New York: Routledge, pp.4-17. Marks, A. (2019) Doctoral business research 1: an introduction to research methods. Edinburgh: Edinburgh Business School, Heriot-Watt University, unpublished. Matzler, K., Grabher, C., Huber, J. and Füller, J. (2013) ‘Predicting new product success with prediction markets in online communities’, R&D Management, 43(5), pp.420-432. Matzler, K., Bailom, F., von den Eichen, S.F. and Anschober, M. (2016) Digital Disruption: Wie Sie Ihr Unternehmen auf das digitale Zeitalter vorbereiten. München: Verlag Franz Vahlen. Matzler, K., Strobl, A. and Bailom, F. (2016) ‘Leadership and the wisdom of crowds: how to tap into the collective intelligence of an organization’, Strategy & Leadership, 44(1), pp.30-35. Mavrodiev, P. and Schweitzer, F. (2021) ‘The ambigous role of social influence on the wisdom of crowds: an analytic approach’, Physica A: Statistical Mechanics and its Applications, 567, Article 125624. Mavrodiev, P., Tessone, C.J. and Schweitzer, F. (2012) ‘Effects of social influence on the wisdom of crowds’. To be published in Proceedings of CoRR [Preprint]. arXiv:1204.3463. May, J. (2012). Schwarmintelligenz im Unternehmen. Erlangen: Publicis Publishing. Mayer-Schönberger, V. and Cukier, K. (2013) Big data: a revolution that will transform how we live, work, and think. Boston: Eamon Dolan/Houghton Mifflin Harcourt. McAfee, A. and Brynjolfsson, E. (2017) Machine, platform, crowd: harnessing our digital future. New York: W.W. Norton & Company. McCaffrey, J. (2016) ‘Test run – introduction to prediction markets’, MSDN Magazine, 31(6). Available at: https://docs.microsoft.com/en-us/archive/msdn-magazine/2016/june/test-run-introduction-to-prediction-markets (Accessed 20 August 2020). McHugh, K.A., Yammarino, F.J., Dionne, S.D., Serban, A., Sayama, H. and Chatterjee, S. (2016) ‘Collective decision making, leadership, and collective intelligence: tests with agent-based simulations and a field study’, The Leadership Quarterly, 27(2), pp.218-241. McHugh, P. and Jackson, A.L. (2012) ‘Prediction market accuracy: the impact of size, incentives, context and interpretation’, Journal of Prediction Markets, 6(2), pp.22-46. McKinsey & Company (2013) Business and web 2.0: an interactive feature. Available at: https://www.mckinseyquarterly.com/Strategy/Innovation/Business_and_Web_20_An_interactive_feature_2431 (Accessed: 23 November 2013). McLellan, E., MacQueen, K.M. and Neidig, J.L. (2003) ‘Beyond the qualitative interview: data preparation and transcription’, Field Methods, 15(1), pp.63-84.

198

McNae, R. (2010) ‘Young women and the co-construction of leadership’, Journal of Educational Administration, 48(6), pp.677-688. Mebert, A. and Lowe, S. (2017) An analysis of W. Chan Kim and Renée Mauborgne's Blue Ocean Strategy. London: Macat International Ltd. Meier, C.J. (2019) K.I. – Wer das Schicksal programmiert. Heidelberg: Polarise. Meissner, D., Polt, W. and Vonortas, N.S. (2017) ‘Towards a broad understanding of innovation and its importance for innovation policy’, The Journal of Technology Transfer, 42(5), pp.1184-1211. Melese, T., Lin, S.M., Chang, J.L. and Cohen, N.H. (2009) ‘Open innovation networks between academia and industry: an imperative for breakthrough therapies’, Nature Medicine, 15(5), pp.502-507. Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S.E., Moore, D., Atanasov, P., Swift, S.A. and Murray, T. (2014) ‘Psychological strategies for winning a geopolitical forecasting tournament’, Psychological Science, 25(5), pp.1106-1115. Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L. and Tetlock, P. (2015) ‘Identifying and cultivating superforecasters as a method of improving probabilistic predictions’, Perspectives on Psychological Science, 10(3), pp.267-281. Mercier, H. and Sperber, D. (2017) The enigma of reason. Cambridge, MA: Harvard University Press. Merriam, S.B. and Grenier, R.S. (eds.) (2019) Qualitative research in practice: examples for discussion and analysis. 2nd edn. Hoboken, NJ: John Wiley & Sons. Merrill, D.C., Ruberry, M.E., Sayin, O., Tunguz, B., Song, L., Alizadeh, E., DeBruin, M.E., Yan, Y., Wilcox, D., Candido, J. and Solecki, B.A. (2019) Systems and methods for providing machine learning model disparate impact information. U.S. Patent Application 16/052,293. Available at: https://patents.google.com/patent/US20190043070A1/en (Accessed: 03 April 2020). Mertens, D.M. and Hesse-Biber, S. (2012) ‘Triangulation and mixed methods research: Provocative positions’, Journal of Mixed Methods Research, 6(2), pp.75-79. Metcalf, L., Askay, D.A. and Rosenberg, L.B. (2019) ‘Keeping humans in the loop: pooling knowledge through artificial swarm intelligence to improve business decision making’, California Management Review, 61(4), pp.84-109.

Meyer-Berhorn, U. (2013) Personal interview with Thomas Wolfram, 29 April. Mezias, J.M. and Starbuck, W.H. (2003) ‘Studying the accuracy of managers' perceptions: a research odyssey’, British Journal of Management, 14(1), pp.3-17. Mezias, J.M. and Starbuck, W.H. (2009) ‘Decision making with inaccurate, unreliable data’, in Hodgkinson, G.P. and Starbuck, W.H. (eds.) The Oxford handbook of organizational decision making. Oxford: Oxford University Press, pp.76-96. Mikalsen, M., Næsje, M., Reime, E.A. and Solem, A. (2019) ‘Agile autonomous teams in complex organizations’, in Hoda, R. (ed.) Agile processes in software engineering and extreme programming – workshops. Cham: Springer, pp.55-63. Miles, M.B. and Huberman, A.M. (1994) Qualitative data analysis: an expanded sourcebook. London: Sage.

199

Millett, S.M. (2011) Managing the future: A guide to forecasting and strategic planning in the 21st century. Axminster: Triarchy Press. Miner, G., Elder IV, J., Fast, A., Hill, T., Nisbet, R. and Delen, D. (2012) Practical text mining and statistical analysis for non-structured text data applications. Waltham, MA: Academic Press. Minto, B. (2009) The pyramid principle. Rev. edn. Upper Saddle River, NJ: Prentice Hall.

Mintzberg, H. (1994) Rise and fall of strategic planning. London: Prentice Hall. Misoch, S. (2015) Qualitative interviews. Oldenburg: De Gruyter. Mitchell, C., Maselli, I., Ray, R.L. and van Ark, B. (2019) ‘C-suite challenge 2019: the future-ready organization’, The Conference Board Research Report 1676. Available at: https://www.ahri.com.au/media/2147/c-suite-challenge-2019-the-future-ready-organization.pdf (Accessed: 05 July 2020). Mitchell, M. (2019) ‘Artificial intelligence hits the barrier of meaning’, Information, 10(2), p.51. Moemken, D.L. (2017) Social capital theory and self-initiated expatriates’ intention to repatriate: German expatriate academics in the United Kingdom. Doctor of Philosophy. University of Kent. Mokyr, J. (2016) A culture of growth: the origins of the modern economy. Princeton: Princeton University Press. Mol, M.J. and Birkinshaw, J. (2009) ‘The sources of management innovation: when firms introduce new management practices’, Journal of Business Research, 62(12), pp.1269-1280. Moore, D.A., Swift, S.A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H.H. and Tenney, E.R. (2017) ‘Confidence calibration in a multiyear geopolitical forecasting competition’, Management Science, 63(11), pp.3552-3565. Morgan, G. and Smircich, L. (1980) ‘The case for qualitative research’, Academy of Management Review, 5(4), pp.491-500. Morrell, K. and Learmonth, M. (2015) ‘Against evidence-based management, for management learning’, Academy of Management Learning & Education, 14(4), pp.520-533. Mu, J., Zhang, J.Z. and Gilliland, D. (2019) No silver bullet with AI: blending in firm culture and capabilities with AI marketing investment for enhancing firm performance. Marketing Science Institute Working Paper 20-109. Boston: MSI. Muntermann, J., Nickerson, R. and Varshney, U. (2015) ‘Towards the development of a taxonomic theory’, Twenty-first Americas Conference on Information Systems. Fajardo, Puerto Rico, 13-15 August. Red Hook, NY: Curran Associates Inc., pp.3536-3550. Napolitano, J. (2018) ‘The big conversations on AI and design at SXSW 2018’, McKinsey & Company, Digital McKinsey. Available at https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/digital-blog/the-big-conversations-on-ai-and-design-at-sxsw-2018 (Accessed: 19 March 2018). Nasher, J. (2018) Convinced!: How to prove your competence & win people over. San Francisco: Berrett-Koehler Publishers. Nasher, J. (2019) ‘To seem more competent, be more confident’, Harvard Business Review, 97(2), pp.74-81.

200

Nassehi, A. (2018) ‘Über Beziehungen, Elefanten und Dritte’, Soziologie – Forum der Deutschen Gesellschaft für Soziologie, 47(3), pp.292-301. Nassehi, A. (2019) Muster – Theorie der digitalen Gesellschaft. München: Verlag C.H. Beck. Nassehi, A. and Saake, I. (2002) ‘Kontingenz: Methodisch verhindert oder beobachtet?. Ein Beitrag zur Methodologie der qualitativen Sozialforschung’, Zeitschrift für Soziologie, 31(1), pp.66-86. Nayak, S.C., Misra, B.B. and Behera, H.S. (2019) ‘Efficient financial time series prediction with evolutionary virtual data position exploration’, Neural Computing and Applications, 31(2), pp.1053-1074. Nguyen, V.D., Bang Truong, H., Merayo, M.G. and Nguyen, N.T. (2019) ‘Toward evaluating the level of crowd wisdom using interval estimates’, Journal of Intelligent & Fuzzy Systems, 37(6), pp.7279-7289. Nutt, P.C. (2007) ‘Intelligence gathering for decision making’, Omega, 35(5), pp.604-622. Önkal, D., Gönül, M.S., Goodwin, P., Thomson, M. and Öz, E. (2017) ‘Evaluating expert advice in forecasting: users’ reactions to presumed vs. experienced credibility’, International Journal of Forecasting, 33(1), pp.280-297. Office of Science & Technology Policy (2019) Implementation of federal prize and citizen science authority: fiscal years 2017-18. Available at: https://www.whitehouse.gov/wp-content/uploads/2019/06/Federal-Prize-and-Citizen-Science-Implementation-FY17-18-Report-June-2019.pdf (Accessed: 03 October 2020). Olanrewaju, ’T., Smaje, K. and Willmott, P. (2014) The seven traits of effective digital enterprises. Available at: http://www.mckinsey.com/insights/organization/the_seven_traits_of_effective_digital_enterprises (Accessed: 06 August 2014). O’Leary, D.E. (2011) ‘Prediction markets as a forecasting tool’, Advances in Business and Management Forecasting, 8, pp.169-184. O’Leary, D.E. (2019) ‘Enterprise crowdsourcing innovation in the big 4 consulting firms’, Journal of Emerging Technologies in Accounting, 16(2), pp.99-118. O’Leary, D.E. (2020) ‘Driving innovation using enterprise crowdsourcing’, Journal of Information Technology Teaching Cases, 10(1), pp.2-10. Oliver, D.G., Serovich, J.M. and Mason, T.L. (2005) ‘Constraints and opportunities with interview transcription: towards reflection in qualitative research’, Social Forces, 84(2), pp.1273-1289. Ord, K., Fildes, R. and Kourentzes, N. (2017) Principles of business forecasting. 2nd edn. New York: Wessex Press. Orrell, D. and McSharry, P. (2009) ‘A systems approach to forecasting’, Foresight, 14, pp.25-30. Orrell, D. and McSharry, P. (2014) ‘A systems approach to forecasting’, in Tashman, L. (ed.). FORESIGHT Forecasting methods tutorials. Medford, MA: International Institute of Forecasters, pp.71-75. Ortmann, A. (2019) ‘Deception’, in Schram, A. and Ule, A. (eds.) Handbook of research methods and applications in experimental economics. Cheltenham: Edward Elgar Publishing, pp.28-38.

201

Osterwalder, A., Pigneur, Y., Smith, A. and Etiemble , F. (2020) The invincible company: how to constantly reinvent your organization with inspiration from the world's best business models. Hoboken, New Jersey: Wiley & Sons, Inc. Ovcharov, E.Y. (2018) ‘Proper scoring rules and Bregman divergence’, Bernoulli, 24(1), pp.53-79. Ozimek, A. (2014) The regulation and value of prediction markets. George Mason University Working Paper. Available at: http://mercatus.org/publication/regulation-and-value-prediction-markets (Accessed: 06 August 2015). Page, L. and Siemroth, C. (2017) ‘An experimental analysis of information acquisition in prediction markets’, Games and Economic Behavior, 101, pp.354-378. Page, L. and Siemroth, C. (2019) How much information is incorporated in financial asset prices? Experimental evidence. SSRN Working Paper. 3130307. Rochester, NY: SSRN. Palak, R. and Nguyen, N.T. (2017) ‘Prediction markets as a vital part of collective intelligence’, INnovations in Intelligent SysTems and Applications (INISTA), 2017 IEEE International Conference. Gdynia, Poland, 3-5 July. New York: IEEE, pp.137-142. Palmer, D. and Hugo, A. (2013) ‘Gamification goes to work’, in White, M.E. and Hillard, R. (eds.) Tech trends 2013 – elements of postdigital. New York: Deloitte Development LLC, pp.60-68. Pan, X., He, S., Zhu, X. and Fu, Q. (2016) ‘How users employ various popular tags to annotate resources in social tagging: an empirical study’, Journal of the Association for Information Science and Technology, 67(5), pp.1121-1137. Pant, P.N. and Starbuck, W.H. (1990) ‘Innocents in the forest: forecasting and research methods’, Journal of Management, 16(2), pp.433-460. Parente, F. and Finley, J.C. (2021) ‘Quantitative and qualitative methods for predicting geopolitical events’ in Sinnott, J.D. and Rabin, J.S. (eds.) The psychology of political behavior in a time of change. Cham: Springer, pp.597-611. Passerini, K., Osatuyi, B. and Stipe, W. (2020) ‘Giving and taking in online communities of practice: the role of geography and culture in knowledge sharing and innovation’, Innovation and Entrepreneurship Theory and Practice, DOI: 10.24251/HICSS.2020.58. Pathak, D., Rothchild, D. and Dudik, M. (2015) ‘A comparison of forecasting methods: fundamentals, polling, prediction markets and experts’, Journal of Prediction Markets, 9(2), pp.1-31. Paul, R. and Weinbach, A. (2015) ‘Using prediction market prices to differentiate factors that influence the highest and lowest priced tickets in dynamic pricing for major league baseball’, Journal of Prediction Markets, 9(2), pp.43-63. Pauw, C. (2018) ‘Prediction markets, explained’, Cointelegraph, 20 February. Available at: https://cointelegraph.com/explained/south-korea-and-crypto-regulationsexplained (Accessed: 31 March 2020). Pearl, J. and Mackenzie, D. (2019) The book of why: the new science of cause and effect. 2nd edn. Munich: Penguin Books. Peeters, T. (2018) ‘Testing the wisdom of crowds in the field: transfermarkt valuations and international soccer results’, International Journal of Forecasting, 34(1), pp.17-29.

202

Pennings, C.L., van Dalen, J. and Rook, L. (2019) ‘Coordinating judgmental forecasting: coping with intentional biases’, Omega, 87, pp.46-56. Petrakis, P.E. and Konstantakopoulou, D. (2016) Uncertainty in entrepreneurial decision making: the competitive advantages of strategic creativity. Heidelberg: Springer Verlag. Pfeffer, J. (2013) ‘You're still the same: why theories of power hold over time and across contexts’, The Academy of Management Perspectives, 27(4), pp.269-280. Pfeffer, J. (2015) Leadership BS – fixing workplaces and careers one truth at a time. New York: Harper Collins Publishers. Phillips, F. and Linstone, H. (2016) ‘Key ideas from a 25-year collaboration at technological forecasting & social change’, Technological Forecasting and Social Change, 105, pp.158-166. Phillips, L. and Rozworski, M. (2019) The people’s republic of Walmart: how the world's biggest corporations are laying the foundation for socialism. London: Verso. Pierce, E. and Thomas, L. (2007) ‘Assessing information quality using prediction markets’, Proceedings of the 12th International Conference on Information Quality. MIT, Cambridge, MA, USA, 9-11 November. Cambridge, MA: MIT, pp.11-25. Piper, K. (2020) ‘Predictions are hard, especially about the coronavirus’, Vox, 8 April. Available at: https://www.vox.com/future-perfect/2020/4/8/21210193/coronavirus-forecasting-models-predictions (Accessed: 6 January 2021). Pisano, G.P. and Verganti, R. (2008) ‘Which kind of collaboration is right for you’, Harvard Business Review, 86(12), pp.78-86. Pittaway, L., Aissaoui, R., Ferrier, M. and Mass, P. (2019) ‘University spaces for entrepreneurship: a process model’, International Journal of Entrepreneurial Behavior & Research, 25(2), pp.1-47. Pluchinotta, I., Kazakçi, A.O., Giordano, R. and Tsoukiàs, A. (2019) ‘Design theory for generating alternatives in public decision making processes’, Group Decision and Negotiation, 28(2), pp.341-375. Polit, D.F. and Beck, C.T. (2012) Nursing research: principles and methods. Philadelphia, PA: Lippincott Williams & Wilkins. Powell, W.A., Hanson, R., Laskey, K.B. and Twardy, C. (2013) ‘Combinatorial prediction markets: an experimental study’, Scalable Uncertainty Management 7th International Conference, SUM 2013. University of Maryland, Washington DC, September 2013. Heidelberg: Springer Verlag, pp.283-296. Prestwich, S.D. (2019) ‘Tuning forecasting algorithms for Black Swans’, IFAC-PapersOnLine, 52(13), pp.1496-1501. Price, E. (2007) ‘Effective process planning and scheduling’, Aspen Technology Publication. Available at: https://www.aspentech.com/publication_files/Control_Engineering_Effective_Process_Planning_Scheduling_1107.pdf (Accessed: 29 April 2016). Price, J.H. and Murnan, J. (2004) ‘Research limitations and the necessity of reporting them’, American Journal of Health Education, 35(2), p.66. Prokesch, T., Heiko, A. and Wohlenberg, H. (2015) ‘Integrating prediction market and Delphi methodology into a foresight support system – insights from an online game’, Technological Forecasting and Social Change, 97, pp.47-64.

203

Puccio, G.J., Burnett, C., Acar, S., Yudess, J.A., Holinger, M. and Cabra, J.F. (2020) ‘Creative problem solving in small groups: the effects of creativity training on idea generation, solution creativity, and leadership effectiveness’, The Journal of Creative Behavior, 54(2), pp.453-471. Puccio, G.J., Murdock, M.C. and Mance, M. (2005) ‘Current developments in creative problem solving for organizations: A focus on thinking skills and styles’, Korean Journal of Thinking and Problem Solving, 15(2), pp.43-76. Qiu, L. and Kumar, S. (2017) ‘Understanding voluntary knowledge provision and content contribution through a social-media-based prediction market: a field experiment’, Information Systems Research, 28(3), pp.529-546. Qiu, L., Rui, H. and Whinston, A. (2013) ‘Social network-embedded prediction markets: the effects of information acquisition and communication on predictions’, Decision Support Systems, 55(4), pp.978-987. Qiu, L., Rui, H. and Whinston, A. (2014) ‘Effects of social networks on prediction markets: examination in a controlled experiment’, Journal of Management Information Systems, 30(4), pp.235-268. Raeside, R., Peisl, T. and Canduela, J. (2019) ‘The importance of trust for interface quality and acceptance’, European Conference on Software Process Improvement. Edinburgh, United Kingdom, 18-20 September. Cham: Springer, pp.672-681. Razmerita, L., Kirchner, K. and Nabeth, T. (2014) ‘Social media in organizations: leveraging personal and collective knowledge processes’, Journal of Organizational Computing and Electronic Commerce, 24(1), pp.74-93. Reade, J.J. and Vaughan Williams, L. (2019) ‘Polls to probabilities: comparing prediction markets and opinion polls’, International Journal of Forecasting, 35(1), pp.336-350. Reeves, M., Love, C. and Tillmanns, P. (2012) ‘Your strategy needs a strategy’, Harvard Business Review, 90(9), pp.76-83. Reichheld, F.F. (2003) ‘The one number you need to grow’, Harvard Business Review, 81(12), pp.46-55. Remidez, H., Stodnick, M. and Beldona, S. (2019) ‘Exploring the use of prediction markets as digital games to support learning in a project management context’ in Tan, W.H. (ed.) Design, motivation, and frameworks in game-based learning. Hershey, PA: IGI Global, pp.239-260. Restocchi, V., McGroarty, F. and Gerding, E. (2018) ‘The temporal evolution of mispricing in prediction markets’, Finance Research Letters, doi:10.1016/j.frl.2018.08.003. Restocchi, V., McGroarty, F. and Gerding, E. (2019) ‘The stylized facts of prediction markets: analysis of price changes’, Physica A: Statistical Mechanics and its Applications, 515(1), pp.159-170. Richard, M. and Vecer, J. (2021) ‘Efficiency testing of prediction markets: Martingale approach, likelihood ratio and Bayes factor analysis’, Risks, 9(2), pp.31-50. Rieg, R. and Schoder, R. (2010) ‘Forecasting accuracy: comparing prediction markets and surveys - an experimental study’, Journal of Prediction Markets, 4(3), pp.1-19. Rieg, R. and Schoder, R. (2011) ‘Corporate prediction markets: pitfalls and barriers’, Foresight, Issue 21(Spring), pp.35-40.

204

Riekhof, H.C. and Brinkhoff, S. (2014) Absatzprognosen: eine empirische Bestandsaufnahme der unternehmerischen Praxis. PFH Forschungspapiere/Research Papers, No. 2014/04, PFH Private Hochschule Göttingen. Riekhof, H.C., Riekhof, M.C. and Brinkhoff, S. (2012) Predictive Markets: Ein vielversprechender Weg zur Verbesserung der Prognosequalität im Unternehmen?. PFH Forschungspapiere/Research Papers, No. 2012/07, PFH Private Hochschule Göttingen. Ries, E. (2017) The startup way: how modern companies use entrepreneurial management to transform culture and drive long-term growth. New York: Currency. Rigby, D.K. (2013) ‘The future of shopping’, in Best of Bain – from the pages of Harvard Business Review. Boston: Bain & Company. Rigby, D.K. (2015) Management tools 2015: an executive's guide. Boston: Bain & Company. Rigby, D.K. (2017) Management tools 2017: an executive's guide. Boston: Bain & Company. Rigby, D.K. and Bilodeau, B. (2013) Management tools & trends 2013. Boston: Bain & Company. Rigby, D.K. and Bilodeau, B. (2015) Management tools & trends 2015. Boston: Bain & Company. Rigby, D.K. and Bilodeau, B. (2018) Management tools & trends. Boston: Bain & Company. Rigby, D.K., Elk, S. and Berez, S. (2020) Doing agile right – transformation without chaos. Boston: Harvard Business Review Press. Rigby, D.K., Gruver, K. and Allen, J. (2013) ‘Innovation in turbulent times’, in Maceda, M. (ed.) Best of Bain – from the pages of Harvard Business Review. Boston: Bain & Company. Rinaudo, E.K. (2020) ‘Bias busters: lifting your head from the sand’, McKinsey Insights. Available at: https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/bias-busters-lifting-your-head-from-the-sand (Accessed: 26 October 2020). Robertson, E., O’Grady, Á., Barton, J., Galloway, S., Emmanuel-Yusuf, D., Leach, M., Hammond, G., Thomson, M. and Foxon, T. (2017) ‘Reconciling qualitative storylines and quantitative descriptions: an iterative approach’, Technological Forecasting and Social Change, 118, pp.293-306. Robson, D. (2020) Prediction - can you learn to navigate uncertainty? Available at: https://www.bbc.com/future/article/20200619-why-you-are-better-at-predicting-the-future-than-you-think (Accessed: 22 June 2020). Rötheli, T. (2017) ‘Superforecasting: the art and science of prediction’, Foresight, 19(2), pp.256-258.

Rogers, E.M. (2010) Diffusion of innovations. 4th edn. New York: Simon and Schuster. Rogers, J. and Révész (2020) Experimental and quasi-experimental designs – The Routledge handbook of research methods in applied linguistics. New York: Routledge. Roley, P. (2006) ‘Strategic management for competitive advantage’, Harvard Business Review, 108(9), pp.163-123.

205

Romney, A.K., Weller, S.C. and Batchelder, W.H. (1986) ‘Culture as consensus: a theory of culture and informant accuracy’, American Anthropologist, 88(2), pp.313-338. Rona-Tas, A. (2020) ‘Predicting the future: art and algorithms’, Socio-Economic Review, 18(3), pp.893-911. Rosas, S.R. (2019) ‘Consulting: applying your skills in new arenas’, in Brown Urban, J. and Linver, M.R. (eds.) Building a career outside academia: a guide for doctoral students in the behavioral and social sciences. Washington, DC: American Psychological Association, pp.67-78. Rostami-Tabar, B., Ali, M.M., Hong, T., Hyndman, R.J., Porter, M.D. and Syntetos, A. (2020) ‘Forecasting for social good’. To be published in ACM SIGCAS Computers and Society [Preprint]. arXiv:2009.11669. Rothschild, D. and Wolfers, J. (2013) Forecasting elections: voter intentions versus expectations. SSRN Working Paper 1884644. Rochester, NY: SSRN. Roughgarden, T. (2016) ‘Prediction markets’. CS269I: Incentives in Computer Science Lecture# 18. Stanford University. Available at: https://theory.stanford.edu/~tim/f16/l/l18.pdf (Accessed: 30 March 2020). Roughgarden, T. (2018) ‘Markets, everywhere’. CS269I: Incentives in Computer Science Lecture# 3. Stanford University. Available at: https://theory.stanford.edu/~tim/f16/l/l3.pdf (Accessed: 30 March 2020).

Rozwell, C. (2013) Telephone interview with Thomas Wolfram, 25 January. Ruberry, M.E. (2013) Prediction markets: theory and applications. Unpublished PhD thesis. Harvard University. Rudestam, K.E. and Newton, R.R. (2014) Surviving your dissertation: a comprehensive guide to content and process. 3rd edn. London: Sage Publications. Rumelt, R.P. (2017) Good strategy / bad strategy: the difference and why it matters. 4th edn. London: Profile Books Ltd. Rustler, F. (2019) Denkwerkzeuge der Kreativität und Innovation – Das kleine Handbuch der Innovationsmethoden. 9th edn. Zürich: Midas Management Verlag AG. Saldaña, J. (2015) The coding manual for qualitative researchers. 3rd edn. London: Sage Publications Ltd. Salgado, S.R., Starbuck, W.H. and Mezias, J.M. (2002) ‘The accuracy of managers’ perceptions: a dimension missing from theories about firms’, in Augier, M. and March, J.G. (eds.) The economics of choice, change and organization. Essays in memory of Richard M. Cyert. Cheltenham: Edward Elgar, pp.168-185. Salzsieder, T. (2019) ‘Unsere Formel für den IT-Erfolg’, in IDG Business Media GmbH (ed.) Jahrbuch 2020 – Prognosen zur Zukunft der IT. München: IDG Business Media GmbH, pp.52-55. Santillán-Salgado, R.J., Ulin-Lastra, M.G. and Escobar-Saldivar, L.J. (2018) ‘Polls, prediction markets, and financial variables’, Revista Mexicana de Economía y Finanzas Nueva Época REMEF, 13(3), pp.295-323. dos Santos Pinheiro, L. and Dras, M. (2017) ‘Stock market prediction with deep learning: a character-based neural language model for event-based trading’, in Wong, J.S. and Haffari, G. (eds.) Proceedings of the Australasian Language Technology Association Workshop 2017. Sydney: Australasian Language Technology Association, pp.6-15.

206

Sanzieri, F., Hall, A. and Anderson, P. (2019) ‘Future forecasting wicked problems: a new framework for design’, in Colombo, S., Lim, Y., Bruns Alonso, M., Chen, L.L., Djajadiningrat, T., Feijs, L., Hu, J., Kyffin, S., Özcan, E., Rampino, L., Ramirez, E.R. and Steffen, D. (eds.) Design and Semantics of Form and Movement, DesForm 2019: Beyond Intelligence. Newcastle: Northumbria University, pp.26-34. Scase, R. and Goffee, R. (2017) Reluctant managers (Routledge revivals): their work and lifestyles. London: Routledge. Schadler, T., Bernoff, J. and Ask, J. (2014) The mobile mind shift: engineer your business to win in the mobile moment. Cambridge, MA: Groundswell Press. Schaer, O. (2019) Business forecasting with online buzz. PhD thesis. Lancaster University. Doi: 10.17635/lancaster/thesis/745. Schein, E.H. (2010) Organizational culture and leadership. 4th edn. San Francisco: Jossey-Bass. Scheiner, C.W., Haas, P., Leicht, N. and Voigt, K. (2013) ‘Accessing knowledge with a game - a meta-analysis of prediction markets’, WAINA, 2013, 27th Advanced Information Networking and Applications Workshops. Barcelona, 25-28 March. New York: IEEE, pp.1489-1494.

Scheuss, R. (2012) Telephone interview with Thomas Wolfram, 27 November. Schnell, R. (2019) Survey-interviews. Wiesbaden: Springer Fachmedien. Schoemaker, P.J. and Tetlock, P.E. (2016) ‘Superforecasting: how to upgrade your company’s judgment’, Harvard Business Review, 94, pp.72-78. Schoen, H., Gayo-Avello, D., Takis Metaxas, P., Mustafaraj, E., Strohmaier, M. and Gloor, P. (2013) ‘The power of prediction with social media’, Internet Research, 23(5), pp.528-543. Schoenebeck, G., Yu, C. and Yu, F.Y. (2020) ‘Timely information from prediction markets’. To be published in 4th Workshop on Mechanism Design for Social Good [Preprint]. arXiv:2011.03645. Schrieber, J.M. (2004) The application of prediction markets to business. Master of Engineering in Logistics. Massachusetts Institute of Technology. Schuller, B., Ganascia, J.G. and Devillers, L. (2016) ‘Multimodal sentiment analysis in the wild: ethical considerations on data collection, annotation, and exploitation’, in Dimeji, F. (ed.) Actes du workshop on ethics in corpus collection, annotation & application (ETHI-CA2). Portoroz, Slovénie: LREC, pp.27-35. Schulz, M., Neuhaus, U., Kaufmann, J., Badura, D., Kerzel, U., Welter, F., Prothmann, M., Kühnel, S., Passlick, J., Rissler, R., Badewitz, W., David Dann, D., Gröschel, A., Kloker, S., Alekozai, E.M., Felderer, M., Lanquillon, C., Brauner, D., Gölzer, P., Binder, H., Rohde, H. and Gehrke, N. (2020) DASC-PM v1. 0-Ein Vorgehensmodell für Data-Science-Projekte. DOI: 10.25673/32872. Schumann, S. (2019) Repräsentative Umfrage: Praxisorientierte Einführung in empirische Methoden und statistische Analyseverfahren. 7th edn. Oldenburg: de Gruyter. Schumpeter, J. (1942) ‘Creative destruction’, Capitalism, Socialism and Democracy, 825, pp.82-85.

207

Schwab, A. and Starbuck, W.H. (2013) ‘Why baseline modelling is better than null-hypothesis testing: examples from international business research’, in Devinney, T.M., Pedersen, T. and Tihanyi, L. (eds.) Philosophy of science and meta-knowledge in international business and management. Bingley: Emerald Group Publishing Limited, pp.171-195. Schwab, A. and Starbuck, W.H. (2016) ‘Collegial “nests” can foster critical thinking, innovative ideas, and scientific progress’, Strategic Organization, 14(2), pp.167-177. Seemann, T. (2009) Prediction markets as forecasting tool in decision processes. Dr. rer. pol. Friedrich-Alexander-Universität Erlangen-Nürnberg.

Seemann, T. (2013) Telephone interview with Thomas Wolfram, 22 January. Seiler, D. (2018) E-Mail to Thomas Wolfram, 27 November. Seraei, S.A.P. (2020) ‘KPIs im Content Marketing’, in Wesselmann, M. (ed.) Content gekonnt. Wiesbaden: Springer Gabler, pp.279-294. Servan-Schreiber, E. (2012) ‘Trading uncertainty for collective wisdom’, in Landemore, H. and Elster, J. (eds.) Collective wisdom: principles and mechanisms. Boston: Cambridge University Press. Shams, S.M.R. and Kaufmann, H.R. (2016) ‘Entrepreneurial co-creation: a research vision to be materialised’, Management Decision, 54(6), pp.1250-1268. Shchory, N.B. (2020) ‘Information asymmetries in e-commerce: the challenge of credence qualities’, Journal of High Technology Law, XX(1), pp.1-60. Shen, E.Z., and Winstanley, C.R. (2019) ‘Modeling prediction markets with dynamic Bayesian networks’ Assignment for CS221 – Artificial Intelligence: Principles and Techniques, Masters in Computer Science, Masters in Electrical Engineering, Stanford University. Unpublished. Shrier, D., Adjodah, D., Wu, W. and Pentland, A. (2016) Prediction markets. Cambridge, MA: MIT Connection Science publication. Shuttleworth, M. (2009) Research bias. Available at: https://explorable.com/research-bias#main-content (Accessed: 13 April 2019). Siba, T.K. and Prakash, A. (2017) ‘Block-chain: an evolving technology’, Global Journal of Enterprise Information System, 8(4), pp.29-35. Sibony, O. (2013) Telephone interview with Thomas Wolfram, 03 May.

Sibony, O. (2017) Skype interview with Thomas Wolfram, 14 March. Sibony, O. (2020a) You’re about to make a terrible mistake: how biases distort decision-making and what you can do to fight them. New York: Hachette Book Group. Sibony, O. (2020b) Vous allez redécouvrir le management! Paris: Flammarion. Sibony, O. (2021) 5 idées reçues sur le management. Available at https://www.linkedin.com/feed/update/urn%3Ali%3Aactivity%3A6760233934702874624/ (Accessed: 29 January 2021). Siegel, A. (2018) Using prediction markets to track information flows: evidence from Google. Available at: https://news.ycombinator.com/item?id=17015055 (Accessed: 18 August 2020). Siegel, E. (2016) Predictive analytics: the power to predict who will click, buy, lie, or die. Hoboken, NJ: John Wiley & Sons.

208

Siemon, D. (2019) IT-supported collaborative creativity. Dr. rer. pol. Technische Universität Braunschweig. Silberg, J. and Maryilka, J. (2019) Tackling bias in artificial intelligence (and in humans). New York: McKinsey Global Institute. Silver, N. (2012) The signal and the noise: why so many predictions fail - but some don’t. New York: Penguin Press. Silverman, D. (2013) Doing qualitative research: a practical handbook. 4th edn. London: Sage Publications Ltd. Silverman, D. (2016) Qualitative research. 4th edn. London: Sage Publications Ltd. Silverman, D. (2017) ‘How was it for you? The interview society and the irresistible rise of the (poorly analyzed) interview’, Qualitative Research, 17(2), pp.144-158. Silverman, D. (2019) Interpreting qualitative data. 6th edn. London: Sage Publications Ltd.

Simon, H.A. (2013) Administrative behavior. New York: Simon and Schuster. Simon, H. and Gathen, A. (2010) Das große Handbuch der Strategieinstrumente. Frankfurt: Campus-Verlag. Šindelář, J. (2014) ‘Determinants of sales forecasting performance in retail chains: an empirical study from Central-Eastern Europe’, International Journal of Economic Practices and Theories, 4(1), pp.1-16. Singleton, C., Reade, J.J. and Brown, A. (2019) ‘Going with your gut: the (in) accuracy of forecast revisions in a football score prediction game’, Journal of Behavioral and Experimental Economics. DOI: 10.1016/j.socec.2019.101502. Skiera, B. and Spann, M. (2011) ‘Using prediction markets in new product development’ in Vaughan Williams, L. (ed.) Prediction markets – theory and applications. New York: Routledge, pp.75-86. Skinnarland, S. (2013) Use of progression planning tools in developing collaborative main contractor-subcontractor relationships in Norway. Doctoral Thesis. Heriot-Watt University. Slingerland, E. (2018) ‘Who’s afraid of reductionism? Methodological naturalism and the academic study of religion’, in Blum, J.N. (ed.) The question of methodological naturalism. Leiden: Brill Publishers, pp.167-205. Smaje, K. (2020) How six companies are using technology and data to transform themselves. Available at: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/how-six-companies-are-using-technology-and-data-to-transform-themselves (Accessed: 30 August 2020). Smith, J.E. and Saritas, O. (2011) ‘Science and technology foresight baker's dozen: a pocket primer of comparative and combined foresight methods’, Foresight: The Journal of Future Studies, Strategic Thinking and Policy, 13(2), pp.79-96. Snowberg, E. and Wolfers, J. (2010) ‘Explaining the favorite–long shot bias: is it risk-love or misperceptions?’, Journal of Political Economy, 118(4), pp.723-746. Snowberg, E., Wolfers, J. and Zitzewitz, E. (2013) ‘Prediction markets for economic forecasting’, in Elliott, G. and Timmermann, A. (eds.) Handbook of economic forecasting (Volume 2A). Amsterdam: Elsevier, pp.657-687.

209

Sohrabpour, V., Oghazi, P., Toorajipour, R. and Nazarpour, A. (2020) ‘Export sales forecasting using artificial intelligence’. To be published in Technological Forecasting and Social Change [Preprint]. DOI: 10.1016/j.techfore.2020.120480. Solaimani, S., Talab, A.H. and van der Rhee, B. (2019) ‘An integrative view on lean innovation management’, Journal of Business Research, 105, pp.109-120. Solomon, M. (2006) ‘Groupthink versus the wisdom of crowds: the social epistemology of deliberation and dissent’, The Southern Journal of Philosophy, 44(S1), pp.28-42. Sopra Steria (2020) Potenzialanalyse Cloud in Europa. Available at https://www.soprasteria.de/docs/librariesprovider2/sopra-steria-de/publikationen/managementkompass/managementkompass-cloud-in-europa-expos%C3%A9.pdf?sfvrsn=247837dc_2 (Accessed: 12 August 2020). Soukhoroukova, A., Spann, M. and Skiera, B. (2012) ‘Sourcing, filtering, and evaluating new product ideas: an empirical exploration of the performance of idea markets’, Journal of Product Innovation Management, 29(1), pp.100-112. Spann, M. and Skiera, B. (2003) ‘Internet-based virtual stock markets for business forecasting’, Management Science, 49(10), pp.1310-1326. Spiegelhalter, D. (2019) The art of statistics – learning from data. London: Penguin Random House. Spiegler, S.V., Heinecke, C. and Wagner, S. (2019) ‘The influence of culture and structure on autonomous teams in established companies’, International Conference on Agile Software Development. Montréal, QC, Canada, 21-25 May. Cham: Springer, pp.46-54. Spielmann, K. (2016) ‘I got algorithm: can there be a Nate Silver in intelligence?’, International Journal of Intelligence and CounterIntelligence, 29(3), pp.525-544. Sprenger, R.K. (2014) Mythos Motivation: Wege aus einer Sackgasse. 20th edn. Frankfurt am Main: Campus Verlag GmbH. Spyrou, S. (2013) ‘Herding in financial markets: a review of the literature’, Review of Behavioral Finance, 5(2), pp.175-194. Starbuck, W.H. (1993) ‘Strategizing in the real world’, International Journal of Technology Management, 8(1-2), pp.77-85. Starbuck, W.H. (2006a) ‘Strategizing realistically in competitive environments’, in Starbuck, W.H. (ed.) Organizational realities: studies of strategizing and organizing. Oxford: Oxford University Press, pp.349-357. Starbuck, W.H. (2006b) The production of knowledge: the challenge of social science research. Oxford: Oxford University Press. Starbuck, W.H. (2009) ‘Perspective—cognitive reactions to rare events: perceptions, uncertainty, and learning’, Organization Science, 20(5), pp.925-937. Starbuck, W.H. (2013) ‘James Gardner March: founder of organization theory, decision theorist, and advocate of sensible foolishness’, European Management Journal, 31(1), pp.88-92. Starbuck, W.H. (2014a) ‘Why corporate governance deserves serious and creative thought’, Academy of Management Perspectives, 28(1), pp.15-21. Starbuck, W.H. (2014b) ‘Five stories that illustrate three generalizations about radical innovations’, Industrial Marketing Management, 43(8), pp.1278-1283.

210

Starbuck, W.H. (2015) ‘Karl E. Weick and the dawning awareness of organized cognition’, Management Decision, 53(6), pp.1287-1299. Starbuck, W.H. (2017) ‘Organizational learning and unlearning’, The Learning Organization, 24(1), pp.30-38. Starbuck, W.H. and Hedberg, B. (2001) ‘How organizations learn from success and failure’, in Dierkes, M., Berthoin Antal, A., Child, J. and Nonaka, I. (eds.) Handbook of organizational learning and knowledge. Oxford: Oxford University Press, pp.327-350. Starbuck, W.H. and Mezias, J.M. (1996) ‘Opening Pandora's box: studying the accuracy of managers' perceptions’, Journal of Organizational Behavior, 17(2), pp.99-117. Steinhöfer, D. (2020) Liberating Structures: Entscheidungsfindung revolutionieren. München: Verlag Franz Vahlen Stephens-Davidowitz, S. (2017) Everybody lies: big data, new data, and what the internet can tell us about who we really are. New York: Dey Street Books. Stanko, M.A. and Henard, D.H. (2016) ‘How crowdfunding influences innovation’, MIT Sloan Management Review, 57(3), pp.15-17. Stanko, M.A. and Henard, D.H. (2017) ‘Toward a better understanding of crowdfunding, openness and the consequences for innovation’, Research Policy, 46(4), pp.784-798. Staples, M. (2020) ‘COVID-19: Implications for business’, McKinsey Insights. Available at: https://www.mckinsey.com/business-functions/risk/our-insights/covid-19-implications-for-business?cid=other-eml-alt-mip-mck&hdpid=e9f0ec3f-fee7-4cd5-ab6d-f525c6299e13&hctky=2707809&hlkid=4ef03585b0484a16b91b12b4b64e9ba9 (Accessed: 05 November 2020) Stershic, A. and Gujral, K. (2020) ‘Arbitrage in political prediction markets’, Journal of Prediction Markets, 14(1), pp.69-104.

Stier, W. (1999) Empirische Forschungsmethoden. Berlin: Springer Verlag. Straubhaar, T. (2019) Die Stunde der Optimisten: So funktioniert die Wirtschaft der Zukunft. Hamburg: Edition Körber. Strijbis, O. and Arnesen, S. (2019) ‘Explaining variance in the accuracy of prediction markets’, International Journal of Forecasting, 35(1), pp.408-419. Struhl, S. (2017) Artificial intelligence marketing and predicting consumer choice: an overview of tools and techniques. London: Kogan Page Publishers. Štrumbelj, E. (2016) ‘A comment on the bias of probabilities derived from betting odds and their use in measuring outcome uncertainty’, Journal of Sports Economics, 17(1), pp.12-26. Studer, S. (2021) ‘Generation Xi – special report Chinese youth’, The Economist, 51(4), pp.1-12. Sung, M.C., McDonald, D.C., Johnson, J.E., Tai, C.C. and Cheah, E.T. (2019) ‘Improving prediction market forecasts by detecting and correcting possible over-reaction to price movements’, European Journal of Operational Research, 272(1), pp.389-405. Sunstein, C.R. (2006) ‘Deliberating groups vs. prediction markets (or Hayek's challenge to Habermas)’, Episteme: A Journal of Social Epistemology, 3(3), pp.192-213.

211

Sunstein, C.R. and Hastie, R. (2015) Wiser: getting beyond groupthink to make groups smarter. Boston: Harvard Business Review Press. ‘Superforecaster’ (2020) Wikipedia. Available at: https://en.wikipedia.org/wiki/Superforecaster (Accessed: 06 March 2020). Suran, S., Pattanaik, V. and Draheim, D (2020) ‘Frameworks for collective intelligence: a systematic literature review’, ACM Computing Surveys (CSUR), 53(1), pp.1-36. Surowiecki, J. (2004) The wisdom of crowds. New York: Doubleday. Syed, M. (2019) Rebel ideas: the power of diverse thinking. New York: Penguin Random House. Sykes, S. (2021) ‘Gamification with Stephen Sykes’, Amazing Outcomes Podcast, 15 February. Available at: https://web.microsoftstream.com/video/7ea06548-35fa-4d04-aeb7-ab0565ded64e?referrer=https:%2F%2Fvg.lightning.force.com%2F (Accessed: 17 February 2021). Syperek, S.J. (2020) Kundenbefragung vom Profi. Available at: http://www.kundenbefragung-vom-profi.de/was-kostet-mich-das/ (Accessed: 28 September 2020). System1 (2020) Our mission. Available at: https://system1group.com/work/our-mission#:~:text=To%20empower%20every%20marketer%20to,profitable%20growth%20for%20their%20business (Accessed: 09 April 2020). Taherdoost, H. (2018) ‘A review of technology acceptance and adoption models and theories’, Procedia Manufacturing, 22(2018), pp.960-967. Tai, M. (2020) Exploring avenues toward improving prediction markets as tools to forecast paper replication probability. Bachelor thesis. Harvard College. Takac, C., Hinz, O. and Spann, M. (2011) ‘The social embeddedness of decision making: opportunities and challenges’, Electronic Markets, 21(3), pp.185-195. Takács, K., Bravo, G. and Squazzoni, F. (2018) ‘Referrals and information flow in networks increase discrimination: a laboratory experiment’, Social Networks, 54, pp.254-265. Taleb, N.N. (2012) Antifragile: things that gain from disorder. New York: Random House. Tello, C.N. (2018) ‘Knowledge generation and exchange of ideas – a proposal to strategize an idea market in Europe by using digital platforms and artificial intelligence’, Der Donauraum, 58(3-4), pp.141-154. Teschner, F. and Rothschild, D. (2013) ‘Simplifying market access: a new confidence-based interface’, Journal of Prediction Markets, 6(3), pp.27-41. Tetlock, P.E. and Gardner, D. (2015) Superforecasting: the art and science of prediction. New York: Crown Publishers. Tetlock, P.E., Mellers, B.A., Rohrbaugh, N. and Chen, E. (2014) ‘Forecasting tournaments: tools for increasing transparency and improving the quality of debate’, Current Directions in Psychological Science, 23(4), pp.290-295. The Economist (2013) ‘Schumpeter – it’s complicated’, 23 November, p.67.

The Economist (2019) ‘AI at Amazon – the learning machine, 13 April, p.61. The Economist (2020) ‘Polling error redux – whiffing twice’, 7 November, p.38.

212

The Journal of Prediction Markets (2020) Available at: http://ubplj.org/index.php/jpm (Accessed: 22 July 2020). The World Bank (2010) Theories of behavior change. Washington, DC: External Affairs Vice Presidency. Thicke, M. (2017) ‘Prediction markets for science: is the cure worse than the disease?’, Social Epistemology, pp.1-17. Thom, N. and Brem, A. (2019) ‘Vom Betrieblichen Vorschlagswesen zum integrierten Ideenmanagement – Ein- und Ausblicke aus zwei Forschergenerationen’ in Landmann, N. and Schat, H-D. (eds.) Ideen erfolgreich managen. Wiesbaden: Springer Gabler, pp.9-16. Thompson, D.N. (2012) Oracles: how prediction markets turn employees into visionaries. Boston: Harvard Business Review Press Thorén, K. and Vendel, M. (2019) ‘Backcasting as a strategic management tool for meeting VUCA challenges’, Journal of Strategy and Management, 12(2), pp.298-312. Tiberius, V. and Rasche, C. (2011) ‘Prognosemärkte’, Zeitschrift für Planung & Unternehmenssteuerung, 21(4), pp.467-472. Tiberius, V., Siglow, C. and Sendra-García, J. (2020) ‘Scenarios in business and management: the current stock and research opportunities’, Journal of Business Research, 121, pp.235-242. Tieben, H. (2017) Prediction Markets. Literaturüberblick und vergleichende Analyse zu Crowdvoting. Bachelor Wirtschaftsinformatik. Universität Paderborn. Tindale, R.S., Winget, J.R. and Hinsz, V.B. (2020) ‘Distributed cognition in teams is influenced by type of task and nature of member interactions’ in McNeese, M., Salas, E. and Endsley, M.R. (eds.) Foundations and theoretical perspectives of distributed team cognition. Boca Raton: CRC Press, pp.91-106. Tischler, R., Fuchs, C. and Engel, A. (2018) Predictive Planning and Forecasting hebt die Unternehmensplanung auf die nächste Stufe. Kolbermoor: CXP Group. Tobaccowala, R. (2020) Restoring the soul of business: staying human in the age of data. Nashville, TN: HarperCollins Leadership. Tomo, A., Mangia, G., Consiglio, S. and Canonico, P. (2019) ‘Innovation in knowledge-based professional service firms. An integrated conceptual model’, Technology Analysis & Strategic Management, 31(9), 1118-1136. Tourish, D. (2019) Management studies in crisis: fraud, deception and meaningless research. Cambridge: Cambridge University Press. Travers, M. (2006) ‘Postmodernism and qualitative research’, Qualitative Research, 6(2), pp.267-273. Treves, L. (2017) The maturity of United Kingdom (UK) accounting actors in Software as a Service (SaaS) and their readiness for automated integrations. Master’s Degree Program in Supply Management. Lappeenranta University of Technology. Turner, D.W. (2010) ‘Qualitative interview design: a practical guide for novice investigators’, The Qualitative Report, 15(3), pp.754-760. Turner, E. (2017) Session 2: disruption in business. Available at: http://idedisrupt.com/speakers/elliot-turner/ (Accessed: 20 August 2020).

213

Turner, S., D’Lima, D., Hudson, E., Morris, S., Sheringham, J., Swart, N. and Fulop, N.J. (2017) ‘Evidence use in decision-making on introducing innovations: a systematic scoping review with stakeholder feedback’, Implementation Science, 12(1), pp.145-152. Tziralis, G. and Tatsiopoulos, I. (2006) ‘Prediction markets: aspects on transforming a business game into a valuable information aggregation tool’, Proceedings of the Future Business Technology Conference (FTC) 2006. Imperial Hotel, Athens, Greece, 17-19 April. Basel: Springer Nature Switzerland AG, pp.35-39. Tziralis, G. and Tatsiopoulos, I. (2007) ‘Prediction markets: an extended literature review’, Journal of Prediction Markets, 1(1), pp.75-91. Ungar, L., Mellers, B., Satopää, V., Baron, J., Tetlock, P., Ramos, J. and Swift, S. (2012) The Good Judgment Project: a large scale test of different methods of combining expert predictions. AAAI Technical Report FS-12-06. Available at: http://people.uvawise.edu/pww8y/Supplement/-ConceptsSup/Orgs/DecMking/MassDecmking/GoodJudgmentProj/MethodsCombineExpertPredictions.pdf (Accessed: 05 July 2020). Van Bruggen, G.H., Spann, M., Lilien, G.L. and Skiera, B. (2010) ‘Prediction markets as institutional forecasting support systems’, Decision Support Systems, 49(4), pp.404-416. Vandenberg, R.J., Richardson, H.A. and Eastman, L.J. (1999) ‘The impact of high involvement work processes on organizational effectiveness: a second-order latent variable approach’, Group & Organization Management, 24(3), pp.300-339. Van den Broeke, M., De Baets, S., Vereecke, A., Baecke, P. and Vanderheyden, K. (2019) ‘Judgmental forecast adjustments over different time horizons’, Omega, 87, pp.34-45. Vanhaverbeke, W. and Peeters, N. (2005) ‘Embracing innovation as strategy: corporate venturing, competence building and corporate strategy making’, Creativity and Innovation Management, 14(3), pp.246-257. Varma, G.K. (2013) ‘Managing risk using prediction markets’, Journal of Prediction Markets, 7(3), pp.45-60. Vaughan Williams, L. (ed.) (2011) Prediction markets – theory and applications. New York: Routledge. Vaughan Williams, L. and Reade, J.J. (2016) ‘Prediction markets, social media and information efficiency’, Kyklos, 69(3), pp.518-556. Vaughan Williams, L., Sung, M. and Johnson, J. (2019) ‘Prediction markets: theory, evidence and applications’, International Journal of Forecasting, 35(1), pp.226-270. Vayghan, L.A., Saied, M.A., Toeroe, M. and Khendek, F. (2018) ‘Deploying microservice based applications with Kubernetes: experiments and lessons learned’, 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). San Francisco, CA, 2-7 July. New York: IEEE, pp.970-973. Veatch, D. (2016) Mid-level procurement manager leader development: a collective case study. Doctor of Business Administration. Liberty University. Vecchiato, R. (2012) ‘Environmental uncertainty, foresight and strategic decision making: an integrated study’, Technological Forecasting and Social Change, 79(3), pp.436-447. Vecchiato, R. (2019) ‘Scenario planning, cognition, and strategic investment decisions in a turbulent environment’, Long Range Planning, 52(5), Article 101865.

214

Verreynne, M.L., Ho, M. and Linnenluecke, M. (2018) ‘Editorial for the special issue on: organizational resilience and the entrepreneurial firm’, International Journal of Entrepreneurial Behavior & Research, 24(7), pp.17-28. Verschuren, P.J. (2001) ‘Holism versus reductionism in modern social science research’, Quality and Quantity, 35(4), pp.389-405.

Vincent, J. (2014) Telephone interview with Thomas Wolfram, 30 April. Wagner, B. (2013) Crowdsourcing – Eine Welt voller Ideen!. Unpublished Master of Science. Hochschule Esslingen. Waitz, M. and Mild, A. (2020) ‘Comparing trading behaviour and profit composition in prediction markets’, Journal of Prediction Markets, 14(2), pp.3-26. Walker, E. and Dewar, B.J. (2000) ‘Moving on from interpretivism: an argument for constructivist evaluation’, Journal of Advanced Nursing, 32(3), pp.713-720. Wallace, T. (2006) ‘Forecasting and sales & operations planning: synergy in action’, Journal of Business Forecasting Methods and Systems, 25(1), pp.16-21. Wang, K. (2013) An explorative study of applying prediction markets in course evaluation. Unpublished Master of Science Thesis. University of Oulu. Watkins, J.H. (2007) ‘Prediction markets as an aggregation mechanism for collective intelligence’, Lake Arrowhead Conference. UCLA, Los Angeles, 25-29 April. Amsterdam: Elsevier, pp.4-12. Webb, A. (2014) ‘What strategists need: a meeting of the minds’, McKinsey Quarterly, 50(3), pp.9-23. Webster, A. and Gardner, J. (2019) ‘Aligning technology and institutional readiness: the adoption of innovation’, Technology Analysis & Strategic Management, 31(10), pp.1229-1241. Wegener, R. and Sinha, V. (2013) The value of big data: how analytics differentiates winners. Boston: Bain & Company Global Technology Practice. Weigle, B.D. (2007) Prediction markets: another tool in the intelligence kitbag. U.S. Army War College Research Project. Available at: https://apps.dtic.mil/dtic/tr/fulltext/u2/a469632.pdf (Accessed: 7 July, 2020). Weijers, D. (2021) ‘“Just wrong”,“disgusting”,“grotesque”: how to deal with public rejection of new potentially life-saving technologies’ in Steff, R., Soare, S. and Burton, J. (eds.) Emerging technologies and international security: machines, the state and war. Milton Park: Routledge, pp.254-272. Weinstein, R. (2020) ‘How to write a manuscript for peer review’, Journal of Clinical Apheresis, 35(4), pp.358-366. Weiss, C.E., Raviv, E. and Roetzer, G. (2018) ‘Forecast combinations in R using the ForecastComb package’, R Journal, 10(2), pp.262-281. Weldon, L. (2016) ‘2016 CIO agenda: building the digital platform’ [Online webinar]. Available at: http://www.gartner.com/webinar/3176617/player?commId=184885&channelId=5500&srcId=null (Accessed: 18 March 2016). Weller, S.C. (2007) ‘Cultural consensus theory: applications and frequently asked questions’, Field Methods, 19(4), pp.339-368.

215

Welsch, C. (2010) Organisationale Trägheit und ihre Wirkung auf die strategische Früherkennung von Unternehmenskrisen. Wiesbaden: Gabler Fachverlag. West, G.B. (2018) Scale: the universal laws of growth, innovation, sustainability, and the pace of life in organisms, cities, economies, and companies. London: Weidenfeld & Nicolson. Whetten, D.A. (1989) ‘What constitutes a theoretical contribution?’, Academy of Management Review, 14(4), pp.490-495. Wiig, K.M. (1993) ‘Thinking about thinking – how people and organizations create, represent, and use knowledge’, KM Series – Knowledge Management Foundations, 2, pp.104-106. Wilken, K. and Kirfel, N. (2020) Agile Transformation: Erfolgreich in der VUCA-Welt. Available at: https://www.cassini.de/inspire/agile-transformation-der-vuca-welt-begegnen (Accessed: 28 September 2020). Williamson, R.C. (2014) ‘The geometry of losses’, Conference on Learning Theory (COLT 2014). Barcelona, Spain, 13-15 June. New York: Association for Computational Learning, pp.1078-1108. Willis, S. (2018) ‘Culture – a space oddity’, The Economist – 1843 Magazine, Feb/March 2018, pp.18-22. Wiltbank, R., Dew, N., Read, S. and Sarasvathy, S.D. (2006) ‘What to do next? The case for non-predictive strategy’, Strategic Management Journal, 27(10), pp.981-998. Wimmer, E. (2019) The In.Car.Nation Code. München: e&Co. Publishing. Wimsatt, W.C. (2006) ‘Reductionism and its heuristics: making methodological reductionism honest’, Synthese, 151(3), pp.445-475.

Winkler, D. (2020) Telephone Interview with Thomas Wolfram, 20 July. Woerle, S.M. (2013) ‘Wanna bet there will be war? A time-series analysis of prediction markets during the Libya conflict 2011’, Journal of Prediction Markets, 7(3), pp.1-28. Woike, J.K. and Hafenbrädl, S. (2020) ‘Rivals without a cause? Relative performance feedback creates destructive competition despite aligned incentives’, Journal of Behavioral Decision Making, DOI: 10.1002/bdm.2162. Wolfers, J. and Zitzewitz, E. (2006) Five open questions about prediction markets. NBER Working Paper (No. w12060). Cambridge, MA: National Bureau of Economic Research. Wong, D.H., Yap, K.B., Turner, B. and Rexha, N. (2011) ‘Predicting the diffusion pattern of internet-based communication applications using bass model parameter estimates for email’, Journal of Internet Business, 9, pp.26-60.

Wooldridge, A. (2011) Masters of management. New York: HarperCollins. Woolf, S. and Danahy, E. (2017) ‘An active learning approach to transcript coding for education research’, 2017 International Conference on Computational Science and Computational Intelligence (CSCI). Las Vegas, USA, 14-16 December. New York: IEEE, pp.1176-1179. Wulf, T., Meissner, P., Brands, C. and Stubner, S. (2013) ‘Scenario-based strategic planning: a new approach to coping with uncertainty’, in Schwenker, B., Wulf, T. and Krys, C. (eds.) Scenario-based strategic planning: developing strategies in an uncertain world. Wiesbaden: Springer Fachmedien, pp.43-66.

216

Xiao, S., Witschey, J. and Murphy-Hill, E. (2014) ‘Social influences on secure development tool adoption: why security tools spread’, Proceedings of the 17th ACM conference on Computer supported Cooperative Work & Social Computing. Baltimore, MD, USA, 15-19 February. Raleigh, NC: North Carolina State University, pp.1095-1106. Yaniv, I. (2004) ‘Receiving other people’s advice: influence and benefit’, Organizational Behavior and Human Decision Processes, 93(1), pp.1-13. Yeung, S., Downing, N.L., Fei-Fei, L. and Milstein, A. (2018) ‘Bedside computer vision-moving artificial intelligence from driver assistance to patient safety’, The New England Journal of Medicine, 378(14), pp.1271-1273. Yi, Y.Y., Jackson, J.D., Park, J.S. and Probst, J.C. (2006) ‘Understanding information technology acceptance by individual professionals: Toward an integrative view’, Information & Management, 43(3), pp.350-363. Yin, R.K. (2017) Case study research and applications: design and methods. 6th edn. London: Sage publications. Yu, O. (2019) ‘Technology innovation investment portfolio planning: a systems approach with application examples’, in Chen, J., Brem, A., Viardot, E. and Wong, P.K. (eds.) The Routledge companion to innovation management. Milton Park: Routledge, pp.560-589. Zachar, I., Szilágyi, A., Számadó, S. and Szathmáry, E. (2018) ‘Reply to Garg and Martin: the mechanism works’, Proceedings of the National Academy of Sciences, 115(20), DOI: 10.1073/pnas.1805021115. Zhang, T., Fletcher, P.O., Gino, F. and Bazerman, M.H. (2015) ‘Reducing bounded ethicality: how to help individuals notice and avoid unethical behavior’, Organizational Dynamics, 44(4), pp.310–317. Zinger, J.A. (2018) Corporate prediction markets for business decisions: new applications, challenges and limitations. Waltham, MA: Bentley University. Zwass, V. (2010) ‘Co-creation: toward a taxonomy and an integrated research perspective’, International Journal of Electronic Commerce, 15(1), pp.11-48.

217

Appendix A: IARPA Data and Analysis

The researcher undertook an analysis of data from the IARPA forecasting tournament.

The investigation was based on data published by the winner of the tournament from

Harvard, supplanted by data categorisations created by the researcher, and using R coding

to achieve the results.

‘Logistics’ – File Shares

The researcher’s files (code for inspection and use together with the data needed to

execute the analysis) are available in the Mendeley data repository at:

http://dx.doi.org/10.17632/3wtx4f943y.2

Data from Harvard University (‘Good Judgment Project Dataverse’) is available at: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BPCDH5 Overview

One reason why the researcher chose to look at the published IARPA data was that the

(accuracy) performance of the markets could be compared to another approach, so-called

survey beliefs or expert-based responses (i.e. individual, not market-based, estimates also

captured on a platform, cf. Burgman et al., 2011). “Individuals … estimated the

probability of each event, updating their predictions when they felt the probabilities had

changed” (Ungar et al., 2012, p.37), so both PMs as well as ‘survey data’ contained a

large multitude of records per individual question.

Secondly, “drawing on data from a four-year forecasting tournament that elicited over

400,000 probabilistic predictions on almost 500 geopolitical questions” (Atanasov et al.,

2020, p.19), and employing several thousand forecasters (Mellers et al., 2014; Atanasov

et al., 2017) happened in a real-world setting producing verifiable probabilistic forecasts.

“The tournament’s winner, the Good Judgment Project, outperformed the simple average

of the crowd by (a) designing new forms of cognitive-debiasing training, (b) incentivizing

rigorous thinking in teams and prediction markets” (Atanasov et al., 2017, p.2) which was

also of interest to the researcher.

A limitation for achieving meaningful results might be “that the forecasts were mainly on

geopolitical events only a few years in the future … [and] evidence from the GJP may

not generalize to forecasting other types of events (e.g. technological progress and social

consequences) or events further in the future” (Kokotajlo and Grace, 2019a; 2019b).

218

Still, based on this quite large empirical study the researcher was confident to be able to

elicit resilient and reliable results which was also corroborated by Parente and Finley

(2021) who analysed various technologies frequently used to assess future trends, like

scenarios and alternative geopolitical events.

Graphs with a higher Level of Detail

This part expands upon some of the graphs presented in section 6.3.3 by showing all data

points and how they are spread out, not only the ‘summary’ figures plus outliers provided

by the standard Box Plot representation.

The adding-on of this completeness of data offered in the graphs is referred to as ‘Jitter’.

The jitter obscures the boxes from the graphs somewhat but demonstrates overall

accuracy even better as the data-points shown cluster around a high and accurate

probability of the events.

Figure A.1 Trades from All Markets Plus ‘Jitter’ (Developed by the Researcher from Good Judgment Project, 2016)

The above graph expands on the detail from Figure 6.3.

219

Figure A.2 Trades from All Markets per Category ‘Type’ plus ‘Jitter’ (Developed by the Researcher from Good Judgment Project, 2016)

cf. Figure 6.4

Figure A.3 PMs versus ‘Surveys’ Plus ‘Jitter’ (Developed by the Researcher from Good Judgment Project, 2016)

cf. Figure 6.6

All graphs were produced by coding the analysis in the R programming package.

220

Appendix B: Feedback on the Researcher’s IARPA Data Analysis

The researcher’s examination of PM accuracy had been lauded to have taken a new and

intriguing approach, being very comprehensive and thorough; and in this way considered

one of the best which two scholars had come across. It had even been called extraordinary

in its assessment and results. During his own research the researcher had had discourses

with the two academics who had praised his work and sought their feedback, one of whom

is a guest researcher lecturing on innovation management at the Zeppelin University,

Friedrichshafen plus the managing director of an innovation consulting company, the

other habilitated at the Mathematics Institute of the University of Bonn (Koch, 2020;

Ivanov, 2021).

221

Appendix C: Interviewee Information Letter

The interview partners were informed about the interview subject, its background and

intention by sending out information beforehand – either in English or German, as shown

below.

English Version:

The Relevance of Prediction Markets for Corporate Forecasting

The topic of my doctoral thesis at Heriot Watt University are prediction markets (or information markets) in the field of corporate planning, to what extent companies make use of them and whether or how they can profit from them. I would like to conduct an interview even if the tool / approach is not known, in such a case I will briefly explain the concept and then discuss the assessment of the possible usefulness of such a method. First of all, a short explanation of what prediction markets are: Information markets are (artificial) securities markets used to derive information from the prices of securities whose liquidation values are contingent on future events. Users sell the virtual shares in case they consider them to be overvalued and buy if they consider them undervalued. As a result of the market dynamic, the trading price reflects the traders’ aggregated beliefs about the outcome of the future event. They are mostly used as a novel instrument for forecasting and have demonstrated their potential to outperform traditional forecasting instruments (surveys, polls, extrapolation, etc.). The interview will be conducted using an interview guide. From an ethical point of view, I would like to point out that an interviewee can ask to end the interview or drop out of the research at any time. Thank you for agreeing to be interviewed as part of my research project. I am looking forward to the interview. Yours sincerely,

222

German Version:

The Relevance of Prediction Markets for Corporate Forecasting

Das Thema meiner Doktorarbeit an der Heriot Watt University sind Prognosemärkte (Vorhersage-, bzw. Informationsmärkte) im Umfeld Unternehmensplanung, inwieweit und in welchem Umfang Unternehmen darauf zurückgreifen und ob bzw. wie sie davon profitieren können. Ich möchte ein Interview auch dann führen, wenn das Tool / der Ansatz nicht bekannt ist, ich werde in einem solchen Fall das Konzept kurz erläutern, um dann die Einschätzung der möglichen Brauchbarkeit einer solchen Methode zu diskutieren. Vorab schon eine kurze Erklärung, was Prognosemärkte sind: Informationsmärkte sind (künstliche) Wertpapiermärkte, die dazu dienen, Informationen aus den Preisen von Wertpapieren abzuleiten, deren Liquidationswerte von zukünftigen Ereignissen abhängen. Nutzer verkaufen die virtuellen Aktien, wenn sie diese für überbewertet halten, und kaufen, wenn sie sie für unterbewertet halten. Als Ergebnis der Marktdynamik spiegelt der Handelspreis die aggregierten Überzeugungen der Händler über den Ausgang des zukünftigen Ereignisses wider. Sie werden meist als neuartiges Instrument zur Vorhersage verwendet und haben ihr Potenzial bewiesen, traditionelle Prognoseinstrumente (Umfragen, Abstimmungen, Extrapolation usw.) zu übertreffen. Das Interview wird mit Hilfe eines Interviewleitfadens geführt werden. Aus ethischer Sicht möchte ich Sie darauf hinweisen, dass Sie jederzeit darum bitten können, das Interview zu beenden oder aus der Forschung auszusteigen. Vielen Dank, dass Sie sich bereit erklärt haben, im Rahmen meines Forschungsprojekts interviewt zu werden. Ich freue mich auf das Gespräch. Mit freundlichen Grüßen

223

Appendix D: Interview Schedule – Pilot, Main Set, Extended Interviews

The thesis incorporated information from a pre-study as secondary data, a pilot, a main

study, and additional interviews on the extension of PMs with respect to innovation

management as depicted in Table D-1.

Table D-1 Interview / Qualitative Data Collection Phases

Phase Activity Number of Interviews (65 Overall)

Phase 1 Collation of Secondary Data 35

Phase 2 Pilot 13

Phase 3 Main Set of Interviews 7

Phase 4 Extended Interviews (Innovation Management) 10

For the last three phases of the data collection, i.e. the ‘non-secondary’ interviews, an

interview schedule presenting the interview date, the interview type, and its duration is

shown below for each phase, only displaying job title or similar for the interview partners

to preserve anonymity.

Table D-2 Interview Schedule Pilot Interviews

Interview Date Interview Type – How the Interview Was Conducted Interview Duration Interview Partner

November 23rd, 2018 Video Conference 2 hours Principal Scientist

November 26th, 2018 Per Telephone 1 Hour Director of Research

November 26th, 2018 Video Conference 45 minutes Professor

January 25th, 2019 Per Telephone 1.25 Hours Director of Business Development

February 2nd, 2019 In Person 2 hours Professor

March 15th, 2019 Per Telephone 1 Hour Director of Business Development

April 2nd, 2019 In Person 3 Hours Professor

May 3rd, 2019 Per Telephone 45 Minutes Director Innovation

July 22nd, 2019 Per Telephone 1 Hour Head of Economic Research

October 22nd, 2019 Per Telephone 1 Hour Client Manager Underwriting

November 11th, 2019 In Person 2 hours

Senior Program Manager

January 17tht, 2020 Video Conference 2 hours President and CEO

January 31st, 2020 In Person 1.5 Hours Senior Consultant Strategy & Processes

224

Table D-3 Interview Schedule Main Set

Interview Date Interview Type – How the Interview Was Conducted Interview Duration Interview Partner

March 17th, 2020 Video Conference 2 hours President and CEO

March 17th, 2020 Video Conference 1.5 hours Professor

March 23rd, 2020 Per Telephone 45 minutes Managing Director

March 27th, 2020 In Person 1.75 Hours CEO and Founder

April 10th, 2020 Video Conference 45 Minutes Executive Director

April 10th, 2020 Video Conference 45 Minutes Principal Consultant

April 13th, 2020 In Person 1.25 hours Board Member of an Association

Table D-4 Interview Schedule Extended Interviews

Interview Date Interview Type – How the Interview Was Conducted Interview Duration Interview Partner

April 29th, 2020 Per Telephone 1.5 hours General Manager Germany and Austria

May 4th, 2020 Per Telephone 1.5 hours Professor

May 7th, 2020 Per Telephone 45 minutes Professor

May 8th, 2020 In Person 45 minutes Head of IoT Program

May 12th, 2020 Video Conference 2 hours Founder/Managing Director

June 12th, 2020 Video Conference 2 hours Research Fellow

July 2nd, 2020 In Person 1.25 hours Head of Team Business Intelligence

July 14th, 2020 Per Telephone 30 minutes Managing Director

July 28th, 2020 Video Conference 45 minutes Professor of Economics

July 31st, 2020 Per Telephone 1.25 hours Head of Corporate Strategy