PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... · primary research 1 . so who is the best predictor in the audience? prize: a real crystal

THE FUTURE PREDICTING

PRIMARY RESEARCH

1

SO WHO IS THE BEST PREDICTOR IN THE AUDIENCE?

PRIZE: A REAL CRYSTAL BALL

PART 1: MAKES SOME PREDICTIONS

IN SEPT 2013 WE ASKED A GROUP OF 400 UK PANELIST TO PREDICT THE SELLING PRICE OF THE IPAD MINI 3 WEEKS PRIOR TO ITS LAUNCH. HOW CLOSE DID THEY GET?

WHAT IS YOUR PREDICTION? WITHIN 10%, WITHIN 5%, WITHIN 3%, WITHIN 1%

*Note we told them the existing selling price of the old model

PREDICT THE PRICE OF 100ML OF CHANEL PERFUME?

In €

I AM GOING TO TOSS A COIN WILL IT BE HEADS OR TAILS

ADDITIONAL QUESTION…

PREDICT HOW MANY SAID HEADS?

WHAT PROPORTION OF WINE DRINKERS IN THE UK

PREFER RED WINE? BASED ON A POLL OF 400 WINE DRINKERS IN UK WHO WERE ASKED IF THEY PREFER RED OR WHITE

WINE

WHAT % DRINK RED?

PREDICT HOW MANY RESEARCHERS CHECK THEIR EMAIL BEFORE BREAKFAST? BASED ON POLL OF ATTENDEES AT ESOMAR CONGRESS 2014

DO YOU CHECK YOUR EMAILS BEFORE BREAKFAST?

England v Montenegro +3 +2 +1 0 -1 -2 -3

Germany v Rep. Ireland +3 +2 +1 0 -1 -2 -3

PREDICT WHAT MARGIN OF VICTORY OUR UK PANELISTS PREDICTED FOR THESE 2 FOOTBALL MATCHES

WILL THE MARKET RESEARCH INDUSTRY BE BIGGER OR SMALLER IN 10 YEARS TIME?

The CXO Advisory group gathered 6,582 buy or sell predictions from 68 different investing gurus made between 1998 and 2012, and tracked the results of those predictions. How accurate were they?

WHAT % WERE CORRECT?

SWAP YOUR QUIZ SHEET WITH THE PERSON NEXT TO YOU READY FOR MARKING

BACKGROUND Gamification à More prediction protocols in surveys à Fostered an interest in the science of prediction à Led to a series of dedicated prediction experiments à Exploration of the world of prediction market trading à Prediki

•  30+ Primary research experiments

•  500+ Predictions analysed

•  60+ Prediction markets v traditional research comparisons

THE TYPES OF EXPERIMENTS WE HAVE RUN

•  Betting on the future of brands •  Predicting why people buy things •  Predicting the behaviour of other people •  Predicting the price of things •  Predicting the election prospects of political parties •  Predicting football match results •  Predicting the outcomes of TV game shows •  Predicting the success of adverts •  Predicting future sales of products •  Predicting the future more generally

SO WHAT HAVE WE LEANT ABOUT PREDICTION?

20%

42% 43%

32% 30%

38%

11%

21% 21% 22% 22%

29%

0% 5%

10% 15% 20% 25% 30% 35% 40% 45% 50%

Consumer purchasing estimates

Observed behaviour of

others

Observed opinion

Forecast Price prediction

Guesswork

Correct prediction Random chance

WHAT ARE WE GOOD AT PREDICTING AS INDIVUALS?

SOME OF US ARE BETTER AT PREDICTING

0%

20%

40%

60%

80%

100%

120%

140%

160%

180%

score 1 score 2 score 3 score 4 score 5 score 6 score 7

Index of Prediction performance over 7 waves of experiments

top 100

bottom 100

!"#$%&'%()  *+,-%'. = /)0("1,'%()  ×200("'  ×345#&'%6%'.×(1−8%00%&+-'.)  ×  :,)$(1)#;;

Note: Not prediction accuracy is not directly dependent on sample size

One person with access to the right information and able to objectively process it can be just as good as much larger groups. e.g. Nate Silver: Correctly predicted the outcome of all 52 states in the 2012 UK election

16 IS A CROWD Jed Christianson, University of Birmingham calculates

LESS ABOUT SAMPLE SIZE MORE ABOUT SAMPLE DIVERSITY & INTELLIGENCE

Jed Christianson, estimates that the quality of consolidated group predictions does not improve much beyond 16 participants, if they are all well informed.

..AND HOW YOU AGGREGATE CROWD WISDOM

MEAN, MODE, MEDIAN V TRADING & DOUBLE AUCTION TRADING

1906 Plymouth County fair

Actual weight = 1198 lb Median average guess = 1207lb Error = <1%

WHAT WE KNOW

THE WISDOM OF CROWDS

Using larger samples and median aggregation techniques can help consolidate wisdom in situations where information and knowledge is dissipated.

CROWD WISDOM IS BASED ON FILTERING THE SIGNAL FROM THE NOISE

Each person’s prediction is made up of 2 components: information & error.

If each individual’s judgement is independent & unbiased then the error will largely cancel itself out and the aggregation process then distils off the

inherent knowledge.

Actual selling price = £319

Median average guess = £316 Error = 1%

SCORE: 1 POINT

2013 GMI online sample

WHAT WE KNOW

THE WISDOM OF CROWDS

WITHOUT COLLECTIVE KNOWLEDGE CROWDS CAN BE PLAIN IGNORANT

WARNING

2014 GMI online sample

Actual weight = 550kg

AN UNWISE CROWD

Median average guess = 350kg Error = 36%

Where there is no wisdom to consolidate, crowds just generate noise.

CROWD WISDOM CAN BE A BIT BEHIND THE TIMES

-44%

-34% -33% -30%

-28% -26% -25% -25%

-17% -14%

2% 5%

9% 9%

21% 23%

-50%

-40%

-30%

-20%

-10%

0%

10%

20%

30%

Examples of price predicition errors

We measured an average 9% price lag on aggregated price predictions – our memory of price is based on historical purchase activity which is often lower that current pricing.

94% 87%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

Women Men

Price prediciton accuracy

Men are less price savvy than women

Crowd €80 Actual €120

YOUR PREDICTIONS?

SCORE: +/-$10 2 POINTS +/-$20 1 POINT

Aggregating crowd wisdom: “If each individual’s judgement is independent & unbiased then the error will largely cancel itself out”

THE PROCESS OF MAKING PREDICTIONS IS LITTERED WITH COGNITIVE BIASES

68% predict heads DUE TO ORDER BIAS

HEADS OR TAILS?...YOUR PREDICTIONS

SCORE: +/-5% 2 POINT +/-10% 1 POINT

WITH NO INFORMATION TO GUIDE US TINY NUDGES CAN HAVE BIG EFFECTS ON OUR PREDICTIONS

HOW THIS EFFECT CORRUPTS PREDICTIONS….

54%

46% 46%

54%

0%

10%

20%

30%

40%

50%

60%

White Red

What percentage of people do you think prefer white wine?

What percentage of people do you think prefer red wine?

V

A 20% shift in prediction based simply on naming the colour of wine in the question!

IN THE ABSENCE OF KNOWLEDGE WE PREDICT THE MAJORITY OF OTHER PEOPLE WILL DO & THINK THE SAME AS US

- 10 20 30 40 50 60 70 80 90

100

Believers

Believe this How many people believe this

- 10 20 30 40 50 60 70 80 90

100

Non believers

Don’t believe How many people don't believe this

those holding minority opinions assume more people agree with them than those holding the majority opinion: is this the definition of delusion?

WHAT WILL THE WORLD BE LIKE IN 2050?

55% 68%

CHECKING EMAIL YOUR PREDICTIONS

SCORE: +/-5% 2 POINT +/-10% 1 POINT

60%

20%

I check my emails before breakfast

I don't check my emails before breakfast

Prediction of how many other people at Lightspeed GMI check their emails

before breakfast*

*Source: office poll!

80% of Researcher said they check their email before breakfast

OUR EMOTIONS REALLY CAN DOMINATE & BADLY DISTORT OUR PREDICTIONS

England v Montenegro +3 +2 +1 0 -1 -2 -3

Germany v Rep. Ireland +3 +2 +1 0 -1 -2 -3

PREDICT WHAT SCORES OUR UK PANELISTS PREDICTED!

SCORE: CORRECT = 1 POINT PER QUESTION

0%

10%

20%

30%

40%

50%

60%

new by 3 new by 2 new by 1 draw liv by 1 liv by 2 liv by 2

Newcastle Liverpool

0%

10%

20%

30%

40%

50%

chel by 3 chel by 2 chel by 1 draw car by 1 car by 2 car by 3

Chelsea Cardiff

0% 5%

10% 15% 20% 25% 30% 35% 40% 45% 50%

man by 3 man by 2 man by 1 draw south by 1

south by 2

south by 3

Southhampton Man U

0% 5%

10% 15% 20% 25% 30% 35% 40% 45% 50%

man by 3 man by 2 man by 1 draw south by 1

south by 2

south by 3

Southhampton Man U

FOOTBALL SCORE PREDICTIONS

These are the comparative football score predictions of competing fans.

0%

10%

20%

30%

40%

50%

60%

70%

80%

Conservative Government

Labour Pary Government

Liberal Democrats Government

Conservative & Liberal Coalition

Labour and Liberal Coalition

Conservative, Liberal, & UKIP

Coalition

Conservatives Labour LiberalDemocrats UKIP

PREDICT WHO WILL FORM THE NEXT UK GOVERNMENT: BY PARTY AFFILIATION

The issue of emotional bias is particularly significant in the political arena where voting intention and predictions on who is likely to form a future government almost completely correlate.

DIFFERENCES BETWEEN PREDICTING WHAT OTHERS WILL DO

V

WHAT I WILL DO

SOCIAL COGNITIVE BIASES RENDER PREDICTIONS ABOUT OUR OWN BEHAVIOUR PARTICULARLY DIFFICULT

Will you tidy up after the meeting? Yes = 50%

TIDIED UP =13%

Predict how many will tidy up? = 15%

…we are much better at predicting the behaviour of others than our own behaviour. We like to think well of ourselves!

WE ARE OFTEN TOO TIED UP IN THE DETAIL TO SEE THE BIGGER PICTURE

WILL THE MARRIAGE LAST? Yes/No Parents are much better than the married couples at predicting this

Source: Queens University Canada

0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99

1

Experts Dillettantes (non

experts)

Chimps (random guesses)

Future Predictions accuracy

CA

LIBR

ATI

ON

Highly recommended reading

PHILIP TETLOCK STUDYING 15,000 GEO-POLITICAL PREDICTIONS

0

0,01

0,02

0,03

0,04

0,05

0,06

Experts Dillettantes (non experts)

Chimps (random guesses)

Descrimination

Philip Tetlock

EXPERT POLITICAL JUDGEMENT

What he describes as “Dillettantes”, smart people with no specific expertise in the topic, prove to be more effective at making geo-political predictions than political pundits/experts.

HOW MANY INVESTMENT GURUS 6,500 STOCK MARKET PREDICTIONS WERE CORRECT?

SCORE BELOW 50% = 1 POINT

48% 2% less accurate than a coin toss!

WE ARE UNABLE TO SEPARATE THE SIGNAL FROM THE NOISE

Will the Market Research industry be bigger or smaller in 10 years time? Yes/No?

I don’t know but we in the market research industry are not probably the best people to ask.

HOW TO WORK THE CROWD! EXPLORING THE BEST TECHNIQUES TO USE TO EXTRACT RELIABLE PREDICTIONS

SWAP BACK YOUR QUIZ SHEETS

10 VOLUNEERS

WHICH OF IS THE MOST POPULAR AD?

BET 1, 2, 3, 4, 5 POINTS

WHICH OF THESE CANDLES SELLS THE MOST

BET 1, 2, 3, 4, 5 POINTS

WHICH OF THESE MUGS SELLS THE MOST?

BET 1, 2, 3, 4, 5 POINTS

Clue: These are mostly purchased as gifts for other people Clue 2: these are sold to English people!

Camping Cooking Gardening DIY (Home improvement)

-200%

-150%

-100%

-50%

0%

50%

100%

150%

200%

250%

300%

350%

Evaluation of 20 different ads Monadic rating Performance prediction: Net Investment

0.89 correlation 5x differentiation

ADVICE: MAKE IT REWARDING

All the research out there show us that prediction made under competitive conditions where you can have the opportunity to win or lose, results in participants investing more thought and effort and this in tern improves the quality of predictions.

WEIGHT BASED ON CONFIDENCE

33% 37% 36%

44%

0% 5%

10% 15% 20% 25% 30% 35% 40% 45% 50%

Total Guess Have a Hunch Fairly Sure Very Confident

Prediction accuracy

In our analysis of 500 predictions, where in each case we asked respondents how confident they were, we found the predictions from the most confident groups to be slightly more accurate.

MONEY BET IS A PROXY TO CONFIDENCE

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

-3 -2 -1 +1 +2 +3

Bet amount: correlation with outcome

We also found an interesting relationship between money bet and prediction accuracy. A non-linear relationship. Small bets were less accurate than medium bets but larger bets were not more accurate than medium bets. The most accurate bets overall were in fact those that bet small amount against.

EXPLOIT PREDICTION MARKET TRADING

•  Prediction markets trading is a process where participants are given money to buy and sell shares in the outcome of an event with a chance to win real money.

•  The most famous example being the Iowa Electronic Market a prediction market set up to predict the outcome of elections around the world: ◦  In 451 out of 596 head to head comparisons, prediction

market trading by groups on average of less than 20 people has more accurately predicted the result that the most accurate local opinion polls.

•  We conducted 32 head to head experiments to compare the prediction from prediction markets trading to predictions made in traditional surveys

•  A survey based approach with random cells* of 15 participants who were asked to predict which products would sell more vs 15 people prediction markets trading – asked to buy or sell variable amounts to create a confidence weighted market

TESTING OUT USING PREDICTION MARKETS FOR MORE TRADTIONAL MARKET RESEARCH PURPOSES

37%

55%

65%

0%

10%

20%

30%

40%

50%

60%

70%

Random guess Micro survey (sample of 15)

15 people prediction market trading

HEAD TO HEAD COMPARISON

Source: GMI/Prediki based on 32 direct head to head comparisons

Comparing the performance of small head to head samples, prediction markets trading groups did more accurately make correct predictions. The trading process more effectively aggregating group wisdom

SOME ISSUES THOUGH

OPINIONS IN PREDICTION MARKET TRADING CAN QUICKLY BE SET IN STONE IF NO NEW INFORMATION IS ADDED

ADDING MORE PEOPLE AFTER A CERTAIN POINT DOES NOT CHANGE THE RESULT

When we used larger samples of respondents in prediction markets, adding more respondents did not improve the quality of predictions – predictions rather rapidly got fixed in stone. The betting intentions of the earlier participants was having a significant impact on the later participants.

37%

55%

65% 69%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Random guess Micro survey sample of 15

15 people prediction

markets trading

Standard survey: sample of 200

HEAD TO HEAD COMPARISON

The aggregated prediction of larger samples of people made standard surveys proved to be more accurate. These predictions were all made in isolation from the influence of other people’s opinion.

INFORMATION SHARING IS KEY PREDICTION MARKETS FEED OF INFORMATION

•  These initial experiments we did not allow the participants in the prediction markets trading process to share opinions all they could see was what others had bet.

•  We then conducted a second series of experiments where we injected some intelligence by encouraging participants to discuss their decisions and allowed them all to see what other people thought.

MARKETS WOULD REACT WHEN WE ADDED INFORMATION

Self generating clues

The participants began to self issue clues that would help the group collectively make more thoughtful predictions. The example above show how the market reacted to thoughtful comments made about which product they felt would sell more.

THINK OF A QUESTION AS A CONUNDRUM: INFORMATION

PROVIDES CLUES TO HELP PEOPLE SOLVE THE PROBLEM

DIALECTICAL BOOT STRAPPING ENCOURAGING CROWDS TO SELF-GENERATE THE INSIGHTS

NEEDED TO SOLVE PREDICTION CONUNDRUMS

Example = Board room decisions

Useful reference: Herzog and Hertwig (2009)

37%

55%

65% 69%

81%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Random guess Micro survey (sample of 15)

15 people Micro prediction market

trading

Standard survey (sample of 200)

Micro prediction market with shared information & free

comments

THE VALUE OF ADDING INFORMATION TO PREDICTIVE MARKET TRADING SYSTEM

In this second series of experiments the prediction accuracy of these small group of individuals sharing information in prediction markets moved ahead of the predictions from larger samples of respondents in standard surveys

THE PROBLEM OF HERDING EFFECTS

Information cascade

However there predictions were still not perfect, stray comments from individuals could occasionally steer the prediction markets off course.

STUDYING THE IMPACT OF NUDGE EFFECTS: THE INFLUENCE ONE PERSON’S OPINION HAS ON ANOTHER

2%

6%

11%

15%

20%

Personal preferences Self evident predictions (e.g. ad

evaluation)

Factual (requiring knowledge)

Inverted personal preference (e.g. pedicting relative levels of dislike)

Complex estimates

Nudge influence by prediction task

Source: GMI research 2014

We conducted some unique experiments to measure the influence that one persons opinion could have on another and found the less certain people are and the harder the prediction is, the more we rely on other people’s opinions

OUR SOLUTION = DIVIDING UP THE HERD

A simple solution to this is to run multiple prediction markets in isolation from each other and then aggregating their predictions.

37%

55% 65% 69%

81% 93%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

Random guess Micro survey (sample of 15)

15 people Micro

prediction market trading

Standard survey

(sample of 200)

Micro prediction

market with shared

information & free comments

Aggrigation of 3 divided

herds

HEAD TO HEAD COMPARISON

By doing this we were able to improve the prediction accuracy from prediction markets even further.

Please note these experiments must be treated only as anecdotal evidence to illustrate these issues. This is only one small series of experiments. A lot more work is need to properly validate these findings.

EFFECTIVE USE OF PREDICTION MARKETS

•  Incentivise - ideally with real money! •  Allow active & dynamic trading •  16 is a crowd •  Share as much information as possible •  A moderator is important to stimulate debate and share

information •  Divide the herd: run multiple micro markets

SAMPLING 100’S SMALLER SMARTER GROUPS BEING ASKED SMARTER QUESTIONS & SHARING THOUGHTS & OPINIONS

SO WHO IS THE BEST PREDICTOR IN THE AUDIENCE?

PRIZE: A REAL CRYSTAL BALL