Speaker Duncan Stuart, Kudos OrganisaPonal Dynamics, New Zealand Part 1: Session 3: Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (London) BUMP BUMP A New Metric Inspired by Neural Networks
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 1
12/5/10 1 Speaker Duncan Stuart, Kudos OrganisaPonal Dynamics, New Zealand Part 1: Session 3: Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (London)
BUMP BUMP -‐ A New Metric Inspired by Neural Networks
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 2
Background
1. A quick framing discussion – the context of where this comes from.
2. A swi[ overview of Neural Networks. What they are, how they work.
3. A couple of examples (disguised) of things we’ve done with NNs....and then the BUMP discovery. We weren’t looking for it – but there it was.
4. QuesPons. Duncan Stuart
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 3
HOW THIS PRESENTATION FITS IN WITH NEW MR
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 4 SLIDE 4
Where NN analysis fits in to New MR
This schemaPc describes what’s possible in market research.
Old MR basically lives in the lower le[. The discovery and analyPc dimension is preay basic. We ask simplisPc quesPons, then we treat these to basic analysis – o[en merely descripPve. Pies, bars and crosstabs.
Consequently (and also independently) the types of decision that come out of this research are also preay basic.
The value fronPer is around about where the middle curve is. Analyse and Plan.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 5 SLIDE 5
Old MR lives in the boaom le[. New MR in the top right.
This schemaPc describes what’s possible in market research.
New MR is about pushing the fronPer further. Anyone with SurveyMonkey, and email list and Excel can do most of the things back in the red square. Old MR has become commodiPsed. More to the point, if everybody does it, it then doesn’t offer the user any great compePPve advantage.
New MR is about using available technology, and some professional brains, to operate in a much richer arena. Instead of the somewhat descripPve “what is” research, it takes us into the world of “what if.”
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 6 SLIDE 6
The world of modelling.
• Acknowledges that the world is complex. For example all consumers won’t respond en masse in the same direcPon.
• Acknowledges that consumer aftudes make up only part of the story. For example the economy may tank. The context may change.
• Acknowledges that business decisions always involve risk.
• Old MR doesn’t really work with these assumpPons. – It usually works with mean scores, or simple linear regressions: in others kind
of treaPng the market as homogenous.
– Research designs stay within the paradigm of market research. Did you see the ad? How strongly did you prefer Brand C?
– We are asked to make a call. This pack or that? Launch or not launch? And we don’t usually employ very sophisPcated means of evaluaPng those decisions.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 7
WHAT ARE NEURAL NETWORKS?
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 8 SLIDE 8
Neural Networks
• NNs are a computer emulaPon of the mind. A computer learning process that searches for answers through thousands (maybe millions) of trial and error calculaPons.
– A very typical example is the use by banks, to ascertain the risks in mortgage lending. Rather than have the bank manager make a decision, big banks have a NN system.
– A typical result is that your local bank manager gets it wrong 12% of the Pme while NNs, working blindly off the data, get it wrong only 7% of the Pme.
– In other words 41% fewer defaults.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 9 SLIDE 9
Regular analyPcs are not so predicPve.
– Regular analyPcs tend to simplify and are engineered around averages and simple regressions. They employ a few variables (lifestage, credit raPng and current income) and cannot really cope with, say 80 variables which interact with each other. (Variable X is condiPonal upon Variables Y and Z)
– For example in the bank loan applicaPon do you have a fatal disease? 99.9% of people say no. So as a variable it might be discarded as useless. But if you DO have a fatal disease...what’s your credit risk now?
– Or some variables are condiPonal upon others: do you have life insurance? may be totally irrelevant unless you answered yes to QuesPon 10.
– In other words many variables are not important unless they are. A linear formula (regressions, correlaPons and other techniques) don’t really tell us what’s going on case by case. They explain the average story, but not case by case.
– NN’s get over this problem...
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 10 SLIDE 10
Neural networks can handle complexity.
• Computer learning – with the NN trying itera5on aOer itera5on, walking around the data landscape tweaking the algorithm (maybe millions of 5mes) un5l it can find no more accurate way of explaining the data. With computers, massive non-‐linear, mul5variate algorithms are no big problem.
• Several outputs. 1. A ranked list of the variables that have
greatest “acPvaPon” or effect on the predicted behaviour. (Great when you pile 80 variables into the mix.)
2. Every respondent or case gets a unique ac#va#on score.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 11 SLIDE 11
Let’s show a simple example. This was energy drinks.
AcPvaPon score too low. Probability that respondent does not buy Brand X.
Threshold
AcPvaPon score higher than 0.72. They probably Do buy Brand X.
1. We used tracking data – training the NN on 1,000 respondents and trying the results on another 1000. 2. There were 80+ variables to determine who and who doesn’t drink energy drinks. Variables include
demographic, aftude, lifestyle interests, surfing, nightclubs, etc. 3. We went home. The NN worked overnight and by morning gave every respondent a simple acPvaPon score
based on the complex algorithm it had generated.
Tested 92% accurate
• At first we used the ac8va8on score as the sort of measure you use with cross tabs. Our interest was in what it was that drove ac8va8on.
• But then....
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 12 SLIDE 12
Then for some reason we ploXed the distribu5on of ac5va5on scores for the respondents.
Threshold
We got a bell curve that leaned over to the right. Don’t forget, this has been derived from a hugely complex mulPvariate process. We let NN do the interrogaPon...
ACTIVATION SCORE
AcPvaPon score too low. Probability that respondent does not buy Brand X.
AcPvaPon score higher than 0.72. They probably Do buy Brand X.
Tested 92% accurate
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 13 SLIDE 13
Well this was fascina5ng...a whole heap of people live near the threshold!
Threshold ACTIVATION SCORE
A big groundswell of people live right next to the Berlin Wall.
They look very “BUMP-‐able”
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 14 SLIDE 14
Bump. How close are users to the Berlin Wall? How easily could they be bumped over?
Berlin Wall 0.72
Low Bump Index. Blue populaPon lives miles from the wall.
A zillion dollar campaign won’t convert them.
High Bump Index. Grey populaPon is
very bumpable. All they need is a
nudge.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 15 SLIDE 15
A bi-‐polar Bump situa5on....Perhaps your market is more polarised?
Berlin Wall 0.72
Note to marketers – don’t waste your 8me on these Low Bump
people!
Note to marketers – focus on these High Bump people.!
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 16 SLIDE 16
Now we can play what-‐if?
• IdenPfy the people who live near the wall.
• Start tweaking the variables that most drive their acPvaPon score.
• (For example in the energy drinks case we examined what would happen if they got a job, or if they suddenly lost interest in night-‐clubbing, or if they suddenly rated Brand Z as parPcularly cool.)
• And by tweaking the data and then re-‐running the algorithm we could see how the acPvaPon scores changed – case by case: how many people jumped over the line? Or not. (Or jumped back the wrong way!)
Berlin Wall
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 17 SLIDE 17
Some things I’ve learned about this approach.
• Generally very accurate and trustworthy. 90%+ accuracy is preay standard.
• Tells us which variables are driving the ac5va5on scores. Which are worth “playing with” in our modelling.
• Usually reveals that if ‘what-‐if’ happens, then there are two or more countervailing trends. In other words it let’s us see complexity.
• And by delivering us a meta-‐measure such as BUMP Index we can s5ll think in simple terms even if the algorithms that gives us the ac8va8on scores are gigan8cally complex. It helps us zoom in on some very useful informaPon but in a very explainable way. E.g. 20% of the market is just half a step from buying your product, and here’s who they are.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 18 SLIDE 18
Summary
As next generaPon researchers I hope we’re consciously moving upwards and rightwards on this model.
Neural Networks help us quite easily work with the rich, mulP-‐variate, curvilinear nature of forecasPng – and BUMP, apart from being a cool liale measure in itself, shows us that the outer fronPers of research value can be reached and communicated quite simply.
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 19
DUNCAN STUART AND RAY POYNTER
Q & A
KUDOS ORGANISATIONAL DYNAMICS Speaker Duncan Stuart, Kudos Organisa5onal Dynamics, New Zealand Part 1: Session 3 Convenor Greg Coops, Chair Ray Poynter, schedule = 5:26am to 5:52am (GMT/London)
SLIDE 20
Thank you!
Duncan Stuart FMRSNZ
duncan@kudos-‐dynamics.com
Telephone 64 9 366 0620
www.kudos-‐dynamics.com
Funds from all projects go directly to the language school we built and support in Siem Reap Cambodia. www.savong.com