The 2013 Market Research Technology Survey · sponsored the survey and provided financial and technical assistance that support its administration. This is acknowledged in the survey’s

The 2013 Market Research Technology Survey

Full report

Tenth annual survey by meaning ltd

Report written by Tim Macer and Sheila Wilson, meaning ltd

independent advice on technology for research

Confirmit 2013 Annual MR Technology Survey: Full report | 2 of 85

Contents

1 Background to the survey 8 1.1 The Sample 8 1.2 Demographics 9

Part 1: New questions 10

2 Smartphone and tablet devices 11 2.1 Mobile participants on smartphones 13 2.2 Platforms supported for mobile 15 2.3 Mobile platforms – regional and company variations 17

3 Voice of the Customer 19 3.1 Level of involvement by MR companies 21 3.2 Data collection methods used 22 3.2.1 Global variations 23 3.3 Services offered 24 3.4 Challenges faced 26 3.5 Special skills developed internally 28 3.6 Additional skills resourced externally 30 3.6.1 Internal up-skilling and externally sourced skills compared 31 3.7 Future positioning of market research within VoC & CEM 32

4 Text analysis and coding 34 4.1 Unstructured text: predicted trends 36 4.2 Text analysis methods currently used 37 4.2.1 Text analysis methods in use: 2013 compared with 2011 38 4.3 Predicted changes in methods applied in the future 40 4.3.1 Text analysis methods future predictions: 2013 versus 2011 41 4.4 Perceived effort in analysing open text 42 4.5 Difficulties in analysing unstructured text 44 4.5.1 Differences by region and company size 45

5 Ten Years of Technical Innovation 46 5.1 Greatest positive influences 47


5.1.1 Regional differences 48 5.2 Greatest disruptive influences 49 5.2.1 Regional differences 50 5.3 The technology that offers the greatest advantage 51 5.4 Looking back from the future 53

Part 2: Trends 55

6 Research activities 56 6.1 Research mix – Seven-year trend 57 6.2 Company revenues from research in 2013 58 6.3 Volume of work 59 6.4 Research modes offered – mainstream modes 61 6.5 Research modes offered – by region 63 6.6 Research modes offered – minority modes 64 6.7 Changes in the amount of work 66

7 Sources of online sample 67 7.1 Proportion of online sample in use - trend 68 7.2 Proportion of online sample in use in 2013 69 7.3 Online sample in use - trend 70

8 Mixed mode 71 8.1 Importance of mixed mode 72 8.2 Use of integrated platforms 73 8.3 Level of mixed mode support required 74

9 Analysis and results reporting 75 9.1 Distribution methods in use 76 9.2 Distribution methods in use, by company size 77 9.3 Change in demand for high tech delivery 79 9.4 Importance of producing cross tabs in volume 81

10 Changing software 82 10.1 Planning on changing software in 2013 83 10.2 Planning on changing software - trend 84 10.3 Software type to change 85


List of tables Table 1 Sample composition: company size by region 9

Table 2 Sample composition: responsibility and seniority 9

Table 3 Companies able to support mobile interviewing in one form or another 15

Table 4 Platforms supported for mobile browser-based self-completion interviewing 17

Table 5 Platforms supported for self-completion interviewing using a mobile app 17

Table 6 Categorisation of the difficulties reported in analysing large amounts of open text 45

Table 7 The primary positive influences in the last 10 years: top three by global region 48

Table 8 The primary disruptive influences in the last 10 years: top three by global region 50

Table 9 The technological innovation that is forecast to be most influential over the next 10 years. 54

Table 10 Proportion of quantitative research by mode 60

Table 11 Mobile self complete – proportion of work compared with the proportion of research companies that offer it as a service – by year 64

Table 12 Anticipated change in demand for digital dashboards over coming year 80

List of charts Figure 1 Respondents taking online surveys on smartphones: 3-year trend 13

Figure 2 Respondents taking online surveys on smartphones: current year breakdown 14

Figure 3 Mobile platforms supported for mobile interviews or interviewing 16

Figure 4 Level of involvement by research companies in VoC and CEM programmes 21

Figure 5 Data collection methods applied to VoC or CEM work – overall 22

Figure 6 Data collection methods applied to VoC or CEM work – by global region 23

Figure 7 The Range of VoC and CEM-oriented services provided by research companies 24

Figure 8 The greatest challenges faced by research companies when doing VoC or CEM work26

Figure 9 Additional skills companies have developed to handle VoC or CEM work 28

Figure 10 Additional skills resourced externally for VoC or CEM work 30

Figure 11 Additional skills resourced externally for VoC or CEM work 31

Figure 12 Predictions on the roles research companies may perform in future 32

Figure 13 Predicted change in how much unstructured text in surveys in the next 1-2 years 36

Figure 14 Text analysis methods – all methods and main method currently used 37

Figure 15 Text analysis methods – all methods used, two-year comparison 38


Figure 16 Text analysis methods – main method used, two-year comparison 39

Figure 17 Text analysis methods likely to make greater use of in the future, compared with current methods used 40

Figure 18 Text analysis methods likely to make greater use of in the future: two-year comparison 41

Figure 19 Perceived effort in deploying the tools available to analyse unstructured text: variations by region and company size 42

Figure 20 Perceived effort analysing unstructured text: 2011 compared with 2013 43

Figure 21 Difficulties experienced in analysing large amounts of unstructured text 44

Figure 22 The greatest positive influences on market research in the last ten years 47

Figure 23 The greatest disruptive influences on market research in the last ten years 49

Figure 24 Proportion of company revenues that derive from research activities, annual trend57

Figure 25 Proportion of company revenues that derive from research activities, by region and company size 58

Figure 26 Volume of work by mode 60

Figure 27 Percentage of market research firms using the main modes of research – by year 62

Figure 28 Percentage of market research firms using the main modes of research – by region in 2013 63

Figure 29 Percentage of market research firms using the minor modes of research 65

Figure 30 Anticipated changes in the amount of work over next three years 66

Figure 31 Proportion of online sample in use 68

Figure 32 Proportion of online sample in use, by region and company size 69

Figure 33 Online sample sources in use 70

Figure 34 Importance of mixed mode 72

Figure 35 Use of integrated platform versus switching between different platforms 73

Figure 36 Level of mixed mode 74

Figure 37 Distribution methods in use 76

Figure 38 Distribution methods in use, by company size – for 2013 and 2012 78

Figure 39 Anticipated change in demand for high tech delivery methods 79

Figure 40 Importance of being able to produce volumes of printed cross tabs 81

Figure 41 Planning on changing software, by region and company size 83

Figure 42 Planning on changing software, by year 84

Figure 43 Software type wishing to change 85



Acknowledgements

We are greatly indebted to Confirmit for the assistance they have provided in supporting the survey financially, and also technically, for providing us with the use of their software for this survey, and for their significant contribution they have made to the content and design of the survey.

We are also most grateful for the support provided by JMRA (http://www.jmra-net.or.jp/index-e.html) and Quirk’s Publishing (http://www.quirks.com) in publicising this survey among their own members and subscribers, and inviting participation. This has been especially helpful in increasing the coverage of this survey among groups that had proved harder to reach in the past.

We also thank Ascribe (http://www.goascribe.com) for their generosity in coding the open-ended questions in this survey, and Lloyd Irving and Alex Brezuica of Magnify Solutions (http://magnifysolutions.co.uk) for programming the survey for us.

Tim Macer and Sheila Wilson, meaning ltd London, May 2014


1 Background to the survey

Meaning ltd has been carrying out this annual survey since 2004, making this the tenth successive year of the survey. The survey provides a unique set of information and insights into the interplay of technology and methodology within the market research industry. It provides a snapshot of current usage and attitudes and predictions from practitioners, and identifies trends from a number of tracking questions that are asked repeatedly each year.

In addition to identifying general trends, each year, the survey also focuses on several areas of topical interest. For the 2013 survey, these topics are:

• Smartphones and tablets – their use in surveys and the extent to which market research companies support their use

• Customer satisfaction and voice of the customer

• Text analytics

• The last ten years and the next ten years of technological innovation

For 2013, we have updated the title, from ‘software survey’ to ‘technology survey’. This recognises the fact that this study has for many years looked at much more than software, by exploring the wider influences that technology exerts on the industry in driving innovation and change. It does not signify a change in emphasis in the project: it merely better reflects what the survey has long been about.

This year (as in several previous years), the survey software company Confirmit has kindly sponsored the survey and provided financial and technical assistance that support its administration. This is acknowledged in the survey’s full title, which is the 2013 Confirmit Market Research Technology Survey by meaning.

1.1 The Sample The 2013 survey comprises a sample of 240 market research companies globally, and only of individuals who are responsible for, influential in or aware of technology decisions within each company. The sample is managed to ensure representation of three global regions: North America, Europe and Asia Pacific according to the relative amount of research carried in these regions from data published by ESOMAR.

The survey consists of a self-completion interview on the Web, comprising around sixty questions and timed to last approximately fifteen minutes. As it is difficult to identify the target individual within a research company, sample is obtained from a variety of sources:

§ Participants who agreed to be re-contacted from the previous year’s survey

§ Sample compiled by meaning ltd

§ Sample provided by the survey’s sponsor, Confirmit


§ Sample from 2010, 2011 and 2012 surveys

§ Emailed invitations by JMRA (Japan Market Research Association) to an open survey link.

§ Several links inviting participation from mailshots and on the homepage of the research publication Quirk’s Marketing Research Review to an open survey link.

All emailed invitations are de-duplicated across all sources prior to invitations being sent, and de-duplication is again applied to completed interviews for the small number of cases where more than one response has been received from the same company within that country.

Responses from open survey links accounted for 4% of the sample. These were additionally asked to provide company information, and this was used to verify their eligibility for the survey – i.e. that they were a research company.

The nature of the sampling method, and its eligibility criteria (i.e. senior IT decision makers within research organisations) makes it impossible to provide a meaningful estimate of response rate, because invitations are sent to individuals without prior knowledge of their eligibility, and those who are not eligible are unlikely to respond to the survey invitation.

1.2 Demographics This report is based on 240 participants in unique research companies within each country and covers 35 countries. The composition of the demographics referred to in this report is as follows:

Size Total N America Europe Asia Pacific Small 130 54% 43 49% 70 61 % 17 45 % Medium 69 29% 29 33% 28 24% 12 32 % Large 41 17% 15 17% 17 15% 9 24 % Total 240 100% 87 100% 115 100% 38 100%

Table 1 Sample composition: company size by region

The declared revenue of the organisation defines company size. ‘Small’ is intended to reveal the different needs of companies unlikely to have specialist in-house technical staff. The smaller proportion of companies in the ‘large’ (or over $25m turnover) category is only a reflection of the pyramid that exists of company size, with a smaller number of large companies globally.

The survey is only asked of senior individuals who are involved in IT decision-making within their respective company. The level of responsibility and role of the participants is shown below:

Role N % Level of responsibility N % Research 105 44% Board Member 74 31% IT 32 13% Primary IT decision maker 94 40% Data processing 32 13% IT Decision influencer 72 30% Business or Operational 71 30%

Table 2 Sample composition: responsibility and seniority


Part 1: New questions

Each year, in addition to a number of unchanging tracking questions, the survey focuses on several areas of topical interest. For the 2013 survey, four related topics have been explored.

The first question in the section on smartphones and tablets has been asked now for the last three years, but is presented here in context with three questions on mobile devices. The questions on text analytics and coding reprise a set of questions asked in 2011. Hence, for both of these, there are cross-references to the results recorded previously. All the other questions are new.

Topics in this section

§ Smartphone and tablet devices § Voice of the Customers and Customer Experience Management § Text analytics and coding § Technological innovation: the last ten years and the next ten


2 Smartphone and tablet devices

Smartphones and tablets have effectively put powerful computers – more powerful than those being used exclusively in workplaces just a decade ago – into the hands of the public, and thereby, the potential research participant. They are transforming market research in two ways, first in an opportunistic way, by allowing surveys to be designed specifically for deployment on these devices, and secondly in an unintended or consequential way, as participants decide for themselves to take online surveys using the browser on their smartphones, whether that was the research designer’s intention or not.

The 2012 survey included two questions on the topic of participants taking online surveys on mobile devices, and those first of those questions is repeated here. The 2013 survey went on to include two questions on companies’ preferences for browsers versus apps for delivering mobile surveys. Continuing our on-going examination on the growth of mobile, the 2013 survey looked at companies’ support for the different device architectures or platforms.

There are effectively five competing platforms or operating systems to be found on mobile devices: Google’s open Android platform; Apple’s proprietary iOS used in iPhones and on iPads; Microsoft’s variants of its Windows OS for mobile and tablet, then there are devices that still use Nokia’s Symbian technology, which is now very much in decline, and even abandoned now by Nokia, but inconveniently for researchers, still to be found on an important minority of handsets still in use. The fifth – BlackBerry – also appears to be waning in popularity, but still has a much more significant share of the market.

For software developers, each platform presents its own set of challenges, who find it difficult to create a meaningful core of software which can be ported and adapted on each platform, which means there is considerable development effort to create applications that support the same functions and provide the same experience to users on different platforms. Most therefore compromise by choosing to support only some – which in the context of survey research, has far reaching implications in terms of potential coverage bias. Add to which, is known that each platform tends to appeal to slightly different demographic groups.

Questions

1. Thinking just about your web surveys, what proportion of participants are taking these surveys on small format mobile devices such as smartphones?

2. If you create online surveys to be taken on a smartphone’s web browser, which of these mobile devices can you support?

3. If you create a survey for delivery via a mobile app, which of these devices can you support?

4. If you create a mobile CAPI survey to be administered on a tablet, which of these devices can you support?


Key findings

Mobile participation is a disrupter that should not be ignored… but often is

§ For the third year we asked companies how many participants they now see attempting to take online surveys on smartphones – in each year it has been growing

§ The average proportion, reported across all companies, is now 16.4%

§ It’s highest in Asia Pacific, at 21.1%

§ Yet 37% of those interviewed could not give us a figure

More firms are ready for mobile using browsers than apps

§ 70% of companies report they can run surveys on mobile using the mobile’s built-in browser

§ Only 42% report they can deploy surveys to a mobile app

Android and iOS rule the roost on mobile

§ Android and iOS are supported by the vast majority of companies whether it is a browser or an app they are using

§ Windows phone and BlackBerry lagging behind; Symbian rarely supported

Tablet interviewing is also making a showing

§ One in three firms report they can handle mobile interviewing (CAPI) on tablets

§ Android dominates on tablet, with iOS following


2.1 Mobile participants on smartphones

à The number of participants taking online surveys on their mobiles continues to grow

à The highest proportions are reported by large companies and in the Asia Pacific region

à One in three of those interviewed did not know how many online surveys are attempted from smartphones

For the last three years, we have asked firms to estimate how many participants in their online surveys are taking them, or attempting to take them on a mobile device. The question has not varied in that time, but the response recorded has increased in each year.

Q: Thinking just about your web surveys, what proportion of participants are taking these surveys on small format mobile devices such as smartphones?

Figure 1 Respondents taking online surveys on smartphones: 3-year trend

Our survey also allowed companies to record that they did not know what the proportion was. In 2013, on a base of 224 companies answering the question, 82 (37%) reported they did not know what the number was. This is an increase from the previous year when 30% of the 233 companies asked could not give an answer.

Overall, the proportions reported by those who did provide an estimate (which is an average of all of the estimates given) is now 16.4% – almost one in six of all survey starts.

There are some variations by region and company size. Medium-sized and large companies tend to experience more smartphone participation than small ones,

16.4%!

13.1%!

6.7%!

2013!

2012!

2011!


Q: Thinking just about your web surveys, what proportion of participants are taking these surveys on small format mobile devices such as smartphones?

Figure 2 Respondents taking online surveys on smartphones: current year breakdown

Whether these variations are real, or can be attributed to under- or over-reporting by the participants in our survey is hard to tell. We know from other published sources, that the proportion varies greatly by the sample source being used. It tends to be lower when working with panels whose members tend to be habituated to taking surveys on PCs and laptops. Conversely, it tends to be higher among samples derived from customer lists and any sample where there is a large proportion of participants under the age of 25.

Of course, the most reliable method is always to examine this figure for any survey. As a matter of relevant interest, mobile participation on this survey was 5%, with a further 2% on tablet.

15.8%!

15.4%!

21.1%!

13.5%!

18.7%!

18.4%!

Americas!

Europe!

Asia Pacific!

Small!

Medium!

Large!

2013!N=142!


2.2 Platforms supported for mobile

à Most companies support mobile interviewing on browsers

à Fewer than half have access to a mobile app

à Android and Apple’s iOS dominate the platforms supported for app, browser and tablet, with Android in the lead

à Windows phone in 3rd place with much more support on browser than as an app

In last year’s survey we explored the need companies saw for software for mobile surveys delivered as Apps or Browsers. Specifically, we asked which method was considered the most viable. Twice as many companies cited browsers rather than apps, though one third of the sample reported both were equally viable.

This year, we asked about the specific platforms that companies were supporting or could support if they were to undertake mobile research in three flavours: self-completion mobile using the device’s browser, self-completion using an app, and interviewer administered, using a tablet.

The companies reporting they were able to support each platform were as follows:

Browser App Tablet

Companies supporting this method 168 100 78

% of full sample (N=240) 70% 42% 33%

Table 3 Companies able to support mobile interviewing in one form or another

The proportion supporting each method therefore followed a similar pattern to the reported ‘viability’ of either method last year, with many more favouring browsers over app. However, around two of every five companies (42%) are capable of delivering surveys via apps – which is a large proportion, considering that this method is still relatively fresh to research.


Q1 If you create online surveys to be taken on a smartphone's web browser, which of these mobile platforms can you support?

Q2 If you create a survey for delivery via a mobile app, which of these devices can you support?

Q3 If you create a mobile CAPI survey to be administered on a tablet, which of these devices can you support?

The base (N) is those offering each method N = 168 (Browser); 100 (App); Tablet (78)

Notes:

1. Symbian is absent from tablet as it was not offered as an answer choice (it does not apply to tablets)

2. Windows Phone only applies to browser and app; for tablet, the platform presented as an answer choice was Windows.

Figure 3 Mobile platforms supported for mobile interviews or interviewing

96%! 97%!93%!93%!

88%!73%!

67%!

45%!

53%!44%!

26%! 25%!32%!

15%!

5%! 1%! 3%!

Browser! App! Tablet!

Android!

Apple iOS!

Windows Phone!

BlackBerry OS!

Symbian!

Other!


2.3 Mobile platforms – regional and company variations

à A very consistent pattern across regions

à Support for BlackBerry stronger in N. America and weaker in Europe

à Large companies shunning BlackBerry in their mobile apps

Total N

America Europe Asia

Pacific Small Medium Large

Total (N) 168 100%

61 100%

78 100%

29 100%

76 100%

56 100%

36 100%

Android 96% 97% 96% 97% 96% 95% 100%

Apple iOS 93% 95% 94% 90% 93% 93% 94%

BlackBerry OS 44% 56% 36% 41% 46% 45% 39%

Symbian 32% 33% 31% 31% 34% 29% 31%

Windows Phone 67% 72% 67% 59% 64% 66% 75%

Other 5% 5% 4% 7% 3% 9% 3%

Table 4 Platforms supported for mobile browser-based self-completion interviewing

Total N

America Europe Asia

Pacific Small Medium Large

Total (N) 100 100%

34 100%

43 100%

21 100%

41 100%

29 100%

27 100%

Android 97% 97% 98% 95% 95% 97% 100%

Apple iOS 88% 97% 86% 77% 91% 83% 89%

BlackBerry OS 26% 35% 20% 23% 26% 37% 15%

Symbian 15% 9% 16% 23% 16% 13% 15%

Windows Phone 45% 50% 39% 50% 40% 47% 52%

Other 1% 3% 0% 0% 0% 3% 0%

Table 5 Platforms supported for self-completion interviewing using a mobile app


The share of each platform varies little by region, despite the low base sizes recorded in some of instances. There is similar consistency by company size, with some notable, though still weak differences.

Large companies are slightly less likely to support BlackBerry than other companies on browser; they are also much less likely to support BlackBerry as an app. than other companies.

The reverse is true for Windows phone, which they are more likely to support than other companies. Small companies are the least likely to support Windows phone.


3 Voice of the Customer

One area of conventional research where technology has facilitated considerable change is in the area of customer research. It has been an enabler, in allowing research data to be merged and analysed alongside data extracted from corporate databases or gathered from new sources such as from social media. Reporting systems and dashboards have also transformed the communication of customer feedback to boardrooms and to operational managers. It has been a disrupter, by allowing other data providers to enter the territory and claim it for their own who make scant use of conventional research methods.

Despite this, the capabilities that research companies offer (technologies, methods and skills) are complementary to the demands of the emerging disciplines of ‘Voice of the Customer’ (VoC) and ‘Customer Experience Management’ (CEM). As a growth area in its own right, it is considered by many to be an opportunity for research companies. However, some take a different position, and assert that it is poor substitute for proper research, and lacks the rigor that only research can bring.

We therefore chose to look this year at the impact of VoC and CEM on research companies’ technology and methodologies, as well by asking about the opportunities and challenges. We wished to understand the extent to which market research is responding to the challenges, and what that means for the technology that underpins research, as it too has to meet these challenges.

Questions

1. Is your company involved in any customer satisfaction, 'Voice of the Customer' or 'Customer Experience Management' programs for your clients?

2. When doing Voice of the Customer or Customer Experience Management work, what data collection methods do you apply?

3. When doing Voice of the Customer or Customer Experience Management work, which of these services do you provide?

4. What are the three greatest challenges you face when doing this kind of 'Voice of the Customer' work for your clients?

5. In which of these areas have you developed skills to do this kind of work (either through specialist recruitment, or by developing staff internally)?

6. In which of these areas do you partner with other specialists outside your organization to provide these skills?

7. What roles do you see your company providing in future in relation to 'voice of the customer' and customer experience management programs?


Key findings

Many research firms are embracing VoC and CEM work

§ Seven out of every ten research firms are involved; four out of ten consider it a major or growing part of their business

§ Almost all of these firms provide full customer satisfaction surveys, and a majority use the Net Promoter Score in various ways

§ A minority include other feedback gathering methods outside of surveys

A lot of the services offered by MR companies in this area are still very conventional MR offerings

§ The range of services offered is relatively undeveloped

§ Very few embrace the non-traditional sources of customer behaviour data such as call centre logs or emails that the CEM data integrators use

It’s an area of work research companies find quite challenging, from many angles

§ Other data providers make for aggressive competitors

§ Other data not derived from surveys is often of poor quality

§ It’s hard to get the mix of reporting right

It also calls on a broad spread of new skills… § Some skills, such as statistical modelling and business consulting, firms see are

worthwhile developing in house

§ Other skills may be either outsourced or developed in house, such as graphic design or social media analysis

§ Companies are also recognising there are technical skills to develop, such as database management, knowledge management and software engineering

Research companies see a future role for themselves providing high quality data and also expertise in research methods – but not in the driving seat

§ Fewer companies saw research taking on the lead role in implementing VoC and CEM initiatives, or as innovators in new ways to capture customer feedback

§ Some remain sceptical that this is an area research should attempt to cultivate


3.1 Level of involvement by MR companies

à Seven out of ten research firms undertake some customer satisfaction, VoC and CEM work

à Only four in ten consider it a major or a growing part of their business

à Companies in Asia Pacific do the most while companies in Europe are least involved

Q1. Is your company involved in any customer satisfaction, 'Voice of the Customer' or 'Customer Experience Management' programmes for your clients? N = 240

Figure 4 Level of involvement by research companies in VoC and CEM programmes

16%!

15%!

14%!

24%!

12%!

23%!

17%!

23%!

29%!

18%!

21%!

15%!

28%!

37%!

14%!

12%!

15%!

18%!

12%!

13%!

22%!

18%!

21%!

15%!

24%!

26%!

13%!

2%!

29%!

24%!

38%!

13%!

35%!

23%!

22%!

Total!

N America!

Europe!

Asia Pacific!

Small!

Medium!

Large!A major part of our work!

Increasing part of our work!

A minor but constant part of our work!

Occasionally involved!

Not involved in this work!


3.2 Data collection methods used

à Customer sat studies are used by almost all companies working in this area of research

à IVR surveys are more popular than quick surveys on mobile devices

à NPS is widely practiced both on its own and in the context of a longer survey

à NPS less popular but communities more so in Asia Pacific

We were interested to know the extent to which research companies were reaching out beyond conventional customer satisfaction surveys for their Customer Experience Management work in the ways in which they collected the data. We were also interested to know the impact of the Net Promoter Score methodology (which, within the research profession, has its promoters and detractors).

Q. When doing Voice of the Customer or Customer Experience Management work, what data collection methods do you apply? N = 170

Figure 5 Data collection methods applied to VoC or CEM work – overall

94%!

39%!

31%!

30%!

21%!

17%!

11%!

4%!

Customer satisfaction surveys!

NPS surveys limited to the one or two NPS questions!

NPS surveys with additional questions!

Interactive voice response surveys!

Comment cards or short feedback!

Micro-surveys on mobile devices!

Customer communities or forums!

Others!


Overall, it appears it is the conventional customer satisfaction survey that continues to dominate. We listed other methods too which are widely used within the VoC and CEM world, which do not have necessarily the same sample-based rigour of the ‘c sat’ survey, but while lacking depth, can offer breadth, and be more immediate in identifying issues and providing early feedback. Beyond NPS, the uptake was fairly low – 21% working with comment cards, ‘How did we do’ surveys or other quick feedback methods, and only 17% using micro-surveys on mobile devices, which can be triggered by an email, scanning a QR barcode, or, in retail, an invite on a till receipt.

Customer communities and forums were cited by only 11% of companies. This low uptake is consistent with previous findings from this study when we have looked specifically at the prevalence of communities or MROCs. In our 2012 study, we found only 16% of research companies were operating any research communities. We were expecting a slightly higher level of utilisation in CEM and VoC work, because of its relevance to the sector. We therefore consider it surprising that only 11% of firms report using communities in this context.

3.2.1 Global variations There was a high degree of consistency across the most common channel – the customer satisfaction survey, which is also the most recognisable conventional research method. After that, a pattern emerged that uptake in Europe and North America were broadly similar for the other methods cited, but in each case uptake in Asia Pacific was much lower.

The exception to this – though based on quite small numbers – is in the prevalence of customer communities. However, a greater prevalence of communities in the Asia Pacific region has been observed in previous years of this study, in different questions.

Q. When doing Voice of the Customer or Customer Experience Management work, what data collection methods do you apply? N = 170

Figure 6 Data collection methods applied to VoC or CEM work – by global region

94%!

44%!

35%!

33%!

21%!

23%!

9%!

5%!

94%!

45%!

32%!

31%!

24%!

18%!

11%!

4%!

91%!

15%!

18%!

21%!

15%!

3%!

15%!

3%!

Customer satisfaction surveys!

NPS surveys limited to the one or two NPS questions!

NPS surveys with additional questions!

Interactive voice response surveys!

Comment cards or short feedback!

Micro-surveys on mobile devices!

Customer communities or forums!

Others!

N America!

Europe!

Asia Pacific!


3.3 Services offered

à Range of services offered is relatively undeveloped

à Other available sources of customer measurement data are rarely used

The aim of this question was to probe the extent to which market research companies’ services diverged from their conventional offerings and into the more holistic requirements of a balanced customer experience management programme. Providing context, and linking attitudes and sentiment to behaviour and external changes is an important aspect of VoC and CEM programmes. Equally, so is communicating the data effectively to as wide an audience as possible, so appropriate actions can be taken at every level in the organisation.

Q. When doing Voice of the Customer or Customer Experience Management work, which of these services do you provide? N = 170

Figure 7 The Range of VoC and CEM-oriented services provided by research companies

Just over half of the companies (52%) say they do report on satisfaction data alongside data from other sources, but only 44% are providing this in dashboard format.

52%!

44%!

40%!

25%!

18%!

9%!

9%!

8%!

10%!

Reporting satisfaction data with other data!

Dashboards drawing data from different sources!

Real-time customer service alerts!

Social media monitoring!

Automated data feeds into customer's reporting system!

Analysis of call center logs!

Analysis of customer email traffic!

Analysis of website logs!

Something else!


Another point of difference is the need to work with data at an individual level. If an interview identifies a customer issue that has not been resolved, there needs to be a quick referral to the relevant team to deal with. In practice, this means an automatically triggered alert – possibly delivered by email – with relevant responses from the individual disclosed to the customer service representative, so that the problem can be discussed in full.

Research professional bodies today recognise that direct follow-up in this context is not a breach of research ethics. CEM proponents argue that not having a closed loop so that a customer issue raised in a survey does not get dealt with is not just bad business, but is also unethical from the customer’s perspective.

Our study shows only 40% of companies are offering this level of closed-loop intervention using customer service alerts.

Another key difference between what the CEM industry does, and what market research customer satisfaction has traditionally done, is the integration of customer satisfaction scoring with measurement data and performance data captured from elsewhere around the business. Our question included several of these potential sources of data that the CEM consultants tend to make use of: call centre logs such as customer service call transcripts, customer emails, website logs, as well as social media monitoring. Whilst one quarter of firms (25%) report they do include some social media monitoring, the other forms of data were barely mentioned.

Overall, it would appear that the opportunities to meet and out-perform what the CEM consulting firms offer are being missed by research companies, who appear not to be moving beyond the comfort zone of their conventional survey processes.


3.4 Challenges faced

à Aggressive competition from other data providers comes top, followed by data quality issues with non-survey data

à Overall there is a very broad spectrum of challenges that companies cited

Our question asked companies to rank the top three out of eleven different challenges we had identified, which you can see in the accompanying chart.

CEM and VoC as disciplines were born outside of the research industry, and research is clearly still having to assert its position in a marketplace that is not its natural home. Aggressive competition was given as a first choice and also a top three choice by more firms than any of the other challenges.

Q. What are the three greatest challenges you face when doing this kind of 'Voice of the Customer' work for your clients? N = 155

Figure 8 The greatest challenges faced by research companies when doing VoC or CEM work

26%!

13%!

10%!

12%!

3%!

5%!

7%!

7%!

7%!

4%!

3%!

5%!

37%!

33%!

32%!

28%!

23%!

22%!

22%!

19%!

19%!

16%!

16%!

7%!

Aggressive competition from other knowledge or data providers!

Poor quality or incomplete external data to work with!

Creating reporting systems that provide real-time operational data and management overviews!

A lot of the data is in the form of comments or free text!

Integrating our systems with our clients' systems!

Having the expertise to deal with sources of data we have not collected ourselves!

The client's demands are not compatible with market research methods!

Don't have the right tools or technology for this kind of work!

Integrating our data streams with those of our client!

Uncooperative attitude of corporate IT departments!

Ethical issues or conflicts with professional standards for market research!

Something else!

Top mention!

All mentions!


Researchers are used to working with relatively small amounts of carefully collected data, over which care is exerted to ensure it is accurate and internally consistent. The opposite is the case with much operational data, which is often incomplete and inconsistent, and the ‘bigger’ the data, the worse it gets. The problems of working with ‘poor quality and incomplete external data’ was a close second to the ‘aggressive competition’ challenge.

The first two challenges cited are therefore largely outside the control of the research company and can be considered to ‘go with the territory’. The third, however, is one that can be addressed by technology and capability development – that is ‘Creating reporting systems that provide real-time operational data and management overviews’. These are the kinds of dashboard and drill-down systems, which provide individualised perspectives on the data for different managers within an organisation, which CEM providers and data integrators are often adept at providing. It is an area where there are very few research data-specific tools at present, and many mainstream research reporting systems are not built with this objective in mind.

The next challenge was dealing with text comments – an activity that research has had decades of experience with, but appears not to be adapting well to the additional demands that volume or scale bring. We look more specifically at text analytics in Section 4 Text analysis and coding on page 34, where it is clear that research companies are still heavily reliant on old, manual methods, and remain sceptical of automated, or auto-assisted methods for handling free text responses.

However, in this question, having the right tools came relatively far down the list of challenges.

Interestingly, ethical issues were only seen as the key challenge by 3% of companies working in the VoC and CEM sector, though the demands of clients to do things that were not compatible with research was the top challenge for 7% of companies. Taken together, this means one in ten firms are finding the realities of fitting a research-inspired ethos to this kind of work to be a significant challenge.


3.5 Special skills developed internally

à A very broad range of new skills get mentioned

à Most research companies have focused on developing statistical dexterity and business consulting acumen

à Technology skills, including database design, knowledge management and software engineering are favoured by one in four to one in three firms

We mentioned eleven areas of skill that VoC and CEM work can demand, and which the data integrators and knowledge management consulting companies are often adept in. From a research industry perspective it is encouraging to see that companies are investing in the internal skills they need to compete in this area.

Q. In which of these areas have you developed skills to do this kind of work (either through specialist recruitment, or by developing staff internally)? N = 170

Figure 9 Additional skills companies have developed to handle VoC or CEM work

57%!

49%!

41%!

37%!

35%!

31%!

31%!

29%!

25%!

24%!

24%!

2%!

Statistical modelling/ statistical analysis!

Business consulting!

Customer experience management!

Database management/ database programming!

Text analysis/ text mining!

Customer communities!

Data mining!

Information design/ knowledge management!

Software engineering!

Social media analysis!

Graphic design!

Other skills!


The most highly prized skill to acquire it emerges is in statistical modelling and analysis (57%) followed closely by business consulting (49%).

Text analysis and text mining was fifth in order of mention, but cited by 35% of companies, which relates to our previous comment on the challenges perceived from having more data as free text available than in traditional quantitative market research surveys.

Technological skills are also being developed. Of the three technical skills we mentioned, database management and related programming skills was top of the list, with 37% of companies mentioning it. Skills in information design and knowledge management – a relatively new discipline – have been developed by 29% of companies. But full-blown software engineering was the preserve of the minority – only 25% of firms have taken that route – 46% of large firms, 26% and 18% of medium-sized and small firms respectively.

In general, and not surprisingly, it is the larger research companies that are investing most heavily across the range of skills. Although smaller research companies are clearly making a significant investment too.

47%!45%!

33%!

26%! 27%!

17%!

24%!

18%! 18%!

11%!

14%!

66%!

53%!51%!

47%!

34%!

42%!40%!

34%!

26%! 26%!

30%!

66%!

56%!

47%!

50%!

59%!

50%!

34%!

50%!

44%!

56%!

38%!



Customer experience

management!

Database management/

database programming!



Data mining! Information design/

knowledge management!



Graphic design!

Small! Medium ! Large !


3.6 Additional skills resourced externally

à Statistical modelling, in addition to being developed internally, is frequently outsourced

à Graphic design skills outsourced more than developed internally

à Social media analysis features highly as a ‘bought in’ skill

à Internal skill development in general appears to be winning over outsourcing

Q. In which of these areas do you partner with other specialists outside your organization to provide these skills? N = 170

Figure 10 Additional skills resourced externally for VoC or CEM work

29%!

27%!

27%!

24%!

23%!

21%!

19%!

16%!

15%!

12%!

9%!

4%!

Graphic design!

Statistical modeling or statistical analysis!




Text analytics and text mining!

Database management or database programing!


Data mining!

Information design or knowledge management!


Other skills (please specify)!


Overall, less outsourcing was reported than the internal development of skills, which to us is a surprising finding. There are only two skill areas where firms report they are outsourcing more than internally developing those skills, and these are graphic design and social media analysis – where firms appear to prefer to work with external partners.

3.6.1 Internal up-skilling and externally sourced skills compared For ease of comparison, the two previous questions are presented together, listed in alphabetical order, to show where companies are favouring internal skill development over hiring in the necessary skills. Overall this also shows that firms are tending to develop skills internally rather than seek those skills externally.

Q1. In which of these areas have you developed skills to do this kind of work (either through specialist recruitment, or by developing staff internally)? Q2. In which of these areas do you partner with other specialists outside your organization to provide these skills?

N = 170

Figure 11 Additional skills resourced externally for VoC or CEM work

49%!

31%!

41%!

31%!

37%!

24%!

29%!

24%!

25%!

57%!

35%!

2%!

24%!

16%!

9%!

15%!

19%!

29%!

12%!

27%!

23%!

27%!

21%!

4%!




Data mining!

Database management/ database programming!

Graphic design!

Information design/ knowledge management!





Other skills! Internal! Outsource!


3.7 Future positioning of market research within VoC & CEM

à Research data and research methods top the list of what MR sees it can bring to the VoC/CEM table

à A minority take the view that MR can develop a leading role in consulting or innovating in the field

Our last question in this section explores where companies see the future may lie for research within VoC and CEM. This was asked of all companies, not just those currently providing customer satisfaction or CRM work.

Top of the list comes what is the most conventional output from the research industry – providing reliable survey data (cited by 62% of firms). The second though, is an innovation, in companies seeing a role for providing expertise in research methods, rather than necessarily doing the research, which appeals to half of those surveyed (49%)

Q. What roles do you see your company providing in future in relation to 'voice of the customer' and customer experience management programs?

Figure 12 Predictions on the roles research companies may perform in future

62%!

49%!

38%!

29%!

28%!

27%!

26%!

20%!

7%!

Providing reliable survey data!

Providing expertise in research methods!

Providing expertise in data reliability!

An innovator in new ways to gather customer feedback!

A leading role in operating these programs!

An integrator of different sources of data!

A leading role in consulting and advising on these programs!

Providing expertise in other methods (beyond research)!

Something else!


Possibly more disappointing was that fewer companies saw a future for market research being a centre of innovation in the field of gathering customer feedback in new ways (only 29%), or taking on a lead role in consulting and advising on VoC and CEM programs (26%).

The majority of the 7% who responded “something else” said they did not see a role for MR in this activity, or that it was something they would not wish to focus on.


4 Text analysis and coding

Two years ago, we asked companies about the extent they were exploiting what technology has to offer in the area of computerised analysis and interpretation of text. As we are focusing on the growth and the impact of the Voice of the Customer movement on research, where a lot of the inputs come in the form of unstructured text, we thought we would repeat some of the questions from 2011, to see if things had moved on.

At that time, it appeared market research was still in an early adoption phase of technology for text analytics and machine-based text processing. It had aspirations to do more, but everyday practice showed these technically-driven innovations were not making much impact.

Questions

1. We would like you to think about the amount of work you do where there is a large amount of unstructured text that you need to analyse. How do you predict this will change over the next 1-2 years?

2. What methods do you ever use when analysing large amounts of unstructured text?

3. Which of those methods do you use most frequently when analysing large amounts of unstructured text?

4. Which of these methods are you likely to make greater use of in the future?

5. With the tools that you use, how easy or difficult do you consider it is to deliver insights from unstructured text from online qualitative or social media research activities?

6. What are the difficulties you experience when analysing large amounts of unstructured text?

Key findings

Market research companies are handling ever more unstructured text

§ But much of it is being analysed manually

§ Word clouds popular – but are they really useful?

Little growth in the use of high tech text analysis methods since 2011

§ Manual coding is still the most frequently used method for over half of companies

§ Semi-automated coding is used by just over 1 in 5 companies as the main method

§ With the exception of word clouds, all other methods are used by less than 5% of companies as their main method

Text analytics and sentiment analysis are predicted to grow in the future


§ Companies are more cautious about committing to using automated or even semi-automated coding solutions in the future

Maybe we should blame the tools?

§ Companies in all regions and of all sizes agree: analysing unstructured text is difficult!

§ They are concerned about the quality of the analysis or outputs from text processing methods when applied to coding


4.1 Unstructured text: predicted trends

à Most companies, and especially large ones, anticipate the amount of text they are handling will grow

à Overall, one in four companies predict it will grow strongly in the next 1-2 years

à In North America and among large companies, that rises to one in three predicting strong growth

The great majority of companies expect there to be more or even much more text to handle in future. A minority think the future that will be broadly similar to the volumes handled now and only a small handful of firms are anticipating any decline.

Large companies are anticipating the strongest growth – 37% foresee a strong increase, and 86% are anticipating some level of increase. The strength of the predictions seems to follow company size, with fewer medium sized (74%) and smaller companies (60%) anticipating growth and more predicting a level course.

Companies in Europe are also predicting less growth than their counterparts in Europe or Asia Pacific – only 18% in Europe anticipating strong growth, versus 35% in North America, for example, and a similar pattern at other reported levels of activity.

Q1. We would like you to think about the amount of work you do where there is a large amount of unstructured text that you need to analyse. How do you predict this will change over the next 1-2 years? N=240

Figure 13 Predicted change in how much unstructured text in surveys in the next 1-2 years

25%!

35%!

18%!

26%!

21%!

28%!

37%!

43%!

38%!

43%!

55%!

39%!

46%!

49%!

28%!

26%!

34%!

16%!

35%!

23%!

15%!

3%!

1%!

5%!

3%!

5%!

3%!

Total!

N America!

Europe!

Asia Pacific!

Small!

Medium!

Large!

Increase strongly! Increase slightly! Remain the same! Decrease!


4.2 Text analysis methods currently used

à Manual coding still dominates text analysis

à Word clouds are a primary tool for one in ten firms

à Automated text methods have gained some ground in the last 2 years as complementary methods

In market research, most unstructured text is still read by humans and usually coded manually, using techniques that are largely unchanged since the dawn of modern market research in the 1940s. The only concession made to modern technology is that very often, coding is collated directly into an Excel spread sheet, and then merged back into the data, rather than physical codes being applied to printed questionnaires. So coding may have gone paperless, but it still relies on time-consuming and costly human effort for 56% of the companies interviewed that mentioned this as their principal method for handling unstructured text.

Q2. What methods do you ever use when analysing large amounts of unstructured text? Q3. Which of those methods do you use most frequently when analysing large amounts of unstructured text? N = 240

Figure 14 Text analysis methods – all methods and main method currently used

Semi-automated coding, whereby software is used to help coders organise their work better – and often allows some element of mass coding of identical responses, or the use of searching and grouping to reduce human time and effort. These are used by 22% of companies routinely, as their principal method, and by 55% of companies for at least some of their work.

83%!

55%!

48%!

31%!

23%!

20%!

20%!

17%!

6%!

3%!

56%!

22%!

9%!

2%!

3%!

1%!

3%!

1%!

1%!

3%!

Manual coding!

Semi-automated coding!

Word clouds!

Word frequency distributions!

Automated coding!

Sentiment analysis!

Text analytics/text mining!

Topic analysis!

Crowdsourcing!

Other!

Ever use!

Main method!


We included Word Clouds as a method, as we had when we originally asked this question in 2011. We contend that Word Clouds are a presentation method rather than an analytical method. They are highly problematic as an analytical method, as they omit a lot of detail that can only be found by reading a word in context. One simple example reveals the problem. In a customer satisfaction survey, “like” may suggest a positive response if it were presented in a Word cloud, but not if part of the phrase “don’t like” or “not much to like”, and it has entirely different meanings (as a homonym) in “designed for people like me”, or “there was no Like button”. It is therefore worrying to see that the method referred to by 48% of companies, and by 9% as a principal method, should be the Word Cloud.

Other more advanced methods are mentioned very infrequently – such as sentiment analysis, text mining and topic analysis.

4.2.1 Text analysis methods in use: 2013 compared with 2011 Compared to the responses two years previously, the changes are modest, though it does appear that the automated methods are gaining more popularity as one of the methods being used – even if they are not being applied as the principal method. As this survey is based on a relatively small sample, individual percentage point differences need to be treated with some scepticism, but an overall pattern has emerged here of growth in these methods.

Q2. What methods do you ever use when analysing large amounts of unstructured text? N = 230 (2011), 240 (2013)

Figure 15 Text analysis methods – all methods used, two-year comparison

Semi-automated coding has grown by 4%, word frequency distributions by 11%, text analytics has grown by 4% and topic analysis by 7%. Only sentiment analysis has not registered growth – it has dropped by 6%.

84%!

51%!

38%!

20%!

17%!

26%!

16%!

10%!

2%!

1%!

83%!

55%!

48%!

31%!

23%!

20%!

20%!

17%!

6%!

3%!

Manual coding!


Word clouds!


Automated coding!

Sentiment analysis!


Topic analysis!

Crowdsourcing!

Other methods!2011!

2013!


However, comparing the principal methods used, as reported in 2011 and 2013 (see Figure 16), it does not look as if any new methods are displacing manual coding – and in fact fewer companies report they are using semi-automated coding now than they were two years ago.

Q3. Which of those methods do you use most frequently when analysing large amounts of unstructured text? N = 230 (2011), 240 (2013)

Figure 16 Text analysis methods – main method used, two-year comparison

52%!

32%!

2%!

5%!

4%!

2%!

1%!

1%!

0%!

1%!

56%!

22%!

9%!

2%!

3%!

1%!

3%!

1%!

1%!

3%!

Manual coding!


Word clouds!

Word frequency

Automated coding!

Sentiment analysis!


Topic analysis!

Crowdsourcing!

Other!2011!

2013!


4.3 Predicted changes in methods applied in the future

à Text analytics and sentiment analysis predicted to grow

à Automated coding has not gained credibility yet

Q4. Which of these methods are you likely to make greater use of in the future? N = 240

Figure 17 Text analysis methods likely to make greater use of in the future, compared with current methods used

Asked to predict the tools our participants could see their companies using in the future, and a picture emerges that automated methods will become much more dominant. Manual coding shrinks to 20%, and is overtaken by semi-automated coding (38%), followed by text analytics and sentiment analysis (both 33%) and regrettably, more word clouds (25%).

83%!

55%!

48%!

31%!

23%!

20%!

20%!

17%!

6%!

3%!

20%!

38%!

25%!

18%!

33%!

27%!

33%!

12%!

8%!

1%!

12%!

Manual coding!


Word clouds!

Automated coding!



Sentiment analysis!

Topic analysis!

Crowdsourcing!

Other!

None of these!Current!Future!


4.3.1 Text analysis methods future predictions: 2013 versus 2011

Q4. Which of these methods are you likely to make greater use of in the future? N = 230 (2011), 240

(2013)

Figure 18 Text analysis methods likely to make greater use of in the future: two-year comparison

Again, this question was also asked in 2011, and the above chart shows the difference between the predictions made then, and in this study. While the predictions are broadly similar, it appears that sentiment analysis has gained credibility as a viable method in the last two years, with 35% considering they would make greater use of it. That and word frequency distributions are the only method that appears to have gained ground. Automated coding has taken a notable tumble – down from 33% in 2011 to 18% in 2013. All the others have lost one or two percentage points, which should be considered as to be within the margin of error in this study. (We are not able to calculate a reliable margin of error due to the sampling method we use, but consider it prudent to consider 2-3% to amount to ‘no change’).

29%!

43%!

27%!

33%!

38%!

17%!

20%!

14%!

5%!

1%!

7%!

20%!

38%!

25%!

18%!

33%!

27%!

33%!

12%!

8%!

1%!

12%!

Manual coding!


Word clouds!

Automated coding!



Sentiment analysis!

Topic analysis!

Crowdsourcing!

Other!

None of these!2011!2013!


4.4 Perceived effort in analysing open text

à More companies find it is difficult to analyse open text than those finding it easy

à This situation has not changed since reviewed in 2011

à Large companies and companies in North America both consider it to be more of a problem

Q5. With the tools that you use, how easy or difficult do you consider it is to deliver insights from unstructured text from online qualitative or social media research activities? N = 240

Figure 19 Perceived effort in deploying the tools available to analyse unstructured text: variations by region and company size

Generally, this chart shows that many firms encounter a level of difficulty that exceeds what you would expect if the question were asked of mature and widely-used technology – e.g. online surveys, or cross-tabs. Those reporting difficulty (41%) outweigh those finding it easy (30%) by 11 negative percentage points (subtracting 30% from 41%).

Large companies seem to be finding it harder than smaller ones – they are perhaps more acutely aware of the difficulties when dealing with larger volumes and more people performing the task. They also are more likely to contend with the problem of dealing with multi-lingual studies and how or where to translate these texts. The difference for large companies, using the same ratio, is –17%. It is also perceived more of a problem in North America where the gap is –22%; less so in Europe where difference narrows –3%, but is still negative.

5%!

3%!

4%!

13%!

5%!

6%!

5%!

25%!

25%!

27%!

21%!

23%!

30%!

24%!

28%!

22%!

36%!

21%!

34%!

20%!

24%!

33%!

43%!

24%!

40%!

32%!

32%!

42%!

8%!

7%!

10%!

5%!

7%!

12%!

5%!

Total!

N America!

Europe!

Asia Pacific!

Small!

Medium!

Large!

Very easy! Moderately easy! Neither easy nor difficult! Moderately difficult! Very difficult!


Remarkably, this picture is unchanged since 2011, as the overall totals come to the same, within half a percentage point (see Figure 20).

Overall, this indicates an area of activity where the tools or methods being deployed are suboptimal, if not dysfunctional.

Q5. With the tools that you use, how easy or difficult do you consider it is to deliver insights from unstructured text from online qualitative or social media research activities? N = 230 (2011), 240 (2013)

Figure 20 Perceived effort analysing unstructured text: 2011 compared with 2013

6%!

5%!

25%!

25%!

28%!

28%!

33%!

33%!

8%!

8%!

2011!

2013!

Very easy! Moderately easy! Neither easy nor difficult! Moderately difficult! Very difficult!


4.5 Difficulties in analysing unstructured text

à ‘Not having the right tools’ is the highest first choice among the difficulties reported

à Companies are also critical of the poor quality results they obtain from automated methods

à The time required to analyse these data is also a critical factor

This question was first asked in 2011, but as an open question. The code frame used in 2013 was derived from the responses given in 2011.

A very wide variety of difficulties were expressed. These appear to cluster into three groups (see Table 6, below), when categorised by order of magnitude of the reported problem. In the same table, we also provide the complete text of the answer choices presented to survey, which have been abbreviated in Figure 21.

Q6. What are the difficulties you experience when analysing large amounts of unstructured text? N = 230

Figure 21 Difficulties experienced in analysing large amounts of unstructured text

1%!

1%!

3%!

1%!

5%!

7%!

6%!

26%!

17%!

12%!

20%!

3%!

13%!

15%!

15%!

16%!

27%!

33%!

36%!

37%!

38%!

47%!

Other!

Clients distrustful of automated methods!

Tools not relevant to market research data!

Skills hard to find!

Researchers distrustful of automated methods!

Badly organized raw data!

Clients unwilling to pay!

Do not have right tools!

Poor quality raw data!

Takes too much time!

Poor quality from automated methods!

Top 3 reasons!

Main reason!


As with other “issues” questions, this question was presented as a ranked order question, with everyone able to choose up to three reasons, placed in their preferred order of priority.

Problem reported Scale of the problem

• The quality of analysis from automated methods is not good enough

• It takes too much time • The raw data are often of poor quality to start with • We do not have the right tools for effective text analysis

The main problem for many

• Clients are unwilling to pay for the work required • The raw data are often badly organized or structured,

making them difficult to handle

A problem for many but the main problem for only a few

• Researchers are distrustful of automated methods • It requires specialized skills that are hard to find • The analysis tools on offer are not relevant to market

research data • Clients are distrustful of automated methods

A problem for a few and the main problem for very few

Table 6 Categorisation of the difficulties reported in analysing large amounts of open text

4.5.1 Differences by region and company size There are three interesting differences in the responses according to company size.

Large companies were more critical of the quality of results from automated methods that small companies (28% of large companies, 18% of medium and 18% small).

30% of small companies reported they did not have the right tools, against 22% of medium-sized companies and 19% of large ones.

Although the numbers involved are smaller, so some caution should be applied in interpreting these, a mere 2% of small companies reported clients were unwilling to pay for textual data analysis, against 13% of medium-sized companies and 8% of large companies.

The differences by global region were insignificant, with the solitary exception of not having the right tools – cited by 34% of companies in Asia Pacific, against 26% in North America and 23% in Europe.


5 Ten Years of Technical Innovation

Questions

1. This is the tenth year of this annual survey. Looking back on the technological changes over the last ten years in market research, which of these developments do you consider have had the greatest influence?

2. Which of these technological or environmental changes do you consider have been the most disruptive for market research?

3. If you were to start your own research company tomorrow, and you were to buy one piece of software or technology, what would you choose to give your new company the greatest business advantage?

4. Imagine you see a copy of the results of this survey in ten years time. What do you think is the biggest technological advance for research that we would be reporting in the year 2023?

Key findings

A high level of agreement on what the positive developments have been

§ Our panel has judged this to have been the decade of the mobile device – it was picked by a majority of those responding

§ Beyond that, opinions split between whether it is big data – which was next in line for people’s first choice, or social media and the social web.

A clear majority see ‘DIY’ surveys having been the most disruptive development

§ Two-thirds picked DIY surveys and 30% ranked it top

§ Beyond that, privacy concerns and mobile displacing fixed line telephony took second and third position, with some interesting regional variations behind them

§ The rise of social media also pulled rank as a disrupter as well as more benign influencer


5.1 Greatest positive influences

à It’s mobile, smartphones and tablets

à …followed by social media and the social web

à …then multi-modal research, and then ‘big data’

à Though, when looking only at participants’ first choices alone, ‘big data’ came second

It was hard to choose our list of influences – there have been so many advances over the last decade. However, we deliberately excluded online research simply because its advent precedes 2004, when our first survey was carried out, by many years, even though our annual survey has, over that period, charted the inexorable rise of online surveys and many other online methods too, and it has probably been the decade in which online became truly mainstream.

Q1. This is the tenth year of this annual survey. Looking back on the technological changes over the last ten years in market research, which of these developments do you consider have had the greatest influence? N = 230

Figure 22 The greatest positive influences on market research in the last ten years

50%!

9%!

8%!

12%!

1%!

7%!

0%!

0%!

1%!

1%!

0%!

5%!

5%!

80%!

48%!

35%!

29%!

20%!

17%!

17%!

10%!

9%!

7%!

5%!

7%!

5%!

Smartphones, tablets and mobile technology!

Social media and the social web!

Multi-modal research!

Big data!

Text analysis and text mining technologies!

Cloud computing!

Computer-based graphics!

Geolocation technologies!

Digital video technology!

Biometrics (e.g. facial recognition) and

Voice recognition!

Something else!

Nothing/not stated!

Top mention!

Other mentions!


Nevertheless, we were surprised to see some innovations that failed to capture the imagination of all but a handful of our participants: cloud computing, computer-based graphics among them, which we argue have been highly influential in how research technology has developed over that period.

5.1.1 Regional differences There were few notable differences by global region or by company size. The first two ranking positions are identical by region, but each diverges for the third position, which is seen as ‘multi-modal research’ in North America, ‘Social media and the Social web’ in Europe and ‘Cloud computing’ in Asia Pacific.

N America Europe Asia Pacific Smartphones, tablets and mobile technology

46% Smartphones, tablets and mobile technology

53% Smartphones, tablets and mobile technology

53%

Big data 13% Big data 11% Big data 11%

Multi-modal research 10% Social media/social web 10% Cloud computing 11%

Table 7 The primary positive influences in the last 10 years: top three by global region


5.2 Greatest disruptive influences

à DIY surveys tools are causing the greatest anguish

à Privacy concerns worry companies in Europe and Asia Pacific, much less so in North America

à Mobiles displacing fixed line telephony is of widespread concern, most acutely in North America and Asia Pacific

The disruptors have divided opinion to a much greater extent than the more benign influences just described, though DIY surveys commanded 30% of the first place votes as the top disrupter, and was mentioned by 63% overall. Privacy concerns and mobiles replacing fixed lines effectively tied for second place – though we note some interesting regional differences below.

Big data does not seem to loom large in people’s minds as a disrupter. It earlier earned its place as a high-ranker in the positive influencers. So it seams research companies are viewing it more as friend than foe.

Q2. Which of these technological or environmental changes do you consider have been the most disruptive for market research? N = 230

Figure 23 The greatest disruptive influences on market research in the last ten years

30%!

18%!

18%!

8%!

11%!

5%!

1%!

4%!

2%!

63%!

43%!

42%!

33%!

29%!

20%!

18%!

9%!

7%!

DIY (Do It Yourself) survey tools!

Privacy concerns and do-not-call lists!

Mobile phones displacing fixed line telephony!

The rise of social media!

Smartphones and the mobile web!

Cyber-crime and internet fraud!

Text mining and sentiment analysis!

Big data!

Something else (please specify)!

Top mention!

Other mentions!


5.2.1 Regional differences Whilst there were few notable differences by global region or by company size with the positive influences, more differences were apparent with the disruptive influences. Here we will focus just on the top mention – the one each firm ranked in first position.

While North America and Europe both agree the foremost disrupter has been the DIY survey (with a 32% and 33% share of the poll, respectively), in Asia Pacific, it is privacy and do-not-call lists that of primary concern (though less strongly, with 26% of the vote). Europe follows closely behind Asia Pacific on this one – it is ranked second overall, with 23% choosing it. Whereas North America seems generally unconcerned by privacy concerns – it is sixth in the ranking, picked by only 7%.

North America and Asia Pacific agree on the disrupter that appears in second place – mobile phones displacing fixed lines, which has massive implications for CATI interviewing. This is apparently less of a concern in Europe – where fewer than half the number of companies rated this as their primary disrupter, though it did make it to third place still.

N America Europe Asia Pacific

DIY (Do It Yourself) survey tools

32% DIY (Do It Yourself) survey tools

33% Privacy concerns and do-not-call lists

26%

Mobile phones displacing fixed line telephony

26% Privacy concerns and do-not-call lists

23% Mobile phones displacing fixed line telephony

21%

Smartphones and the mobile web

13% Mobile phones displacing fixed line telephony

10% DIY (Do It Yourself) survey tools

18%

Table 8 The primary disruptive influences in the last 10 years: top three by global region


5.3 The technology that offers the greatest advantage

à Almost everyone has a different opinion

à Some common ground: many wanted easy-to-use, integrated, web-based platforms

This was not only an entirely open-ended question, but it was written as a “gamified” question, in order to stimulate a wider range of responses. The actual wording of the question was:

If you were to start your own research company tomorrow, and you were to buy one piece of software or technology, what would you choose to give your new company the greatest business advantage?

It was asked in this way in order to free participants of the constraints that might restrict choices, such as company policies, or the influences of others in arriving at a choice. The scenario provided was intended to invoke a personal choice – i.e. what would you buy, if it was your money.

Such questions take more effort to answer. 28 participants (12%) gave no answer and a further 39 (16%) said they did not know what they would choose. However, 173 participants (72%) responded with a very broad spectrum of responses.

Many of the participants cited particular brands of technology they would buy but many instead gave a wish list of features or benefits they would like to have - with ease of use, integrated platforms and web-based software being some of the most frequently mentioned desires.

A number of people gave relatively expansive responses, which may provide some inspiration to software developers! For example:

“Mobile-first integrated community (qual)/structured data collection (quant) with associated mobile first app (supporting all major smartphone platforms), including back-end community management tools with strong social-media based collaboration capabilities to support analyst/end-client collaboration, and good "DIY"/end-user-oriented toolset.”

“Online survey software that uses responsive design to format automatically on any screen, with an app option, supported by a direct feed into an easy-to-define client portal that integrates other data sources.”

“Online reporting and analysis software that provides ability to incorporate multiple data sources, slice and dice and to create controllable levels of information to a client organisation.”

“An integrated CATI/CAWI system that allows one-click set-up of screens for different modes, with shit-hot real time online reporting.”

“A graphical package that has the ability to transform hard data into visual images, which can be easily revised and finalized.”

The wide range of opinions and ideas was striking:

“Online community panel software that would mix qual, quant techniques and capture passive data at a very low cost, and would work across all devices.”


“Email survey with automatic data analysis with crosstab functions that automatically highlights statistically significant differences.”

“Online data capture package with real time reporting dashboards with alerts. Real time reporting and dashboard capabilities give clients management information at their fingertips.”

“A CAPI technology with GPS-enabled time stamping apps and audio/video recording abilities for data collection so that interviews can be validated and QC problems are reduced”

“Integrated mobile survey platform that works across all operating systems with good engaging survey designs”

“Data mining software – many corporate clients have no expertise and technology to analyse corporate data resources spanning across the organization.”

“Neuroscience capable software/technology”

“Software that would be interesting for respondents to use so they would freely give their opinions on a variety of topics and software that would produce data that was easy for the client to understand and use in their business.”

Relatively few participants focussed on cost or efficiency issues, but they were occasionally mentioned:

“Cost effective and mixed mode data collection. I would look to fixed prices for online survey as opposed to per complete charges.”

“Efficient project management system to manage various data collection tools, providers and global teams.”

For some, however, ambitions were much more modest:

“Excel would cover most needs of a new company at a low cost. Very potent and flexible.”


5.4 Looking back from the future

à Real time data collection and analysis is expected to be the next game-changer

à Mobile data collection, big data and data integration all expected to continue in influence

à Augmented reality and thought/emotion detection also tipped as a future trend setters

The last question in this section focusing on ten years of innovation was another ‘gamified’ question. The aim of the question was to identify what the future for technology holds – by asking them to imagine what this report might look like in ten years time.

This time, participants were asked:

Imagine you see a copy of the results of this survey in ten years time. What do you think is the biggest technological advance for research that we would be reporting in the year 2023?

58 (24%) said they did not know, and a further 16 (7%) left no answer, leaving 166 (69%) who had suggestions to make on the subject. The responses were translated into English where necessary, and coded.

Predictions were split between continued dominance of trends in evidence today and some more exotic outcomes.

In the ‘more of the same’ category are mobile data collection (cited by 22), big data analysis (18), social media mining (13) and data/text mining (11).

In the more exotic terrain, lies the use of augmented reality (mentioned by 12), and measurement or monitoring of people’s thoughts, emotions and behaviour (9). Yet the overall leader was shifting to real time data collection and analysis, which could be described as “always on” research. Rather fewer predicted that automated analysis and reporting would be the great game changer (6 mentions) along with the use of biometrics (4) and photo and video analysis (also 4 mentions). Voice analysis was mentioned by three participants, as was neuroscience, and the notion that research is simply integrated into everyday life.

Rather more pessimistically, seven participants predict that in 10 years, the big story will be the death of conventional or “old school” data collection, via the survey, although that could be a consequence of research being fully integrated into everyday life, as mentioned by some.

Two technologies that have generated a lot of interest in recent years were mentioned relatively infrequently, considering the attention they have received: geolocation-based research was mentioned by five participants, and gamification by only two.


Number of mentions

Real time/live date collection/analysis 23

Mobile data collection 22

Big data analysis 18

Social media mining 13

Augmented Reality 12

Integration of data from multiple resources 12

Data/text mining/analysis 11

Online/web-based research/reporting 10

Thought/emotion/behaviour monitoring 9

Death of basic/old school research/data collection 7

Automated analysis/reporting 6

Geolocation research/technology 5

Automation (unspecified) 4

Biometrics 4

DIY based reporting 4

Photo/video analysis 4

Cloud computing 3

Neuroscience 3

Research integrated into daily life 3

Voice analysis 3

Gamification 2

Making internet accessible anywhere/everywhere 2

Quantum computing 2

Ubiquitous access to big/small data 2

Table 9 The technological innovation that is forecast to be most influential over the next 10 years.


Part 2: Trends

There is a central core of questions in this survey that we have been asking for several years – and some since the inception of this project in 2004. With several years of data, several clear and stable trends have emerged. This shows continued consistency in the survey as a whole, since the sample each year is made up largely of different companies and individuals than in previous years.

Topics covered in this section

§ Research activities § Sources of online sample § Mixed-mode § Analysis and results reporting § Changing software


6 Research activities

Questions

§ Volume of work that derives from quantitative, qualitative or other research activities?

§ Volume of work by research mode (CATI, CAWI etc)?

§ Anticipated changes in the amount of work?

Key findings

Mobile research growing rapidly

§ The number of companies offering mobile research (self completion, but not including SMS) is increasing rapidly. When we first measured this in 2009, 6% of companies offered this as a service. In 2013 it is 19%.

Volumes of paper and CATI research dwindling, Web strong

§ Web research represents just over half (51%) of all quantitative research and this proportion may now be stabilising.

Mobile volumes very small but growing fast

§ Some of paper and CATI work seems to be switching to mobile, where the volume is still very small but is growing rapidly.

Research companies expecting Web and mobile research to be main growth areas

§ The participants in this survey say that mobile and Web research are the areas they expect to grow the most (the latter to some extent contradicting our observation above)

Mixed mode projects starting to slacken?

§ Although volumes of mixed mode research have remained stable, the number of companies offering it was increasing, but in 2010 started decreasing

IVR and SMS very low volume, no sign of growth

§ Both these modes are languishing at around 0% and have been at around 0% to 1% for the duration of the time we have collected data


6.1 Research mix – Seven-year trend

à Almost a quarter of revenues derive from qualitative

à Qualitative vs. quantitative split is very stable over the years

à Research firms do very little non-research work

Over the years, the proportion of quantitative research revenues is to qualitative is remarkably stable. Quantitative represents around 70% and qualitative between 20% and 24%. There is a discontinuity in the chart, at 2009, because “non-research activities” was added as a new answer category.

Q: Please indicate the approximate proportion of your company revenues that derive from quantitative, qualitative or other research activities.

N = 240

Figure 24 Proportion of company revenues that derive from research activities, annual trend

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

2007! 2008! 2009! 2010! 2011! 2012! 2013!

Quantitative research!

Qualitative research!

Other!

Other research!

Non-research activities!


6.2 Company revenues from research in 2013

à Small and medium-sized companies do more qualitative work, compared to larger companies

à The mix of work varies little between global regions

These results are much the same for all regions. However large companies do more quantitative and less qualitative work than small and medium-sized companies.

Q: Please indicate the approximate proportion of your company revenues that derive from quantitative, qualitative or other research activities.

N = 240

Figure 25 Proportion of company revenues that derive from research activities, by region and company size

69%! 69%! 71%! 66%! 66%! 68%!79%!

23%! 24%! 22%!25%! 26%! 24%!

14%!

4%! 4%! 5%!4%! 4%! 5%! 5%!3%! 4%! 2%! 6%! 4%! 4%! 2%!

Total! N America! Europe! Asia Pacific! Small! Medium! Large!

Quantitative research! Qualitative research! Other research! Non-research activities!


6.3 Volume of work

à Web + CATI + paper = 82% of work – but that proportion is gradually declining

à Web was growing until 2011 but seems to have reached a plateau

à Steep decline in paper, gradual decline in CATI

à Self-completion on mobile devices is very small in volume but growing; mCAPI also small but growing.

The volume of Web research has risen by 11 percentage points between 2006 and 2013. In the same time, paper research has declined by almost exactly the same amount.

The volume of CATI work has also declined a little over the years, from 27% in 2006 to 22% in 2012. Despite the large increase in the number of companies offering laptop/tablet CAPI, it remains at 6% of total quantitative research volume in 2012. In 2006, it was 5%.

For details on the more minor modes, see the Figure 26 overleaf.


Q: Focusing on your quantitative research activities, please indicate the approximate proportion of your work represented by each of these interviewing modes or combinations. N = 240

Figure 26 Volume of work by mode

2006 2007 2008 2009 2010 2011 2012 2013

N 213 233 215 188 213 230 250 240

Web 40% 43% 48% 46% 47% 51% 51% 51%

CATI 27% 25% 26% 23% 27% 23% 21% 22%

Paper 21% 19% 14% 16% 13% 14% 10% 10%

CAPI total 5% 7% 5% 8% 5% 6% 9% 9%

Laptop or tablet CAPI 5% 5% 4% 5% 3% 5% 6% 6%

mCAPI 0% 2% 1% 3% 2% 1% 3% 3%

Mixed mode total 7% 6% 6% 6% 6% 6% 6% 6%

Mixed mode CATI and web

4% 4% 4% 3% 4% 5% 4% 4%

Any other mixed mode

2% 2% 2% 3% 2% 1% 2% 2%

Self-completion on mobile devices (not SMS)

0% 0% 1% 1% 0% 2% 2%

IVR 0% 1% 1% 1% 1% 1% 0%

SMS text messaging (self completion)

1% 0% 0% 0% 0% 0% 0% 0%

Table 10 Proportion of quantitative research by mode

0%!

10%!

20%!

30%!

40%!

50%!

60%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Web!

CATI!

Paper!

laptop/tablet CAPI!

Mixed mode!

mCAPI!


6.4 Research modes offered – mainstream modes

à More companies offering CAPI research now

à Fewer companies doing paper-based research

à Online research practised by most – but not all firms

à Mixed mode becoming less important

à CATI operations starting to run down?

CAPI seems to be making inroads at a time that paper-based research is clearly on the decline, although over half of companies (53%) still offer paper research in 2013.

Mixed mode research grew rapidly from 2006 to 2010, with the number of companies offering it increasing from 41% to 54%, but now the opposite appears to be happening, with a steady annual decline that reached 42% in 2013. As we have already seen in section 6.3 volumes of mixed mode are low (6%) and have remained stable. Perhaps this decline in the number of companies offering this service is simply a reflection of the relatively small amount of work available.

The proportion of companies offering CATI research has hovered around 70% since we started gathering data. However, it does look like this is a very slow decline in the number of companies providing CATI. Since 2009, there has been a very small dip every year in the CATI figure.

The results for this are derived from the question in the previous section (‘Please indicate the approximate volume of your work represented by each of these interviewing modes’). It is calculated by counting the number of people who give a non-zero response.


Q: All mentions of each research mode N = 240

Figure 27 Percentage of market research firms using the main modes of research – by year

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

90%!

100%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Web!

CATI!

Paper!

Mixed mode!

CAPI!


6.5 Research modes offered – by region

à CATI, paper and CAPI more frequently offered in Asia Pacific

à Web and mixed mode broadly similar in all regions

In Asia Pacific, nearly as many companies offer CATI as Web research. However, volumes of CATI are the same as elsewhere in the world, so, presumably, research companies in Asia Pacific will gradually cut their investments in CATI.


Figure 28 Percentage of market research firms using the main modes of research – by region in 2013

90%!93%!

90%!87%!

67%!

61%!

69%!

74%!

53%!

39%!

58%!

66%!

45%!

35%!

48%!

63%!

42%!45%!

40%! 40%!

0%#

10%#

20%#

30%#

40%#

50%#

60%#

70%#

80%#

90%#

100%#

2013# N#America# Europe# Asia#Pacific#

Web#

CATI#

Paper#

Any#CAPI#

Any#mixed#mode#(net)#


6.6 Research modes offered – minority modes

à Mobile self-complete becoming mainstream

à IVR is dwindling

à mCAPI stable

à Mixed mode: CATI & Web trend is unclear, other types of mixed mode declining

Here, we look at the percentage of companies offering the more specialised modes of research, for some of which we only started collecting data in 2009.

Mobile self-complete is rapidly becoming a common offering among research companies – just 6% offered this mode in 2009 compared with 19% in 2013. Although, as reported earlier in this chapter, volumes of research conducted in this mode are still very low, however there is a very clear growth.

2008 2009 2010 2011 2012 2013

N 215 188 213 230 250 240

Mode offered 0.2% 5.9% 7.0% 10.9% 18.5% 19.6%

Proportion of work - 0.5% 1.0% 0.5% 1.9% 2.3%

Table 11 Mobile self complete – proportion of work compared with the proportion of research companies that offer it as a service – by year

IVR (interactive voice response) research is down to 4% from its 2009 ‘peak’ of 12%.

We measure laptop and tablet CAPI separately from mobile CAPI (mCAPI), and can see that the growth in CAPI in the previous few years (reported in the previous section) is entirely from laptop and tablet CAPI, although we assume that it is actually tablets, and not laptops, that are fuelling the growth.

We also divide mixed mode into two groups: CATI and Web mixed mode and ‘other’. There seems to be a slow decline in mixed mode overall, and this looks like it may be coming from the ‘other’ group rather than CATI and Web.



Figure 29 Percentage of market research firms using the minor modes of research

0%!

5%!

10%!

15%!

20%!

25%!

30%!

35%!

40%!

45%!

50%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Mixed mode CATI & Web!

laptop/tablet CAPI!

mCAPI!

Mixed mode other!

Self complete - mobile!

IVR!

SMS!


6.7 Changes in the amount of work

à Participants expect main areas for growth to be mobile and web

à Paper, CATI, IVR expected to decline

The fact that the participants in this survey expect mobile self-complete to grow fairly modestly is to be expected, although it seems quite pessimistic when considering the growth observed between the 2011 and 2013 studies.

Q What changes do you foresee in the amount of work you will handle over the next three years in these research activities? N = 240

+2= major growth; +1= modest growth; 0= no change; -1 = decline

Figure 30 Anticipated changes in the amount of work over next three years

-1!

-0.5!

0!

0.5!

1!

1.5!

2!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Web! Self complete - mobile! mobile CAPI! Mixed mode CATI & Web!

laptop or tablet CAPI! SMS! Other mixed mode! IVR!

CATI! Paper!


7 Sources of online sample

Questions

§ Which sources of online sample do you use?

§ Proportion of online samples from each of five identified sources

Access panels favoured over in-house panels

§ Over the years, there has been a trend towards the use of access panels and away from in-house panels.

§ The proportion of client sample in use has remained steady.

§ Large companies are best at sourcing sample from their own panels but even they source more than half of their sample externally


7.1 Proportion of online sample in use - trend

à Researchers becoming more reliant on access panels

à Own panels supplying lower proportion of sample

These figures provide evidence for our observation that researchers are finding it increasingly difficult to source sample for online surveys, either in the quantity or quality they require. The volume of client sample has never declined, despite predictions to the contrary in previous years in this study. The amount of sample from access panels has slowly risen, whereas the volume from companies’ own panels has gradually declined.

Q: What proportion of your online samples comes from each of these sources? N = 240

Figure 31 Proportion of online sample in use

0%!

5%!

10%!

15%!

20%!

25%!

30%!

35%!

40%!

2006! 2008! 2009! 2010! 2011! 2012! 2013!

Third party/access panels!

Sample provided by client!

Own panels!

Specialist sample providers!

Others!


7.2 Proportion of online sample in use in 2013

à Large companies best at supplying own sample

à Even large companies source nearly three-fifths (59%) of sample externally

à Europe makes most use of own panels

The results for large companies are markedly different to those of small and medium-sized companies. Although, it is probably to be expected that due to their greater resources, large companies manage to source more sample from their own panels and are therefore less reliant on client data, access panels and sample providers.

Looking at the global regions, North American companies make greater use of access panels, and source less of their sample from own panels.

Q: What proportion of your online samples comes from each of these sources? N = 240

Figure 32 Proportion of online sample in use, by region and company size

36%!

44%!

33%!

30%!

34%!

45%!

29%!

27%!

28%!

23%!

38%!

30%!

26%!

19%!

22%!

16%!

29%!

14%!

17%!

20%!

40%!

12%!

12%!

11%!

15%!

15%!

6%!

11%!

3%!

1%!

4%!

3%!

3%!

3%!

1%!

2013!

North America!

Europe!

Asia Pac!

Small!

Medium!

Large!

Third party/access panels! Sample provided by client! Own panels!

Specialist sample providers! Others!


7.3 Online sample in use - trend

à Substantial growth in use of access panels since 2004

à Slightly more companies using client-provided sample

Most (83%) research companies use sample provided by their clients, perhaps indicating a growth in customer satisfaction work, over market-wide or nat, rep. studies. Three-quarters (75%) use access panels and this has risen from just over half (54%) in 2004. There has been no change in the proportion of companies using specialist sample providers and own panels.

Q: Which sources of online sample do you use? N = 240

Figure 33 Online sample sources in use

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

90%!

100%!

2004! 2005! 2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Sample provided by client!

Third party/access panels!

Specialist sample providers!

Own panels!

Others!


8 Mixed mode

Questions

§ How important is mixed mode support in data collection software?

§ Do companies use integrated or disparate software for mixed-mode work?

§ What mixed mode capabilities are required?

Most companies think mixed mode an important capability

§ Most (85%) of our participants think that the capability to conduct mixed mode research is an important feature when looking for new software – this has changed little since 2008.

Integrated platforms increasingly the norm

§ There is a clear upward trend over recent years towards companies using integrated mixed mode platforms for their data collection.

Sticking to simpler forms of mixed mode requirements

§ Only just over a fifth (22%) require mixed mode with switching (the most advanced type of multimode research), and this has remained constant since 2005.

However, number of companies offering mixed mode research is declining

§ From 2006 to 2010, the number of companies offering mixed mode research was increasing buts since 2010 it has decreased back to 2006 levels. (See section 6.4)


8.1 Importance of mixed mode

à 85% of respondents think mixed mode is important

à Around one third think it is essential

This has been the case since 2008 and there is no sign that there is any weakening in this response.

Q: If you were choosing new software, or reviewing your current solution, how much importance would you place on the tool’s ability to mix and combine different data collection modes? N = 240

Figure 34 Importance of mixed mode

27%!17%!

35%!30%! 28%! 31%!

31%!43%!

31%!

30%! 33%!32%!

29%! 25%!

26%!28%! 25%! 22%!

6%! 9%!3%! 6%! 8%!

7%!

3%! 5%! 2%! 4%! 3%! 5%!4%! 2%! 3%! 2%! 2%! 3%!

2008! 2009! 2010! 2011! 2012! 2013!

Don't know!

Unimportant!

Fairly unimportant!

Moderately important!Very important!


8.2 Use of integrated platforms

à A clear trend towards integrated platforms

This chart clearly shows that in the last eight years, there has been a clear and gradual shift away from switching between platforms and towards integrated software platforms. In 2013 nearly three-fifths use integrated platforms.

Q: Do you use an integrated software platform for your multimode interviewing, or do you need to switch between different software platforms to combine modes? N = 240

Figure 35 Use of integrated platform versus switching between different platforms

38%!44%! 50%!

59%! 53%!60%! 57%! 59%!

63%!56%! 50%!

41%! 47%!40%! 43%! 41%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Integrated platform! Switch between different platforms!


8.3 Level of mixed mode support required

à Having a common authoring platform is the most popular mixed mode requirement

à A very consistent 21-25% of companies need support, for multi-modal projects, where interviews switch between modes

It is clear that there is no increase in demand for multi-mode with switching, to support multi-modal research projects. The requirement for this has remained at around a quarter of companies since we first asked this question in 2005. What is less clear is what is happening with common authoring and mixed modes in parallel (Mixed mode in parallel allows companies to use the same tool for all their research modes but not for individual interviews to embrace more than one mode). It looks as if the former could be becoming more important the later declining, but it is too early to be sure.

Q: What level of mixed mode or multi-mode capabilities do you require? N = 240

Figure 36 Level of mixed mode

35%! 38%!27%! 27%! 28%!

41%! 43%! 43%!

40%! 40%!49%! 51%!

39%!

36%! 35%! 35%!

25%! 23%! 24%! 21%!34%!

23%! 22%! 22%!

2005! 2006! 2007! 2008! 2010! 2011! 2012! 2013!

Common authoring! Mixed modes in parallel! Multi-mode with switching!


9 Analysis and results reporting

Questions

§ What proportion of projects involves each of the main deliverables or distribution methods?

§ What changes are anticipated in the demand for each delivery methods?

§ How important is the ability to produce volumes of cross-tabular reports in future?

Large companies using more high tech reporting

§ As we also saw in 2012, large companies are going high tech with their results distribution, with a swing towards the use of dashboards and interactive analysis and away from ‘low tech’ methods such as Acrobat PDF files. This was not the case in 2011.

Medium companies starting to go higher tech

§ In 2013, it looks as if medium sized companies are starting to follow the same path as large companies - switching from low tech to high tech delivery methods

Large companies more optimistic about future of high tech reporting

§ Overall, in 2011, 2012 and 2013 our participants anticipate a modest increase in demand for all newer technology-based methods for delivering results, although in all three years large companies are particularly optimistic.

Bad news for trees?

§ Over 80% of the participants in this survey said that the ability to print out large volumes of tables is ‘essential’ or ‘moderately important’. This has changed little since 2004.

§ However, in 2013 almost no projects (4%) are actually delivered as printed tabs


9.1 Distribution methods in use

à Microsoft PowerPoint is leading delivery method – by far

à Since 2006 printed tabs have gone from mainstream to exceptional

à Dashboards may be on the up

There is a clear trend showing the decreasing use of Microsoft Word for report delivery. It also looks as if more projects are being delivered on dashboards but it is too soon to be sure. (And this hypothesis is supported elsewhere in this study, and reported in sections 9.2 and 9.3 of this report).

Printed tables have quickly shifted from a mainstream client deliverable to a specialist one. In 2006, they were used for 23% of projects and in 2013, this had reduced to 4%. However, their use internally, we believe, is undiminished.

Q: What percentage of projects currently involves the following deliverables or distribution methods to the client? N = 240

Figure 37 Distribution methods in use

0%!

10%!

20%!

30%!

40%!

50%!

60%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

MS PowerPoint!

MS Excel!

Acrobat PDF!

Online static reports!

MS Word!

Interactive analysis!

Printed tabs!

Digital dashboards !


9.2 Distribution methods in use, by company size

à Large companies ditching low tech delivery methods in favour of high tech

à Medium companies following swiftly in large companies’ footsteps?

In 2012 it was noticeable that large companies were starting to make greater use of the more high tech methods of results delivery, such as dashboards and interactive analysis. This change has become more pronounced in 2013, but this is due to sizeable reductions in their use of ‘low tech’ delivery methods such as Acrobat PDF files, Excel, Word and online static reports.

In fact, large companies are now using interactive analysis and dashboards more than Microsoft Word and PDF files. Dashboards are very rarely used in small companies (4% of projects) but are used by over one in eight (13%) of projects in large companies. There is also a difference in the use of interactive analysis – 9% at small companies compared with 19% at large companies. These are more resource-intensive methods that demand both the technical infrastructure and in-house skills, which larger companies are more likely to have.

In 2013, it is striking how many medium-sized companies appear to be catching up with large companies in swapping low-tech delivery methods in favour of high tech. In 2013, the use of dashboards within medium-sized companies has risen sharply. Similarly, there has been a big reduction in medium-sized company’s use of PowerPoint and also a reduction in Excel delivery and online static reports.


Q: What percentage of projects currently involves the following deliverables or distribution methods to the client? N = 240

N=250

Figure 38 Distribution methods in use, by company size – for 2013 and 2012

53%! 55%!

49%!

54%!

24%! 26%!22%! 22%!

19%!23%!

16%!

10%!12%!

16%!

22%!

12%! 14%! 11%!

8%!11%! 9%! 10%!

19%!

5%! 4%! 6%!8%!

4%!

12%! 13%!

2013! Small! Medium! Large!

MS PowerPoint! Microsoft Excel documents! Acrobat PDF! Online static reports!

MS Word! Interactive analysis! Printed tabs! Digital dashboards !

53%!51%!

58%! 56%!

29%! 29%! 29%! 29%!

21%! 24%!

15%!

22%!19%! 20%!

27%!

14%! 14%!14%!

12%!10%! 8%! 7%!

21%!

4%!8%! 8%!6%! 3%! 7%!

13%!

2012! Small! Medium! Large!

MS PowerPoint! Microsoft Excel documents! Acrobat PDF! Online static reports!

MS Word! Interactive analysis! Printed tabs! Digital dashboards !


9.3 Change in demand for high tech delivery

à Modest increase in demand for all high tech delivery methods is anticipated

à Dashboard delivery remains a hot future topic

Companies of all sizes anticipate a modest increase in demand for all the high tech delivery methods. Generally speaking, larger companies are slightly more optimistic. The same relationship was apparent in 2011 and 2012.

Q: What kind of change in demand do you anticipate for the following research delivery methods over the coming year? N = 240

Figure 39 Anticipated change in demand for high tech delivery methods

However in 2013, when looking solely at dashboards, we also see that medium-sized companies are expecting increases in demand similar to those of large companies. This was not the case in previous years.

21%!

15%!

23%!

15%!

24%!

50%!

45%!

46%!

47%!

47%!

27%!

38%!

31%!

38%!

28%!

2%!

2%!

1%!

0%! 20%! 40%! 60%! 80%! 100%!

Fixed reports online!

Clients create analyses online!

Info portals!

External portals (eg Sharepoint)!

Digital dashboards!

Major increase! Modest increase! No increase! Decrease!


Region Company size

Total 2013

N America

Europe Asia Pacific

Small Medium Large

N 240 87 115 38 130 69 41

Major increase 24% 26% 22% 26% 15% 35% 34% Modest increase 47% 45% 48% 50% 43% 49% 56% No increase 28% 29% 30% 24% 42% 15% 10% Decrease 0% 0% 1% 0% 0% 1% 0%

Mean score 1.0 1.0 0.9 1.0 0.7 1.2 1.2

Table 12 Anticipated change in demand for digital dashboards over coming year


9.4 Importance of producing cross tabs in volume

à Nearly everyone needs to produce cross tabs in volume

Even though very few projects are now delivered on printed tables, most of our survey participants (82%) say that is essential or moderately important to be able to produce cross tabs in volume. This can be explained by a continuing need to use tables as a primary tool and resource for analysis, and also as a means to cross-check other deliverables, as a quality control method. The need for cross-tab packages is not about to disappear.

Q: When considering analysis and reporting tools for the future, how important is it that these should be able to produce volumes of cross-tabular reports? N = 240

Figure 40 Importance of being able to produce volumes of printed cross tabs

59%!55%! 55%! 52%!

57%!50%! 53%! 52%! 53%! 50%!

27%!27%!

33%! 36%! 28%!34%! 30%! 33%! 34%!

32%!

13%!14%!

10%! 11%! 13%! 13%! 16%! 13%! 10%!15%!

1%! 4%! 2%! 1%! 2%! 3%! 1%! 2%! 2%! 4%!

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

90%!

100%!

2004$ 2005$ 2006$ 2007$ 2008$ 2009$ 2010$ 2011$ 2012$ 2013$

Essential! Moderately important! Relatively unimportant! Not something future tools should provide!


10 Changing software

Questions

§ What desire exists to replace the software currently in use, in the near future?

§ What type of software is being considered for replacement?

§ What are the reasons for changing software?

Key findings

Big appetite for new research software

§ Over a third of respondents say that their companies are planning on changing their research software over the next two years.

§ Demand is particularly strong in Asia Pacific for both 2012 ad 2013

All types of research software on shopping list

§ Between 59% and 69% of all of those companies who wish to change their research software want to change core research software modules (data collection, data processing, analysis and report publishing).


10.1 Planning on changing software in 2013

à Strong appetite for new software

à Especially in Asia Pacific and among large companies

We see this question as a barometer both of research companies’ willingness to invest money in as well as their satisfaction with their existing tools. There are some wide regional variations, with 50% of those in Asia Pacific planning on buying new software against 32% and 33% in Europe and North America. Similarly, large companies seem to have much bigger plans for software purchases compared with small and medium companies.

Q: Are you considering changing the software you are using for your data collection or data analysis in the next one to two years? N = 240

Figure 41 Planning on changing software, by region and company size

35%! 33%! 32%!

50%!

34%! 33%!44%!

24%! 24%! 23%!

29%!

24%! 28%!

20%!

40%! 43%! 45%!

21%!

42%! 39%! 37%!

2013! N America! Europe! Asia Pac! Small! Medium! Large!

Yes! Not sure! No!


10.2 Planning on changing software - trend

à 2013 is a relatively favourable year for software developers with new tools to offer

With over a third of companies saying they plan to buy new software, the current year (i.e. the period that follows the 2013 survey) appears to offer more opportunities to software developers for new business than some recent years.

Q: Are you considering changing the software you are using for your data collection or data analysis in the next one to two years? N = 240

Figure 42 Planning on changing software, by year

26%! 22%!

40%!32%!

25%!

46%!35%! 35%!

25%! 34%!

18%!26%!

23%!

20%!

24%! 25%!

49%! 44%! 42%! 43%!52%!

35%!40%! 40%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013!

Yes! Not sure! No!


10.3 Software type to change

à For the first time, data analysis tools are at the top of the shopping list

à Closely followed by report publishing and data collection

Various questions within this survey suggest a very recent and sudden shift in activity in analysis and reporting, and this just adds weight to it. This is the first year that data collection software has not been the most sought after new software.

The high scores across all categories also indicate that many companies are considering replacing tools in more than one area.

The lower scores for panel management tools is simply an indication that they are specialised tools which are not used universally. In that context, demand for these also appears to be strong.

Q: What software are you considering changing? N 2010 = 54; 2011 = 105; N 2012 = 85

Figure 43 Software type wishing to change

43%!

65%!

48%!

67%! 67%!

30%!

70%!

61%!

65%! 65%!

31%!

69%!

51%!

57%!

62%!

32%!

64%!

59%!

69%!66%!

Panel management! Data collection! Data processing! Data analysis! Report publishing and delivery!

N 2010 = 54, 2011 = 105, 2012 = 85, 2013 = 85 !

2010! 2011! 2012! 2013!

The 2013 Market Research Technology Survey · sponsored the survey and provided financial and technical assistance that support its administration. This is acknowledged in the survey’s

Documents