FocusVision 2014 MR Technology Report - 3.2 · London, April 2015 . FocusVision 2014 Annual MR Technology Report | 7 of 92 1 Management summary This annual survey of over 200 research

FocusVision 2014 Annual MR Technology Report

Full report

Eleventh annual survey by meaning ltd

Report written by Tim Macer and Sheila Wilson, meaning ltd

independent advice on technology for research

FocusVision 2014 Annual MR Technology Report | 2 of 92

Contents

1 Management summary 7 2 Background to the survey 9

2.1 The Sample 9 2.2 Demographics 10 2.3 Trends reported for 2014 11 2.4 New topics covered in 2014 11

3 Trends in research activities and modes 12 3.1 Research mix – Eight-year trend 14 3.2 Company revenues from research in 2014 15 3.3 Volume of work 16 3.3.1 Trend in Web volume – by region 18 3.3.2 Trend in CATI volume – by region 19 3.3.3 Trend in paper volume – by region 20 3.4 Research modes offered – mainstream modes 21 3.5 Research modes offered – by region 23 3.5.1 CATI research – trend by region 23 3.6 Research modes offered – minority modes 25 3.6.1 ‘Mobile self complete’ trend - by region 26 3.7 Mobile participation in online surveys 28

4 Making surveys mobile-friendly 30 4.1 Is mobile considered at the research design stage? 32 4.2 Company’s general approach to support for mobile participation 34 4.3 Technological support for mobile-friendly surveys 36 4.3.1 Capabilities for creating surveys to be taken in both modes 37 4.3.2 Capabilities for switching 38 4.3.3 Observations 39 4.4 Importance of combined mobile and PC support online survey technology 40 4.5 Attitudes towards accommodating mobile participation 42 4.6 Conclusions 44 4.7 A metadata postscript 44

5 Trends in online sample sources 46 5.1 Proportion of online sample in use - trend 47 5.2 Proportion of online sample in use in 2014 48


6 Trends in research software demands 49 6.1 Planning on changing software in 2014 50 6.2 Planning on changing software - trend 51 6.3 Software type to change 52

7 Shadow IT and alternative technologies 53 7.1 Technologies in use 55 7.2 Company approach to allowing non-standard software 59 7.3 Company policies for alternative technology from clients or suppliers including Shadow IT 61 7.4 Use of systems controlled by suppliers or clients 64 7.5 Attitudes towards alternative technology 65 7.6 Conclusions 66

8 Trends in analysis and results reporting 68 8.1 Distribution methods in use 69 8.2 Distribution methods in use, by company size and region 70 8.3 Importance of producing cross tabs in volume 72

9 Data visualization and dashboards 73 9.1 Projects delivered using data visualization 75 9.2 Ease of achieving client demands 77 9.3 Challenges with providing dashboards 79 9.4 Software used for data visualization 81 9.4.1 Changes since 2011 82 9.5 Who produces data visualizations? 84 9.5.1 Changes since 2011 85 9.6 Words of advice when designing a dashboard 87 9.7 Conclusions 89

10 Conclusions 91


List of tables

Table 1 Sample composition: company size by region 10

Table 2 Sample composition: responsibility and seniority 10

Table 3 Proportion of quantitative research by mode 17

Table 4 Mobile self complete – proportion of work compared with the proportion of research companies that offer it as a service – by year 25

Table 5 Average scores for software capabilities 36

Table 6 Summary of mean scores for rating scales on attitudes towards mobile surveys 43

Table 7 Additional technologies and Shadow IT resources, by region and company size 57

Table 8 External software and technology policies in place, by region and company size 63

Table 9 Attitudes towards alternative technology and Shadow IT by region and company size 66

Table 10 Volume of projects delivered using various types of data visualization: by region and company size 76

Table 11 Ease or difficulty or meeting client demands for various visual presentation formats: distribution by region and company size 78

Table 12 Data visualization tools used: Which other tools are used when Microsoft Excel and PowerPoint are mentioned. 82

List of charts

Figure 1 Proportion of company revenues that derive from research activities, annual trend 14

Figure 2 Proportion of company revenues that derive from research activities, by region and company size 15

Figure 3 Volume of work by mode 17

Figure 4 Trend in Web volume by region 18

Figure 5 Trend in CATI volume by region 19

Figure 6 Trend in CATI volume by region 20

Figure 7 Percentage of market research firms using the main modes of research – by year 22

Figure 8 Percentage of market research firms using the main modes of research – by region in 2013 23

Figure 9 Percentage of market research firms using the mode CATI 24

Figure 10 Percentage of market research firms using the minor modes of research 26

Figure 11 Percentage of market research firms using the mode ‘mobile self complete’ 27

Figure 12 Mobile or smartphone participation in online surveys 28

Figure 13 How often mobile is discussed when planning new research projects 32

Figure 14 Company approach to supporting mobile participation in online surveys 34


Figure 15 Support provided in the survey technology used to create mobile-friendly surveys. 37

Figure 16 Survey technology support for creating online and mobile surveys 38

Figure 17 Survey technology support for participants to switch from a mobile to a PC 38

Figure 18 Importance of dual-mode support within survey tools 40

Figure 19 Research company attitudes towards accommodating mobile participation within online research 42

Figure 20 Proportion of online sample in use 47

Figure 21 Proportion of online sample in use, by region and company size 48

Figure 22 Planning on changing software, by region and company size 50

Figure 23 Planning on changing software, by year 51

Figure 24 Software type wishing to change 52

Figure 25 Additional technologies and Shadow IT resources used within the company 55

Figure 26 Company approach towards using software external or non-standard software 60

Figure 27 Specific company policies towards different kinds of external technology use 61

Figure 28 Specific company policies towards different kinds of external technology use: by company size 62

Figure 29 Changes in the use of suppliers’ or partners’ software and systems in the last 1-2 years 64

Figure 30 Attitudes towards alternative technology and Shadow IT 65

Figure 31 Distribution methods in use 69

Figure 32 Distribution methods in use, by region and company size –2014 71

Figure 33 Importance of being able to produce volumes of printed cross tabs 72

Figure 34 Volume of projects delivered using various types of data visualization 75

Figure 35 Ease or difficulty or meeting client demands for various visual presentation formats 77

Figure 36 Principle challenges in providing dashboard reporting as a part of a research project 79

Figure 37 Software used for data visualization 81

Figure 38 Changes in tools used for data visualization: 2011 and 2014 83

Figure 39 People who produce data visualizations – overall and by company size 84

Figure 40 People who produce data visualizations – 2011 and 2014 compared 85

Figure 41 Advice on how to set up dashboards 87


Acknowledgements

We are greatly indebted to FocusVision for the assistance they have provided in supporting the survey both financially and technically, for providing us with the use of their Decipher online survey software for use on this survey, and for a significant and welcome contribution to the direction and content of the 2014 survey.

Tim Macer and Sheila Wilson, meaning ltd London, April 2015


1 Management summary

This annual survey of over 200 research companies worldwide (215 companies in 2014) shows that research is continuing in its shift towards technology-based methods. Mobile is now growing, after a slow start, and there is also recent growth in CAPI. Paper has continued a long decline and is now close to zero in North America, while it is also declining rapidly in Europe. CATI is also continuing to decline, particularly in North America, while online research is still finding some room to grow in volume. Mixed mode research is showing little sign of growth, and remains at a low level.

Trends in research modes Mobile participation in online surveys is increasing rapidly, as reported by participating companies. Several specific questions this year looked at research companies attitudes and responses towards allowing or even encouraging mobile participation in their online surveys. A majority of research firms are taking steps to support mobile participation with just under a third saying most of their online surveys are mobile-enabled. However, there are also many companies that are taking a more passive approach, and appear to be relying on the technology alone to make their surveys mobile-friendly.

Making surveys mobile-friendly While a large number of firms say they consider the mobile implications of new research projects at the design stage, there are as many who rarely or never do this. Smaller firms are more likely to ignore mobile participation, but firms in the Asia Pacific region are the most likely to take it into account.

Companies are reporting limitations for mobile support in the data collection tools they use, though many find it is moderately easy (though not very easy) to get their survey software to provide a mobile-friendly instance of their online surveys. Most agree, however, that parallel online and mobile support in any modern survey design software is an essential capability, with the highest support for this among larger firms. Opinions are more divided over whether it makes for good research. Many consider it a good way to improve response rate, and also believe its particular modal effects, which can influence research findings, are now well understood.

Technology decisions A long-term trend in companies seeking to replace or renew the software they use to carry out research shows slightly fewer companies in 2014 looking to replace software. For the first time in five years since this question was introduced, report publishing and data delivery tools top the list of what the tools that firms are looking to acquire.

The report reveals a large amount of the technology and tools that researchers are using are no longer acquired through traditional routes or the central control of an IT function. It also reveals that it is widespread practice now for many research companies to be using data collection tools chosen by and controlled by their suppliers, and that many are also using the reporting tools and systems of their clients to disseminate research findings. Furthermore, these practices are on the rise.


Companies are aware of, and concerned about the data security and other risks that ‘Shadow IT’ poses (that is, people using their own devices and downloading software from the internet), and also when technology decisions are taken in a non-centralized way. Most companies have policies or guidelines in place to govern what is permitted and what requires special permission. Not surprisingly, larger firms are taking the policy route, whereas smaller firms are taking a more case-by-case approach to de-centralized technology choices.

While the report reveals good practice among many firms, it also appears that a large number seem unaware of the risks or how best to manage them.

Research delivery and dashboards High-tech delivery methods for research findings are only growing modestly, and PowerPoint remains by far the most popular way to distribute research findings. However, dashboards appear to be entering the mainstream, and some element of online distribution is involved in over a third of projects. Around one in every seven projects now appear to involve some infographic treatment – and that rises to one in five among larger firms.

Dashboards and data visualizations are still prone to being an unprofitable activity, with many firms reporting insufficient budget to deliver what the client wants as the greatest challenge; while the need to customise software – which applies another pressure on cost – is also viewed as a major challenge.

Many are finding it technically challenging to provide dashboards, and often don't have the right technology in place to support their production. Some report that they have overcome these challenges, usually though choosing appropriate tools and building up the necessary skills. Research companies certainly appear to be developing their skills and capabilities in the area of data visualization, with many more reporting the use of graphic artists and visual designers than in when this same question was put in 2011, and fewer relying on research assistants or charting departments to create data visualizations.

The advice from companies providing dashboards is overwhelmingly to keep them simple, focus on client and end-user needs – and to plan them very carefully.


2 Background to the survey

Meaning ltd has been carrying out this annual survey since 2004, making this the eleventh successive year of the survey. The survey provides a unique set of information and insights into the interplay of technology and methodology within the market research industry. It provides a snapshot of current usage and attitudes and predictions from practitioners, and identifies trends from a number of tracking questions that are asked repeatedly each year.

In addition to identifying general trends, each year, the survey also focuses on several areas of topical interest. For the 2014 survey, these topics are:

• Data visualization

• Participation in online surveys using mobile devices, such as smartphones

• Technology used by market research companies that is owned by others – for example, free software downloads, employees’ own tablets or suppliers’ technology

This is the first year that FocusVision, a company providing quantitative and qualitative research solutions, has kindly sponsored. The importance of that contribution is acknowledged in the official name of the survey for 2014, the FocusVision 2014 Annual MR Technology Report.

2.1 The Sample The 2014 survey comprises a sample of 215 market research companies globally, and only individuals who are responsible for, influential in or aware of technology decisions within each company. The sample is managed to ensure representation of three global regions: North America, Europe and Asia Pacific according to the relative amount of research in each region according to ESOMAR.

The survey consists of a self-completion interview on the Web, comprising around fifty questions and timed to last under fifteen minutes. As it is difficult to identify the target individual within a research company, sample is obtained from a variety of sources:

§ Participants who agreed to be re-contacted from the previous years of the survey

§ Sample compiled by meaning ltd

§ Sample provided by the survey’s sponsor, FocusVision

§ Social media activities by FocusVision and meaning using an open survey link

§ Direct email contact by FocusVision to market research companies

De-duplication is applied prior to invitation, across all of the sample sources, and again after fieldwork is completed, to ensure that only one response is counted from each company in each country.

Responses from open survey links accounted for 7% of the sample. These were additionally asked to provide company information, and this was used to verify their eligibility for the survey – i.e. that they were a research company and to apply de-duplication, when necessary.


The nature of the sampling method, and the survey’s strict eligibility criteria (i.e. senior IT decision makers within research organizations) makes it impossible to provide a meaningful estimate of response rate, because invitations are sent to individuals without prior knowledge of their eligibility, and those who are not eligible are unlikely to respond to the survey invitation.

2.2 Demographics This report is based on 215 participants in unique research companies within each country and covers 24 countries.

Of the 215 participants, 87 (40%) are from North America, 94 (44%) are from Europe, and 34 (16%) from what this report refers to as ‘Asia Pacific’, which includes countries in southern and eastern Asia and Australasia.

Low base warning: Although the Asia Pacific region is present in proportion with the sampling frame described above, it does mean there is a small base for comparison between the other global regions, when this occurs. We therefore advise caution in interpreting these results, and recommend they be considered to be only indicative of a possible difference.

The composition of the demographics referred to in this report is as follows:

Size Total N America Europe Asia Pacific

Smaller companies 137 64% 54 39% 65 61 % 18 13 %

Larger companies 78 36% 33 42% 29 37% 16 21 %

Total 215 100% 87 100% 94 100% 33 100%

Table 1 Sample composition: company size by region

The declared revenue of the organization defines company size. ‘Smaller companies’ is intended to reveal the different needs of companies unlikely to have specialist in-house technical staff. The category ‘larger companies’ are those with a turnover of over $50 million or more. (In previous years, the ‘larger companies’ group has been divided into two – medium-sized and large companies, but this year, we had too few companies in the ‘large’ category for reliable analysis.)

The survey is only asked of senior individuals who are involved in IT decision-making within their respective company. The level of responsibility and role of the participants is shown below:

Role N % Level of responsibility N % Research 90 42% Board Member 68 32% IT 23 11% Primary IT decision maker 92 43% Data processing 18 8% IT Decision influencer 49 23% Business or Operational 78 36% Not stated 6 3% Not stated 6 3

Table 2 Sample composition: responsibility and seniority


2.3 Trends reported for 2014

There is a central core of questions in this survey that we have been asking for several years – and some since the inception of this project in 2004. With several years of data, several clear and stable trends have emerged. This shows continued consistency in the survey as a whole, since the sample each year is made up largely of different companies and individuals than in previous years.

Topics covered These tracking questions fall into four general topics, each of which is covered in a separate chapter:

• Research activities (Chapter 3, p. 12

• Online sample sources (Chapter 5 p. 46)

• Demand for software (Chapter 6, 49)

• Analysis and results reporting (Chapter 8, p. 68)

Number of observations (N) Unless otherwise shown, the number of observations (N) for each chart is the full sample size for the relevant year. These are as follows:

2006 213

2007 233

2008 215

2009 188

2010 213

2011 230

2012 250

2013 240

2014 215

2.4 New topics covered in 2014

Each year, in addition to a number of unchanging tracking questions, the survey focuses on several areas of current interest.

For the 2014 survey, current interest topics covered are:

• Making surveys mobile-friendly (Chapter 4, p.30)

• Shadow IT and alternative technologies in use within MR firms (Chapter 7, p. 53)

• Data visualization experiences and practices (Chapter 9, p. 73)

Most of these questions are ‘one off’ in that they have not appeared before in the life of the survey, and probably will not be repeated in next year’s study. Occasionally, however, some questions may be reprised in later years, and there are two questions in Chapter 9 that were also asked in 2011, and those findings are repeated for comparison.

As these current interest topics tend to amplify items covered in the trend chapters, these current interest chapters are placed to follow the most relevant chapter on trends.


3 Trends in research activities and modes

Questions

§ Volume of work that derives from quantitative, qualitative or other research activities

§ Volume of work by research mode (CATI, Web etc.)

§ The proportion of participants each company observes attempting to take online surveys on mobile devices.

Key findings

Research involving mobile devices: tiny marketplaces, but growing

§ Mobile self complete (not including SMS) growing rapidly (the proportion of companies offering this is increasing rapidly. When we first measured this in 2009, 6% of companies offered this as a service. In 2014 it is 27%. The amount of business these companies are doing is small, but growing fast.

§ mCAPI growing? The proportion of companies offering this as a service has grown slightly in 2014. The volume of business is very small, but appears to be growing (1.8% in 2007, 3.6% in 2014)

§ SMS possibly growing, but probably not A higher proportion of research companies in 2014 are offering this service than ever before, but there is no sign of any growth in volumes. We view this with caution.

Web and (to a small extent CAPI) projects replacing CATI and paper

§ As CATI and paper decline in volume, Web and CAPI are increasing. Paper has decreased considerably, and CAPI has increased slightly, so that CAPI has now overtaken paper in volume – 2014 volumes are: Web 53%, CATI 18%, CAPI 10%, paper 9%

§ The number of companies offering a CAPI service has increased from 30% in 2006 to 45% in 2014, although CAPI volumes have increased far more slowly.

Nearly all market research companies offer Web research

§ Since 2007, the proportion of companies offering Web research has been over 90%

North America: the marketplace is gravitating towards Web

§ The volume of Web research in North America (68%) has grown rapidly since we first measured this in 2006 (45%).

§ CATI is the second biggest mode, but volumes are decreasing in North America (2006 28%, 2014 18%).

§ Paper, a fairly significant mode as recently as 2006, now represents only 3% of total volume in North America. No other modes are at more than 5%.


Mixed mode projects starting to slacken?

§ Although volumes of mixed mode research have remained stable, the number of companies offering it was increasing, but in 2010 started decreasing

IVR very low volume, no sign of growth

§ This mode is languishing at around 0% and has been at around 0% to 1% for the duration of the time we have collected data

Mobile participation in online surveys continues to grow

§ Over the last 4 years, firms report that the proportion of participants who are taking online surveys on smartphones and other mobile devices has risen each year from 6.7% in 2006 to 21.4% in 2014

Awareness within companies of the extent of mobile participation remains low

§ Also in common with previous years, almost 30% of the managers interviewed could not give an estimate.


3.1 Research mix – Eight-year trend

à Almost a quarter of revenues derive from qualitative

à Qualitative vs. quantitative split is very stable over the years

à Research firms do very little non-research work

Over the years, the proportion of quantitative research revenues to qualitative is remarkably stable. Quantitative represents around 70% and qualitative between 20% and 24%. There is a discontinuity in the chart, at 2009, because “non-research activities” was added as a new answer category.

Q: Please indicate the approximate proportion of your company revenues that derive from quantitative, qualitative or other research activities.

N = 215 (2014). For other years see note on p. 11.

Figure 1 Proportion of company revenues that derive from research activities, annual trend

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

Quantitative research!

Qualitative research!

Other!

Other research!

Non-research activities!


3.2 Company revenues from research in 2014

à The mix of work varies little between global regions

These results are much the same for all regions. However in previous years, we have found that the largest companies do more quantitative and less qualitative work than small and medium-sized companies. (Due to a lower than usual number of participants from large companies, we had to merge these two size of business into one category called ‘larger companies’)

Q: Please indicate the approximate proportion of your company revenues that derive from quantitative, qualitative or other research activities.

N = 215

Figure 2 Proportion of company revenues that derive from research activities, by region and company size

67%! 61%!73%!

63%! 66%! 67%!

23%!26%!

21%!

24%!24%! 23%!

6%! 6%!5%!

10%! 6%! 6%!4%! 7%! 4%! 4%! 5%!

Total! N. America! Europe! Asia Pac! Smaller co's! Larger co's!

Quantitative research! Qualitative research! Other research! Non-research activities!


3.3 Volume of work

à Web continues to grow

à Paper continues a long-term slide

à CATI showing signs of decline, especially in America

à CAPI growing and overtaking paper

à Mobile self-completion small but clearly growing

The volume of Web research has risen by 13 percentage points between 2006 and 2014.

CAPI volumes are also clearly rising and CAPI has now overtaken paper as the third most important mode by volume of work. As market research companies seek to improve quality and efficiency, Web and CAPI are clearly replacing work that used to be on paper and CATI, as both of these modes, especially paper, have been decreasing in volume for several years. Paper volumes are now very low in North America, as shown in section 3.3.3.

CATI was 27% of total work volume in 2006 and 18% in 2014. It is decreasing in North America in particular, as shown in section 3.3.2, where it has sunk from 28% of overall volume in 2006 to 13% in 2014.

Among the more minor modes, there seems to be some growth in SMS, which has moved slowly from 0-1% to 2% in recent years, and in 2014 is reported as accounting for a little under 4%. Most of the other minor modes remain largely unchanged.

There has been little change over the years in respect of one technology that can be very convenient for market research, in the guise of interactive voice response or IVR technology, despite the dominance of this technology now in other areas of business. It is now widely used in many closed-loop customer feedback programs, for example. When used in business, it usually supplements human contact, either before or after a call, and there is no reason why it could not play a similar role in an interview, once contact has been established.

For details on the more minor modes, see Table 3.


Q: Focusing on your quantitative research activities, please indicate the approximate proportion of your work represented by each of these interviewing modes or combinations.


Figure 3 Volume of work by mode

2006 2007 2008 2009 2010 2011 2012 2013 2014

N 213 233 215 188 213 230 250 240 215

Web 40% 43% 48% 46% 47% 51% 51% 51% 53%

CATI 27% 25% 26% 23% 27% 23% 21% 22% 18%

Paper 21% 19% 14% 16% 13% 14% 10% 10% 9%

CAPI total 5% 7% 5% 8% 5% 6% 9% 9% 10%

Laptop or tablet CAPI

5% 5% 4% 5% 3% 5% 6% 6% 7%

mCAPI 0% 2% 1% 3% 2% 1% 3% 3% 4%

Mixed mode total 7% 6% 6% 6% 6% 6% 6% 6% 5%

Mixed mode CATI and web

4% 4% 4% 3% 4% 5% 4% 4% 4%

Any other mixed mode

2% 2% 2% 3% 2% 1% 2% 2% 1%

Self-completion on mobile devices (not SMS)

0% 0% 1% 1% 0% 2% 2% 4%

IVR 0% 1% 1% 1% 1% 1% 0% 0%

SMS text messaging (self completion)

1% 0% 0% 0% 0% 0% 0% 0% 1%

Table 3 Proportion of quantitative research by mode

0%!

10%!

20%!

30%!

40%!

50%!

60%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

Web!

CATI!

Paper!

Mixed mode!

CAPI!


3.3.1 Trend in Web volume – by region

à Web volumes growing most rapidly in North America

à Web volumes much lower in Europe

In North America, fieldwork that would previously have been on paper or CATI is now often online – saving research companies vast expenses in personnel and also giving them the ability to deliver results more quickly. The chart below shows how Web volumes have increased rapidly in North America since 2006 and the next two charts show the trend for CATI and paper.

The trend for Europe is similar, just not as pronounced. The 2014 volumes for Web are 68% for North America and 46% for Europe. Europe is more culturally diverse than North America and so is the marketplace: Europe makes more use of CATI, paper and CAPI.

This is the first in several charts where we look at regional trends. We do not show Asia Pacific because the sample size for that region is small.



Figure 4 Trend in Web volume by region

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

N. America!

Europe!


3.3.2 Trend in CATI volume – by region

à Volume of CATI work decreasing in Europe and North America



Figure 5 Trend in CATI volume by region

0%!

5%!

10%!

15%!

20%!

25%!

30%!

35%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

N. America!

Europe!


3.3.3 Trend in paper volume – by region

à Volume of paper-based research close to zero (3%) in North America

à Decreasing rapidly in Europe too



Figure 6 Trend in CATI volume by region

0%!

5%!

10%!

15%!

20%!

25%!

30%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

N. America!

Europe!


3.4 Research modes offered – mainstream modes

à More companies now have CAPI capability

à Fewer companies offering paper-based research and CATI

à Online research practiced by most – but not all firms

à Unclear whether mixed mode growing or declining

CAPI is making inroads at a time that paper-based and CATI research is clearly on the decline, although half of companies (50%) still offer paper research and nearly two-thirds (65%) CATI.

Mixed mode research grew rapidly from 2006 to 2010, with the number of companies offering it increasing from 41% to 54%, but now the opposite appears to be happening, with a steady decline that reached 45% in 2014. As we have already seen in section 3.3 volumes of mixed mode are low (5%) and have remained stable ever since we first measured this in 2006. This decline in the number of companies offering this service is probably a reflection of the relatively small amount of work available.

The proportion of companies offering CATI research seems to be in a slow decline. Since the peak of 2009, when 73% of companies had a CATI capability, this figure has dipped by a very small amount each year. As shown in section 3.5.1, this change is due to a decline in CATI in North America. The volume of CATI work has, however, decreased steadily in both North America and Europe since 2005, as shown in section 3.3.2.

The results for this are derived from the question in the previous section (‘Please indicate the approximate volume of your work represented by each of these interviewing modes’). It is calculated by counting the number of people who give a non-zero response.


All mentions of each research mode


Figure 7 Percentage of market research firms using the main modes of research – by year

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

90%!

100%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

Web!

CATI!

Paper!

Mixed mode!

CAPI!


3.5 Research modes offered – by region

à It’s all about Web in North America

à CATI, paper and CAPI still at the forefront in Europe and Asia Pacific

Web research is the dominant mode in all regions, but the contrast between Web and the other modes is very stark North America. In Asia Pacific and Europe, CATI, paper and CAPI are still very much at the forefront, due to different market conditions and cultural differences between the countries in these regions.

Mixed mode is of roughly equal importance in all regions


N = 215

Figure 8 Percentage of market research firms using the main modes of research – by region in 2013

3.5.1 CATI research – trend by region

à Strong downward trend in companies offering CATI in North America

Companies in North America have clearly been divesting themselves of CATI operations but CATI still seems strong in Europe. There has been a slight dip in Europe since 2013, but it is far too early to tell if this is a new trend. As CATI volumes are decreasing (as reported in section 3.3.2), it seems likely that over the coming years, we will also see companies in

93%!95%! 95%!

79%!

65%!

57%!

72%!

62%!

50%!

31%!

62%!

68%!

45%!

30%!

53%!

62%!

45%! 45%! 47%!

41%!

0%#

20%#

40%#

60%#

80%#

100%#

120%#

2014# N.#America# Europe# Asia#Pac#

Web#

CATI#

Paper#

Any#CAPI#

Any#mixed#mode#


Europe exiting the CATI marketplace. This may be a slow process due to the large investment in equipment, premises and staff.

All mentions of CATI


Figure 9 Percentage of market research firms using the mode CATI

40%!

45%!

50%!

55%!

60%!

65%!

70%!

75%!

80%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

N. America!

Europe!


3.6 Research modes offered – minority modes

à Mobile self-complete is now mainstream

à Is mCAPI growing?

à SMS may be growing, but probably not

à IVR is dwindling

à Mixed mode: CATI & Web trend seems stable, other types of mixed mode declining

Here, we look at the percentage of companies offering the more specialized modes of research, for some of which we only started collecting data in 2009.

Mobile self-complete is rapidly becoming a common offering among research companies – just 6% offered this mode in 2009 compared with 27% in 2014. Although, as reported earlier in this chapter, volumes of research conducted in this mode are still very low, however there is a very clear growth.

2008 2009 2010 2011 2012 2013 2014

N 215 188 213 230 250 240 215

Mode offered 0.2% 5.9% 7.0% 10.9% 18.5% 19.6% 26.5%

Proportion of work - 0.5% 1.0% 0.5% 1.9% 2.3% 3.6%

Table 4 Mobile self complete – proportion of work compared with the proportion of research companies that offer it as a service – by year

We measure laptop and tablet CAPI separately from mobile CAPI (mCAPI), and can see that in previous years the growth in CAPI (reported in the previous section) was entirely from laptop and tablet CAPI, although we assume that it is actually tablets, and not laptops, that fuelled the growth. 2014 is different – this year there has been a growth in mCAPI (21% in 2009, 25% in 2014). It is not large and it is too early to be certain that this is a new trend but given the ubiquity of smartphones and the rise of 4G mobile networks, we feel it is certainly one to watch.

We also notice that the proportion of companies offering SMS research has risen sharply in 2014. However, the figures are still very small – it was 6% in 2013 and 10% in 2014. Given that it has been languishing at the 4% to 6% level for several years, there is a strong chance it is a ‘blip‘ but it could also be the start of a new trend.

Very few market research companies (7%) offer an IVR (interactive voice response) service.

We also divide mixed mode into two groups: CATI and Web mixed mode and ‘other’. There seems to be a slow decline in mixed mode overall, and this looks like it may be coming from the ‘other’ group rather than CATI and Web.




Figure 10 Percentage of market research firms using the minor modes of research

3.6.1 ‘Mobile self complete’ trend - by region

à Mobile self-complete has grown rapidly in N. America and Europe

Back in 2009, the mode ‘mobile self complete’ was offered by as many as 12% of companies in North America. In Europe it was practically non-existent (2%). By 2014, both regions are in a similar position, with a third (33%) of companies in Europe and 28% in North America offering mobile self-complete.

With the ubiquity of smartphones, improvement in data handling on mobile phone networks and increasing availability of free Wi-Fi, we can only expect this mode to rise further. It is both a way for market research companies to drive up response rates and also offer their clients new research opportunities.

0%!

5%!

10%!

15%!

20%!

25%!

30%!

35%!

40%!

45%!

50%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

Mixed mode CATI & Web!

laptop/tablet CAPI!

Self complete - mobile!

mCAPI!

Mixed mode other!

SMS!

IVR!


All mentions of ‘mobile self complete’


Figure 11 Percentage of market research firms using the mode ‘mobile self complete’

0%!

5%!

10%!

15%!

20%!

25%!

30%!

35%!

2009! 2010! 2011! 2012! 2013! 2014!

N. America!

Europe!


3.7 Mobile participation in online surveys

à Survey participants taking online surveys on mobile devices continues to grow rapidly

à Managers in many companies seem unaware of the actual numbers

For the last four years, we have asked companies to report how many of their survey participants are attempting to take online surveys on smartphones and other mobile devices. Each year, we have observed that this is a difficult question for many companies to answer, with many responding that they don’t know. In 2014, the number of ‘don’t knows’ for this question was 29%. (In 2013 the number who did not know was even higher, at 37% of the sample).

Among the majority of the sample who were able to provide an estimate, the upward trend has continued from a mean average of 6.7% reported in 2011 to 21.4% in 2014.

Q. Thinking just about your web surveys, what proportion of participants are taking these surveys on small format mobile devices such as smartphones?

Figure 12 Mobile or smartphone participation in online surveys

FocusVision has independently reported the number of actual online survey starts observed on mobile devices on its Decipher online research platform for the past 4 years, which is used by many different research companies1. These data show a very similar pattern of growth over the same period for smartphone participation, which in the report are differentiated as samples drawn from panels and those from customer lists. FocusVision reports 3% smartphone participation on panel surveys in 2011, rising progressively to 7% in 2014, and 6% for customer lists in 2011, rising to 22% in 2014.

1Trend Report: Mobile Participation in Online Surveys, FocusVision. Download available from http://ww2.focusvision.com/wp-content/uploads/2015/03/FV_Dec_MobileUpdateWhitePaper.pdf,

21.4%

16.4%

13.1%

6.7%

2014

2013

2012

2011


There was virtually no differences reported between larger and smaller companies, or between North America and Europe (the sample size for Asia Pacific is too small to consider), demonstrating that this truly is a global phenomenon.

Those in larger companies were much more aware of the issue, with only 11% unable to give an estimate, as opposed to 45% of smaller companies unable to report the value. This probably reflects the reality that many smaller companies will use a third party to carry out their online research, rather than carrying it out directly, and either do not obtain the information, or do not track it across their various different studies.


4 Making surveys mobile-friendly

Since 2011 we have tracked the rise of ‘unintentional’ mobile survey participants – those who choose to respond to an invitation to an online survey on their smartphone, tablet or other web-enabled mobile device. This subject is a contentious one for research, yet it is not one that companies cannot afford to ignore. In 3.7 on page 28, we observe that once again, companies are reporting a large increase in the number of survey starts attempted on mobile devices.

Software providers have been somewhat slow to respond to this challenge in our view. A lot of effort in the period around 2010 and 2011, when the subject started to draw attention, was going into building tools for exploiting mobile as a standalone method, such as with surveys using mobile apps. Some of the online survey tools started to add capabilities to detect the device being used, and deliver a mobile-friendly survey in a largely automated way, while others held back, and left the problem for the user to solve – something that is very difficult to do without software support.

This section explores both the current practices within research companies with regards to allowing mobile participation in online surveys, the demands they have and the extent to which software providers are responding to these.

Questions

1. Extent to which mobile participation is considered at the research design stage

2. Company’s general approach to mobile participation in online surveys

3. Support for dual mode participation (mobile and desktop) in the technology used

4. Importance of dual mode support in online survey technology

5. Research company attitudes towards accommodating mobile participation within research designed for online participation

Key findings

Companies split almost equally between those that routinely discuss mobile participation at the research design stage and those that rarely consider it

§ 48% always or often consider mobile participation

§ 16% rarely or never do, rising to 23% among smaller firms

§ Most likely to be discussed in Asia Pacific (though the finding is on a small base)

The majority of research firms are taking steps to support mobile participation

§ 31% say all of their surveys are enabled for mobile participation and 36% say many of them are.

§ Smaller firms are taking the lead on making most or all of their surveys mobile friendly.


§ Only 4% routinely block mobile participating, but 11% let mobile participants in, without necessarily making their surveys mobile friendly.

Survey software is providing support for mobile-enabled surveys, but it could be better

§ 19% find it difficult or impossible to get their survey software to provide a mobile-accessible survey when designing online surveys

§ A large number of practitioners do not have a detailed understanding of the support their software tools provide for mobile-friendly online surveys

§ Larger firms tend to be more dissatisfied with the support for mobile within the software they use

Mobile support in online survey software is now considered an essential requirement

§ 57% say it is essential that online survey software supports both PCs and smartphones and another 34% say it is very important.

§ Demand is stronger among larger firms.

§ However, smaller firms report they are leading the way in making their surveys mobile-friendly

Practitioners still have some reservations about allowing mobile participation

§ The view that including mobile participants can improve response rates is widely held.

§ Promoters and detractors are evenly split when it comes to the suitability of many research projects for mobile participation.

§ A minority consider that insufficient understanding of the modal effects of mobile participation is a barrier.


4.1 Is mobile considered at the research design stage?

à Companies split almost 50/50 between those who mostly do and those who mostly don’t

à Only one in five almost always take it into account

à Smaller companies are more likely to ignore mobile participation when designing a new survey

à More likely to be considered in the Asia Pacific region

There is a complete spectrum of behavior overall, with almost half (48%) always or often discussing the question of mobile participation when designing online research, and 52% sometimes, rarely or never discussing the matter. Practice therefore seems to be slightly skewed towards not discussing it across the industry.

Q. How often would you say that the question of dealing with survey participants on mobile devices is discussed when designing online research projects?

N = 215

Figure 13 How often mobile is discussed when planning new research projects

There is little difference in the extent to which larger or smaller companies are taking a conscious approach towards discussing mobile (49% ‘always’ or ‘often’ in larger firms against

20%!

22%!

17%!

21%!

20%!

19%!

28%!

28%!

26%!

38%!

28%!

29%!

36%!

37%!

38%!

29%!

30%!

47%!

16%!

14%!

19%!

12%!

23%!

4%!

0%! 20%! 40%! 60%! 80%! 100%!

Total!

N. America!

Europe!

Asia Pac!

Smaller co's!

Larger co's!

Always/almost always! Often! Sometimes! Rarely/never!


47% in smaller firms). However, there are many more people in smaller firms (26%) reporting that they rarely or never discuss it, against almost none (4%) in large firms.

The issue gets more attention in Asia Pacific (59% always or often discuss it) than in Europe (43%) while North America conforms to the norm overall (49% North America, 48% across all companies).

This begs the question as to what happens to those surveys – and subsequent questions go some way to answering this. However, this is not a problem that companies can rely on technology alone to solve for them, and if it is not discussed in the design stage, we cannot imagine how a good outcome can be achieved.

In our 2012 report, we asked companies what they considered to be the natural limit for an interview carried out by telephone, on the web and on mobile. Taking the median of all the times given, this indicated that firms thought online surveys should be no longer than 18 minutes, but that mobile surveys should be no more than seven minutes long – less than one half the time.

If nothing else is considered, the length of the interview is something that firms should be discussing, given that, later in this report, firms are telling us that one in five online survey starts are now on a mobile device (see page 28).

There are other difficult issues that the published literature has identified that make mobile surveys perform very differently to conventional online ones, inter alia:

1. The difficulties of presenting wordy questions, grids and very long answer lists on a small smartphone display, and related to that, the need to avoid response bias from predefined answers that are not as accessible as others a smartphone on such lengthy questions.

2. The demographic profile of those participating on mobile versus a desktop – the former tending to be younger, less affluent, and in North America, from ethnic minorities

3. The length of time it takes to complete the same survey – typically longer on a mobile.


4.2 Company’s general approach to support for mobile participation

à Two thirds of firms report they support mobile in many or all of their surveys

à Support is more prevalent in North America and among larger firms

à Smaller firms taking the lead on making most of their surveys mobile friendly

This question aimed to uncover research companies’ general approach or policy with regard to allowing participation in online surveys using smartphones, from catering for it and welcoming it, through to actively preventing it.

Q. Which of these most closely reflects your approach to participants wishing to take online surveys on mobile devices?

N = 215

Figure 14 Company approach to supporting mobile participation in online surveys

The same diversity of approach observed in the previous question was apparent here, although the overall approach is more skewed towards providing support for mobile participation. Very few companies (4% in total) put a block on mobile participation, whereas two thirds (67%) claim to provide support for all or many of their surveys. A further one in six (17%) take an ‘ask and we will do it’ approach.

31%!

33%!

31%!

26%!

36%!

22%!

36%!

40%!

34%!

32%!

30%!

47%!

17%!

10%!

19%!

29%!

15%!

22%!

11%!

13%!

12%!

6%!

15%!

5%!

4%!

3%!

4%!

6%!

4%!

4%!

0%! 20%! 40%! 60%! 80%! 100%!

Total!

N. America!

Europe!

Asia Pac!

Smaller co's!

Larger co's!All surveys can be taken on a mobile device or PC!

Many surveys can be taken on a mobile device or PC!

We only provide mobile support if required!

Don’t support mobiles but we don’t prevent it!

Prevent participation from mobiles!


Support is strongest in North America (74%) and among larger firms (69%) in terms of those reporting support for ‘all’ or ‘many’ of their surveys, however it is smaller firms that are taking the lead on making all or most of their surveys mobile friendly: 36% against 22% of larger firms.

This behavior contrasts with the previous question – it would seem although conversations are not always talking place about how to handle mobile at the research design stage, companies are relying on default policies and also technology to ensure that surveys delivered are largely workable on a smartphone.

There is, in our view, a strong risk that this approach is over-optimistic, because some issues, such as interview length and question wording, cannot be dealt with by technology or general policies alone, and we suspect that it is too easy to assume that surveys are ‘mobile friendly’ when they are not, because they have not been designed or tested in that format.

Anecdotally, we are aware of survey participants still frequently finding they are having to pinch and scroll to be able to take a survey on their mobile device – and expressing astonishment when they can’t.


4.3 Technological support for mobile-friendly surveys

à It is moderately easy rather than very easy to produce mobile-friendly surveys from today’s online survey tools

à Larger firms report more limitations in their tools

à Smaller firms less aware of more advanced mobile-friendly requirements

The purpose of these two questions was to assess the extent to which technology providers are meeting the needs of research companies in providing tools that allow for survey designs to be executed on both traditional PC browsers and within the browser of a smartphone or other mobile device.

The two questions explore different aspects of the set of features that need to be built into any software capable of working in both environments.

The first is that surveys need to be able to detect the device being used, and render the questions in a way that will fit well on the screen and use only the capabilities of that device. In our view, this is an essential technical requirement and one that many software providers are addressing, some more effectively than others.

The second, more advanced point is about reducing respondent burden and increasing convenience by allowing the same participant to switch from one format to another during the interview. It also allows researchers to create more advanced research designs, such as for diary surveys, where diary entries are completed on a smartphone, but the participant may be expected to complete any pre- or post-survey questionnaire on large format device.

The pattern of answers to both questions is very similar with more finding the software easy rather than difficult to operate. If both questions are treated as 4-point rating scales (scoring 1 for very easy and 4 for very difficult) the mean scores are as follows:

N Mean score Don’t know

Q1. How easy is it to create a survey that can be taken on both PCs and mobile devices with the survey technology you use?

215 1.9 22 10%

Q2. How easy is it with the survey technology you use to create surveys that participants can start on a mobile device and then switch to a PC (or vice versa)?

215 2.0 49 23%

Table 5 Average scores for software capabilities

The major point of difference is that a large number of participants (23%) were unaware of the more advanced capability of being able to switch surveys, which appears to indicate this is a technique that is not widely used or encouraged.


Q1. How easy is it to create a survey that can be taken on both PCs and mobile devices with the survey technology you use?

Q2. How easy is it with the survey technology you use to create surveys that participants can start on a mobile device and then switch to a PC (or vice versa)?

N=215

Figure 15 Support provided in the survey technology used to create mobile-friendly surveys.

In only a third of cases (29% and 28%) is the software considered very easy, though a large number find it moderately easy (42%) for creation and somewhat fewer (30%) for the more technically advanced switching capability. But 19% in both cases report the software as being difficult or even impossible to use for this purpose.

We will next look at differences by global region and company size for these two questions.

4.3.1 Capabilities for creating surveys to be taken in both modes

Intriguingly, more of the larger companies report their software difficult to use – 27% versus 14% of smaller firms, and they are also more likely to know the capabilities, with only 3% not knowing, against 15% of smaller firms.

In global terms, firms in North America were also more likely to find their software not up to the task (22% versus 16% in Europe). Those in Europe were also less likely to know about the capabilities (17% versus 5% in North America), perhaps demonstrating less experience in this area, as well as less awareness.

28%

29%

30%

42%

14%

16%

5%

3%

23%

10%

0% 20% 40% 60% 80% 100%

Switching

Create for both

Very easy Moderately easy Moderately difficult V. difficult/impossible Don’t know


Q. How easy is it to create a survey that can be taken on both PCs and mobile devices with the survey technology you use?

N=215

Figure 16 Survey technology support for creating online and mobile surveys

4.3.2 Capabilities for switching

Q. How easy is it with the survey technology you use to create surveys that participants can start on a mobile device and then switch to a PC (or vice versa)?

N=215

Figure 17 Survey technology support for participants to switch from a mobile to a PC

It appears that less generally is understood about support for participants choosing to start a survey on a mobile device then continuing or finishing it on a different device. This may not be a capability that is required from a research design perspective, but it is an important consideration when the aim is to make surveys as respondent-friendly as possible.

29%

31%

27%

32%

31%

27%

42%

43%

40%

44%

41%

44%

16%

20%

12%

18%

12%

23%

3%

2%

4%

2%

4%

10%

5%

17%

6%

15%

3%

0% 20% 40% 60% 80% 100%

Total

N. America

Europe

Asia Pac

Smaller co's

Larger co's

Very easy Moderately easy Moderately difficult V. difficult/impossible Don’t know

28%

29%

29%

24%

31%

22%

30%

31%

27%

38%

26%

37%

14%

15%

13%

15%

10%

21%

5%

2%

10%

4%

6%

23%

23%

22%

24%

28%

14%

0% 20% 40% 60% 80% 100%

Total

N. America

Europe

Asia Pac

Smaller co's

Larger co's

Very easy Moderately easy Moderately difficult Very difficult or impossible Don’t know


Apart from the higher prevalence of don’t know responses, the pattern of answers is broadly similar to those for the previous question on creating surveys to run in both environments. Once again, larger firms seem to be least satisfied with the support in the software, and those in Europe are slightly ahead of the others in recording it is difficult or impossible to do.

4.3.3 Observations

The prevalence of so many ‘don’t knows’ on the last two questions is cause for concern, as it implies the extent to which technology is solving these problems is not well known by some practitioners. Put in the context of the first question in this section, which reveals that mobile enablement is not necessarily being discussed to the extent it should during the design stage, the lack of awareness on the capabilities of the survey platform being used among quite a large number of practitioners again points to this being a subject that is not being discussed as much as it should.

It also implies that the hope expressed by companies that many of their surveys are mobile-friendly may be over-optimistic.

We have already commented that technology alone cannot create mobile-friendly surveys, but the scores above also seem to show that technology is not making it as easy as it should be.

Perhaps it is unrealistic to expect that software can make the task of creating a mobile-friendly version of standard online surveys ‘very easy’ – though 29% say they have tools that do this. But again it is of concern that one in five professional researchers (19%) are reporting that it is difficult or impossible to get the survey platform they use to create an online survey that will function satisfactorily on a mobile device.

This is a finding that both research companies and software developers need to pay attention to.


4.4 Importance of combined mobile and PC support online survey technology

à Almost all companies consider support for both mobile and PC support is very important in any survey tool

à Larger companies more likely to consider such support is essential

Q: How important is it that online research software should support both PCs and mobile devices?

N=215

Figure 18 Importance of dual-mode support within survey tools

The purpose of this question was to understand the extent to which ‘mobile friendly’ is becoming a deciding factor in any company’s choice of online research software. Overall, 91% considered this to be essential or very important.

The differences between regions are very slight: Asia Pacific companies seem less likely to consider it an essential requirement (but this is based on only 34 companies).

This gives online survey software providers as well as panel providers a very clear steer that their technology will be rejected, and they will lose business, if they do not invest in building the capabilities within their software to support both desktop or laptop PCs and smartphones or other mobile devices. The preceding two questions have shown that there is still work to do in much of the software, with a small minority stating that their software

57%!

57%!

60%!

47%!

54%!

62%!

34%!

36%!

30%!

41%!

36%!

29%!

9%!

7%!

10%!

12%!

9%!

9%!

1%!

1%!

0%! 20%! 40%! 60%! 80%! 100%!

Total!

N. America!

Europe!

Asia Pac!

Smaller co's!

Larger co's!

Essential! Very important! Useful, but not important! It is not required!


makes building mobile-friendly online surveys easy, and around one in five finding it difficult or impossible with the tools they use.


4.5 Attitudes towards accommodating mobile participation

à Supporting mobile is seen as a way to improve response

à The ‘unsuitability of mobile’ argument persists, though with only a slender majority

à The argument that modal effects aren’t well enough understood is rejected, but again by a slender margin

In order to assess the companies’ attitudes towards mobile participation in online research, and its acceptability or even desirability, participants were asked to rate their agreement with three statements, the first which addresses the principal benefit claimed for researchers in facilitating mobile participation in online surveys and the other two refer to two commonly cited barriers to mobile survey participation:

1. “Allowing mobile participation in online research can help to improve response rates.”

2. “A large amount of online research is not suitable for participants to complete on a mobile device.”

3. “The effect of including mobile participants on the data is not yet well enough understood to recommend this approach.”

A five-point scale was used with score of 1 signifying full agreement and a score of 5 full disagreement.

Q: To what extent do you agree or disagree with these statements…?

N = 211a 208b 206c

Mean average. Excludes don’t know responses.

Figure 19 Research company attitudes towards accommodating mobile participation within online research

Overall, 74% agreed with the notion that accommodating mobile participation in online surveys will improve response rates, with 43% expressing the strongest agreement; 19%

2.1

2.6

3.4

1 3 5

Allowing mobile participation in online research can help to improve response rates (a)

A large amount of online research is not suitable for participants to complete on a mobile device (b)

The effect of including mobile participants on the data is not yet well enough understood... (c)

Agree Disagree


disagreed, but only 6% disagreed strongly. This argument appears to have been won quite decisively.

The two concerns, however, generated a more balanced response, one concern weakly held still, and the other weakly rejected.

50% still consider that much online research cannot successfully be translated into a mobile-friendly format, 19% holding that view strongly, while 24% disagree, 6% disagreeing strongly – but a further quarter (26%) occupy the middle ground. A substantial minority of the industry therefore seems to have found ways to overcome this barrier – possibly by making their research designs less extravagant in their use of the screen, less wordy and less time-consuming, for example.

The balance has tipped the other way with regards to concerns that the modal effects of mixing large- and small-format interviews can affect the data in ways that are not understood. There is now a large corpus of research on research that documents different modal effects, some of which also sets out strategies that can be adopted to mitigate or control for these differences. 49% disagree that not enough is known, 20% strongly disagreeing, against 20% agreeing (with only 4% agreeing strongly). Here, though, there is a larger group sitting on the fence – with 31% choosing the center of the scale.

The summary of mean average scores given, broken down by region and company size is shown in Table 6.

Total

N. America

Europe

Asia Pac

Smaller co's

Larger co's

Allowing mobile participation in online research can help to improve response rates

2.08 2.14 2.00 2.15 2.01 2.21

A large amount of online research is not suitable for participants to complete on a mobile device

2.62 2.68 2.58 2.55 2.55 2.73

The effect of including mobile participants on the data is not yet well enough understood to recommend this approach

3.45 3.66 3.36 3.13 3.51 3.35

Table 6 Summary of mean scores for rating scales on attitudes towards mobile surveys

It appears that larger companies are more accepting (or more adventurous, according to your perspective) of making online surveys mobile friendly, along with those in North America, but again, the differences are modest.

Overall, it looks as if the industry is moving towards accepting the merits of designing online surveys to be mobile friendly too, though still expressing some entirely reasonable caution about its applicability and the effects it can have on findings.

However, harking back to our first question, if discussions are not always routinely taking place at the design stage as to how to treat the 20% or more participants who are now likely to try taking a survey on a smartphone, then the reservation that many research projects are not suitable for mobile access will become self-perpetuating.


4.6 Conclusions

Mobile-friendly surveys are still a work-in-progress for the industry. The answers to our survey questions lead us to these conclusions:

• The important matter of how mobile participants are to be handled is not being discussed at the research design stage as much as it should

• A majority of firms now aspire to delivering surveys which are mobile friendly but they may not be achieving this to the extent they believe they are

• Many firms seem too reliant on the technology and/or default policies to deliver mobile-friendly surveys

• Some felt that their software platforms provide good support for mobile surveys, but others, particularly larger companies, find the task is made too difficult by the software

• Practitioners are well aware of the benefits of including mobile participants as a way to increase response rates, which have been a critical concern for the industry for many years.

• Practitioners still have reservations about the method, though most appear to accept that the modal differences and the effects these can have on data are now well understood.

• It is still a majority held view that many research projects are simply not suitable for mobile access. Given the current direction of travel in the outside world, this is an increasingly unsustainable position.

The research industry cannot afford to keep thinking of mobile participation in its online surveys as a problem, even less, a problem to be ignored. It is an opportunity to reach respondents in difficult target groups and those who would otherwise not participate. There is urgent work to be done in providing better support in the technology to facilitate this – and there is difficult work to do in persuading research clients, and some researchers, to create surveys that will still behave well in front of a participant using a smartphone.

Social media, e-commerce, and the very brands that many research projects are asking participants about have migrated to mobile, and so too must market research.

4.7 A metadata postscript

In case you were wondering, our survey was completely mobile-friendly and has been since 2011. We did not advertise this in the invitation in 2014, which, for most participants, was an email link.

This year five per cent of our participants chose to take the survey on a smartphone and two per cent on a tablet. This is a population which will all have access to a PC, and therefore, we consider the response by mobile to be quite high – especially since many will also be aware that surveys on mobile devices can take longer than on a desktop.


We advertised the survey would take typically 12-15 minutes to complete. The median interview length for those participating on a standard PC was 14 minutes and one second; for those on a smartphone it was 18 minutes 43 seconds, so it did take 34% longer.


5 Trends in online sample sources

Question

§ Proportion of online samples from each of five identified sources

Access panels favored over in-house panels

§ Over the years, there has been a trend towards the use of access panels and away from in-house panels.

§ The proportion of client sample in use has remained steady.

§ Larger companies are best at sourcing sample from their own panels but even they source nearly two-thirds of their sample externally


5.1 Proportion of online sample in use - trend

à Researchers becoming more reliant on access panels

à Own panels supplying lower proportion of sample

These figures provide evidence for our observation that researchers are finding it increasingly difficult to source sample for online surveys, either in the quantity or quality they require. The volume of client sample has never declined, despite predictions to the contrary in previous years in this study. The amount of sample from access panels has slowly risen, whereas the volume from companies’ own panels has gradually declined.

Q: What proportion of your online samples comes from each of these sources?


Figure 20 Proportion of online sample in use

0%!

5%!

10%!

15%!

20%!

25%!

30%!

35%!

40%!

45%!

2006! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

Third party/access panels!

Sample provided by client!

Own panels!

Specialist sample providers!

Others!


5.2 Proportion of online sample in use in 2014

à Larger companies best at supplying own sample

à Even large companies source nearly two-thirds (65%) of sample externally

The results for larger companies are noticeably different to those of smaller companies. Although, it is probably to be expected that due to their greater resources, larger companies manage to source more sample from their own panels and are therefore less reliant on client data, access panels and sample providers.

Looking at the global regions, North American companies make greater use of access panels, and source less of their sample from own panels.

Q: What proportion of your online samples comes from each of these sources?

N = 215

Figure 21 Proportion of online sample in use, by region and company size

22%!

17%!

22%!

34%!

16%!

32%!

39%!

49%!

32%!

32%!

40%!

37%!

9%!

8%!

11%!

6%!

10%!

8%!

26%!

24%!

30%!

22%!

30%!

20%!

4%!

2%!

6%!

6%!

5%!

3%!

2014!

N. America!

Europe!

Asia Pac!

Smaller co's!

Larger co's!

Own panels! Third party/access panels! Specialist sample providers!

Sample provided by client! Others!


6 Trends in research software demands

Questions

§ What desire exists to replace the software currently in use, in the near future?

§ What type of software is being considered for replacement?

Key findings

Big appetite for new research software among larger companies

§ Over a third of larger companies say that they are planning on changing their research software over the next two years.

§ A quarter of smaller companies have plans for new software purchases

All types of research software on shopping list

§ Between 63% and 73% of all of those companies who wish to change their research software want to change core research software modules (data collection, data processing, analysis and report publishing).


6.1 Planning on changing software in 2014

à Strong appetite for new software

à Especially among larger companies

We see this question as a barometer both of research companies’ willingness to invest money in as well as their satisfaction with their existing tools. There is relatively little regional variation in 2014. However, larger companies seem to have much bigger plans for software purchases compared with smaller companies.

Q: Are you considering changing the software you are using for your data collection or data analysis in the next one to two years?

N = 215

Figure 22 Planning on changing software, by region and company size

30%! 29%! 28%!36%!

25%!37%!

22%! 27%!18%!

24%!

23%!

22%!

48%! 45%!54%!

39%!

52%!41%!

2014! N America! Europe! Asia Pac! Smaller co's! Larger co's!

Yes! Not sure! No!


6.2 Planning on changing software - trend

à Good opportunities for software developers with new tools to offer

With three out of every ten companies saying they plan to buy new software and over half not ruling it out, the current year (i.e. the period that follows the 2014 survey) appears to offer good opportunities to software developers for new business, although not quite as strong as in some of the preceding years.

Q: Are you considering changing the software you are using for your data collection or data analysis in the next one to two years?


Figure 23 Planning on changing software, by year

26%! 22%!

40%!32%!

25%!

46%!

30%!35%!

30%!

25%! 34%!

18%!26%!

23%!

20%!

22%!

25%!

22%!

49%! 44%! 42%! 43%!52%!

35%!

48%!40%!

48%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

Yes! Not sure! No!


6.3 Software type to change

à For the first time, report publishing and delivery tools come top of the shopping list

à Firms are also on the look-out for core data collection and data analysis platforms

Various questions within this survey over the last two to three years suggest recent shifts within data collection (increasing use of mobile devices) and also a gradual shift towards higher tech results delivery methods, so the results in this question add weight to our hypotheses.

The high scores across all categories also indicate that many companies are considering replacing tools in more than one area.

The lower scores for panel management tools probably reflects the situation that these are specialized tools which not every company needs. In that context, demand for these therefore appears to be strong.

Q: What software are you considering changing?

N 2010 = 54; 2011=105; 2012=85; 2013=85; 2014=62

Figure 24 Software type wishing to change

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2010 2011 2012 2013 2014

Report publishing and delivery

Data collection

Data analysis

Data processing

Panel management


7 Shadow IT and alternative technologies

Shadow IT, sometimes also called ‘stealth IT’ is the growing phenomenon within organizations where the IT software or hardware being used by staff is not technology the organization has provided. It may be entirely unapproved and unofficial, in the case of a member of staff attaching their own iPad, or syncing their smartphone, or storing company files on one of the popular consumer file-sharing services like Dropbox. It could be that the technology is accepted but not chosen by the organization, as is the case if the organization makes use of software that is controlled by a supplier, such as a data collection platform, or a client, such as a reporting system.

With the burden of responsibility on all research companies to protect confidentiality, the growth in research industry standards for data security, and a complex web of legal responsibilities around the world, the appearance of so many ‘back doors’ through which data can escape unprotected is cause for great concern.

On the other hand, technology moves fast, and many of the cloud-based tools that consumers and business people are turning to can feed creativity and allow people to move quickly to tackle new problems and be innovative in their work.

It is a topic that is little discussed, which is why we considered asking some specific questions about, what we referred to in the survey as ‘Alternative technology’ in order to measure the extent to which research companies are rising to the challenge.

Questions

1. The range of alternative technologies in use

2. Company approach to alternative technology and Shadow IT

3. Policies in operation

4. The extent to which IT systems controlled by clients or suppliers are in use

5. Attitudes to the use of Shadow IT and alternative technology

Key findings

Much of the technology in use in MR firms is no longer under central organizational control

§ 82% of companies are using data collection, analysis or other software operated by their suppliers and 48% are working with systems that their clients control

§ 24% of companies are using research software selected and purchased out of project or departmental budgets.

§ 23% report that Internet software downloaded by employees is in use in their firms and 13% report employees connecting their own hardware (though we believe this may under-report the phenomenon).


Many companies have controls in place to allow localized software choices to be made

§ Only 15% of firms report they have no restrictions, though only 4% of larger firms

§ 30% have guidelines and 29% only permit use in exceptional circumstances

§ Large firms tend to use guidelines more, in preference to operating on a ‘by exception’ approach, which is favored slightly more by smaller companies.

Policies are widely used to manage the risks of using alternative technology

§ But between a third and a half of businesses do not have policies to cover one or more risks.

§ 75% of firms control downloading software from the Internet (which rises to 97% of larger firms.

§ Only 72% of larger firms and 52% of smaller firms have policies relating to cloud-based services where there can be a risk of data leakage.

MR companies using their suppliers’ or clients’ technology is on the increase

§ 34% report it is growing while 45% report that it is unchanged.

§ Virtually no-one reports it is declining.

Practitioners are concerned about the growth of alternative and Shadow IT within the organization

§ Security and data protection risks are considered the most concerning.

§ The idea that giving users the freedom to choose their own software is a good way for firms to innovate is largely rejected.

§ There are also concerns that companies can lose their competitive edge if made to use data collection or reporting systems chosen by their clients.


7.1 Technologies in use

à Most MR companies now work using software controlled by their research suppliers

à One third of firms report their data using reporting systems under their suppliers’ control

à 23% of firms use DIY survey tools selected by client

This question presented a wide range of possible technologies that may be in use within the company – essentially in four groups: systems operated by research suppliers, systems operated by clients, software chosen at a local or departmental level and Shadow IT, of which two specific options were provided (web software downloads and ‘bring your own device’).

Q: What additional technology resources do you make use of in your company?

N=215

Figure 25 Additional technologies and Shadow IT resources used within the company

We were careful to refer to ‘research supplier’ in the first grouping, otherwise any cloud-provided software or service could fall into this category.

10%

13%

23%

28%

24%

13%

23%

34%

48%

23%

54%

47%

62%

82%

Something else

‘Bring your own device’ - hardware owned by employees but used to connect to company systems

Free or very low cost software downloaded from the internet by employees

Any 'Shadow IT' (net)

Software selected and purchased out of project or local departmental budgets

Other kinds of software operated by a client

DIY survey software operated by a client (eg SurveyMonkey, Zoomerang)

Reporting software (including dashboards and data portals) operated by a client

Any software operated by a client (net)

Other kinds of software operated by a research supplier

Data analysis or reporting software operated by a research supplier

Panel management software operated by a research supplier

Data collection software operated by a research supplier

Any software operated by a research supplier (net)


Three of these four categories were expanded in the question asked, to make the responses more specific. The four groups are represented as the darker blue bars in the chart in Figure 25– three of these were constructed as a net total of the specific categories within the group, which are shown as the light blue bars.

The pattern of responses shows that all kinds of software and technology used within research companies’ core area of research service activity has not been chosen by the organization as a whole. Using the software that a research supplier provides, in particular, is very common, reported by 82% of companies.

Over the years, many panel companies have taken on an enhanced role not only in providing panel but also in carrying out the fieldwork using its own software, and we believe the very high figure of 62% (the highest in those reported) reflects this situation. It also shows the extent to which research companies access these systems directly, to direct how the sampling is done, to supervise the progress of work, extract data and perform some preliminary analysis, all using the provider’s systems. However, this is not something that is only experienced by smaller research companies who may not have their own in-house technology. The figures for smaller and larger companies are virtually the same: 81% for smaller companies, 83% for larger companies.


Total N.

America Europe Asia

Pac Smaller

co's Larger

co's

Any software operated by a research supplier (net) 82% 83% 80% 85% 81% 83%

Data collection software operated by a research supplier 62% 60% 62% 71% 58% 71%

Panel management software operated by a research

supplier 47% 53% 55% 56% 50% 62%

Data analysis or reporting software operated by a

research supplier 54% 52% 41% 50% 45% 51%

Other kinds of software operated by a research supplier 23% 39% 30% 32% 24% 51%

Any software operated by a client (net) 48% 54% 43% 47% 41% 60%

Reporting software (including dashboards and data

portals) operated by a client 34% 25% 21% 26% 19% 32%

DIY survey software operated by a client (e.g.

SurveyMonkey, Zoomerang) 23% 30% 20% 15% 22% 26%

Other kinds of software operated by a client 13% 22% 23% 24% 21% 26%

Software selected and purchased out of project or local

departmental budgets 24% 24% 20% 26% 16% 35%

Any 'Shadow IT' (net) 28% 30% 29% 24% 26% 32%

Free or very low cost software downloaded from the

internet by employees 23% 16% 12% 12% 11% 18%

‘Bring your own device’ - hardware owned by employees

but used to connect to company systems 13% 13% 12% 15% 8% 21%

Something else 10% 7% 15% 6% 12% 8%

Table 7 Additional technologies and Shadow IT resources, by region and company size

However, a small difference opens up when it comes to using the client’s software, which smaller companies seem to have resisted more than larger companies: 41% for smaller firms versus 60% for the larger firms.

There is also an important difference between smaller and larger companies in the extent to which some software decisions are devolved to a departmental or local level, which has a rational explanation. For smaller companies, the figure is 16%, for larger ones, more than double, at 35%.

Many smaller firms will not have much in the way of a departmental structure, and decisions by a department are more likely to become company-wide decisions, or supported by any central IT function, if one exists. Removing smaller companies from the picture, where 16% report software being procured at a departmental level, we consider the more interesting figure is that 35% of larger companies report that some software decisions are taking place at a departmental level, or by a project team, selected for and funded by a project.

Shadow IT, however, is no respecter of global region or company size. It is prevalent overall in 28% of companies, and is slightly higher in North America (30%), and also in larger companies (32%). Our estimate relies entirely on a reported total from one individual within


an organization, and is therefore prone to underreporting, since this is a covert or ‘shadow’ activity.

Whereas companies choosing to, or being forced to use a provider’s or a client’s technology systems may exert some influence on that company’s performance, its autonomy or its distinctiveness in the market – which may carry with it some moderate commercial risk – the rise of Shadow IT brings with it a tangible threat that data security and that confidential research data could be disclosed. This has serious professional, ethical as well as commercial consequences.

All risks can be managed to some extent. Our next question focused on how company attitudes and policies may be used to control these risks.


7.2 Company approach to allowing non-standard software

à Most companies have some controls in place to allow localized software decisions

à Larger firms more likely to have a set of guidelines

à Smaller firms more likely to follow a case-by-case approach

Using software that has not been procured centrally carries that risk that it may not comply with legal or corporate requirements. For companies in Europe, a cloud-based solution may not comply with the legal requirement that any personally identifiable data does not reside outside of the European Union, for example. At the other end of the spectrum, the software may be of poor quality, and not carry out calculations as accurately as users believe. The central procurement and control of IT allows companies to carry out more extensive due diligence to avoid such pitfalls.

There are essentially two different scenarios to consider. First, non-standard software may have been selected by a department or project team described in the previous question, where several people are involved and there is an element of oversight. In the second situation, software may be introduced by an individual who decides to download some software from the Internet, or increasingly, access and upload data to on the Internet to a Cloud-based tool, and then possibly share it with colleagues too. The first situation may sometimes be considered an example of Shadow IT; the second is a prima facie example of it.

Our question encompassed both of these situations: we asked simply:

What best describes your company’s approach to individuals or teams using software which has not been procured centrally?

The answer options provide a range of control regimes, from permissive to restrictive:

• Allow without restrictions

• Allow within some general guidelines

• Allow only in specific defined situation

• Only allow in exceptional circumstances

• Not allowed

• It varies

The responses overall and by company size are shown in Figure 26. There were no notable differences in the responses by global region.

Overall, only 15% report no restrictions: smaller companies are more permissive (20%), larger firms less likely to have no restrictions (4%). Larger companies were also a little more likely to have guidelines in place – 35% report control through guidelines, against 28% of


smaller firms, as opposed to control through an overall restriction, but then allowing specific exceptions. This case-by-case approach is followed by 24% of larger firms, but by 33% of smaller firms. A minority of firms (13%) takes the route of banning it altogether.

Q: What best describes your company’s approach to individuals or teams using software which has not been procured centrally?

N=215

Figure 26 Company approach towards using software external or non-standard software

15%20%

6%

30% 28%35%

29% 33%24%

13% 15% 10%

7% 7%8%

6% 7%3%

0%

20%

40%

60%

80%

100%

Total Smaller companies Larger companies

Don’t know

It varies

Not allowed

In specific situations or exceptional circumstances

Allow within general guidelines

No restrictions


7.3 Company policies for alternative technology from clients or suppliers including Shadow IT

à A majority of companies have policies in place, but many do not

à Downloading software from the internet appears to be well-covered

à The risks from cloud-based solutions and ‘bring your own devices’ are less well covered

Having looked at the general approach to the specific case of localized or departmental decision-making over software, we turned to the other areas of risk, to see whether companies had specific policies in place to deal with these.

Q. Do you have any company policies relating to… ?

N=215

Figure 27 Specific company policies towards different kinds of external technology use

The five specific areas of risk or concern were elaborated in the wording of the question as follows:

1. Accessing software or systems operated by your suppliers 2. Accessing software or systems operated by your clients 3. Downloading software from the Internet onto company systems 4. Accessing cloud-based services (e.g. Google Drive, Microsoft Cloud, Dropbox) 5. Employees connecting their own devices (e.g. tablets, smartphones, laptops)

Overall, more than half of companies report having some policies in place. Three quarters (75%) report they have a policy to cover downloading software from the Internet, but only one half have policies concerning accessing systems provided either by suppliers or clients.

52%

51%

75%

62%

61%

Accessing software or systems operated by suppliers

Accessing software or systems operated by clients

Downloading software from the Internet

Accessing cloud-based services

Employees connecting their own devices


Again, there are important distinctions by company size. Our sample also includes very small companies of only one, two or three individuals, so for these, policies may not be necessary.

Almost all larger companies have a policy covering downloading software from the Internet (97%) whereas only 63% of smaller companies do, and the same pattern is repeated for every other of the risk areas we asked about.

Q. Do you have any company policies relating to… ?

N=215

Figure 28 Specific company policies towards different kinds of external technology use: by company size

An important observation is while companies are targeting ‘Downloading software from the internet’ through policies, the almost equal risk of ‘Accessing cloud-based services’ is relatively lightly policed through policy. The specific risks are of data leakage through employees choosing to use cloud-based services such as Dropbox and Google Drive (which we mentioned specifically in our question). Only 72% of larger companies report policies for this risk, and that drop to 57% among smaller companies.

There are also gaps in how companies are addressing the risks from employees connecting their own devices (‘BYOD’ or ‘bring your own device’): 77% of larger firms have policies, only 52% of smaller ones to. Connecting an unauthorized device to a network provide another opportunity for data to be siphoned off in a way that is untraceable.

There are also some curious regional variations regarding policies around accessing clients’ or suppliers’ systems – these policies are more common in Europe and in Asia Pacific than they are in North America. In our earlier question (see Table 7), there was little reported difference in the prevalence of using supplier systems between North America and other regions, though companies in North America appeared to be more often connected to their clients’ systems (see Table 7 on p. 57, and refer to “Any software operated by a client”). Overall, as shown in Table 8, companies in Asia Pacific (at least, those we interviewed) appear more advanced in their policy-making.

Total

N. America

Europe

Asia Pac

Smaller co's

Larger co's

72%

71%

97%

72%

77%

41%

39%

63%

57%

52%




Accessing cloud-based services


Small companies Large Companies


Total

N. America

Europe

Asia Pac

Smaller co's

Larger co's


52% 45% 54% 65% 41% 72%


51% 46% 51% 62% 39% 71%


75% 72% 76% 82% 63% 97%

Accessing cloud-based services 62% 60% 61% 74% 57% 72%


61% 60% 60% 68% 52% 77%

Table 8 External software and technology policies in place, by region and company size


7.4 Use of systems controlled by suppliers or clients

à Suppliers or clients insisting that MR companies use their systems is on the increase

à Larger firms are experiencing it more than smaller firms

We were interested to know if the specific phenomenon of the research company making use of research technology provided and controlled by its suppliers, or its research clients, was on the increase. Over a third (34%) of companies say it is, and larger companies seem to be experiencing it more, with 38% reporting an increase. However, the increase reported was, for the most part, a gradual increase; and 45% of companies say it is not changing. Virtually no one reported a drop.

Q: Would you say the number of clients, suppliers or other partners requesting that you use software or systems that they control, has increased over the last 1-2 years?

N=215

Figure 29 Changes in the use of suppliers’ or partners’ software and systems in the last 1-2 years

7%

7%

5%

27%

23%

33%

45%

46%

44%

3%#

1%#

5%#

19%

22%

13%

0% 20% 40% 60% 80% 100%

Total

Smaller co's

Larger co's

Strongly increasing Gradually increasing Remaining the same Decreasing Can’t say


7.5 Attitudes towards alternative technology

à Practitioners are cautious about alternative technology

à A majority see ‘BYOD’ and software downloads at work as a security and data protection risk

à Concerns over MR firms losing competitive advantage if clients/suppliers oblige them to use their systems

Our final question in this section attempted to understand the concerns or feelings that practitioners had about the developments they were reporting, in the growth of the use of alternative technologies, not procured centrally, and the rise of Shadow IT within their organizations. Three statements were provided, and participants were asked on a five-point scale to indicate agreement or disagreement. These statements are based on concerns, which are often reported in the research media, discussion forums and at industry events:

1. “Allowing individuals and teams to use whatever software and technology solutions they prefer is a good way for research companies to innovate.”

2. “Research companies can lose their competitive advantage if they have to use their client’s data collection or reporting system rather than their own.”

3. “The security and data protection risks are too great to let individuals choose their own software or connect their own devices to company systems.”

The mean average score is presented for each in Figure 30, with the position of the “slider” representing graphically the average response given. The scale runs from 1 to 5, with a midpoint representing neither agreement nor disagreement at 3.

Q. To what extent do you agree or disagree with these statements… ?

N=215

Figure 30 Attitudes towards alternative technology and Shadow IT

Overall, the item where most concern is expressed is over the security and data protection risks arising from the use of Shadow IT, with a score of 2.46. 56% expressed agreement and 24% expressed disagreement

3.06

2.80

2.46

1 3 5

Allowing individuals to use to use whatever technology they prefer is a good way for research

companies to innovate

Research companies can lose their competitive advantage using client data collection or reporting

systems rather than their own

Security and data protection risks are too great to let individuals choose their software or connect

their own devices to company systemsAgree Disagree


There is much weaker agreement over the risk that companies can lose their competitive advantage if research buyers insist that they use their own technology, rather than technology the research company has selected or developed. It is a strategy that some research buyers are adopting to ensure that they are less dependent upon one supplier of their research. In this, the average score was very close to the midpoint of 3, at 2.8. However, 42% of companies agreed, 30% disagreed and 25% were neutral.

There was a moderate rejection of the idea that allowing teams to chose and use their own software was a good way for research companies to innovate – as we presume our participants could think of better and perhaps less risky ways for a company to innovate. The average score was barely beyond the midpoint, at 3.2. Behind this average, 32% agreed, 37% disagreed and 30% were neutral.

Company size made almost no difference to the scores, but there is an interesting regional variation on the statement that elicited the strongest response – the risks over BYOD and individuals downloading their own software. Those in North America appear to be more tolerant than those in Europe, and those in Asia Pacific (where there is a low base) judged the risk to be greater.

Mean Summary

Total N.

America Europe Asia Pac Smaller

co's Larger

co's

Total 215 87 94 34 137 78

“Allowing individuals and teams to use to use whatever software and technology solutions they prefer is a good way for research companies to innovate”

3.1 2.9 3.1 3.3 3.1 3.0

“Research companies can lose their competitive advantage if they have to use their client’s data collection or reporting system rather than their own”

2.8 3.0 2.7 2.5 2.8 2.8

“The security and data protection risks are too great to let individuals choose their own software or connect their own devices to company systems”

2.5 2.7 2.4 2.2 2.5 2.4

Table 9 Attitudes towards alternative technology and Shadow IT by region and company size

7.6 Conclusions

It is no longer the case that all the technology in use within research companies is chosen centrally, and this carries considerable risks for companies, in particular the risk of data security breaches. Many companies have policies and procedures to manage the risks (though we did not go as far in our questioning to establish how effective these procedures are). However, it is worrying that many companies do not appear to have strong protections in place.


It is natural that larger firms will be more diligent over having formal policies than smaller ones, where informal policies can be just as effective. Nevertheless, looking just at the larger companies in our sample, it is clear there are areas that are not necessarily being considered. Virtually every larger firm had a policy over downloading software from the Internet (which has been a problem for many years, and the risks largely center around downloading viruses or Trojan Horses).

However, a much more contemporary threat is the Cloud-computing solution that can be accessed for a few dollars a month on a corporate credit card. These products are becoming very prevalent, and provide useful services for the busy and resource-stretched researcher. For example, tools for creating word clouds, or visualizing data, performing translations, or those we mentioned in the survey, for storing data and exchanging files. Far fewer of the larger firms had policies in this area, and barely one half of the smaller ones did.

Research companies need to be highly vigilant over what is happening to their data. Even though most of the data that may be uploaded to one of these services may not contain anything that is either personal or commercially sensitive, there is always the possibility that something that should not get through does.

The report reveals good practice among many firms, who are clearly being diligent and rational in dealing with this phenomenon – which does bring benefits with it too. We are concerned about the firms for whom this does not seem to be a priority.


8 Trends in analysis and results reporting

Questions

§ What proportion of projects involves each of the main deliverables or distribution methods?

§ How important is the ability to produce volumes of cross-tabular reports in future?

Larger companies using more high tech reporting

§ We also saw in 2012 and 2013, larger companies are going high tech with their results distribution, with a swing towards the use of dashboards and interactive analysis and away from ‘low tech’ methods such as Acrobat PDF files. This was not the case in 2011.

Good news for trees?

§ Most of the survey participants (86%) said that the ability to be able to produce tables in volume is ‘essential’ or ‘moderately important’. This has changed little since 2004.

§ However, in 2014 almost no projects (6%) are actually delivered as printed tabs

Market researchers moving away from low tech distribution methods, except Microsoft PowerPoint

§ Microsoft PowerPoint is every market research company’s favorite results delivery tool and every year, it is used to deliver the results on around half of all projects

§ Other ‘low tech’ results delivery methods are becoming less popular – printed tables, Microsoft Word and probably Acrobat PDF


8.1 Distribution methods in use

à Microsoft PowerPoint is leading delivery method – by far – and no sign that this will change

à Growth in other tech-based delivery methods largely static

à Other ‘low tech’ methods on the wane: printed tabs, Microsoft Word clearly becoming minority methods, Acrobat PDF possibly also

With the very notable exception of Microsoft PowerPoint, the lower tech distribution methods seem to be on the wane.

Printed tables, or bulk cross-tabular reports, the oldest and most low-tech deliverable, have shifted from a mainstream client deliverable to a specialist one. In 2006, they were used for 23% of projects and in 2014, this had reduced to 6%. However, their use internally, we believe, is undiminished. Microsoft Word, another ‘close to paper’ method seems to be on a clear decline and Acrobat PDF is starting to look like it is moving the same way.

Although there are some indications, as discussed overleaf, that the industry is adopting the more high tech methods such as interactive analysis and dashboards.

Q: What percentage of projects currently involves the following deliverables or distribution methods to the client?


Figure 31 Distribution methods in use

0%!

10%!

20%!

30%!

40%!

50%!

60%!

2006! 2007! 2008! 2009! 2010! 2011! 2012! 2013! 2014!

MS PowerPoint!

MS Excel!

Acrobat PDF!

Online static reports!

MS Word!

Interactive analysis!

Printed tabs!

Digital dashboards !


8.2 Distribution methods in use, by company size and region

à Larger companies probably more likely to use higher tech distribution methods – dashboards and interactive analysis

à Few distribution methods used routinely in Asia Pacific

In 2012 and 2013, we saw signs that the largest market research companies were moving towards the higher tech results delivery methods, such as interactive analysis and dashboards. This led to a corresponding drop in the use of lower tech methods such as Microsoft Word, printed tables and Acrobat PDF. In 2014, this same pattern seems to be true among larger companies but the results are not as clear. The reason for this is almost certainly because fewer large companies took part in the 2014 study. It could, of course, be that there is no trend towards high tech delivery methods but, since moving to more high tech delivery methods makes logical sense for market research companies with regard to customer service, we anticipate that this trend does continue in 2015.

There is some regional variation. Notably, in Asia Pacific most results distribution methods are rarely used, except for Microsoft PowerPoint, Excel and online static reports.


Q: What percentage of projects currently involves the following deliverables or distribution methods to the client?

N = 215

Figure 32 Distribution methods in use, by region and company size –2014

49%! 48%!52%!

20%!22%!

18%!19%!16%!

25%!

14%! 16%!

11%!12%! 14%!9%!9%!

8%!12%!

6%! 7%!4%!

5%! 4%!7%!

3%! 3%!4%!

2014! Smaller co's! Larger co's!

MS PowerPoint! MS Excel! Online static reports!

Acrobat PDF! MS Word! Interactive analysis!

Printed tabs! Digital dashboards! No answer!

49%!

58%!

46%!

37%!

20%!

15%!

26%!

18%!19%! 19%!18%!

23%!

14%! 12%!

19%!

6%!

12%!11%!

16%!

5%!9%! 9%!11%!

6%!6%! 8%!4%!

4%!5%! 5%!

6%! 4%!3%! 2%! 3%!

2014! N. America! Europe! Asia Pac!

MS PowerPoint! MS Excel! Online static reports!

Acrobat PDF! MS Word! Interactive analysis!

Printed tabs! Digital dashboards! No answer!


8.3 Importance of producing cross tabs in volume

à Nearly everyone needs to produce cross tabs in volume

Even though very few projects are now delivered on printed tables, most of our survey participants (86%) say that is essential or moderately important to be able to produce cross tabs in volume. This can be explained by a continuing need to use tables as a primary tool and resource for analysis, and also as a means to crosscheck other deliverables, as a quality control method. The need for cross-tab packages is not about to disappear.

Q: When considering analysis and reporting tools for the future, how important is it that these should be able to produce volumes of cross-tabular reports?


Figure 33 Importance of being able to produce volumes of printed cross tabs

59%!55%! 55%! 52%!

57%!50%! 53%! 52%! 53%! 50%! 49%!

27%!27%!

33%! 36%! 28%!34%! 30%! 33%! 34%!

32%! 37%!

13%!14%!

10%! 11%! 13%! 13%! 16%! 13%! 10%!15%!

12%!

1%! 4%! 2%! 1%! 2%! 3%! 1%! 2%! 2%! 4%! 2%!

0%!

10%!

20%!

30%!

40%!

50%!

60%!

70%!

80%!

90%!

100%!

2004$ 2005$ 2006$ 2007$ 2008$ 2009$ 2010$ 2011$ 2012$ 2013$ 2014$

Essential! Moderately important! Relatively unimportant! Not something future tools should provide!


9 Data visualization and dashboards

This report has considered different aspects of data visualization and dashboards before: most recently in 2011. For this year’s study, we repeated two of the questions asked in 2011, to see the extent to which practice is evolving, and also added some new questions to explore how the industry is responding to both the opportunity and the challenge that adding data visualization into the reporting process brings.

Questions

1. The amount of work that involves different kinds of visual reporting

2. How easy or difficult it is to meet client demands for visual presentation formats

3. The software utilized in producing data visualizations

4. The challenges in providing dashboard reporting solutions as a part of a research project

5. Who is involved in producing data visualizations

6. Advice on designing a dashboard to present survey data

Key findings

Data visualization is moving into the mainstream

§ Companies report 27% of their projects are now delivered with some dashboard element to them

§ Infographics are now used in around 15% of reports

Producing data visualizations used to be really difficult – it’s now only fairly difficult

§ 20% report providing an online dashboard is easy, 27% say they are difficult.

§ 24% find producing infographics easy, 32% say they are hard.

§ More have mastered getting static charts and visualizations online: 50% say it’s easy and only 9% say it’s difficult.

It’s difficult to produce good dashboards due to software and project budget constraints

§ 51% report their software needs customization in order to deliver a dashboard

§ 45% cite not having enough budget to deliver everything the client asks for

§ 35% point to not having the right skills in-house skills to create dashboards

Firms are using tools other than PowerPoint and Excel for their data visualizations, though it can be hard to tell


§ 75% mostly use what Excel and PowerPoint provide and 13% don’t use anything but Excel or PowerPoint

§ 20% mostly rely on the capabilities of their core cross-tab or reporting software

§ Most companies use several tools in combination – and a wide variety of tools are used

Companies are making greater use of graphic artists and data design specialists than they were when visualizing data

§ Companies calling on a wide range of talent to produce visualizations

§ In 2011 only 13% of firms used graphic artists and only 8% used visual designers; in 2014, 33% report using graphic artists and 17% visual designers.


9.1 Projects delivered using data visualization

à Over a quarter of projects are now delivered as online dashboards with some visual component

à Infographics are used in 15% of reports

Our first question asked participants to give a percentage value for the amount of projects that involved four different kinds of visual presentation: static reports presented online, online dashboards, interactive charts, and (typically static) customized Infographics in reports. We appreciate that there may be some degree of overlap between some of these categories, but they serve to provide a basic taxonomy of the kinds of graphical deliverables that research buyers are seeking today.

Q. Now thinking specifically about data visualization, approximately what percentage of the projects you deliver require data to be presented visually in any of these formats?

N = 205

Figure 34 Volume of projects delivered using various types of data visualization

Generally, the frequency with which each deliverable is produced tends to reflect the cost and complexity behind each one. Static charts or graphs presented in a web portal or some other online context feature in 37% of projects. These represent outputs that may be produced in many software packages, or even produced in Microsoft Office. This delivery method appears to be slightly more prevalent among larger companies and also in North America.

Online dashboards, which are much more technically demanding, but for which there appears to be a growing appetite among research buyers, are delivered in over a quarter of projects. The share is 27% overall, with very little variation between larger and smaller companies. Demand is much lower in Europe, at just 23%, higher in North America, with 29%, and higher still in Asia Pacific.

Interactive charts, where the user is able to apply different filters or selection criteria, and see the effect displayed immediately, now account for 18% of reports, with higher demand reported in Asia Pacific and from larger companies.

15%

18%

27%

37%

Custom infographics in reports or online

Interactive charts presented online

Online dashboards that include some data visualisation

Static charts or graphs presented online


A similar pattern is reported for custom created Infographics, where a visual designer or graphic artist would work with a data specialist to produce a one-off graphical storyboard – the most labor-intensive, through not necessarily the most technically demanding of the four deliverables listed here. Overall, Infographics now appear to be used in 15% of projects. Smaller companies report using them on only 11% of their projects while for larger companies it is virtually double, at 21%.

Total

N. Americ

a Europe Asia

Pac Smaller

co's Larger

co's

Static charts or graphs presented online 37% 42% 33% 37% 32% 45%

Online dashboards that include some data visualization 27% 29% 23% 33% 26% 28%

Interactive charts presented online 18% 16% 17% 27% 15% 24%

Custom Infographics in reports or online 15% 14% 15% 19% 11% 21%

Total 205 84 89 32 129 76

Table 10 Volume of projects delivered using various types of data visualization: by region and company size

There is also a pattern that larger companies provide more data visualizations across all of these formats, with the single exception of online dashboards that feature some data visualization – where the number is effectively the same (28% versus 26%).

It is probably fair to conclude that, at this point, dashboards specifically, and more ambitious data visualizations than those achieved in Excel and PowerPoint, have entered the mainstream.

As such, in the context of this report, it is important to know whether the technology required to do this is judged to be up to the task. The next few questions explore this by looking at the challenges faced.


9.2 Ease of achieving client demands

à Providing static online charts easiest, but still found difficult by a few

à Firms finding Infographics the hardest to produce

à Dashboards are also tricky – only one in five consider them relatively easy to produce

Among bloggers, industry media columnists and even speakers at industry events, it is a commonly voiced concern that research buyers want to have more visual and more interactive presentation of their data, but are unwilling to allow additional time or budget. We therefore specifically asked the extent to which companies were able to produce these deliverables with ease (or not) within the limits a typical project budget imposes.

We would like to know how easy or difficult it is to meet your client’s demands for each of these visual presentation formats within the limits of a fixed project budget, where 1 is very easy and 10 is very difficult?

N = (dashboards) 191; (Infographics) 192; (interactive charts) 190; (static charts) 195

Figure 35 Ease or difficulty or meeting client demands for various visual presentation formats

We were anticipating that the more ‘advanced’ methods would be rated as progressively more difficult, but we were surprised that each of these, beyond static charts online, achieved a very similar average score, at the cusp of where easy turns into difficult.

Closer examination of the distribution of the highest and lowest scores (the lowest and highest three scores have been grouped together in Table 11) shows that there is some polarization in the responses. One quarter of companies assert that interactive charts and also custom Infographics are, overall, easy to do (scoring 1, 2 or 3 on the ten-point scale) and rather more – 28% for interactive charts, and 32% for Infographics, place them in the difficult category (a score of 7 or higher).

4.0!

5.5!

5.6!

5.7!

1! 5.5! 10!

Static charts or graphs presented online!

Interactive charts delivered online!

Custom infographics in reports or online!

Online dashboards that include some data visualisation!

Easy Difficult!


There is a larger middle ground in the case of dashboards, with fewer firms finding them easy (only 20%) and many more (27%) finding them difficult.

Static charts Interactive charts

Infographics Dashboards

Easy (3 lowest scores) 50% 25% 24% 20%

Neither 41% 46% 44% 53%

Difficult (3 highest scores) 9% 28% 32% 27%

Mean score 4.0 5.5 5.6 5.7

Table 11 Ease or difficulty or meeting client demands for various visual presentation formats: distribution by region and company size

From this it is clear that these are not activities that can easily be absorbed into project budgets, and they are therefore adding pressure to research companies. However, there is a minority of research companies who either use software or have developed processes such that these tasks can be performed with relative ease.


9.3 Challenges with providing dashboards

à Main challenges: in-house limitations – money, technology and skills

à Among client concerns are misinterpretation of results and clients who over-complicate the dashboard

Our next question examines the specific pressures research companies face – and for this question we chose to focus on what we understood to be an area many firms find difficult, and which the previous question has shown, is an area that the fewest companies considered easy to do.

What do you consider to be the principle challenges for research companies in meeting client demands to provide a dashboard reporting solution as a part of a research project?

N = 215

Figure 36 Principle challenges in providing dashboard reporting as a part of a research project

Participants were allowed to choose up to three challenges from a list of possible challenges, and to put their primary concern first. The chart in Figure 36 shows the responses sorted in

4%

4%

5%

6%

6%

8%

9%

12%

23%

24%

4%

29%

31%

24%

19%

27%

21%

35%

51%

45%

No answer

Demands to over-complicate dashboards

Standard software unsuitable for MR data

Ensuring data is error-free

Other delivery methods more suitable

Users misinterpreting results

Gaining acceptance among end-users

In house skills to create dashboards

Research software needs customisation

Not enough budget

Top 3 concerns

Primary concern


the order from high to low of the primary challenge cited. The specific answers given have been abbreviated in this chart. The complete text that participants responded to is:

• Not enough budget to deliver everything the client wants • Gaining widespread acceptance among end-users • Research software not flexible enough to meet all demands without customization • Industry-standard software does not meet the demands of market research data • The risk of users misinterpreting or misusing the results • Ensuring that all combinations of the data are consistent and error-free • Having the right skills in-house to create dashboards • Demands to over-complicate dashboards so they fail to provide a simple overview • Clients wanting dashboards when other delivery methods are more suitable

Not surprisingly, given the effort required that we have observed in the previous question, the primary concern is over not having enough money in the project budget to do what the client wants, cited by one in four (24%) as their primary concern, and by almost half (45%) as one of their three challenges.

The need to customize research software to accommodate dashboards was given as a concern by more than half of the sample (51%), and it ranked second after budget concerns, as a primary concern.

The third principal challenge cited relates to skills. Another question in this section asked specifically about those involved, and the answers to that question bring this challenge into focus, as it is often not those who may not have the necessary skills who are carrying out these tasks.

From our fairly lengthy list of nine possible challenges, only one was chosen by fewer than 20% of those answering: and that is that other online delivery methods might be more suitable. The wide range of secondary or tertiary challenges selected really serve to underline that creating dashboards is a complex task that carries with it a wide range of potential difficulties.

One point of interest to the more technically minded is the answer “Standard software is unsuitable” (the precise wording used was “Industry-standard software does not meet the demands of market research data”). From our own knowledge of the software products available on the market, both those designed for business intelligence users, and those specifically created for market research, we know that tools the IT industry tends to use are not at all adept in handling survey data, and to overcome these inadequacies requires a great deal of highly skilled technical effort. This was a primary concern for only 5% of companies, but 31% did mention this as one of their concerns, placing it fourth. We suspect that it does not have a higher position because many firms are passing over these more generalist tools, in favor of those specifically created for survey data.


9.4 Software used for data visualization

à Microsoft Excel and PowerPoint are ubiquitous

à Specialist dashboard software now used by over a third of companies

à But more are using the charts and dashboard capabilities within their crosstab software

Our next question focuses more generally on data visualization, not just visualization expressed through dashboards.

What software does your company use for data visualization?

N = 215

Figure 37 Software used for data visualization

Though Microsoft’s Excel and PowerPoint are ubiquitous, it is surprising that they are cited quite so frequently in the domain of products used to create interactive or online data visualizations. It is possible to produce a range of fairly basic visualizations from within these tools, but they are limited in scope. Since they do not contain the means to create Dashboards or interactive charts published on a web portal, we have to interpret their presence here as being tools used in the more elementary requests for data viz. It is almost like adding ‘value for money’ to a purchasing intention question in a consumer study – to get it out of the way before looking at what the real drivers are.

For Note: percentages shown are of the number of companies (i.e. based on N=215)

12%

9%

4%

9%

19%

75%

30%

24%

16%

23%

41%

19%

Other tools

Specialist Digital Dashboard creation software

Specialist Charting software (e.g. Dundas, Chart FX)

Market research chart automation software (e.g. E-tabs, Rosetta Studio)

Charting or dashboard capabilities within the survey tabulation software that your company uses

Charting capabilities within Excel and PowerPoint

Usually Sometimes


Table 12 we have cross-tabulated the ‘usually’ and ‘sometimes’ users of Excel and PowerPoint with the other product categories, to show the inter-relationship between any one type of software used and any other. The table only combines ‘usually’ or ‘sometimes’. The percentage is on the total sample, so each cell shows the proportion of companies with that combination of tools.

N=215 Tab

software Chart

automation Specialist charting Dashboard Other

Nothing else

Excel or PowerPoint 58% 32% 20% 33% 40% 13%

Tab software

23% 14% 26% 28% >0.5%

Chart automation

15% 19% 15% -

Specialist charting 14% 13% -

Dashboard 17% -

Other >0.5%

Note: percentages shown are of the number of companies (i.e. based on N=215)

Table 12 Data visualization tools used: Which other tools are used when Microsoft Excel and PowerPoint are mentioned.

This inter-relationship table reveals several interesting findings. First, only 13% don’t appear to use anything else, so therefore 87% of companies are using at least one other tool in combination. Secondly, apart from tabulation software (which clearly has not been acquired specifically to deal with data visualization, but is being exploited, like Excel and PowerPoint, because it is there) there is no other tool that has a dominant position – but there are a lot of tools used in combination with each other.

Overall, 48% of companies are using ‘other’ tools – there is little observable difference between those that ‘usually’ and those that ‘sometimes’ use Excel and PowerPoint. There is no one killer application, it seems, and different companies are finding a solution using a combination of different tools.

There is also a long tail of “others”, which we captured as a verbatim response, when given. The answers given here are also very diverse, and included several mentions of graphic programs such as Illustrator and InDesign (which are appropriate when creating Infographics). One specific product was mentioned by 5% of companies: Tableau, which is a specialist business reporting tool rather than a market research specific product.

9.4.1 Changes since 2011

à Situation virtually unchanged since 2011

The same question was asked in 2011. To create a comparison, we have combined together the responses for software ’usually’ used and software’ sometimes’ used, to create a net use measure.

Due to limitations of the sample, we will discount minor variations of two or three per cent as these are likely to be due to natural variation in the sample. Only two items showed a larger change – but the difference is only 4% in each case, and so is hardly indicative. These


were for the use of charting/dashboard capabilities within the firm’s main analysis package, and chart automation software – possibly indicating a very small rise since 2014.

The differences were even smaller when comparing just the ‘usually use’ responses from 2014 with 2011.

What software does your company use for data visualization? Showing net use: ‘usually’ and ‘sometimes’ combined

N = 230 (2011); 215 (2014)

Figure 38 Changes in tools used for data visualization: 2011 and 2014

42%

33%

20%

32%

60%

94%

44%

34%

20%

28%

56%

97%

Other tools

Specialist Digital Dashboard creation software

Specialist Charting software (e.g. Dundas, Chart FX)

Market research chart automation software (e.g. E-tabs, Rosetta Studio)

Charting or dashboard capabilities within the survey tabulation software that your company uses

Charting capabilities within Excel and PowerPoint

2011 2014


9.5 Who produces data visualizations?

à Researchers do the most, but graphic artists are involved a third of the time

à Larger companies involve more people in producing visualizations

Producing data visualization calls on a very wide range of skills: data or IT specialists are often involved in getting the data in a format where they can be visualized efficiently; visual design specialists are needed to create an effective design that can communicate the results well, and researchers also need to be involved, to select the most relevant data.

Who typically produces any data visualizations required?

Figure 39 People who produce data visualizations – overall and by company size

This question does not show an estimate of the volume of projects treated this way – it is the frequency of mention of each group of people by each participant, as to whether they are or are not involved in producing visualizations within their company. Therefore, the 33% mention of graphic artists implies that 67% of companies are not using graphic artists.

The dominance of researchers in the activity can be interpreted in one of two ways – they are functioning as part of a team, or they have to produce the visualizations for themselves. We suspect both situations apply to some extent.

What is clear is that there are many more people with different skills involved in producing visualizations – and there is no group where you could consider their involvement is exceptional. However, the specialists are mentioned less frequently, which implies that many

70%

33%

26%

21%

17%

15%

69%

28%

23%

14%

9%

10%

72%

40%

32%

35%

32%

24%

Researchers

Graphic artists

Programmers or IT specialists

Research assistants

Visual designers

Charting specialists

Total Smaller companies Larger companies


research companies are relying on the visual design talents of their researchers or IT specialists to produce their data visualizations.

When examined by size of company, it is also clear the extent to which large companies are able to involve more kinds of skills in the process than smaller companies are.

The very large difference on research assistants can probably be explained by that role not existing in smaller companies, so researchers are unlikely to have a research assistant to call on. However, there are consistently fewer mentions in every category among smaller companies – except for the omnipresent role of the researcher.

9.5.1 Changes since 2011 This is another of the questions we also asked in 2011, and this time there are a few notable differences between the responses then and now.

Who typically produces any data visualizations required?

N = 230 (2011); 215 (2014);

Figure 40 People who produce data visualizations – 2011 and 2014 compared

First is that graphic artists are being involved much more, as are visual designers. To some, these groups of people may overlap, but we have found both of these titles in circulation and they appear to represent different roles: graphic artists having a foundation in drawing and illustration (such as in advertising), visual designers working more in the area of industrial design, or computer interface design. What they have in common is that they are not the conventional people you would find in a research team.

Secondly, there are some groups who are involved less often: research assistants most notably, but also charting specialists. This may also indicate increased recognition that when high quality visual treatment is required, these skills are not typically found in the people hired to work in these roles.

73%

13%

31%

43%

8%

23%

4%

70%

33%

26%

21%

17%

15%

3%

Researchers

Graphic artists

Programmers or IT specialists

Research assistants

Visual designers

Charting specialists

Other

2011 2014


Programmers and IT specialists have experienced a more modest drop – probably reflecting an on-going need for their involvement in order to prepare the data or present the results in an online environment.


9.6 Words of advice when designing a dashboard

In order to gain some insight into what practitioners saw as the fundamental principles that apply when designing dashboards for market research data, we asked what single piece of advice participants would give to anyone tasked with designing a dashboard. This was asked as an open question, and we have categorised the unstructured responses received into groups. The question was optional, and 143 of the 215 participants responded.

Among these, 43 (31% of those answering) emphasised simplicity – keeping the dashboard as simple as possible. Of those, 27 used the exact words “Keep it simple” in English or the direct equivalent in their own tongue. Three more expressed it in the four letters KISS (“Keep it simple, stupid”, it refers to the KISS principle of design that emphasises avoiding unnecessary complexity). Some also expressed this idea using the phrase “less is more”. These responses point to the tendency we too have observed, that dashboards fail to provide a simple overview when they try to do too much, often through trying to accommodate a long list of demands from the client or end users, or sometimes to appease the demands of a particularly influential stakeholder.

This one response overshadowed all the others given. However, there is rich learning in the 22 other groups of answers we identified. The twelve most frequently given are shown in Figure 41.

What one piece of advice would you give to anyone tasked with designing a dashboard to present survey data?

N = 143

Figure 41 Advice on how to set up dashboards

Only three other pieces of advice were given by more than 10% of those responding. First among these is to understand the client needs fully which included suggestions about working

31%

10%

9%

9%

5%

4%

4%

3%

3%

3%

3%

3%

Keep it simple

Understand your client's needs

Focus on the end-user

Make it easy to use/user-friendly

Visually engaging/focus on the

Avoid them

Understand the research

Focus on accuracy/reliability

Plan carefully before implementing

Avoid function creep

Reveal the story

Use a trusted supplier


closely with them, agreeing a specification and project goals, and looking at the business needs. The second was to consider fully the end user or consumer of the dashboard, and what he or she would require, or would be able to understand. The third, which also expresses elements of the latter, was to ensure that the dashboard design was easy to use, easy to understand or intuitive to the end user. If we combine the latter two, end-user considerations were cited by 25 participants (17% of those answering).

Six participants (4% of those answering) recommend avoiding creating dashboards altogether.

The other seven shown in the chart (with our interpretation in parentheses) are:

• Ensure the dashboard is visually engaging and focus on choosing effective graphical elements.

• Recognise the research objectives and remain true to the research or the principles of research.

• Focus on accuracy/reliability (as errors are not easily corrected and will undermine the credibility of the dashboard).

• Plan carefully before implementing (possibly by adopting a formal implementation plan, prototyping and agreeing the design before going ahead).

• Avoid function creep (and resisting the pressure to add more and more features to the dashboard so that it no longer meets the goal of being simple and at a high level).

• Reveal the story or ensure there is a narrative present within the dashboard, which includes observations about designing dashboards that encourage end-users to explore further.

• Use a trusted supplier, both from the perspective of guaranteeing safe delivery and also with some elements of ‘don’t try to do this yourself…’).

Beyond those shown in the chart, the other advice given (again, with our interpretation in parentheses) is to:

• Consider automation/updating of data (so that tracking studies or external feeds of data can be added regularly).

• Build in flexibility (such as to allow the dashboard to change in response to new demands, or to allow the end-user to tailor the dashboard to what they need).

• Build up your own knowledge or expertise first (it’s a different style of presentation, and not one market researchers are necessarily used to).

• Design in stages and build up slowly, rather than try to start with everything • Educate the end-users of the dashboard, who will need help in understanding the data

and how to interpret it • Focus on the data, which included both the preparing of the data and restructuring it,

and also the concept of ‘being true to the data’ (which relates closely to the ‘being true to the research’, mentioned above.)

• Focus on trends – that dashboards appear to work best when presenting trended data

• Reach out beyond classic market research thinking (i.e. look to how dashboards are used in other industry sectors rather than treating the dashboard as an interactive cross-tab or online PowerPoint presentation. This also relates to the point above on building up knowledge and experience).

• Allow enough time (to go through the process of agreeing requirements and designing with the end-user in mind – it takes much longer than many people anticipate).

• Don't overdo the visuals (resist graphics that don’t actually convey understanding any better than figures or words might)

• Test thoroughly (providing a dashboard is not unlike developing a new piece of software, and there is great potential for problems to arise with both the functionality provided and also the data being presented).


There was virtually no difference in the popularity of these responses by either company size or global region.

9.7 Conclusions

Producing data visualizations that go beyond mere charting, providing interactive charts, online dashboards with visual components and Infographics are now widely established in the canon of deliverables that research companies are expected to provide.

Some of these channels, such as dashboards, or Infographics, which would have been considered exotic a few years ago, are now in the mainstream and are an everyday occurrence within many research companies.

When this topic was last explored in 2011, a significant concern expressed then was that companies did not have the right tools or the skills to be able to turn out the kinds of visualizations clients were demanding.

In 2014, the greatest concerns voiced were in relation to having sufficient project budget to meet client needs, and also the needs to customize. Though having the right skills was also mentioned as a concern, it was well down the list, and this appears to be because some progress has been made in acquiring the right skills. In the specific case of dashboards, which are a pressure point for many research companies, it is a tale of over-stretched budgets and expensive demands that require customizations to the software.

It is clear from the changes taking place in who is carrying out visualizations, between 2011 and 2014 that it is now recognised as a multi-disciplinary activity, and that companies are making greater use of artists and design experts than they were, as a way of ensuring they have the right skills. The industry appears to have risen to the challenge and is finding the talent it needs to do visualization work properly.

The picture is less rosy when it comes to the technology, however. Some companies report good experiences with the software they use making it relatively easy to produce what they need; more, however, report it is still too difficult to produce dashboards, deliver interactive charts online or create Infographics.

With companies still finding it convenient to a large extent to rely on their core tabulation or analytical software, one argument is for those software providers to build in more dashboard and data visualization capabilities into those tools. Alternatively, the most effective remedy for research companies may be to invest in better tools more suited to the task, which they can integrate into their work stream.

There appears to be a depth of experience in providing dashboards among many research companies of all sizes, however, judging by the richness and diversity of advice provided – the most prevalent being to recognize that an important goal is for any dashboard to be simple.

Simplicity, however, is usually only achieved through an iterative process of refinement. The experiences reported in designing a dashboard also reveal the classic tension that exists in market research in balancing what the client demands and what the researcher judges to be an appropriate professional solution, or ‘good research’.

It highlights the need for dashboard designers to feel empowered to say ‘no’. This can not only help to ensure that dashboards ultimately fulfil their goal of providing an overview of


widely accessible, actionable insights, but may also overcome the problems of insufficient budget and cost over-runs cited in another question in this section. It is not a problem that is confined to creating dashboards, however. These are the same tensions that give rise to surveys continuing to be longer than most researchers consider are sustainable.


10 Conclusions and recommendations

Throughout this report, we have made a number of conclusions and recommendations. This short chapter brings these together in summary form, as eleven key points to address (if these are not already being addressed) for technology specialists within research companies, and a handful more specifically addressed to companies providing technology or technology-led services to research companies.

Flexing the modes 1. Review the capabilities and technological support you have across the full range of

research modes, in order to maximise coverage and response. Though mixed-mode research is a powerful way to achieve both, it remains static (see pp. 17, 21 and 25). Individually though, different research channels are showing signs of growth, such as CAPI and SMS and mobile self-completion (pp. 17, 25 and 26). Anticipate that the mix of work is likely to carry on changing in coming years.

2. In the drive for greater efficiency, there could be an overlooked role for IVR in the mix of modes you offer – particularly if combined with other modes, where there is initially some personal contact. The contrast between its low level of use within market research and in near ubiquity in business is striking (p. 16)

Becoming mobile friendly 1. The universal trend for people to access the web from their smartphones is

relentless in its advance, and market research has to move quickly from viewing this as a nuisance to considering it an opportunity and a benefit (see p. 28). Make sure you are among the growing number of firms that discusses the implications of mobile when designing each new piece of research (p. 32). The answer may not always be that the survey is suitable for mobile, but assuming it isn’t is no longer tenable as a default position (p. 44).

2. Ensure that the technology you are using, or that your provider is using, is capable of delivering mobile-friendly surveys, and that it is also easy for your team to test how your surveys will render across a range of different devices (p. 44).

3. Keep an eye on the stats, and ensure you know how many survey starts are attempted from mobile devices (p. 28). Use this as a means to educate everyone, clients included, in the merits of accommodating mobile participants – which includes shorter surveys for those on the move. (p. 44)

‘Shadow IT’ and localized IT decision-making 1. Acknowledge that individuals will wish to bring in their own devices to connect to

your networks, and that departments and individual teams will increasingly want to purchase and use software they have chosen (see p. 55). Set up appropriate policies and the means to seek approval for exceptions (pp. 59 and 61) to manage the risks effectively.

2. Above all, consider the risks of data leakage, as this has data protection as well as research confidentiality implications, and could cause major financial and reputational damage (p. 66).


3. Review the use of clients’ or suppliers’ external systems within your company to ensure these too meet your requirements from a data protection perspective (p. 64).

Data visualization and dashboard provision 1. Invest in appropriate technology: many companies report that the task is difficult with

the tools they are using (see pp. 77 and 79) – yet the tools most commonly used are not specialized in providing dashboards or visualisations (p. 81). Are your employees equipped with the best tools to present data? Are they making do with PowerPoint and Excel, when other tools or approaches could be much better?

2. Recognize that data viz, dashboard design and Infographic creation are multi-disciplinary activities. Up skill by having graphic artists and specialists in visual design on staff, or on call, to help with the design – follow the lead companies are reporting by involving more of these kinds of people between 2011 and 2014 (p. 85).

3. When asked to provide a dashboard, aim for simplicity, (p. 87) and plan the work carefully with your client to reduce the risk of functional creep and demands to add more complexity (see pp. 79 and 89).

Recommendations for technology providers 1. Recognise that, even if the mixed-mode survey may not be growing in volume, there

is greater diversity of modes being used across all surveys and this trend is growing. Therefore ensure that data collection tools can support this diversity, either directly, or through establishing links and partnerships with providers specialized in, for example, mobile apps, SMS or even IVR (see p. 16 and 25).

2. Clients now expect their online survey software platforms (or providers) to generate surveys that are also mobile-friendly, with the minimum of intervention (p. 40). In particular, ensure that clients also have the means to test these surveys for themselves.

3. Be aware that research companies are looking for better solutions in the area of data presentation (pp. 52 and 73). They are likely to be looking for solutions that offer, among other things, simplicity in presenting and visualizing data, flexibility and ease of use.

4. If you are providing cloud-based data storage as a part of your services, inform clients about the data security protection measures you have in place, and provide information where those data are stored (see pp. 59 and 61) and who else is allowed to access them. Clients will also benefit from knowing what they need to do, on their part, to ensure their data remain secure.

FocusVision 2014 MR Technology Report - 3.2 · London, April 2015 . FocusVision 2014 Annual MR Technology Report | 7 of 92 1 Management summary This annual survey of over 200 research

Documents