Best Practices in Mobile Research

8/3/2019 Best Practices in Mobile Research

1/14

Best practices in mobile researchManuel Zahariev, Chris Ferneyhough, Chris Ryan

ESOMAR

Online Research, Chicago, October 2009


2/14

Best practices in mobile research

Manuel Zahariev, Chris Ferneyhough and Chris Ryan

Vision Critical, Canada

Sonia Bishop

Angus Reid Forum

INTRODUCTION

For the past few years, online data collection for marketing research has been established as a robust, mainstream

methodology. Until very recently, data collection using mobile technology was primarily confined to SMS surveys and

intercept-based research.

SMS-based surveys consist of a series of exchanges between a survey system and the respondent, where the system sends

a question in an SMS message and the respondent types their choice in a message. A typical exchange is illustrated below:

System: Does our mobile web site provide you with the information you were looking for?

REPLY Y FOR Yes OR N FOR No

Respondent: Y

The system continues the survey by sending the respondent another question. SMS surveys have low resilience to

respondent errors and are applicable to very short surveys.

Mobile devices are also used to augment traditional intercept survey scenarios. Responses are automatically entered in a

database, either through a wireless network connection while the survey is being filled or at a later time, when the device is

synchronized. It could be argued that mobile-augmented intercept increases the quality of the responses through the removal

of conversion errors and through application-based enforcement of the survey logic.

Recent advances in mobile phone technology make it possible to consider the delivery of online surveys directly to mobile

devices. A new generation of so-called smartphones have web-browsers, email access and the ability to install and run

applications.

Globally, an increasing number of cellular phone operators offer Internet access; users can browse the web and access email

Title: Best practices in mobile research

Author(s): Manuel Zahariev, Chris Ferneyhough, Chris Ryan

Source: ESOMAR

Issue: Online Research, Chicago, October 2009

Downloaded from warc.com

2


3/14

on phones with these capabilities. New phones from a number of mainstream manufacturers have the ability to connect to Wi-

Fi networks. Wi-Fi points of presence are increasing in coverage and represent, in many cases, faster and less expensive

alternatives for accessing the Internet than the access provided by wireless carriers.

As a result of these developments, access to the Internet happens increasingly through mobile devices in a number of

geographies. Respondents to existing online panels already receive invitations to surveys on their mobile device; in many

cases, these invitations link to surveys designed for consumption on a desktop or laptop computer, and on a mobile device

can lead to inefficient, painful or completely broken respondent experiences.

For example in the screening study for this project, conducted on national panels provided by Vision Critical, a percentage of

respondents had received their invitation to a survey on their mobile device, reported by geography United States: 1.9%,

United Kingdom: 1.2%, Canada: 3.8%. Obviously, results will vary depending on the composition of the panel. Specifically, our

Canadian panel has a higher proportion of higher income and employed members who tend to be more tech-savvy, so the

higher proportion in Canada should be interpreted with caution.

In this paper, we will be focusing on best practices associated with data collection for web-based online surveys where

respondents use mobile devices.

DEVICES

There are significant differences between the respondent experience in an online computer environment (further referred to as

online in this paper) and in a mobile environment. The online respondent experience on a computer cannot be directly

transferred to mobile devices.

For computers, there are a small number of primary, cross-standardized hardware architectures; two operating system families

(Windows and Mac OS) cover most of the market; four browsers with comparable and advanced feature sets (InternetExplorer, Firefox, Safari and Google Chrome) cover over 99% of Internet users, with reasonable standardization and

comparable, complete feature sets. Generally, online surveys do not push the limits of the computers they are running on.

By contrast, there is a wide variety of mobile devices, from more than a dozen major manufacturers, in a variety of models. It is

reported that there are over 1,600 different models of smartphones and new models are brought to market frequently. Screen

sizes, resolution, processing power, memory, and input modalities vary widely, even between devices of the same

manufacturer. Operating systems and application programming environments are also very different. For example, Nokia

supports three families of operating systems on their devices, with significant variation even between minor versions of the

same family and at least five different programming platforms. Mobile browsers are a lot more variable, with capabilities

generally lagging well behind computer-based counterparts. Significant differences are also common in rendering onlinecontent, adherence to web standards and support for dynamic page content through client-side scripting. Support for rich

content browser plug-ins such as Adobe Flash is very restricted, with significant variation in the supported feature set between

devices.

Assumptions can safely be made about the existence of a client-side scripting engine (JavaScript), support for cookies, font

colours and images in a computer-based browser. These assumptions do not hold on mobile devices; for example, JavaScript

is disabled by default on a variety of RIM Blackberry devices; many users disable the automatic image download capability in

the browser.


3


4/14

As a result, providing a uniform and robust respondent experience across a variety of devices, on different browsers, with

different display and input capabilities is the most significant challenge of mobile data collection for marketing research.

For this study, we used sample from Vision Critical's national panels in the United States, United Kingdom and Canada. These

panels are used by clients via omnibus and field & tab services, syndicated studies, public affairs and custom research. Panelmembers are drawn from across the full breadth of the adult population and profiled against a range of demographic,

consumer, social and attitudinal characteristics. Data quality is assured by the application of rigorous panel management

practices and recruitment quality standards. Since these panels are operated on Vision Critical's industry online research

platform Sparq, they can take advantage of the mobile delivery capabilities of the platform.

To assess the situation and identify panellists with smartphones, we conducted a profiling study across three national panels

(Canada, United States and United Kingdom). Best practices for profiling include:

l Create and maintain a comprehensive list of phone manufacturers;

l Create and maintain a comprehensive and up to date list of phone models by manufacturer;

l Identifying if the smartphone is for business or personal use;

l Determine the email address associated with the smartphone; this can be different than the email address registered with

the panel for online access;

l Assess interest in the ability to conduct surveys via their smartphone;

l Determine the panelist's data plans (can influence willingness to do mobile surveys);

l Determine the panelist's capability and use of internet browsing on smartphone.

In profiling our three national panels in Canada, United States and United Kingdom, we discovered the following differences in

the incidence of devices (see Table 1).

Table 1

Seven manufacturers cover more than 10% of panellists with a mobile phone in at least one of the markets; maximum

penetration in any country is just over 36% for Nokia in the United Kingdom and 31% for BlackBerry in Canada. The challenge

is substantial, since the list of devices for each manufacturer can be quite sizeable; for example, there are more than 30

models of BlackBerry smartphones, and hundreds of Nokia phone models.

QUESTION TYPES SUPPORTED


4


5/14

For this test, we did not support Vision Critical visual questions, grids, or other-specify types. There were two main reasons for

this. First, in terms of usability, a longer process of design and testing is required to tailor the question types to the small

screen. Mobile devices have their own design constraints and strengths, and simply porting over existing designs or

implementations would not result in a good respondent user experience in many cases. Incorporating user input techniques

such as multi-touch is ultimately desirable but will take a significant amount of effort for a relatively small portion of

respondents currently using an iPhone (see Figure 1).

Figure 1

Second, the implementation environment differs, particularly across platforms. For example, there is no support for Adobe

Flash on any of the phone platforms except Nokia, which prevents deployment of most of Vision Critical's visual question types

on those platforms. JavaScript is turned off by default on a number of BlackBerry devices, precluding the normal behaviour of

other-specify questions (enabling the textbox when the other button is selected and disabling the textbox when any otherbutton is selected). In order to accommodate as wide a range of mobile devices as possible, a lowest common denominator

approach to question design was taken. This was still effective, as there are tradeoffs between the increased usability issues

for the more sophisticated question designs, and users know how to deal with the standard widget set.

The small screen environment requires a dedicated survey implementation. Surveys designed and developed for desktop and

laptop browsers will generally work on mobile devices, but the user experience is not optimal. For example, the screen shot in

Figure 2 shows a question with a default browser-based skin, displayed in Mobile Safari on the iPod touch (the iPod touch and

iPhone share the same user experience). The text is unreadable and the targets are too small to hit.


5


6/14

Figure 2

The user can read and complete the question by zooming in, but this would be required for every question, and the zooming

and scrolling would become laborious and likely result in high drop rates.

Similarly, if the question is too long or answers are verbose, the mobile user will have to scroll to see all content and access

the Next button. On some devices, it is not obvious that there is additional content that can be accessed by scrolling (see

Figure 3).

Figure 3

Grid questions were broken down as single-choice questions, one per screen, and reduced in number: one grid question with

six items in an online survey would become six different questions in a mobile survey. Figures 3 and 4 show one of these

disassembled grid questions in a desktop browser and in Mobile Safari on iPhone OS.

Figure 4

Open-ended questions (see Figure 5) were used but limited to one or two per survey, to reduce effort in manual text entry

using small or on-screen keyboards.

Some common-sense guidelines here include:

Figure 5

l Limit the amount of text in the question. The more text the less space for your answer categories, and the more scrolling

required. It is also important to try to keep the Next button above the fold (see Figure 6).


6


7/14

Figure 6

l If possible, don't state your scale in the question.

l Edit to remove superfluous words: get to the point. The example in Figure 7 shows how shortening responses makes the

difference between the Next button being above the fold or not.

Figure 7

l Limit the number of answer categories; each one will take up a line of precious space. It is strongly advised to have no

more than ten answer categories.

l Limit the amount of text in each individual answer category or wrapping of text will occur, taking up two lines instead of

one and thereby reducing the number of answer categories (see example above).

METHODOLOGY

Vision Critical conducted a series of surveys to further test the mobile survey space. The main objectives of the surveys were:

l Compare online data with mobile data is the data comparable despite the different mode of data collection? If the data

is the same, it opens the door for multi-modal studies

l Compare results by type of device do different devices with different user experiences affect the data?

l Compare the experience across different regions are there different issues by region?

l Use the study to further develop best practices.


7


8/14

Three countries are included in the research: Canada, United States and United Kingdom. The sample population was

smartphone users, notthe general population. The demographic profile of smartphone users is different from that of the

general population (for example, higher income and more education).

Panelists on our Canadian, US and UK access panels were screened to identify the ownership and use of smartphones both

in terms of brand and model and whether it was their business or personal phone. Furthermore, to respect our current

relationships with panelists, we inquired into their willingness to take part in surveys via their smartphone as we could not

presume to contact them in this manner without their consent.

The United States and United Kingdom involved a mobile only survey. That is, panelists with smartphones were invited to

complete a survey on their mobile phone. They could not complete the survey on their desktop computer. If they attempted the

survey via their desktop, they were directed to complete via their mobile phone. We designed the study in this manner in order

to have a clean sample of people who completed the survey via a single mode (see Table 2).

Table 2

The Canadian sample of smartphone users was split into two separate samples: 500 smartphone users were directed to a

mobile survey and 500 smartphone users were directed to an online survey. The two samples were balanced on gender and

age and results were weighted with the same weighting scheme (by gender and age). As the incidence of iPhone users is

lower than for Blackberry and other devices, iPhone users were oversampled to have a sufficient number of completes among

iPhone users (see Table 3).

Table 3

Based on the research objectives, it was determined that the more important objectives of the research were to examine data

comparability between online and mobile surveys. If the data is the same, it opens the door for multi-modal studies.

In Canada and the United Kingdom, response rates to the mobile survey among iPhones users are slightly higher than for

users of Blackberry devices and other brands. In the United States, response rates for Blackberry and iPhone are similar (see

Table 4).


8


9/14

Table 4

iphone users tend to be: technologically savvy, early adopters, high users of mobile applications, physically and emotionally

attached to their phone and willing to use it in every way possible. Their enthusiasm for their phone would partially explain the

higher response rates. Data (discussed later in this report) indicates that iPhone users are more likely to have enjoyed the

experience of doing the mobile survey and to want to do more surveys on their phone. An additional explanation for iPhone

users enjoying the experience more could be that the iPhone does a better job of replicating the desktop Internet experience.

The iPhone offers a larger mobile screen than most BlackBerry devices (with the exception of the BlackBerry Storm), and

provides the ability to rotate the device into a portrait or landscape view, and provides the ability to easily zoom in and zoom

out. While this is no way is an exact replica of the desktop or laptop Internet experience, it is closer to the experience than

most BlackBerry devices allow.

SURVEY FINDINGS

Compare Online Data with Mobile Data

As when online became a mode of data collection, many questions have arisen about the comparability of data collected via

desktop to that collected via telephone or mail. The same question will arise with mobile surveys. Keeping sample

characteristics and survey questions consistent, is the data similar for both online and mobile?

The survey topic covered the innocuous but topical subject of the economy. The data revealed remarkable similarities in the

results. Both the single choice questions on the Canadian economy and personal finances were almost identical. The multi-choice responses were also very similar with no statistically significant differences. In total, six single choice questions and one

multi-choice question on the economy were included in the survey. There were no statistically significant differences in results

across any of these questions (see Figure 8).

Figure 8: Consistent Result for Online and Mobile

Other questions related to:


9


10/14

l Change in the price of consumer goods over the next six months (online vs. Mobile) Up (40% vs. 38%), Stay the same

(49% vs. 52%), Down (8% for both);

l Change in the economy over the next three months (online vs. Mobile) Improve (39% vs. 41%), Stay the same (51% vs.

48%), Decline (7% vs. 9%);

l Canadian economy compared to other economies (online vs. Mobile) Better (76% vs. 78%), Same (21% vs. 18%),

Worse (1% for both);

l When the Canadian economy will be officially out of recession (online vs. Mobile) 2009 (16% for both), 2010 (53% vs.

54%), 2011 or later (23% vs. 21%).

One open ended question was included related to the economy. What is the best piece of advice you've heard about how to

personally adjust to living in a recession?

Several analyses were conducted to assess the use of open ended questions in mobile surveys beyond that of obtaining some

sound advice on cutting spending, reducing debt and living with your means. The amount of content provided for the online

and mobile verbatim question was remarkably similar: the average number of characters was 44 for online vs. 42 for mobile.

It is interesting to note some differences by country for the mobile verbatim responses: those in the United Kingdom are similar

to Canada at 39 characters while US respondents are less wordy in their responses at 32 characters.

Furthermore, the format of the content (phrases, sentence structure) is similar; mobile respondents do not abbreviate their

response beyond that which online respondents do. There was also no difference in the proportion of respondents who

provided a response of none, na or some other non-response.

Compare Results by Type of Device

In light of the many different devices and variability in usability across devices, one research objective was to determine if any

differences emerged related to device. There were no statistically significant differences in the survey results for questions

relating to the economy.

While the sample size for other types of devices is small, it is evident, even directionally, that the results are similar the one

exception being that iPhone users are less likely than Blackberry users to have refinanced their mortgage (4% vs. 11%)!

The Mobile Survey Experience

A series of questions were asked near the end of the survey about their experience completing the survey. This was asked of

both mobile and online respondents. The vast proportion of respondents across all countries and online and mobile agreed

with each of the following statements:

l This survey was easy to complete.

l I enjoyed doing this survey.

l I found it convenient to do this survey.


10


11/14

However, the strength of this agreement (strongly agree) does tend to be lower for the mobile sample within Canada. As a

note, the US and UK panels are younger panels which are most likely still experiencing a honeymoon period (see Figure 9).

Figure 9: Attitudes Towards the Survey-%Agree

The more important factor is whether the option of mobile surveys encourages more participation in the survey process. For all

regions investigated, the answer is yes with 36% to 66% stating they would be more likely to participate in future surveys if

they had the option of completing via their phone. These results would suggest that the ability to complete a survey on a

smartphone is an attractive option for panelists (see Figure 10).

Figure 10: If I Had the Option of Completing Surveys Via My Phone, I Would Be

In Canada, iPhone users' enthusiasm for their phone comes through once again as they are the most likely (45%) to take part

in more surveys if they had the option of completing via their phone (see Figure 11).


11


12/14

Figure 11: If I Had the Option of Completing Surveys Via My Phone, I Would Be

There were few differences in the demographic profile of smartphone users by device. The small sample size among those

with other brands of smartphones made it difficult to detect significant differences, however, there is direction evidence that

they are younger and more male. iphone users are significantly younger than Blackberry users (see Figure 12).

Figure 12: Fairly Similar Demo Profiles by Device Canada

SURVEY LENGTH

The mobile survey requires more time to complete. An analysis of the average time to complete indicated the online version

took an average of two minutes compared to four to six minutes for the mobile version.

Keeping the questionnaire short is key; all mobile respondents who completed a follow-up online survey indicated the surveywas just the right length.

CONCLUSIONS

As Internet users migrate towards mobile usage of the Internet and decreased usage of the traditional desktop Internet, it will

be important for research companies to ensure they have the means and wherewithal to reach respondents on their mobile

device or risk losing out on data collection from an important segment of the market. Equally important will be ensuring that the

user experience for completing mobile questionnaires is positive to ensure that respondents will continue to complete

questionnaires on their mobile device.


12


13/14

Encouragingly, the research we conducted suggests that when mobile data collection best practices are followed, and when

proper sampling techniques are utilized, data collected by mobile devices is consistent with data collected using a traditional

online approach.

At a minimum, the mobile questionnaire design best practices should include:

1. Design the questionnaire based on the question types supported (single choice, multi-choice, open ends).

2. Design the questionnaire differently (shorter, more concise). Have less text in the questions and answer categories.

3. Limit survey length to 15 questions (although in theory higher incentives can be used for longer surveys).

Equally important are the back end best practices, which should include:

1. Minimize the skin to maximize the screen size/space

2. Profile your panel for Smartphones

Brand and model of device

Business or personal phone

Email address for Smartphone

Interest/willingness to do surveys via phone

3. Identify devices supported in your data collection software

Major brands are Blackberry, iPhone, Nokia, Motorola, Samsung

But there are many more and many models

4. Manage deployment process

Process for occasions when email address for mobile surveys is different from main panel email address

5. Determine incentive amount and process

6. Error messages in online studies often rely on JavaScript which can't be done for mobile

Copyright ESOMAR 2009ESOMAR

Eurocenter 2, 11th floor, Barbara Strozzilaan 384, 1083 HN Amsterdam, The Netherlands

Tel: +31 20 664 2141, Fax: +31 20 664 2922

www.warc.com


13
http://www.warc.com/http://www.warc.com/


14/14

All rights reserved including database rights. This electronic file is for the personal use of authorised users based at the subscribing company's office location. It may not be reproduced, posted on intranets, extranets

or the internet, e-mailed, archived or shared electronically either within the purchasers organisation or externally without express written permission from Warc.


14

Best Practices in Mobile Research

Documents