Edmedia2011 online.surveys

Online Surveys: Best Practices and Ethics

Dr. Valerie IrvineCo-Director, TIE Research LabUniversity of Victoria, Canadahttp://[email protected]: _valeriei

Acknowledgement to Dr. John Williams, University of Northern Iowa

Some Advantages…

• Large Sample Sizes• High Statistical Power• Diverse and Distant Geographical Areas• External Validity and Generalizability• Relatively Easy (relatively!!!)

Use of Internet for Research• From 1998 to 2003, 425% increase in listed studies on

one website (Birnbaum, 2004)

• Google “Online Experiment” • 42,100 pages in 2007• 107,000,000 in 2011• Explore some, but note not to be a participant if you

are clicking through it

Online vs. Traditional

• Are online studies fundamentally different than traditional studies?• Direct contact between researcher and participant?• Use of computers?• Age verification?• Electronic data transmission potentially less

confidential?

Ethical Considerations• Ethical considerations are generally the same for online

studies as for traditional studies.

• Various ethical guidelines still apply• For example, US Code of Federal Regulations:

Protection of Human Subjects or Canada SSHRC Tri-Council Guidelines for Research Ethics

• Guild Guidelines• American Psychological Association’s Ethical

Principles of Psychologist’s and Code of Conduct• Institutional/Ethical Review Boards (IRB or ERB)

Lack of Direct Contact Beneficial?

• May increase response rate because of increased perception of anonymity

• May increase ability to discontinue a study because of less social pressure

Lack of Direct Contact Detrimental?• Harder to judge understanding of consent

form or debriefing• Harder to provide immediate clarification

of participant questions• Harder to verify age

• These problems however are not unique to online studies and also occur in mailed surveys, telephone interviews, and group testing

Seven Ethical Issues

• Informed Consent• Deception• Debriefing• Right to Withdraw• Security of test materials• Confidentiality and Anonymity• Avoiding Harm

Informed Consent

• There is a difference between asking for consent and documenting consent• Asking for consent is easy, an informed

consent statement on a webpage will do.• Documenting consent is more involved.

Options to Documentation of Consent• ERBs typically require some indication of consent.

• A webpage with “I consent” and a “I do not consent” options (buttons) may be sufficient.

• Emailed consent may be adequate

Deception

• The issue with deception in online studies is related to debriefing• Participants may not read or understand debriefing

information• Researchers may have difficulty addressing emotional

turmoil related to deception.

• Studies with minimal deception • Weigh risks vs. possible harm• If justified for other methods with limited contact than

probably justified for online studies

Right to Withdraw

• In-person studies may exert social pressure for participants to stay.

• Online studies may exert little pressure to remain and easy ways to exit a study should be provided• “Quit the Study” links on each page

Withdrawing Data

• Withdrawing data is more problematic• Some studies may collect data throughout the

study. Withdrawing data from these studies will require manual deletion

Withdrawing Data

• Studies which collect data upon completion of the study may have links to exit and not submit data if the participant so choses• E.g., surveymonkey • Watch out for inadvertently losing participants

should they miss clicking the submit button***

Withdrawing Data

• The consent form could tell participants that if they withdraw by clicking a “Quit the study” button, they will be asked whether they would like to withdraw their data, but if they withdraw by closing the browser window, the data they have already submitted will be analyzed.

Confidentiality and Anonymity• Keeping Data Secure

• The steps for data security in online studies is the same as with traditional studies• Restrict access to data• Separate identifying information

• Issue with U.S. Patriot Act• See surveymonkey’s response• See Mt. Alison’s University’s response

Confidentiality and Anonymity• Keeping Data Secure

• Restricting Access• Requires physical security of the server• Requires electronic security of the server collecting

data and the computer storing the data

• Identifying Information• Names, email address, addresses, or even usernames or

locations• Identifying information should be stored in separate files,

databases, or computers

Confidentiality and Anonymity• The level of security needed with directly

depend upon the sensitive of the information collected and the amount of identifying information.• Data can be encrypted on the server and during

transmission.• Data values and labels can be assigned in a way that is

meaningless to others without the key• <input type=”radio” name=”Item3” value=”b”>• rather than• <input type= “radio” name= “How often have you stolen a

car?”• value= “More than once”>

Confidentiality and Anonymity• Public use computers may reduce confidentiality

and anonymity.• Pages may be cached on the computer.• Others using the computer may be able to view study

information in URLs.• Corporations and businesses have legal rights to

search and view information on the computers owned by them. Participants using these computers may be revealing information about themselves.

• IP addresses are sometimes considered identifying information.

Confidentiality and Anonymity• Establishing Identity

• Asking for information that only the participant knows (i.e. student number)

• Sending individualized participation invitations• Recipient of invitation may NOT be the individual expected

Avoiding Harm

• Because much online research is widely advertised, consider additional risks to participants from other countries and cultures. • Everyday topics, words, and pictures may be

distressing or illegal in the participant’s country. • Forewarn potential participants of such content

through the use of appropriate study titles and descriptions.

Avoiding Harm• Researchers should provide detailed information

regarding threats to confidentiality, if interception of the material could result in severe social or legal consequences for participants in some countries.

Measurement• T = O + E

(true score equals observed score plus error)

• Double-barrel items: Agree or Disagree to “I like cats and dogs”

• Numerical ranges: Your Age: 19-25, 25-35, 35-50, 50-65

Measurement• Presentation of Likert Scales

• Horizontal and keep ‘don’t know’ out of it for spatial response formation

• Provide definitions to avoid confusion• Put the definition BEFORE the question

Getting Read• Avoid high-risk words that may send it to

SPAM• Send HTML and plain text email invitations• Insert a large START HERE link• Have a familiar person or person of status

send the email• Do NOT harvest email addresses from the web• Be sure to send reminder prompt

Common Mistakes in Conducting Surveys (online)

• Increasing risk of losing participants with submit button • (e.g., saying “thank you for your participation”

ABOVE the submit button)• In some delivery modes, naming two fields the

same• Ability to check more than one option (radio vs.

checkboxes)

Common Mistakes in Conducting Surveys (online)• Survey webpages being long and overwhelming

• Participants will skip questions or enter false data• Use “skip logic” to shorten # of questions

• Avoid stupid questions in online versions• “I completed this survey:

• By paper and pencil• Online”

• Use of open-ended when structured answers could prevent long data coding time

Pilot it first

• Complete a few online surveys first to see pitfalls and good practices

• Get friends and colleagues to pilot the survey BEFORE you deploy it

Examples of Online Research• http://psych.hanover.edu/research/exponnet.html

• See attitudes toward online counselling (surveymonkey)

• http://psychexps.olemiss.edu (needs plug-ins….)

References

Birnbaum, M. H., & Reips, U. -D. (2005). Behavioral research and data collection via the Internet. In R. W. Proctor and K.-P. L. Vu (Eds.), The handbook of human factors in Web design (pp. 471-492). Mahwah, New Jersey: Erlbaum.

Buchanan, T., Johnson, J. A., & Goldberg, L. E. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115-127.

Ess, C. (2007). Internet Research Ethics. In A. Joinson, K. McKenna, T. Postmes, and U.-D. Reips (Eds.), The Oxford Handbook of Internet Psychology, pp 487-502. New York: Oxford University Press.

Krantz, J. H. & Dalal, R. (2000). Validity of web-based psychological research. In M.H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. 35 – 60). San Diego, CA: Academic Press.