Mechanical Turk: Current Issues and Solutions JOSEPH K. GOODMAN BASED IN-PART ON: GOODMAN AND PAOLACCI (2017), “CROWDSOURCING CONSUMER RESEARCH,” JOURNAL OF CONSUMER RESEARCH, 44 (JUNE), 196-210. AVAILABLE AT WWW.JOSEPHKGOODMAN.COM Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
31
Embed
Mechanical Turk: Current Issues and Solutions · Mechanical Turk: Current Issues and Solutions JOSEPH K. GOODMAN BASED IN-PART ON: GOODMAN AND PAOLACCI (2017), “CROWDSOURCING CONSUMER
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Mechanical Turk:Current Issues and SolutionsJOSEPH K. GOODMAN
B A S E D I N - PA R T O N : G O O D M A N A N D PA O L A C C I ( 2 0 1 7 ) , “ C R O W D S O U R C I N G C O N S U M E R R E S E A R C H ,” J O U R N A L O F C O N S U M E R R E S E A R C H , 4 4 ( J U N E ) , 1 9 6 - 2 1 0 . AVA I L A B L E AT W W W. J O S E P H KG O O D M A N . C O M
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Special Thanks to my CoauthorsGABRIELE PAOLACCI
ERASMUS UNIVERSITY
SCOTT WRIGHT
PROVIDENCE COLLEGE
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
Agenda
Basics of Mechanical Turk: Issue or Non-issue?
What about editors and reviewers?
Solution to Many Problems: TurkPrime√Screen based on demographics or any other criteria that you choose
√Provide real random completion codes and automatically approve
√Block repeat participants
√Calculate Completion & Bounce rates
√MicroBatch (or HyperBatch)
√Block duplicate IPs
√Anonymize worker IDs
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
MTurk Myth or Truth?1. Not real people2. Low quality data3. Uneducated4. Poor/low income5. Internet geeks6. The lab is still more reliable7. They don’t pay attention8. Mostly Indian9. Mostly non-native English speakers10. Editors/reviewers won’t accept papers with only MTurk data
◦ Surveyed marketing researchers at the top 150 schools (CB and non-CB)◦ Asked about their use and opinions of MTurk
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
?
4.68
3.644.00
4.68
3.15
3.96
4.72
3.764.04
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Papers should not have alltheir data from Mturk
Mturk studies should be re-run using a non-Mturk
sample.
Mturk samples are betterthan student samples.
All P's (including quant and CCT) ERB AEs/Editors
Neitheragree nordisagree
Stronglydisagree
Strongly agree
What Do Researchers Think about MTurk?
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
AE’s & Editors are Not All Convinced
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
0.110.07 0.05
0.200.18
0.15
0.24
0.14
0.05 0.04
0.23
0.140.11
0.30
0.16
0.08 0.080.12
0.08 0.08
0.40
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
1StronglyDisagree
2 3 4Neither agreenor disagree
5 6 7Strongly agree
Dis
trib
uti
on
"Papers should not have all their data from Mturk"
All P's (including quant and CCT) ERB AEs/Editors
Solution 1: Stop Doing MTurk Studies
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
Solution 2: Get a job at Ohio State(OR OTHER SCHOOL WITH 8000 SUBJECT HOURS/YEAR)
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
◦P’s entered their availability/calendar on Qualtrics
◦ IV: Generate times bounded/not bounded by tasks (btw SS)
◦DV: Choice of MTurk study ◦ 30 minute brief study for $2.50
◦ 45 minute full study for $5.00
◦ Neither
Example2: Coin flip game to measure cheating (Balasubramanian, Bennett, and Pierce 2017)
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
Solution 3: Measure Real Behavior on MTurk
Example3: Take picture of artwork and upload or email it
Example4: Costello and Malkoc (working paper)◦DV: Donate $ or donate time (micro-donations: counting/classifying animals, algae, weather patterns)
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
Solution 4: Go Beyond Surveys
Longitudinal studies◦ 60% retention rates (but need to pay more)
Group interactions
Webcams for facial expression analysis and analyzing emotions (FaceReader)
Physiological measures◦ Measure heart rate (Muender et al. 2016)
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018
Joseph K. Goodman, PhD
RA Worko Classification of contento web searcheso registering for websiteso viewing adso edit transcriptso test softwareo skilled tasks: logo & graphic
design, ad copyo See Fiverr, Upwork, Guru
Issue: Pre-screening & Non-naiveté Workers are usually honest, but will lie to be eligible for participation◦ 48/106 heterosexual workers claimed LGB to be eligible for study
◦ 8% women, 17% men changed gender to be eligible for study
Lying increases with compensation
Lie more about owning a dog than religion or politics
Non-naiveté ◦ We are sampling the same people across studies
Mechanical Turk: Current Issues and Solutions, SCP Pre-Conference 2018