Turk data analysis
Hadas + mitcho
{hkotek, mitcho}@mit.edu
Hackl Lab Turkshop
April 2013
Today..
Goal: prepare data for analysis, get (basic) results.
The Manage Tab
Approve subjects
Get data
Decode your results
R!
R basics
Prepare data for analysis
Calculate accuracy
Discard subjects
Look at results
2
NOT Today..
Statistics
Graphics
…. those need to fit your design and data
There is no ‘one size fits all’
Requires more knowledge (take a stats course!)
3
Materials
These slides and materials on ESSL website:
http://web.mit.edu/hackl/www/lab/turkshop/
Download and unzip “week4-results” in examples.
And if you haven’t done so, install R 3.0 and RStudio 0.97:
http://cran.r-project.org/
http://www.rstudio.com/ide/download/desktop
4
The Manage tab
5
Monitor the progress of your experiment
Approve/reject subjects
Get your data!
The Manage tab
6
Approving/rejecting workers
7
Reject subjects who did not comply with instructions.
Completed more than one survey
(often a requirement for language studies!).
Did not complete survey.
Possibly also:
Failed on ‘catch’ items.
Exhibit guessing behavior.
But NOT: Non-native speakers!
May bias participants into saying they are native speakers.
Approving/rejecting workers
8
Open your .csv results file in Excel.
Custom sort
by WorkerId
Find
duplicates
Conditional formatting:
highlight unanswered
questions;
decide if anyone
missed too many.
What we wanted to test: blocking example
9
Effect of crossing animacy of causee and verb type on
acceptability of causative sentences.
# blocking 1 inanimate-v
That’s the ball that the coach bounced on the floor.
# blocking 1 animate-v
That’s the gymnast that the coach bounced on the floor.
# blocking 1 inanimate-make-v
That’s the ball that the coach made bounce on the floor.
# blocking 1 animate-make-v
That’s the gymnast that the coach made bounce on the floor.
What it looks like from the workers’ end
10
Our Turk survey
11
Forced choice: natural = 1, unnatural = 0
10 items
5 targets
Four conditions each
5 fillers
Two clearly unnatural, three clearly natural
8 lists
56 total participants
Understanding and decoding our results
12
Open the .csv results file in Excel.
What information does it give us?
What does it not have?
Open the .decode.csv file in Excel.
What information does it give us?
Combine the two files using decoder.py!
Now open the decoded file in Excel.
What information does it give us?
Fields in the decoded results
13
WorkerId: Amazon ID for the worker
AssignmentId: unique to that submission
AssignmentStatus: Submitted, Approved, Rejected
WorkTimeInSeconds
ListNumber, PresentationOrder
Section, Item, Condition
These came from the decode file
field_N: your actual stimuli
In our template skeletons:
Choice: forced choice value
Extras: english, foreignlang, numanswered, useragent
R
14
Open source statistics software
Free! So lots of people use it.
Designed specifically for statistical data analysis
Generally treats everything as “data made up of multiple observations”
as we will see
Really, a programming language of its own
It’s a little weird and annoying
RStudio: a better interface for R
Open RStudio. Let’s play with some R! [Tutorial]
Writing an analysis script
15
In an analysis script, we (might) want to:
1. Read in the results
2. Filter out parts of them
3. Check filler accuracy and disqualify participants
4. Recode some of the data
5. Look at the results
6. (Compute statistics)
7. (Make pretty charts)
Writing a script forces us to precisify what we are doing.
It also makes the procedure reproducible.
Eliminates Reduces human error
Writing our analysis script
16
In RStudio:
Start a new script and save it in the same folder as the results.
Set working directory to Source File Location.
[Tutorial]
Pre-baked version in blocking-analysis.R