Nicole Ronald Experimentation CRICOS Provider: 00111D | TOID: 3059
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 1/22
Nicole Ronald
Experimentation
CRICOS Provider: 00111D | TOID: 3059
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 2/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Understanding of experimental research design inIT/IS/CS settings
- A brief refresher of research questions
- Different types of human experiments
- An overview of CS experiments
2
Today’s aim
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 3/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Surveys are useful when you need to understand what iscurrently happening (or could happen)
- Experiments change something and then measure
(Gray distinguishes between descriptive and analyticalsurveys)
3
How do experiments differ to surveys?
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 4/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
4
Process
Identify issue
Reviewliterature/theories
Develop
hypotheses
Identifyin/dependent
values
Conduct study
Analyse Accept or rejecthypotheses
Report
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 5/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
• “What is happening”
Descriptive
• “What is happening compared to what should happen”
Normative
• “What is the relationship/strength between X and Y”
Correlative
• “What impact does a change in X have on Y”
Impact
5
Question types
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 6/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Testing algorithms, e.g., a new searching algorithm
- Testing ideas in a simulated world, e.g., giving someentities different information
- Testing parameters, e.g., for an algorithm, in a simulation
- A lab situation is usually an experiment
6
When would experimentation be used?
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 7/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Dependent variable- Outcome
- Independent variable- Treatment
7
Variables
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 8/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Experimental- Randomise allocation to groups- Variables can be manipulated/control group
- Quasi-experimental- Group membership cannot be randomised, or is pre-
existing- Variables can be manipulated/control group
- Non-experimental- Group membership cannot be randomised, or is pre-
existing
- Variables cannot be manipulated 8
Experimental design
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 9/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Two groups, randomised:- One receives treatment, the other not (control)
- Both are evaluated before and after treatment
- Example: time-to-completion of IT troubleshooting tickets- Need to check that times in both groups are similar; also
control for experience, age etc.
- Experimental group receives training, control does not9
Experimental group with control
Pre-test (t1) Treatment (t2) Post-test (t3)Experimental Yes Yes Yes
Control Yes No Yes
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 10/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Two groups, randomised:- One receives treatment, the other not (control)- Only one group measured beforehand
- Avoid influencing outcome by pretesting
10
Four-group design
Pre-test (t1) Treatment (t2) Post -test (t3)
Experimental Yes Yes Yes
Experimental No Yes Yes
Control Yes No YesControl No No Yes
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 11/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Two groups:- Could be pre-determined, by class/department- One receives treatment, the other not (control)
- Both groups measured beforehand, hopefully equal
11
Quasi-experimental with control
Pre-test (t1) Treatment (t2) Post-test (t3)
Experimental Yes Yes Yes
Control Yes No Yes
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 12/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- One group, observe afterwards
- Either unable to observe beforehand (e.g., unexpecteddisaster) or just didn’t
- Example: student evaluations
- Can also use a control group
12
Non-experimental
Treatment (t1) Post-test (t2)Experimental Yes Yes
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 13/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Studies over time
- Cross-sectional: same measurement over time
- Longitudinal: same subjects and measurements over time
13
Longitudinal and cross-sectional studies
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 14/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Internal- Assume only independent variable influences
dependent variable
- Threats: selection, external maturation/events, dropout,pretesting, sharing info
- External- Effectiveness of generalising
- Threats: people, places, time
14
Validity
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 15/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Similar principles:- Control and experimental algorithms- Different treatments or scenarios
- Fewer ethical issues (if not accessing personal data)
15
Algorithmic experiments
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 16/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Representation of the world
- Can simulate many objects
- Useful when real-world experiments impractical
16
Simulation
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 17/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Base case (if using simulation then verify against realworld)
- Altered case
17
Experimental design in IT/IS/CS
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 18/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Repeatable- Get same results with same code/data
- Reproducible- Get same results with method
- Stochastic vs. deterministic
18
Replication
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 19/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
1. Experiments require careful design
2. Form research questions carefully
3. Think about validity4. Think about how your simulations/experiments relate to
real world
19
Takeaways
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 20/22
Experimentation
Nicole [email protected]
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 21/22
Swinburne
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN
Experimentation
- Discuss experimental paper
- Design an experiment with human subjects
- Design an experiment using computers/equipment
21
Today’s tasks
8/18/2019 Experiments(1)
http://slidepdf.com/reader/full/experiments1 22/22
Experimentation
Nicole [email protected]