See yourself graduate in 2014!
See yourself graduate in 2014!
Methodology and IRB/URRDr. James Lani
Take Away Message
Research design is a blueprint with several
components:
• Research method: Quant/Qual/Mixed
• Operational constructs: How constructs are measured
• Sampling strategy and procedure
What is Research
Research follows the scientific method. What’s the scientific
method?• Formulation of testable questions or hypotheses
• It’s organized knowledge: logical (theory) and evidence based (observable)
• Precise constructs• Can be disproven (falsifiable)• Parsimonious (simplest
explanation)
Methodology: The Cookbook Metaphor
Cooking Researching
Making a stew… Examining research questions…
Ingredients: 2 lbs beef, I clove garlic…
Preparation: Cube beef, mince garlic…
Cooking Instructions: Bake at 350 for one hour
Celebrate: You’ve made a replicable) stew!
Ingredients: 20 participants, 15 item questionnaire, semi-structured interview.
Preparation: Administer questionnaire before and after lecture; semi-structure interview with participants for 20 minutes using a tape recorder.
Data analysis plan: Conduct dependent sample t-test; transcribe interviews then thematize participants responses.
Celebrate: You’ve conducted a (replicable) research study!Where the recipe can be replicated
Methodology Essential Ingredients
Restate research
question and hypotheses
Overview
Research Design
Participants
Materials/ Instruments
Data Collection
Procedures
Data Analysis Plan
Sample Size
Limitations
Methods Quantitative Qualitative MixedYou can count it Not quantitative Both
Goal
Tend to answer “What questions” (What is relationship or differences) or “When questions” (when is theory supported…after intervention
Tend to answer “Why questions” (understand why people feel that way) or “How questions” (explore how they see things) Both
Research methods/
Strategies of data collection
Experimental (random assignment) and Non-experimental methods (no control group), Observations (time participant)…
Semi-structured interviews, Archive data, Observations (write down positive and negative feeling words)… Both
Operationalize variables
Define constructs/ instruments
Define constructs/ instruments Both
Sampling strategy
Discuss sampling process Discuss sampling process
Both
Theoretical (or Conceptual) Framework
Theory is a systematic explanation of behavior of phenomena.
• Theory guides analyses• List existing theories and how
your research questions relate to those theories
• E.g., Theory of mind: the ability to understand that others have their own beliefs, desires, intentions. Empathy. Tested by Faux Pas Task (ability to recognize a faux pas). Research question: Does alcohol abuse impact empathy?
Population The population is the group you
want to generalize to.
• Describe characteristics of population
• Why is population relevant to problem (look at other peer reviewed study’s justification)
• Distinguish the population from the sample
Sampling Frame and Sample
Universe: theoretical
population to generalize to
Population: largest target
population from universe you
have access to or “Sampling Frame”
Original Sample
Final Sample
Attrition
Survey Item
Old (one’s own SE)
Change in item
New (perceptions of others’ SE)
1I feel that I have a
number of good qualities.“I have” to “she has” I feel that she has a
number of good qualities.
2I feel I do not have much
to be proud of.“I do not” to “she
does not”I feel she does not have much to be
proud of.
3On the whole, I am
satisfied with myself.“I am” to “she is” and “myself” to “herself”
On the whole, she is satisfied with herself.
…
If you need to amend instrument, use a change matrix; do not create
your own instrument!
Materials: Informed Consent
• State purpose of project
• State procedure and how long it will take
• State voluntary nature of participation
• State risks (if any)
• Have them sign or state that by filing our survey they are agreeing to participate.
Constructs vs. Variables
Th
eore
tical
Em
pir
ical
Construct A: Social Environment
Construct B: Personality
Variable A: Birth-Order
Variable B: Introversion/ extroversion
Constructs vs. Variables Constructs are the invisible abstract things we’re measuring (e.g., personality), while variables are the way we’re assessing (measure/operationalize) those invisible things.
Constructs and variables need to be precise (is personality measured by introversion scale or by conscientiousness scale?)
e.g., Intelligence is a construct, while the number of words remembered is a way of assessing intelligence.
e.g., Personality is a construct, while the scores on an introversion/extroversion test is a way to assess an aspect of personality.
VariablesOperationalize Constructs, Make
Distinction between IV’s/DV’s, and Describe Level of MeasurementExample:
Does Empathy differ by group (alcohol abuse vs. no alcohol abuse)?
Empathy is my dependent variable and Group is my independent variable.
Empathy (my construct) is measured by scores on the Faux Pas task. Alcohol abuse (my construct) is measured by 5 or more drinks in one day.
Empathy is a ratio-level variable measured with scores ranging from 0-25, while my Group variable is a nominal-level (categorical-level) variable because participants re in one of two groups (alcohol abuse group or not in alcohol abuse group).
Reliability and Validity
Valid but not Reliable
Reliable but not Valid
Reliable and Valid
Validity• Internal Validity: IV causes a change in the DV (not
time or other covariates, etc.
• External validity: can be generalized to the population
• Construct validity: does the scale measure the theoretical construct
• Translational validity:• Face validity (items are reasonable)• Content validity (items match the domains of
interest)
• Criterion related validity: measures behave as theory predicts• Convergent validity: how close the variable
aligns with the construct (use EFA)• Concurrent validity: construct relates to
established instruments• Predictive validity: measure can predict an
outcome (GPA→Income)
Reliability Internal consistency: Cronbach alpha. Average inter-item correlation
Inter-rater: if interval, correlate; if dichotomous, kappa
Test-retest: administer same test at two times
Split half: divide instrument into 2 parts and calculated totals, then correlate totals.
Brief Review: Units of Analyses Quantitative (e.g., Age)
Nominal-level (Latin for name). Gender (M/F), Grouped (Old = 65+, middle age = 36-64, young = 35 or younger). Assign any number of groups (old = 1, middle = 2, young = 3).
Ordinal-level is ranked (Latin for showing order). GPA (A-F), or age (group 1 = age 1–15, group 2 = age 16-25, group 3 = age 25-54, group 4 = age 65+)
Interval/Ratio-level (also named scale or continuous; Latin for [equal] space [between numbers]). What is your age today in years? ____ (a number from 1-105)
Types of Methodology ModelsTheories explain phenomena, Models represent Phenomena.
• SEM and Path models• Regression models (linear, logistic,
ordinal)• ANOVA models (repeated-measures)• Time-series• Heirarchical Linear Models (HLM)• Correlational Models
Relationship Among Variables
Employee
Tenure
Leadership
Style
Employee
Satisfaction
Customer
Satisfaction
Mediator
Moderator
Data Collection Method -- Overview
• Describe the procedures used to administer the materials to the participants
• Remember to be as detailed as necessary so someone can literally replicate your study
Data Collection
Qualitative • Procedure for accessing participants• Selection of data collected• Number and duration of interviews• How and when data is collected• How data is recorded (e.g., hand
notes, audiotaped)• Role of researcher-relationship to
researcherQuantitative• Procedure to administer measures
Pilot Test (only if you develop instrument)• Detect potential issues in the
instrument• Allows you to get feedback and to
finalize your survey/interview items • Makes sure participants understand
survey items• Assess typical responses to survey
items or interview questions (were participants comfortable, long-winded, defensive, etc.• Can you access data?
Data Analysis Plan
• Quantitative: Describe the analysis plan used to test each hypothesis, the assumptions of the statistical analyses, and a justification of the appropriateness of the analysis for each research question.• Qualitative: Describe how the data
will be analyzed (or thematized)• Phenomenological• Case Study• Grounded theory
Sample SizeQuantitative
Varies by type of statistical analysis
1. Research questions in statistical language2. Level of measurement of variables3. Determine statistical analysis4. Conduct power analysis
Qualitative
Varies by which qualitative approach is taken
• Phenomenological
• Case Study • Grounded
theory
Assumptions, Limitation, Delimitations
Assumptions: Discuss things out of your control about the population and design, then justify assumptions (e.g., participants will answer honestly.)
Limitations: Are out of your control and describe weaknesses in design, threats to validity (e.g., generalizability).
Delimitations: Are in your control and relate to choices you will make to narrow the scope of the study (e.g., variables, research questions).
Ethical Considerations
• Describe informed consent procedures
• State whether your study will be anonymous or confidential with respect to the participants
• Describe considerations for children or vulnerable participants
Questions &
Answers
See yourself graduate in 2014!
1-1 Personalized Dissertation Consulting
877-437-8600
Thank you for your
participation and attention!
Join us for our next webinar on Wednesday, April 23rd at
8:30pm ET.Results and Discussion Chapters, Editing, and
Defense