I/O chapter 2 by Jason Manaois

Research in Psychology

A Scientific Endeavor

Objectives Describe the goals of psychological research

Learn the process of doing research

Understand the different research methods

Explore the advantages and disadvantages of doing research and its different methodologies

Apply ethical consideration in doing research

Definition RESEARCH A formal process by which knowledge is

produced and understood

GENERALIZABILITY The extent to which conclusions drawn from

one research study spread or apply to a larger population.

Goals of Psychological Research

Description of social behavior Are people who grow up in warm climates different

from those in cold climates?

Establish a relationship between cause & effect Does heat cause higher amounts of aggression?

Develop theories about why people behave the way that they do We dislike Duke students to feel better about

ourselves

Application Creating effective therapeutic treatments, more

successful negotiation tactics, and greater understanding amongst groups of people

Empirical Research Cycle

Empirical ResearchEmpirical Knowledge based on direct

observation

Theory Set of ideas which try to explain what

we observe Theoretical diversity A statement that proposes to explain

relationship among phenomena of interest.

Statement of a ProblemInductive Method A research process in which conclusions are

drawn about a general class of objects or people based on knowledge of a specific member of the class under investigation.

DATA THEORY

Deductive Method A research process in which conclusions are

drawn about a specific member of a class of objects or people based on knowledge of the general class under investigation.

THEORY Collects DATA

Design of the Research StudyResearch Design A plan for conducting scientific research for the purpose of

learning about the phenomenon of interest

Internal Validity The degree to w/c the relationship evidenced among the

variables in a particular research study are accurate or true.

External Validity The degree to w/c the relationship evidenced among the

variables in a particular research study are generalizable or accurate in other contexts.

Concerns

Naturalness of the Research Setting Laboratory vs. Field

Degree of Control Are you able to control or manage the

conduct of the research?

Primary Research

Laboratory ExperimentQuasi-experimentQuestionnaireObservation

META-analysis

Secondary Research

Qualitative Research

Ethnography A research method that utilizes field

observation to study a society’s culture EMIC – insider’s view ETIC – external view

Measurement of Variables

Variables

Quantitative Variables E.g. Age, weight

Categorical Variables E.g. Gender, race

Variables

Independent Variables A variable that can be manipulated to

influence the values of the dependent variable. (the one that you manipulate)

Dependent Variables A variable whose values are influenced by

the independent variables. (the one that you measure)

Examples

Leadership Style

Employee Performance

Employee Performance

Employee Trainability

Variable in I/O

Predictor Variable A variable used to predict or forecast

a criterion variable

Criterion Variable A variable that is a primary object of

a research study; it is forecasted by a predictor variable

Examples

Personality LeadershipStyle

Employee Performance

Employee Morale

Analysis of Data

Descriptive Statistics Mean Median Mode

Variability Range SD Correlation

Conclusions from Research

After collection and analyzing data, the researcher draws conclusions. Answers research hypothesis or research problem.Generalizability of research findings???

Ethical Issues

Right to informed consentRight to privacyRight to confidentialityRight to protection from deceptionRight to debriefing

Another Perspective

The Process of Doing Research

First, select a topic Good theory:

Has predictive power Is simple & straightforward

Then, search the literature Find out what others have

done that may be applicable to your area of interest

The Process of Doing Research

Next, formulate hypotheses Hypothesis: specific statement

of expectation derived from theory State the relationship between

two variables

Variable: can be any event, characteristic, condition, or behavior

The Process of Doing Research

Then pick your research method Experimental vs. correlational (DesignDesign) Field vs. laboratory (SettingSetting)

Finally, collect & analyze yourdata

Let’s take a closer look . . .at variables

Dependent variable (outcome variable) Dependent on the influence of other factor(s) How do we operationalize?

Independent variable (predictor variable) Factor(s) that change the outcome variable How do we operationalize & manipulate? Control group

Let’s take a closer look . . . at research methodsExperimental vs. correlational designs Correlational: observe the relationship

between two variables Describe patterns of behavior

Types include Naturalistic observation Case studies Surveys

Correlational researchAdvantages Sometimes manipulation of variables

is impossible or unethical Efficient – look at lots of data

Disadvantages CANNOT DETERMINE CAUSATION Could be a lurking variable

Experimental ResearchResearcher manipulates one variable (IV) to see effect on other variable (DV) Try to hold everything else constant

True experiments have Random sampling: selecting Ps

randomly from population Random assignment: chance

assignment to condition

Ethics in ResearchShould the study be done? Value vs. potential cost APA guidelines, colleagues

How do we protect Ps? Informed consent Confidentiality & anonymity Debriefing

Organizational Research Methods:

CAUSALITY

What Do We Mean By Causality?

Relationship between two events where one is a consequence of the otherDeterminism: A (cause) leads to B (effect)“In the strict formulation of the law of causality—if we know the present, we can calculate the future—it is not the conclusion that is wrong but the premise”.On an implication of the uncertainty principle. Werner Heisenberg

Heisenberg & Uncertainty Principle

Certain properties of subatomic particles are linked so the more accurately you know one, the less accurately you know the other We can compute probabilities not certainties Argues against determinism

“Physics should only describe the correlation of observations; there is no real world with causality”Heisenberg, 1927, Zeitschrift für Physik

Psychology, like quantum physics, is probabalistic

Cause Versus Effect

Effect of a Cause (Description) What follows a cause?

Cause of an Effect (Explanation) Why did the effect happen?

Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18, 449-484.

Three Elements of Causal Case

Cause and effect are relatedCause preceded effectNo plausible alternative explanations

John Stuart Mill

Experiment

Vary something to discover effects Shows association Shows time sequence Can rule out only some alternatives

Confounds Boundary conditions (generalizability)

Good for causal description not explanationNatural science control through precise measurement Sterile test tubes, electronic instruments

Studying and Performance

Students randomly assigned to study amountTest scores as DVDid studying lead to test results? Encouragement led to test results Impact on studying unclear Effect of studying unclear

What was cause of test results?

Holland, P. W. (1988). Causal inference, path analysis, and recursive structural equations models. Sociological Methodology, 18, 449-484.

Nonexperimental Research Strategy

1. Determine covariation2. Test for time sequence

• Longitudinal design• Quasi-experiment

3. Rule out plausible alternatives• Based on data/theory• Logical

Can Job Satisfaction Cause Gender?

Correlation of Gender and satisfaction = group mean differencesSatisfaction can’t cause someone’s genderSatisfaction can be the cause of gender distribution of a sampleSuppose Females have higher satisfaction than MalesMultiple reasons

Alternative Gender-Job Satisfaction Model

Females more likely to quit dissatisfying jobsDissatisfaction causes gender distributionGender moderates relation of satisfaction with quitting

Sati

sfact

ion

MalesFemales

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

….

.

.

.

More Alternatives1. Women less likely to take

dissatisfying job (better job decisions)2. Women less likely to be hired into

dissatisfying jobs (protected)3. Women less likely to be

bullied/mistreated4. Women given more realistic previews

(lower expectations)5. Women more socially skilled at

getting what they want at work

How To Use Controls

Controls great devices to test hypotheses/theoryRule in/out plausible alternativesBest based on theorySequence of tests

Control Strategy1. Test that A and B are related

• Salary relates to job satisfaction

2. Confirm/disconfirm control variable• Gender relates to both

3. Generate/test alternative explanations for control variable

• Differential expectations• Differential hiring rate• Differential job experience• Differential turnover rate

Validity and Threats To Validity

Validity Interpretation of constructs/results Inference based on purpose

Hypothesized causal connections among constructs

Nature of constructs Population of interest

People Settings

Four Types of Design Validity

Statistical conclusion Appropriate statistical method to make

desired inference

Internal validity Causal conclusions reasonable based on

design

Construct validity Interpretation of measures

External validity Generalizeability to population of interest

Threats to ValidityStatistical Conclusion Statistics used incorrectly Low power Poor measurement

Internal Validity Confounds of IV with other events,

variables Group differences (pre-existing or

attrition) Lack of temporal order Instrument changes

Threats To Validity 2Construct Validity Inadequate specification of theoretical

construct Unreliable measurement Biases Poor content validity

External Validity Inadequate specification of population Poor sampling of population

Subjects Settings

Qualitative Methods

What are qualitative methods Collection/analysis of written/spoken text Direct observation of behavior

Participant observationCase studyInterviewWritten materials Existing documents Open-ended questions

Qualitative Research 2

Accept subjectivity of science Is this an excuse?

Less driven by hypothesisAssumption that reality a social construction If no one knows I’ve been shot, am I really dead?

Interested in subject’s viewpointMore open-endedMore interested in contextLess interested in general principlesFocus more on interpretation than quantification

Analysis

Content Analysis Interviews Written materials Open-ended questions Audio or video recordings Quantifying

Counts of behaviors/events Categorization of incidents Multiple raters with high agreement

Nonquantitative Analysis of case Narrative description

The Value of the Qualitative Approach

What is the value/use of this approach?Is this science?Must everything be quantified?

Qualitative Organizational Research: Job Stress

Quantitative survey dominatesRole ambiguity and conflict dominated in 1980s & 1990s (Katz & Kahn)Dominated by Rizzo et al. weak scales

Keenan & Newton’s SIR

Stress Incident Record Describe event in prior 2 weeks Aroused negative emotion

Top stressful events for engineers Time/effort wasted Interpersonal conflict Work underload Work overload Conditions of employment

Subsequent SIR Research

Comparison of occupations Clerical: Work overload, lack of control Faculty: Interpersonal conflict, time

wasters Sales clerks: Interpersonal conflict, time

wasters

Informed subsequent quantitative studies Focus on more common stressors

Interpersonal conflict Organizational constraints

Cross-Cultural SIR Research

Comparison of university support staffIndia vs. U.S.

Stressor India US

Overload 0% 25.6%

Lack of control 0% 22.6%

Lack of structure 26.5% 0%

Constraints (Equipment) 15.4% 0%

Conflict 16.5% 12.3%

Research As Craft

Scholarly research as expertise not bag of tricksLogical caseGo beyond sheer technique Research not just formulaic/trends Not just using right design, measures,

stats

Developing the CraftExperienceTrying different things Constructs Designs/methods Problems Statistics

ReadingReviewingTeachingThinking/discussingCourses necessary but not sufficientLifelong learning—you are never done

Developing the CraftField values novelty and rigorDon’t be afraid of exploratory research Not much contribution if answer known in

advance

Look for surprisesDon’t be afraid to follow intuitionAsk interesting question without a clear answerFocus on interesting variablesGood papers tell stories Variables are characters Relationships among variables

Construct & External ValidityandMethod Variance

Constructs

Theoretical level Conceptual definitions of variables Basic building blocks of theories

Measurement level Operationalizations Based on theory of construct

What We Do With Constructs

DefineOperationalize/MeasureEstablish relations with other constructs Covariation Causation

External Validity: Population

Link between sample and theoretical populationDefine theoretical populationIdentify critical characteristicsCompare sample to population Employed individuals Do students qualify?

External Validity: SettingLink between current setting and other settings Organization Occupation

Identify critical characteristics of settingsCompare setting to others Lab to field

External Validity: Treatment/IV

Link between current treatment/IV and othersCompare treatment/IV Distance learning vs. traditional

External Validity: Outcome/DV

Link between current outcome/DV and othersWill results in study work similarly in nonresearch condition?Will different operationalizations of outcome have same result? Supervisor rating of performance vs.

objective Safety behavior versus accidents/injuries

When Politics Attack Science

EvolutionIQ and performanceDifferential validity of IQ testsOthers?

Quasi-Experimental Design

What is an experiment? Random assignment Creation of Conditions? Naturally occurring experiment

Quasi-experiment

Design without random assignmentComparison of conditionsResearcher created or existingCan characteristics of people be an IV? Gender Personality

Is a survey a quasi-experiment?

SettingsLaboratory vs. fieldLaboratory Setting in which phenomenon doesn’t

naturally occur

Field Setting in which phenomenon naturally

occurs

Classroom field for educational psychologistClassroom lab for us

Lab vs. Field Strengths/Weaknesses

Lab High level of control Easy to do experiments Limits to what can be studied Limited external validity of

population/setting

Field Limited control Difficult to do experiments Wide range of what can be studied High reliance on self-report High external validity

Lab in I/O Research

What’s the role of lab in I/O research?Stone suggests lab is as generalizeable as field. Do you agree?Stone says I/O field biased against lab. Is it?When should we do lab vs. field studies?

Challenges To Field Research

Access to organizations/subjectsLack of control Distal contact with subjects (surveys) Who participates Contaminating conditions

Participants discussing study

Lack of full cooperationOrganizational resistance to change

Survey Methods & Constructs

Survey methodsSamplingCross-cultural challenges Measurement equivalence/invariance

Survey SettingsWithin employer organizationWithin other organization University Professional association Community group Club

General population Phone book Door-to-door

MethodsQuestionnaire Paper-and-pencil E-mail Web

Interview Face-to-face Phone Video-phone E-mail Instant Message

PopulationSingle organizationMultiple organizations Within industry/section

Single occupationMultiple occupationsGeneral population Employed students

Sample Versus PopulationSurvey everyone in population vs. sample Single organization or unit of organization

Often survey goes to everyone Multiple organizations

Kessler: All psychology faculty Other organization

Professional association Often survey everyone

General population

Sampling DefinitionsPopulation – Aggregate of cases meeting specification All humans All working people All accountants Not always directly measurable

Sampling frame – List of all members of a population to be sampled List of all USF support personnel

Sampling Definitions cont.Stratum – Segment of a populationDivided by a characteristic Demographics

Male vs. female Job level

Manager vs. nonmanager Job title Occupation Department/division of organization

Instrument IssuesLinguistic meaning Translation – Back-translation

Calibration Numerical equivalence Cultural response tendencies

Asian modesty Latin expansiveness

Measurement equivalence Construct validity Factor Structure

What Is A Theory?Bernstein Set of propositions that account for

predict and control phenomena

Muchinsky Statement that explains relationships

among phenomena

Webster General or abstract principles of

science Explanation of phenomena

Types of Theories

Inductive Starts with data Theory explains observations

Deductive Starts with theory Data used to support/refute theory

Common Usage of Theory

Conjecture, opinion, speculation or hypothesis Wikipedia

Advantages

Integrates and summarizes large amounts of dataCan help predictGuides researchHelps frame good research questions

Disadvantages

Biases researchers“Theory, like mist of eyeglasses, obscures facts” (Charlie Chan in Muchinsky)“Facts are the enemy of truth” (Levine’s boss)A distraction as research does not require theory (Skinner)

Hypothesis

Statement of expected relationships among variablesTentativeMore limited than a theoryDoesn’t deal with process or explanation

Model

Representation of a phenomenonDescription of a complex entity or process Webster

Boxes and arrows showing causal flow

Theoretical Construct

Abstract representation of a characteristic of people, situation, or thingBuilding blocks of theories

Paradigm

Accepted scientific practiceRules and standards for scientific practiceLaw, theory, application and instrumentation that provide models for research. Thomas Kuhn

What Are Our Paradigms?

Behaviorism?Environment-perception-outcome approachSurveys

Ethics In Research

Ethical Practices

Conducting Research Treatment of human subjects Treatment of organizational subjects

Data Analysis/InterpretationDisseminating Results Publication

Peer reviewing

Ethical Codes

Appropriate moral behavior/practiceAccepted practicesBasic Principle: Do no harmProtect dignity, health, rights, well-beingCodes APA??

American Psychological Association Code

Largely Practice orientedFive principles Beneficence and Nonmaleficence [Do no

harm] Fidelity and Responsibility Integrity Justice Respect for People’s Rights and Dignity

Standards and practicesApplies to APA membershttp://www.apa.org/ethics/

PreamblePsychologists are committed to increasing scientific and

professional knowledge of behavior and people's understanding of themselves and others and to the use of such knowledge to improve the condition of individuals, organizations, and society. Psychologists respect and protect civil and human rights and the central importance of freedom of inquiry and expression in research, teaching, and publication. They strive to help the public in developing informed judgments and choices concerning human behavior. In doing so, they perform many roles, such as researcher, educator, diagnostician, therapist, supervisor, consultant, administrator, social interventionist, and expert witness.

APA Conflict Between Profession and Ethical Principles

Restriction of Advertising Violation of the law

Maximization of income for membersTolerance of torture Convoluted statements

Other associations manage to avoid such conflicts

Academy of Management CodeLargely academically orientedThree Principles

Responsibility Integrity Respect for people’s rights and dignity

Responsibility to Students Advancement of managerial knowledge AOM and larger profession Managers and practice of management All people in the worldhttp://www.aomonline.org/aom.asp?ID=&page_ID=239

Professional PrinciplesOur professional goals are to

enhance the learning of students and colleagues and the effectiveness of organizations through our teaching, research, and practice of management.

Principles Vs. Practice

Principles clear in theoryEthical line not always clearEthical dilemmas Harm can be done no matter what is

done Conflicting interests between parties

Employee versus organization Whose rights take priority?

Example: Exploitive Relationships

Principle Psychologists do not exploit persons over

whom they have supervisory, evaluative, or other authority

What does it mean to exploit?

Conducting Research

PrivacyInformed consentSafetyDebriefingInducements

Privacy

Anonymity: Best protection Procedures to match data without

identities

Confidentiality Security of identified data

Locked computer/cabinet/lab Encoding data Code numbers cross-referenced to names

Removing names and identifying information

Informed ConsentSubject must know what is involved Purpose Disclosure of risk Benefits of research

Researcher/society Subject

Privacy/confidentiality Who has access to data Who has access to identity

Right to withdraw Consequences of withdrawal

Safety

Minimize exposure to risk Workplace safety study: Control

group

Physical and psychological risk

Debriefing

Subject right to knowEducational experience for studentsWritten documentPresentationSurveys: Provide contact for follow-upProvide results in future upon request

Inducements

Pure Volunteer – no inducementCourse requirement Is this coercion?

Extra creditFinancial payment Is payment coercion?

Successful Research Career

Conducting good research Lead don’t follow

Visibility Good journals Conferences Other outlets Quantity

First authored publications Important more early in career

ImpactGrants

Programmatic

Program of research More conclusive Multiple tests Boundary conditions More impact through visibility Helps getting jobs Helps with tenure/promotion Can have more than one focus

Conducting Successful Research

Develop an interesting question Based on theory Based on literature Based on observation Based on organization need

Link question to literature Theoretical perspective Place in context of what’s been done Multiple types of evidence Consider other disciplines

Conducting Successful Research 2

Design one or more research strategies Lab vs. field Data collection technique

Survey, interview, observation, etc. Design

Experimental, quasi-experimental or observational

Cross-sectional or longitudinal Single-source or multisource

Instrumentation Existing or ad hoc

Conducting Successful Research 3

AnalysisHierarchy of methods simple to complex Descriptives Bi-variable relationships Test for controls Complex relationships

Multiple regression Factor analysis HLM SEM

Conducting Successful Research 4

Conclusions What’s reasonable based on data Alternative explanations Speculation Theoretical development Suggestions for future

ImpactEffect of work on field/worldCitations Sources

ISI Thomson Harzing’s Publish or Perish Others

Self-citation Citation studies

Individuals (e.g., Podsakoff et al. Journal of Management 2008)

Programs (e.g., Oliver et al. TIP, 2005)

Being attacked