Texas A&M University EPSY 642 FALL 2009 Victor L. Willson, Instructor.

Texas A&M University EPSY 642FALL 2009

Victor L. Willson, Instructor

EPSY 642 Meta Analysis of Social Science Research Syllabus FALL 2009 TUESDAYS 4:45 – 7:35 Victor L. Willson, Professor: Office: by appt 704 Harrington 845-0904 /fax: 862-1256 email:v-

[email protected]   Course description: Theory and methodology of current state of the

topic of meta-analysis of research documents. Course objectives: Students will learn meta-analysis methods by

conducting a meta-analysis on a circumscribed topic of their choice. Students will be required to present their findings at the end of the course and produce a research paper on their topic. Students will be encouraged to develop a potentially publishable research article from their work. Components for the project will be reported by students during the semester.

  Texts: Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-

Analysis. Sage. Cooper, H. (2010). Research Synthesis and Meta-

Analysis. 4th Ed. Sage

Date Topic Reading Assignment due

 SEP 1 Introduction: Purposes, Approaches HC 1 Activity: Online Google Scholar search 5 articles: samples description, variable list

8 NO CLASS 15 Problem finding, Study Selection LW2, HC 2-3     22 Study coding, menu elements LW4, 5, HC 4, handout

29 Outcome variable effect computation LW3, HC 6, handouts

 OCT 6 Study Design variables HS 2, HC 5   13 Correlational study analysis LW 6-7   Experimental study analysis LW 7 #1   27 Findings cumulation and reporting HC 6-9, LW 8 #2 handouts & examples NOV 3 Advanced Modeling handouts #3   10-17 Individual meetings #4 Nov 10, #5

Nov 17   24 Presentation of student papers using powerpoint  DEC 1 continuation of presentations as needed

Grades: Based on completion of intermediate assignments (20%) and quality of paper (80%)

U= < 60% S = 61% - 100%   Intermediate assignments: 1  Proposed meta-analysis topic with preliminary list of at least 20 articles 2 Preliminary list of proposed variables to be extracted from articles 3 Proposed Coding menu for variables 4 Preliminary analysis of outcomes using Q method, funnel plot of one

outcome variable 5 Preliminary covariate analysis using QB and QW method (at least 1

covariate)    Project and paper: conduct a meta-analysis using the procedures of this course

on a topic of your choice; limit the number of studies to be finally included to no more than about 20 but at least 8 as appropriate to the topic. Produce a finished paper in APA format summarizing the procedures and results; produce a powerpoint of your paper for presentation.

  In order to receive a grade a presentation must be made on the dates listed;

otherwise an incomplete will be given and a presentation to me and a group of students must be made before a final grade can be submitted. Final papers may be turned in after the last day to turn in grades for the semester with a grade of I given and no penalty, but after a presentation has been made and all intermediate assignments are completed. After one semester a grade of F will be given if these conditions are not met.

Note: The handouts and web-based files used in this course are copyrighted. By “handouts” I mean all materials generated for this class, which includes but is not limited to syllabi, quizzes, exams, lab problems, in-class materials, review sheets, and additional problem sets, in paper or electronic form. Because these materials are copyrighted, you do not have the right to copy the handouts unless I expressly grant permission.

As commonly defined, plagiarism consists of passing off as one’s own ideas, words, writings, etc. which belong to another. In accordance with this definition, you are committing plagiarism if you copy the work of another person and turn it in as your own, even if you should have the permission of that person. Plagiarism is one of the worst academic sins, for the plagiarist destroys the trust among colleagues, without which research cannot be safely communicated.

If you have any questions regarding plagiarism, please consult the latest issue of the Texas A&M University Student Rules, under the section “Scholastic Dishonesty”. Honor Code and refers the student to the Honor Council Rules and Procedures on the web:

http://www.tamu.edu/aggiehonor.) “An Aggie does not lie, cheat or steal, or tolerate those who do.”

Purposes for Meta-Analysis• Cumulate findings of studies on a particular

topic• Examine homogeneity of outcomes• Estimate effects of independent variables on

outcomes in a standardized format– Evaluate moderator and mediator effects on

outcomes– Differentiate different types or classes of

outcome effects

Historical backgroundCriticism of traditional narrative reviews of

researchExasperation in social sciences with

constructs measured different ways in terms of determining consistencies

Need to formulate theoretical relationships based on many studies

History part 2Early 1970s efforts focused on significance

testing and “vote counts” of significanceGlass (1976) presented a method he called

“meta-anlaysis” in Am. Ed. Research Assn. presidential address

Others proposed related methods, but Glass and colleagues developed the most widely used approach (Glass, McGaw, & Smith, 1981)

Meta-Analysis as Survey Research• Research articles as unit of focus• Population defined

– Conditions for inclusion of articles• Data requirements needed for inclusion• Completeness of data available in article or estimable• Publication sources available, selected

• Sample vs. population acquisition– Availability of publications and cost– Time to acquisition (how long to wait for

retrieval)

Strengths of Meta-AnalysisDefinition of effect and effect size beyond

“significant or not”Focus on selection threats in traditional

reviews (bias in selection of articles for review)

Systematic consideration of potential mediators and moderators of effects

Data organization of articles for public review

Weaknesses of Meta-AnalysisMethodologically sophisticated and expensivePotential ignoring of contextual effects not

easily quantified; eg. historical/environmental placement of research

Potential improper mixing of studiesAverages hiding important subgroupingsImproperly weighting studies with different

methodological strength/rigor

Research focus for meta-analysis

Defining and delineating the constructDetermining a research outletMeta-Analysis as an interactive, developing

process

Recent CriticismSuri & Clarke (2009): Advancements in

Research Synthesis Methods: From a Methodologically Inclusive Perspective (Review of Educational Research, pp. 395-430)

They propose 6 overlapping approaches:1. Statistical research syntheses (eg. meta-

analysis)2. Systematic reviews3. Qualitative research syntheses4. Qualitative syntheses of qualitative and

quantitative research5. Critical impetus in reviewing research6. Exemplary syntheses

Some critical comments on Suri & Clarke (2009) Systematic reviews- original Glass criticisms

hold: what is the basis for inclusion and exclusion; why are certain articles privileged?

Qualitative research syntheses- how can these be done with situated contexts, small samples, environmentally-developed variables, sources, etc.? Will there be a review for every reader, or for every researcher? Same limitation as all qual research

Qual syntheses of quant and qual research- potentially doable, with an alternating order: qual first to focus emphases in the quant analysis, or quant first to be validated with the qual studies of particular environments and populations- do they fit/match in reasonable ways?

Critical impetus- code words for critical theory/Marxist etc. Answer is already known, why do the research?

Exemplary syntheses- what is the purpose?

Defining and Delineating the Research Topic• Outcome construct definition

– Importance to the field to know what has been learned– How big is it? How many potential studies? – Conduct preliminary searches using various databases

• Refining the construct– How much resource is available? Eg. 1000 studies = 2-

3 years work – Are there specific sub-constructs more important than

others? Select them or one of them– Are there time-limitations (no studies before 19xx)– Are there too few studies for the given construct,

should it be broadened? Too few-> less than 10?

Defining and Delineating the Research TopicWhat is the typical research approach for the

topic area?All quantitativeAll qualitativeMixed quantitative and qualitative

Are there sufficient quantitative studies to provide evidence for findings?Can qualitative studies be included as a

separate part of the study? How?

Determining Research Outlet• Does the proposed journal

– publish research on the construct?– Publish reviews or meta-analyses?

• Is there a journal devoted to reviews that your project would fit with?

• Has a recent similar meta-analysis been published? If so, will yours add anything new?– Ex. Allen, et al (under review) evaluated articles on

first grade retention after 1990 focusing on the quality of the research design in each study to determine if the effects were different from a fairly recent meta-analysis by Jimerson (2001)

Meta-Analysis as an interactive, developing process• View meta-analysis as evolutionary

– As studies are reviewed and included, purpose and scope may change

• Assume initial conceptualizations about both outcomes and potential predictors may change over time– Definitions, instruments, coding may all change

as studies are found and included

• Plan for revisions to all aspects of the meta-analysis

ACTIVITYGet into groups of 3 or 4Use GOOGLE SCHOLAR to find research

articles from the following:Group 1: elementary school grade retentionGroup 2: teenage depression treatmentsGroup 3: IQ and socioeconomic statusGroups 4+: one of above

Select 5 articles that have empirical dataFor each article

define the population being samples List with specific names the variables used in the study Discuss within your group the similarities and differences of

the populations and variables; consider difficulties you would have in summarizing research for these

Plan to report to the class what you found