PBL in Economics

Effects of Problem Based Economics on High School

Economics Instruction

Neal D. FinkelsteinRegional Educational Laboratory, West (REL West), WestEd

Jason RavitzBuck Institute for Education

AERA Annual Meeting – New Orleans, LAApril 14, 2011

1

Neal D. FinkelsteinSenior Research Scientist

REL West)WestEd

December 8, 2010

2

The case for economic literacy is obvious. High school graduates will be making economic choices all their lives, as breadwinners and consumers, and as citizens and voters. A wide range of people will be bombarded with economic information and misinformation for their entire lives. They will need some capacity for critical judgment. They will need it whether or not they go to college.

James Tobin, Yale Economist and Nobel Laureate Wall Street Journal, July 9, 1986

3

Purpose• Assess student-level impacts of a problem-

based instructional approach to high school economics.

• Tests the effectiveness of Buck Institute for Education curriculum on student learning of economics content and problem solving skills.

• Within-school randomized controlled trial in urban and rural high schools in California and Arizona during the 2007/08 school year.

4

Why Study Economics Instruction?• For decades, economists, educators, and business and

government leaders have advocated for economic literacy as a component in school curricula.

• In general, high school economics does not help students understand our economic system.

• Most teachers are not adequately prepared to teach economics.

• 48 states include content standards in economics. 40 require implementation of standards, 23 require testing, and 17 require an economics course for graduation.

5

6

• Each of five curriculum modules is built around a case study well-suited to student-driven problem solving.

– Teachers use economic problems to help students learn to contextualize, understand, and solve problems.

• Teaching guide lays out all aspects of instruction

– including problem statement, placement in curriculum, concepts taught, objectives, resources materials and do’s and don’ts.

• Teachers in the intervention group receive professional development introducing the curriculum.

– Led by expert teachers with experience using the PBE units

What is Problem-Based Economics Instruction?

Students are often challenged in high school Economics courses because:

O Economics concepts can be abstract

O They don’t see relevance to real life

O They are bored

O All of the Above

7

“Anyone…? Anyone…?”

8

The Curriculum

Organizes curriculum and instruction around• Carefully crafted situations• Adapted from real-world issues• With more than one possible solution

Students work in teams to gather, apply and demonstrate knowledge and skills as they develop and present solutions.

9

• Units on micro- & macro-economics (7 total, 5 used in study)

• Aligned with National Content Standards in Economics and state standards

• Units take from one to three weeks

• Typically taught in stand-alone high school Economics courses

10

HOW EACH PBE UNIT IS STRUCTURED:

1. Project Launch (do NOT pre-teach!)

2. Framing the Inquiry (Know/Need to Know List & Driving Question)

3. Problem-Solving & Learning Activities

4. Presentation, Assessment, and Debrief

11

WHAT IS INCLUDED IN EACH PBE UNIT:

• guidance on teaching with PBL

• “Make More Money?” activity

• step-by-step directions (with Project Log prompts and Content Teaching Notes)

• student handout masters

• content background material for teacher mini-lectures & lessons

• rubrics & exemplars for assessment

• multiple-choice test

12

PBE UNIT: RUNNING IN PLACE

• As grad students in Economics, students are asked for advice by an entrepreneur who wants to start a new shoe company

• Students hear from consumers and producers

• Students learn about factors of production, circular flow

• Students create visuals and make an oral presentation of their recommendations

13

“Making economics more engaging and relevant for students is possible!”

“I have really enjoyed teaching with this curriculum; having a central problem to refer back to throughout the unit helps with teaching the concept.”

“I just finished using PBE for the second time. The material that my kids are putting together is amazing. These are inner-city kids and the results are beyond belief. They love your material. I have been teaching for over 30 years and the lessons are undoubtedly the best I have ever used.”

WHAT TEACHERS SAY ABOUT PBE:

14

Conceptual Framework

15

• Student achievement outcomes are of primary importance and are hypothesized to be mediated by changes in teacher knowledge and pedagogical practice.

16

Logic model for the study of high school instruction with Problem-based Economics

Teacher Professional Development

Classroom Instruction

Student Performance

Content• Review of economic

theory and related applications; curriculum reivew

Pedagogy• Problem-based

approach to teaching and support strategies for students

Curriculum• Fluent presentation

of curriculum; scaffolding for students

Engagement• Problem-solving

strategies that build economic content and reinforce core analytic thinking

Concepts• Broad-based

literacy in micro- and macroeconomic concepts

Problem Solving• Ability to apply

concepts and analytic approaches to real-world problems

Conceptual Framework

Research Questions1. Does PBE change students’ content knowledge in

economics?

2. Does PBE change students’ problem-solving skills in economics

3. Does PBE change teachers’ content knowledge of economics?

4. Does use of PBE change economics teachers’ instructional practices

5. Does the use of PBE change teachers’ satisfaction with teaching materials and methods used to teach economics?

17

Key Outcomes to be Studied and Their Measures

• The primary outcome measure is content knowledge gains for students in economics measured by the Council for Economic Education’s Test of Economic Literacy.

• Student problem-solving skills are measured with open-response performance assessment of applied economic concepts developed by UCLA CRESST.

• Teachers’ content knowledge was also measured using the Test of Economic Literacy. Pedagogical practices and teacher satisfaction were measured using a series of survey measures.

18

Key Study Characteristics

19

Study design Cluster-randomized trial

Unit of assignment Teachers

Statistical power estimatesFor Type 1 error = .05, 80 percent or higher power to detect minimum detectable effect size of 0.18-0.21 at student level and 0.55 at teacher levela

Implementation began Summer 2007

Student measures

Test of Economic Literacy (pre/post)

Student surveys (pre/post)

Performance task assessments

Administered January 2008, June 2008

Administered January 2008, June 2008

Administered June 2008

Teacher measures

Test of Economic Literacy (pre/post)

Teacher surveys (pre/post)

Administered June–August 2007, June 2008

Administered June–August 2007, June 2008

Note: a. The estimates were based on 83 teachers, with an average of 40 students per teacher.

Study characteristics and data collection schedule

Teacher Participants

20

Number of recruited teachers: n = 128 teachers from 106 schools

Excluded: 0 teachers

Number of teachers randomized: n = 128 teachers from 106 schools

Allocated to intervention group: n = 64• 44 teachers from 44 singleton

schools• 20 teachers from 16 schools

with 2 or more participating teachers.

Allocated to control group: n = 64• 46 teachers from 46 singleton

schools• 18 teachers from 15 schools

with 2 or more participating teachers.

Included: n = 42• 28 teachers from 28 singleton

schools• 14 teachers from 11 schools

with 2 or more participating teachers.

Not Included: n = 22• 16 teachers from 16 singleton

schools• 6 teachers from 6 schools

with 2 or more participating teachers.

Baseline Measures (Summer 2007)• Teacher content knowledge in

economics (n = 41)• Pedagogical practices used (n = 39)• Satisfaction with teaching materials

& methods (n = 40)

Outcome Measures (June 2008)• Teacher content knowedge in

economics (n = 38)• Pedagogical practices used (n = 38)• Satisfaction with materials &

methods (n = 37)

Included: n = 41• 31 teachers from 31 singleton

schools• 10 teachers from 9 schools

with 2 or more participating teachers.

Not Included: n = 23• 15 teachers from 15 singleton

schools• 8 teachers from 8 schools

with 2 or more participating teachers.

Baseline Measures (Summer 2007)• Teacher content knowledge in

economics (n = 29)• Pedagogical practices used (n = 25)• Satisfaction with teaching materials

& methods (n = 29)

Outcome Measures (June 2008)• Teacher content knowedge in

economics (n = 34)• Pedagogical practices used (n = 35)• Satisfaction with materials &

methods (n = 35)

Allocation Retention Data Collected & Analyzed

Teacher Participants Providing Student-Level Data

21

Number of teachers recruited and randomized: n = 128 teachers

Allocated to intervention group: n = 64

Attrition: n = 22

Allocated to control group: n = 64

Attrition: n = 23

Number of teachers remained in the study: n = 42

In fall 2007:• 38 teachers returned student-level data• 3 teachers did not return any student-level

data

In spring 2008:• 35 teachers returned student-level data• 6 teachers (including those who dropped

out of the study) did not return any student-level data

Number of teachers remained in the study: n = 41

In fall 2007:• 38 teachers returned student-level data• 3 teachers did not return any student-level

data

In spring 2008:• 29 teachers returned student-level data• 12 teachers (including those who dropped

out of the study) did not return any student-level data

Total number of teachers who provided student-level data in spring 2008: n = 64 teachers

Student Participants

22

Number of students consented to participate in the study & their data available to the research team: n = 4,350 students from 64 teachers

In intervention group (35 teachers):

• Included: n = 2,502• Not Included: n =0

In control group (29 teachers):

• Included: n = 1,848• Not Included: n = 0

Baseline Measures (January 2008)• Student content knowledge in

economicsNon-missing: n = 2,232 (98%)Missing: n = 270 (11%)Number of teachers = 35

Outcome Measures (June 2008)• Student content knowledge in

economicsIncluded: n = 2,178 (87%)Not Included: n = 324 (13%)Number of teachers = 35

• Student problem-solving skillsIncluded: n = 1,918 (77%)Not Included: n = 584 (23%)Number of teachers = 33

Baseline Measures (January 2008)• Student content knowledge in

economicsNon-missing: n = 1,589 (86%)Missing: n = 259(14%)Number of teachers = 29

Outcome Measures (June 2008)• Student content knowledge in

economicsIncluded: n = 1,574 (85%)Not Included: n = 274 (15%)Number of teachers = 29

• Student problem-solving skillsIncluded: n = 1,497 (81%)Not Included: n = 351 (19%)Number of teachers = 29

Experimental Condition Data Collected & Analyzed

Data Analysis Methods

23

• The analyses for this study compare outcomes for students and teachers in the treatment group with their counterparts in the control group after the economics course has been completed.

• For student outcomes, the analyses involve fitting conditional multilevel regression models (HLM) to account for the nesting of students within teachers.

Data Analysis Methods

24

• A random effect for teachers is included in the model to account for the nesting of student observations within teachers. Fixed effects include treatment status, baseline (pre-test) measures of outcome variables, and other student and teacher-level covariates.

• For teacher outcomes, single-level regression models that include treatment status along with certain covariates (such as pre-test measures of outcome variables and teacher background information) are used to estimate program impacts.

25

• Statistically significant finding: students whose teachers had received professional development and support in PBE outscored their control group peers on the TEL by an average of 2.6 test items.

Student-Level Findings

• The outcomes on student measures of problem-solving skills and application to real world economic dilemmas also showed significant differences in favor of the intervention group.

Student-Level Findings

26

27

Teacher-Level Findings

• No statistically significant difference between the interventions and control groups on teachers’ knowledge of economics.

• No statistically significant difference in teachers’ pedagogical style with the survey measures used.

• Statistically significant differences in favor of the intervention group teachers on a measure of satisfaction with the teaching materials and methods.

Effects of Problem Based Economics on High School

Economics Instruction

http://ies.ed.gov/ncee/edlabs/regions/west/pdf/REL_20104012.pdf

28