Comparing the Effects of Two Versions of Professional Development on Science Curriculum Implementation and Scaling- Up Session 43.030 April 13, 2005 American Educational Research Association Annual Meeting Montreal, Canada
Dec 21, 2015
Comparing the Effects of Two Versions of Professional
Development on Science Curriculum
Implementation and Scaling-UpSession 43.030
April 13, 2005American Educational Research Association Annual Meeting
Montreal, Canada
Introduction and Overview
Paul R. Brandon
Curriculum Research & Development GroupUniversity of Hawai‘i at Mānoa
3
Project Purpose• A randomized study of the effects of
variations in professional development (PD) on – program implementation– student achievement– scale-up
• First phase (ending 2/28/06) to prepare for five-year second phase (if funded)
4
FAST
• The project compares two versions of teachers’ PD for Foundational Approaches in Science Teaching (FAST).– An award-winning middle-school inquiry-
based science program– Shown to positively affect achievement and
other student outcomes
5
Phase-I Project Tasks
• Prepare the 5-day PD institute with electronic resource and on-line course
• Prepare instruments for the second phase:– Teacher questionnaire– Teacher log– Observation procedures– Student assessments
6
Focus on Span of PD
• Traditionally, FAST PD is delivered in a 10-day institute.
• We will compare it with an alternative 5-day institute followed by an online course, with an electronic multimedia resource.
7
Phase-II Hypothesis
The 5-day inquiry-based science training institute, followed by an on-line university-credit
course with computer-based multimedia, will have more favorable outcomes than a 10-day
institute without the follow-up university course.
8
Examining the effects of PD on:
• levels of classroom implementation
• intensiveness and extensiveness of long-term use of inquiry-based science (i.e., scale-up)
• levels of student achievement
9
Experimental Design
• Randomized cluster sample, with schools (N = 80) as clusters
• One teacher per school
• Even if attrition is 25%, statistical power will be .80.
10
Merit of the Study
• Addresses at least five deficits in the literature:– Few studies on effects of PD time span– Few studies about using technology in PD– Few randomized studies– Few studies of the effects of PD on student
learning– Few, if any studies, of inquiry science PD
11
Order of Presentations
• Gray, Nguyen, & Speitel: Description of PD • Brandon: Log and questionnaire development
and questionnaire validation• Taum: Observation guide development• Ayala: Student assessment development • Lawton: Comparison of early effects of the two
versions of PD
Introduction and Overview
Paul R. Brandon
Curriculum Research & Development GroupUniversity of Hawai‘i at Mānoa
Developing and Implementing an Alternative Version of FAST
Professional Development
Mary E. GrayThanhTruc T. Nguyen
Thomas W. SpeitelUniversity of Hawai‛i at Mānoa
14
FAST
• Foundational Approaches in Science Teaching
• Inquiry-based, middle school science
• Physical, biological, and earth sciences
• Exemplary program (USDOE, 2001)
15
FAST Professional Development
• All FAST teachers must participate in PD– Certification required to
purchase science curricula
• Content• Inquiry• Standards• Assessment• Classroom organization
and management• Safety
16
The ChallengeTo develop and implement an alternative version of FAST I professional development
Year Round Schedules
17
Considerations
• Current needs of science teachers
• Future possibilities (emerging technologies)
• Unique challenges (the dynamic nature of inquiry)
18
Review of LiteratureProfessional Development
On-Line LearningInquiry Science
19
Review of Literature Professional Development
• Strong subject area content (Kennedy, 1999)
• Focus on higher-order teaching strategies (Porter, Garet, Desimone, Yoon, & Birman, 2000)
– Collaborative, Same subject-grade-school, Active learning, Consistent with teacher goals
20
Review of Literature Professional Development
• Quantity linked to improvements (Radford, 1998; Supovitz, Mayer, & Kahle, 2000; Fishman, Marx, Best, & Revital, 2003)
– science content knowledge, process skills, and attitudes
• Quantity linked to standards based teaching (Supovitz & Turner,2000)
• Problems with researching inquiry-based teaching and student learning (Fishman et al., 2003)
21
Review of Literature On-line Learning
• 3 million adult learners on-line (Waitts & Lewis, 2003)
• Convenience and increased flexibility (Hülsmann,1999)
• Evidence of implementation of various pedagogical approaches (Schlager & Schank, 1997)
22
Review of Literature On-line Learning
• Quantifiable learning outcomes were not significantly linked to technology adoption (Jones and Paolucci,1998).
• Focus on motivation, skills and knowledge, self-directed learning, interactive competence, and technology skills (Kabilan, 2004)
• Increased efficacy and self-perception in teachers (Huai, Braden, White, & Elliot, 2003)
23
FASTPro
24
How are the traditional PD and FASTPro different?
• Two-weeks in duration, face-to-face
• One-week in duration, face-to-face
• Conduct 43 investigations
• Extensive modeling and practicing time
• Conduct 19 investigations
• Electronic resource• WebCT-based
learning community
InquiryInquiry
Teaching and learning science through inquiry is a new experience for many teachers and requires a significant change in attitude and behavior.
26
FASTPro – Addressing Inquiry• Strategies (Loucks-Horsley, Hewson,
Love, & Stiles,1998)
– Examples• Immersion in inquiry
into science• Coaching and
mentoring • Technology for
professional learning
27
FASTPro – Addressing Inquiry• Change (Hord, Rutherford, Huling-Austin, & Hall,1987; Guskey,
2000)
– takes time and persistence• awkwardness and frustration expected
– as teacher’s progress through a change process, their needs for support and assistance change
• Optimal Mix (Guskey, 2000) – A combination of teacher, program and change
agent that will help create a positive relationship for PD to be effective
28
FASTPro
29
FASTStart• One week face-to-face
institute
• Teachers conduct nearly half of actual student investigations
• Use inquiry methodology including modeling, discussion, assessment, and practice
30
31
FASTeR
• Observe interactions of students and teachers
• View and reflect upon investigations not experienced
• See detailed step-by-step procedure suggestions
• Movies and animations
• Web interface– Quicktime and Flash
plugins
• DVD-ROM medium
32
Examples from FASTeR
33
Examples from FASTeR• Slideshow 9, Density
and the Cartesian Diver
• Slideshow 22, Identifying Unknown Substances
34
35
FASTForward
36
Formative Findings
• Suggest that teachers were able to implement successfully and share teaching practice
• Added utility, focused and meaningful to teachers’ own environment
37
Future considerations• Infuse technological
aspects into the FASTStart face-to-face experience
• Address credit equivalencies to enable more robust activity requirements
• Expand to include FAQ’s, indexing, and rich descriptions
• Expand to include a Website as well as DVD-ROM
• Automate some computer assisted instruction and feedback mechanisms
38
Reflections…
39
Reflections…• Teamwork and
flexibility
• Coordination of equipment and resources
40
Future Considerations• Building capacity
• Application to other CRDG quality educational programs
41
Mahalo.
Developing and Implementing an Alternative Version of FAST
Professional Development
Mary E. GrayThanhTruc T. Nguyen
Thomas W. SpeitelUniversity of Hawai‛i at Mānoa
Instrument Development for a Study Comparing Two
Versions of Inquiry Science Professional Development
Paul R. Brandon Alice K. H. Taum
University of Hawai‘i at Mānoa
44
Identifying Constructs
• on teaching science with inquiry methods– Reviewed FAST and other inquiry science documents
and worked closely with inquiry science experts.
• on the context within which inquiry-based science is taught.– Reviewed 55 books and key articles on curriculum
indicators and school effectiveness.
45
Developing the Teacher Log
• Purpose: To identify the extent to which teachers implement the key features of inquiry-based science.
• Reviewed the recent literature on logs.
• Kept the instrument short to avoid overburdening teachers (21 items).
• Is completed immediately after finishing each student science investigation.
46
Focus of the Teacher Log
• Instrument addresses topics such as:– students’ questioning behaviors.– the teacher’s use of questioning strategies.– the teacher’s circulation about the classroom.– teacher-led discussions about variations in
the data.
47
Developing the Teacher Questionnaire
• Purposes: – To obtain information about implementation
that is most appropriately collected once a year.
– To collect information on implementation of investigations not covered on logs.
48
Focus of the Questionnaire
• 150 items address topics such as:– implementation of key features.– the investigations taught during the entire
year.– the adequacy of materials, equipment, etc.– teacher demographics.– teacher attitudes toward science.– teacher participation in science activities
outside the classroom
49
Types of Validation Studies to Date
• Two types of questionnaire-data validation studies conducted to date:– analyses addressing the relationship between
the responses on the log and the questionnaire (“concurrent” validity)
– an analysis examining the theoretical model underlying our study (construct validity)
50
Relationship Between Log and Questionnaire Responses
• Compared results on total scores for five items that the two instruments had in common
• Correlation = .54; effect size = .50
51
Examining the Theoretical Model• Second set of analyses examined whether the
expected relationships were found among some questionnaire variables.
• We regressed a variable measuring program implementation on five independent variables measuring:– teachers’ ongoing learning about science.– the resources available for teaching science.– the number of science courses taken.– the number of years have taught K–12 science– teachers’ attitudes toward teaching science
52
Validity Argument• Teachers’ ongoing learning about science
(including learning that occurred in recent PD) should predict implementation more than– science education background.– years of experience teaching science.– attitudes toward teaching science.– classroom resources available.
53
Results of Second Analysis• Teacher learning about science outside the classroom: • beta = .19 (significant at the .05 level)• Teacher attitudes toward teaching science: • beta = .17 (significant)
• Classroom and school resources for teaching science: • beta = .04 (not significant) • Number of university science courses taken: • beta = -.02 (not significant) • Number of years the teacher has taught K–12 science: beta =
-.02 (not significant)
Instrument Development for a Study Comparing Two
Versions of Inquiry Science Professional Development
Paul R. Brandon Alice K. H. Taum
University of Hawai‘i at Mānoa
Coding Teachers in Science Classrooms using the Inquiry Science Observation
Guide
Alice K. H. TaumCurriculum Research & Development Group
University of Hawai‘i at Mānoa
56
Inquiry Science Observation Guide• Introduction and Overview
– FAST, SCUP Project, Purpose and Development of ISOG
• General coding guidelines• Descriptions for each of the six
Activities• Definitions of Activity details
• Code Sheet • Recording Sheet • Reconciling Recording Sheet
57
Development of the Code Sheet
• 1.5 year process
• Collaborative efforts between researchers at the University of Hawai‘i at Mānoa, Stanford University and Sonoma State University; FAST teachers; curriculum developers, and coders
• 36+ revisions
58
Goal of the Code Sheet • Identify strings of activities/teacher
behaviors using the details in columns A, B, C and D. – questioning strategies used – level of engagement between teacher and
students– creating a profile of teachers implementation
of FAST in the classroom through science inquiry
59
Design of the Code Sheet
• 6 activities.
• 66 possible activity details to plug into the 6 activities.
• Multiple variations between an activity and the activity details which describe an activity string of teachers pedagogical practices.
60
Inquiry Science Observation Code Sheet (ISOCS)
Activity Activity details Activity
descriptors
1 - 6 A B C
D
1. Teacher directs student (s) ______A______ to ______B______ _______C______.
1A1. individually 1A2. in a small group
1B1. record 1B2. discuss 1B3. define 1B4. read materials to whole class
1C1. observations 1C2. predictions/ hypotheses 1C3. procedures 1C4. data 1C5. science (concept, vocab words, mechanics of science, etc…)
1D1. with evidence or examples 1D2. does not apply
2. Through ______A______, the teacher ______B_______ science ______C______.
2A1. direct instruction questioning: 2A2a. rhetorical 2A2b. interactive
2B1. introduces or provides an overview of 2B2. reviews/summarizes 2B3. demonstrates 2B4. collects 2B5. compares/contrasts 2B6. clarifies
2C1. concepts(idea) 2C2. procedures(activity) 2C3. tools/equipment 2C4. investigation (science experiment) 2C5. problem 2C6. goal 2C7. -related safety issues 2C8 data (unknown): 2C8a. differences in 2C8b. relationships between 2C8c. quality of 2C8d. analysis of 2C8e. synthesis of 2C8f. evaluation of 2C9. vocabulary words
2D1. new information 2D2. previously learned information 2D3. investigation 2D4. unit 2D5. does not apply
1. Teacher directs student(s)
_____A_______ to
_____B_____ _____C_____.
61
1. Teacher directs student(s) individually (1A1) to discuss (1B2) procedures (1C3).
1. Teacher directs student(s) _____A_____ to _____B______ _____C_____.
62
Inquiry Science Observation Recording Sheet
• Notes the start time of when the “broader” and “other activities” a teacher is engaged in begins
• Allows for comments to facilitate the reconciliation process
• Designed to record multiple activities occurring at the same time
63
Broader and Other Activities– The broader activity captures the larger activity
occurring:Through interactive questioning (2A2b), the teacher introduces or provides an overview of (2B1) science investigation (2C4).
– The other activities capture the details occurring within the broader activity:Teacher questions students through clarifying (3A1a) questioning and responds to student comment (3B1) by repeating (3C2) the comment and probing further (3C6).
64
65
Reconciling Codes• Two coders are paired to compare their
independent codings, identifying any differences between them.
• Differences are then discussed and each coder provides a rationale for his or her selected code.
• When necessary, a review of the DVD is conducted by the paired coders.
• Coding differences are discussed until consensus is reached between coders.
66
Inquiry Science Observation Reconciling Code Sheet
67
Challenges• greatest challenges:
– establishing universal definitions for terms and phrases
• What constitutes a “procedure”? • What is the difference between a science “term” and a
science “concept”?
– standardizing the recording of activity strings using the Recording Sheet
• broader verses other activities
– focusing on the teacher, rather than on the students
68
What Next?• To date we have:
– videotaped 15 teachers;– 140 videotape cassettes have been digitized
to DVDs;– 116 have been quality-checked for audio
clarity and teacher visibility; and – 8 coders are trained and ready to begin
coding!
Coding Teachers in Science Classrooms using the Inquiry Science Observation
Guide
Alice K. H. TaumCurriculum Research & Development Group
University of Hawai‘i at Mānoa
Developing Student Outcome Measures Frameworks
Carlos C. Ayala Sonoma State University
Scaling Up: Student Measures
The Buck Stops HereThe Buck Stops Here
Carlos Ayala
Let’s Talk Charge
Develop student assessment instruments in order to tease out differences between professional development models.
2 Session Pre Test3 Session Post Test Suite
73
Where we expect differences
• Student’s content knowledge• Student’s science inquiry skills• Student’s views of the nature of science• Student’s efficacy toward science• Student’s motivation to learn science
74
Final Assessment Suite
• Pre-test – 30 item multiple-choice/ short answer test (α= .86)– Attitudinal Survey
• Post-test – 30 item multiple-choice/ short answer test– Attitudinal survey– Mānoa River performance assessment
75
76
Knowledge Type Framework
Type of Knowledge Definition Examples Prompt
Declarative Knowing that Concepts & facts “What is
percolation?”
Procedural Knowing how Actions, steps, “How do
measure how much & procedures
water soil will hold?”
Schematic Knowing why Principles & “How does the water
cycle mental models work?”
77
78
Content Validity
• Reviewed curriculum and created content matrices• Curriculum developers parsed content down• Linked matrices to materials• Created test linking items to matrices• Piloted assessments• Talk alouds
79
• As students become more engaged in the FAST curriculum and the teacher more fully implements the curriculum, students will be more proficient at Science Inquiry.– Design and conduct a scientific investigation– Use appropriate tools to gather, analyze and interpret data– Develop descriptions, explanations predictions and models
using evidence
• Targets based on – Duschl’s Transformations in Three Domains– Pottenger’s Deductive Explanatory Inquiry
Student Science Inquiry
NRC. (1996). National Science Education Standards . Washington D.C.: National Academy of the Sciences.
80
Transformation 1Data to Evidence: Deciding if the data are evidence, irrelevant and/or problematic.
Transformation 2: Evidence to patterns or models decisions about selecting tools for identifying patterns or models
Transformation 3:Patterns and models to explanations. Deciding how the patterns or models lead to explanations.
Reformulation: From explanations to new questions: Deciding what next questions to ask and what new data are needed.
Student Science Inquiry
81
82
• As students become more engaged in the FAST curriculum and the teacher more fully implements the curriculum, students will understand that– anyone can be a scientist.– science knowledge is useful.– science knowledge builds over time.– science is creative.
Nature of Science
Lederman, Abd-El-Khalick, Bell and Schwartz (2002) Views of Nature of Science Questionnaire; Toward Valid and Meaningful Assessments of Learner’ Conceptions of the Nature of Science.
83
• As students become more engaged in the FAST curriculum, the greater control they will feel towards science, science investigations and science knowledge.
– I can make accurate measurements during a science investigation.– I can make appropriate predictions about what will happen during a science investigation.
Self Efficacy
Britner, S. and Pajares F., (2001) Self-Efficacy Beliefs, Motivation, Race and Gender in Middle School Science.
84
• As students become more engaged in the FAST curriculum, patterns of motivation may change.
• Dweck Groups (goals and epistemic beliefs)– Mastery Orientation– Ego Orientation– Helpless Orientation
Motivation
Haydel, A., & Roser, R. (2001). On the links between students' motivation patterns and their perceptions of, beliefs about and performance on different types of science achievement, Multidimensional Approach to Achievement
85
Next Steps
• 760 Tests and Surveys completed• Collect post test data• Run analyses• Provide results to group
Developing Student Outcome Measures Frameworks
Carlos C. Ayala Sonoma State University
The Differential Effects of Two Versions of Professional Development on Teachers’ Self-Efficacy to Implement
Inquiry-Based ScienceBrian Lawton
University of Hawai‘i at Mānoa
88
Purposes
• Effects of both versions of PD institutes on teacher self-efficacy to implement program
• Differences between the two versions
• Factors associated with self-efficacy change
89
Data Collection• Qualitative and quantitative methods used• Self-efficacy scale
– Developed to facilitate focus group discussion• Focus groups
– Conducted immediately following the institutes
– Provide in-depth information about changes in self-efficacy• Five stages of inquiry science
1. Introducing new science investigations2. Facilitating valid experimental design3. Facilitating the investigation process4. Constructing meaning5. Linking knowledge to new situations
• Seven teachers from the 10-day and seven teachers from the 5-day participated in the study
90
Data Collection (cont.)
Self-Efficacy Scale
1. STAGE: Introducing new science investigations - my ability to introduce students to new science investigations by reviewing and tying in previous work.
Now Low High ability ability
Before Low Highability ability
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
91
Data Analysis
• Only presenting focus group results• Analyzed for changes in self-efficacy
– Two categories: • Statements implying an increase in self-efficacy• Statements implying no change in self-efficacy
• Categories analyzed to identify the influencing factors
• The statements and factors were quantified
92
Results, Influencing Factors
• Overall seven factors were identified as influencing self-efficacy.
1 Factors identified as most influential in increasing self-efficacy
2 Factor identified as most associated with no change in self-efficacy
Factors influencing increased self-efficacy
•Content/pedagogical knowledge1
•Program Structure1
•Professional Collaboration
•Experience
Factors influencing no change in self-efficacy
•Practice/feedback2
•Student characteristics
•Experience
•Time issues
93
Results: Changes in Self-Efficacy
Changes in self-efficacy10-day version
5-day version
Statements identified as increases in self-efficacy
11 8
Statements identified as no change in self-efficacy
6 9
94
Discussion
• Group increases in self-efficacy
• Changes in self-efficacy between the groups
• Groups perceived gains in content and pedagogical knowledge
• Importance of 5-day follow-on support
95
Further Study
• A study comparing the groups after the 5-day group has received all their training.
96
Thanks!
The Differential Effects of Two Versions of Professional Development on Teachers’ Self-Efficacy to Implement
Inquiry-Based ScienceBrian Lawton
University of Hawai‘i at Mānoa
Comparing the Effects of Two Versions of Professional
Development on Science Curriculum
Implementation and Scaling-UpSession 43.030
April 13, 2005American Educational Research Association Annual Meeting
Montreal, Canada
99
Questions and Comments• Paul Brandon - Overview• Mary Gray and Truc Nguyen - Description of PD• Paul Brandon - Log and questionnaire development and
questionnaire validation.• Alice Taum –Observation guide development• Carlos Ayala - Student assessment development • Brian Lawton - Comparison of early effects of the two versions
of PD
Paper, Presentation and Contact Info available at:
http://hisii.hawaii.edu/SCUP/research/