PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 1 A. Effect Size Can Tell a Lot! B. Future Ready Core NC Accountability Conference February 2009 Mike Gallagher Accountability Services Dept. NCDPI
Feb 25, 2016
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
1
A. Effect Size Can Tell a Lot!
B. Future Ready Core NC Accountability Conference
February 2009
Mike GallagherAccountability Services Dept.
NCDPI
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
2
Purposes – Two Parts
• Part A. Show how effect size can give important information to compare test results across EOG, EOC and SAT and other tests.
• Part B. Review the background and give sources of information about the Future Ready Core.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
3
Part A: Effect Size Lens• Usefulness: Enables one to compare
across tests and test forms.• Usefulness: Provides solid information
about performance.• Concept: What effect size means.• Calculation: How to figure it.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
4
How does North Carolina improvement on SAT math compare with its improvement on NAEP math?
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
5
SAT Math Score Trend for N.C.
• Between 1990 and 2008, the Math gap between NC and the Nation narrowed from 31 pts. to 4 pts.
• The annual SAT reports on the DPI Accountability web site have lots more information:
www.ncpublicschools.org/accountability
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
6
SAT Math Score Trend
501 501 504 508 512 514 516 518 518 515
470479 482 486
492 496505 507
513 511
440450460470480490500510520530
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008
Mea
n M
ath
SAT
Scor
e
Nation North Carolina
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
7
NC progress on NAEP mathNAEP results from 1992 to 2005Compared with the U.S. average and
with other states, North Carolina went from back of the pack to ahead of the pack. NC was somewhat sluggish 2003 to 2007.
For the NAEP data in the following slides, go to:
http://nces.ed.gov/nationsreportcard/mathematics/
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
8
NAEP Grade 4 MathTrend in Average Scale Score
210215220225230235240245250
1990 1995 2000 2005 2010
NCUS
Cut for Basic = 214
Cut for Proficient = 249
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
9
NAEP Grade 8 MathTrend in Average Scale Score
250255260265270275280285290
1990 1992 1996 2000 2003 2005 2007
NCUS
Cut for Basic = 262
Cut for Proficient = 299
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
10
Finding the Effect Size aka, “Standardized Difference”• Effect size compares two groups
starting with a scale score.• For calculation of effect size, the
average scale score of each group is needed, as well as the standard deviation (SD) of the scores.
• Find the difference of the averages and divide by the SD.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
11
Comparing trends in tests: NAEP and SAT
• SAT math, comparing NC in 1990 and 2008.
…..Eff.Size = (511-470)/100 = 0.41• NAEP Grade 4 math, comparing NC in
1992 and 2007.
…..Eff.Size = (242-213)/30 = 0.97
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
12
Is that a “Big-un?”
General use Effect Size Indexes• 0.20 … Small• 0.50 … Medium• 0.80 … LargeFrom A Power Primer, Jacob Cohen,
Psychological Bulletin, 1992, Vol 112, No. 1, 155-159.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
13
While we’re talking SAT, SAT puts an “official” question of the day
on its web site.The math questions are great thinking
questions and they are mostly within our 8th grade territory. Why not have 8th graders use them as formative assessment for problem solving?
www.collegeboard.com
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
14
Effect Size and NC Tests• Effect size can provide a comparison
across test editions or subjects.• See the attached table for some
examples. …. The 3rd Edition Math EOG had
much more rigor than the 2nd. …. Effect size enables continuous
comparison.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
15
Obtaining Av. Scale Scores• The “disag” report is on-line.• Go to the Accountability web page. www.ncpublicschools.org/accountability
then scroll way down to Data and Reports, chose “State, School System (LEA) and school performance disaggregated data report.”
• Now choose location, test, grade, subj.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
16
Green Book
Find out about getting data from the Green Book and its online brother:
• both in its standard form …• and its interactive, electronic form.http://www.ncpublicschools.org/
accountability/testing/Scroll down to “State Testing Results”
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
17
Questions to pose ..
• In your LEA (or school) which grades, courses, SAT results are stronger?
• In your LEA how do schools compare on average EOG, EOC, SAT?
• How does your LEA (school) compare with similar LEAs (schools)?
• What is the effect of what you’ve been trying to improve?
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
18
Current ResearchThe Problem With “Proficiency”:
Limitations of Statistics and Policy Under No Child Left Behind. By Andrew Dean Ho, Educational Researcher, 2008; 37, 351
• “The limitations [of the Percentage of Proficient Students (PPS) statistic] are unpredictable, dramatic, and difficult to correct.” …. From abstract.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
19
Part BFuture Ready Core
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
20
ACT Study
… found that “whether planning to enter college or workforce training programs after graduation, high school students need to be educated in a comparable level in reading and mathematics.”
Ready for College and Ready for Work: Same or Different? 2006. Available on the web.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
21
Future Ready Core - GoalsOn September 7, 2006, the NC Board of Ed
approved goals: Future Ready Students for the 21st Century.
Note the first goal and the 1st item in that goal.“Goal: NC public schools will produce
globally competitive students. • Every student excels in rigorous and
relevant core curriculum that reflects what students need to know and demonstrate in a global 21st Century environment … ”
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
22
Future-Ready Core Course of Study
The State Board of Education in June 2007 approved the Future-Ready Core Course of Study with the goal that all students would graduate prepared for post-secondary education and careers.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
23
Future Ready Core Timeline
Beginning with the 2009-2010 ninth grade class, all students will be expected to meet the requirements outlined in the Future Ready Core Course of Study.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
24
FRC: Primary Differences.
• An additional math course for those students who previously did not pursue the College/University Prep Course of Study.
• An increase from 20 to 21 for the total number of credits required for graduation.
• A recommended 4 elective credits in a concentrated area.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
25
FRC: Mathematics4 Credits
(Algebra I, Geometry, Algebra II) OR (Integrated Math I, II, III) 4th Math Course to be aligned with the student’s post high school plans A student, in rare instances, may be able to take an alternative math course sequence as outlined under State Board of Education policy. Please see your school counselor for more details.
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
26
FRC - Information
• Information resources are linked from:
www.ncpublicschools.org/gradrequirements
click “Resources.”
PUBL
IC S
CHOO
LS O
F NO
RTH
CARO
LINA
STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION
27
RESOURCES (on dpi web site)• Course & Credit Requirements Chart• Course and Credit Requirements Checklists
for Parents• Graduating Future Ready• Making the Grade – Future Ready
Graduates• Math Sequence• Questions and Answers• Student Accountability Standards Brochure