Incorporating Standardized Tests and Modifying Policies to Measure Program Outcomes Bernice Brennan MSN,RN Andrea Schapire MSN,RN
Dec 29, 2015
Incorporating Standardized Tests and Modifying Policies to Measure Program Outcomes
Bernice Brennan MSN,RN Andrea Schapire MSN,RN
PRESENTATION OBJECTIVES
Historical perspective of standardized testing in nursing education
Overview of recent literature related to standardized testing and policies
Modification of policies/measurements Incorporation of standardized testing
from a student and faculty viewpoint Measurement of SLO(student learning
outcomes) and overall NCLEX results
KEY TERMS
NCLEX-RN scores Standardized Tests Student Learning Outcomes (SLO) External Assessment Programs Non-traditional students Computerized Adaptive
testing(CAT) Benchmarks
Historical Perspective A little bit about nursing history….. https://
www.youtube.com/watch?v=GQ2ni7VOobQ
Professional roots of nursing in the United States since 1870’s when the first school was organized in New York City
Scope of nursing practice for new graduates: Dispensing medications Providing physical and psychologic comfort
to the ill and infirm(Holstein 2006)
Historical Perspective Nursing care was based on standards
recognized across the country and on more stringent educational requirements.
Prior to 1982 in the US, the State Board Test Pool Examination was administered by individual states, then the NCLEX-RN replaced the Test Pool Exam.
The NCLEX exam was the accepted method of mandating that a nurse has met the minimum requirement for safe nursing practice
Historical Perspective 1941-1982 The State Board test pool
Examination (SBTPE)one exam used for all students (paper and pencil)
1978 NCSBN(National Council Licensure Exam) established. This organization renamed exam NCLEX in 1982
1986-1994- pilot studies conducted to determine sound, valid and legality of the NCLEX
Historical Perspective
First NCLEX on-line- April 1st 1994
(155,00 candidates took the NCLEX via computerized testing (CAT)
Review of the Literature An eight year review of the
literature utilizing CINAHL/Medline data bases were retrieved from : (2006-2014)
Temple University Ginsberg Health Science Library
Thanks to Rebecca LloydEducation Services Librarian
Review of the Literature Highlights (2006)Holstein- Standardized Testing
Tools to Support Quality Educational Outcomes- many schools of nursing integrated standardized computerized testing into the curricula to decrease the failure rate in the NCLEX-RN examination. Student’s at risk for failure in NCLEX-RN would have time for remediation
Predictors of success in NCLEX-RN was based on GPA’s in nursing courses, NLN Comprehensive test scores, Mosby’s Assess Test, E2
Review of Literature
Cont’d Holstein: Multiple researchers conducted
studies to identify predictors for NCLEX success but data not consistent, so schools of nursing turned to commercial companies for assistance through computerized testing that simulates the NCLEX-RN
Review of the Literature
(2008)-Richards and Stone: Student Evaluation of a Standardized Comprehensive Testing Program-report high level of satisfaction with testing, favorable SLO’s
Faculty gave points for classes with standardized tests Students given remediation and felt it was helpful
Review of the Literature
(2008) Spurlock,D,R. and Hunt,L,A. A study of the usefulness of the Hesi exam at predicting NCLEX-RN Failure. Many schools of nursing have implemented progression policies based on a single end-of-program predictive test.
Review of the Literature
(2010)- Harding, M. Predictability Associated with Exit Examinations: A Literature review
Only 16 studies have been done on Standardized testing. HESI most widely used standardized test with 86.8% predictability for success in NCLEX. Determined more research needed in this area.
Review of the Literature
(2011)- DeLima, L. Retrospective Study of AD Programs-Standardized test results and pass/fail rate in nursing courses to be the most significant variables identified in profiling students most likely to fail NCLEX. Males and minorities (African-American, Latino and Asian) had lower pass rate than their White counterparts.
Review of the Literature
(2011) Carr- NCLEX-RN pass rate peril: One school’s journey through curriculum revision, standardized testing, and attitudinal change.
Faculty are reluctant to use external exams to evaluate student’s ability and knowledge. Self-made standardized exams (faculty prepared) utilized in each clinical course. Remedial courses implemented for student’s not passing the exams
Review of the Literature
(2011) Alameida, M.D- et.al, Predicting NCLEX-RN Success in a Diverse Student Population- 589 students taking ATI Predictor from 2003-2009. Standardized testing had positive outcomes with first time attempt of NCLEX. Age not a predictor. Men failed exams at higher rates. Cumulative GPA’s and increased grades in nursing had impact on NCLEX success
*MH Pharm and Fund. predicted success.
Review of the Literature (2012) Staykova, P. Community college education through the
looking glass of Associate Degree Nursing- numerous barriers affected students success: inflexible schedules, busy curriculum, and insufficient socioeconomic status, minority students, and immigrants with limited English language skills.
Competent educators must be skilled at integrating assessment tools throughout the curriculum to help students pass NCLEX-RN on the first attempt.
Review of the Literature
(2013)-Ayers,V,D.- Finding the balance in testing-Improving performance and setting consequences: Survey 2013-Deans,Directors and Faculty
Questionaire looked at the barriers/benefits to standardized testing.
Review of the Literature
(2013)Santos,L.T et.al: The use of Standardized Exit Examinations in Baccalaureate Nursing education: looked at ethical, legal, cultural, socioeconomic and technological considerations of standardized testing
Each school needs to determine if one predictor can restrict student from graduating or taking the exam
Legality-high stakes testing Cultural-English as a second language (ESL
students). Socioeconomic Technological (space/students with disabilities)
Review of the Literature (2013) Halstead, J, The NLN’s
fair testing imperative and implications for faculty development:
20% of nursing schools are required to achieve a minimum score on ST to graduate from program and take the licensure examination, regardless of how well they mastered their required nursing courses
Con’td (2013) Halstead Many faculty, even experienced
faculty,find the development of valid and reliable means of evaluating student learning to be a challenge
Faculty must have access to comprehensive testing, administration, and evaluation information before they administer or write policies regarding use of standardized testing
Review of the Literature
(2013) Zweighaft,E. Impact of HESI Specialty exams: The 9th HESI Exit Exam Validity Study- 379 students all random schools and types of programs, HESI E2 can be confidently used to evaluate students for licensure exam readiness
Implementing as the final exam to students, the higher the grade the more successful the student.
Review of the Literature
(2014)- Coons, I,V,C. : Use of standardized tests within nursing education programs-Use of standardized exams has increased over several decades. May be related to satisfy various accreditation requirements (including annual state board of nursing pass rates).
Controversy due to tests being detrimental to minority students
Review of the Literature Cont’d Coons
Experts in the field of measurement need to be consulted by nursing faculty in relation to policies concerning standardized testing
Expectations and goals with implementation of standardized testing must be in line with nursing schools mission ,philosophy and frameworks
Incorporating Standardized Testing Policies
NEHSON: Standardized Testing (In Student Handbook) (2010-2011)
Standardized Testing Complete and achieve the minimal Proficiency
Level 2 for all exams at each nursing level. Remediation with faculty members after all
unsuccessful exams. Remediation of 100% on unsecured exams(Form
B) before repeating the secured exam a second time
Graduation Completion of all standardized tests with a
minimal Proficiency Level 2 Student must achieve 91% predictability on
Comprehensive Predictor Exam Then attend Live Review
Incorporating Standardized Testing Policies
(2012-2013) Standardized Testing Complete and achieve the minimal Proficiency Level 2 for
all exams at each nursing level. Remediation with faculty members after all unsuccessful
exams. Remediation of 100% on unsecured exams(Form B)
before repeating the secured exam a second time Graduation Completion of all standardized tests with a minimal
Proficiency Level 2 Student must achieve 91% predictability on
Comprehensive Predictor Exam Live Review
FEBRUARY 2012
Fair Testing Guidelines implemented (NLN) Feb. 2012
Students notified of testing policies at meet and greet, prior to admission
Reviewed during orientation Freshmen Year
In each course syllabi where standardized testing is to be completed
Remediation plan developed and implemented
Incorporating Standardized Testing Policies
(2013-2014) (Provisional Status) Standardized Testing Complete and achieve the minimal Proficiency
Level 2 for all exams at each nursing level. Remediation with faculty members after all
unsuccessful exams. Remediation of 100% on unsecured exams
(both A and B Forms)before repeating the secured exam a second time.
Focused assessments to be completed for each individual student on unsuccessful proctored exam before taking it the 2nd time.
2013-2014 Focused assessments to be completed for each individual
student on unsuccessful proctored exam before taking it the 2nd time.
No more than one ATI proctored exam per day- can not remediate and take the same day.
Fundamentals assessment unsecured tests done at holiday break- a minimum of 3 hours of remediation to be done prior to taking proctored exam.
A fee to be charged for any exam after the 2nd attempt Graduation Completion of all standardized tests with a minimal
Proficiency Level 2 Student must achieve 94% predictability on Comprehensive
Predictor Exam A readiness program and individualized remediation
developed for each student and detailed report signed prior to graduation.
Live Review Capstone Implemented
2014-2015 (Full Approval-off Provisional)
Standardized Testing Complete and achieve the minimal Proficiency
Level 2 for all exams at each nursing level. Remediation with faculty members after all
unsuccessful exams. Remediation of 100% on unsecured exams (both
A and B Forms)before repeating the secured exam a second time.
Focused assessments to be completed for each individual student on unsuccessful proctored exam before taking it the 2nd time.
No more than one ATI proctored exam per day- can not remediate and take the same day.
2014-2015 (Full Approval-off Provisional)
Fundamentals assessment unsecured tests done at holiday break- a minimum of 3 hours of remediation to be done prior to taking proctored exam.
A fee to be charged for any exam after the 2nd attempt Fair Testing Guidelines implemented (NLN)Feb. 2012 Students notified of testing policies at meet and greet , prior to
admission Reviewed during orientation Freshmen Year In each course syllabi where standardized testing is to be completed Remediation plan developed and implemented Graduation Completion of all standardized tests with a minimal Proficiency Level 2 Capstone Student must achieve 97% predictability on Comprehensive Predictor
Exam A readiness program and individualized remediation developed for
each student and detailed report signed prior to graduation.
Comparative Analysis using Standardized testingClass of 2014-2015
Fundamentals Assessment
CLASS OF 2014 N=42
Individual Score Required
CLASS OF 2015 N=38
Adjusted group score 68.4%
Level 3 80-100%Level 2 68.3-78.3%Level 1 58.3-66.7%Below Level 1 <58.3%
Adjusted Group Score 68.6%
Proficiency Level 3 N=1
Proficiency Level 3 N=1
Proficiency Level 2 N=22
Proficiency Level 2 N=21
Proficiency Level 1 N=17
Proficiency Level 1 N=15
Below Level 1 N=2 Below Level 1 N=1
% of group Above Ind. Mean program54.8%
% of group Above Ind. Mean Program57.9%
Comparative AnalysisClass of 2014-2015
Fundamentals AssessmentCLASS OF 2014 MAJOR
CONTENT AREAS
CLASS OF 2015
Sub-Scale Group Score
Subscale Group Score
Safety Inf. Control 61.2%
Safety Inf. Control 61.8%
Health Pro&Maint 64.3%
Health Pro&Maint 64.3%
Psychosocial Integrity 58.1%
Psychosocial Integrity 58.1%
Basic Care and Comfort 58.7%
Basic Care and Comfort 59.4%
SLOCLASS OF 2014 CLASS OF 2015
THINKING SKILLS THINKING SKILLS
CLIN.JUDGEMENT/CRITICAL THINKING IN NURSING 67.2%
CLIN.JUDGEMENT/CRITICAL THINKING IN NURSING 69.5%
PRIORITY SETTING GROUP SCORE 68.2%
PRIORITY SETTING GROUP SCORE 68.8%
NURSING PROCESS NURSING PROCESS
PLANNING GROUP SCORE 74.8%
PLANNING GROUP SCORE 76.3%
EVALUATIONGROUP SCORE 60.5%
EVALUATIONGROUP SCORE 63.4%
EVALUATION OF DATAClass of 2015
SAFETY AND INFECTION CONTROL INCREASED BY 0.4%
HEALTH PROMO AND MAINT INCREASED BY 0.7%
PSYCHOSOCIAL INTEGRITY INCREASED BY 8.7%
BASIC CARE AND COMFORT INCREASED BY 0.7%
Class of 2015 adjusted score increased 0.2% from 2014 class
Class of 2015 2 less students with Proficiency Level score 1
Class of 2015 Individual Mean for Program increased 3.1%
STUDENT LEARNING OUTCOMESGroup Scores Class of 2015
Clinical Judgment/Critical Thinking
2.3% increase
Priority Setting 0.6% increase
Planning 1.5% increase
Evaluation 2.9% increase
FACULTY VIEWPOINT
All faculty needs to be committed to the process
Why were faculty non-supportive• Lack of technical skills• Unknown concepts• Threat to autonomy of teaching• Unsure how to remediate • Unsure how to evaluate results and
read individual/group reports
INTERVENTIONSFACULTY ORIENTATION
Live in-service from testing vendor representative Faculty supporting other faculty
SCHEDULED WEBNARS FOR ALL FACULTY Monday afternoons (7 topics) One hour in length TOPICS: Anatomy/Physiology Learning Strategies Active Stack Pharm Flash Cards Nursing Touch Manager Certificate Program Dosage Calculations Reporting/Remediation
STUDENT VIEWPOINTSCLASS OF 2014
More work, More stress Did not take it seriously No grade or value to it in their
mindset Timely burden Unable to understand their results Anger
Class of 2015 Implementation Strategies…
Early introduction to test taking skills Test reviews (theory )and how it is
looked at from a test taking strategy Positive reinforcement to students
explaining standardized testing is a skill for many
Faculty cheerleading students to success
Remediation for those identified early on with difficulty
Class of 2015 Implementation Strategies…
Students oriented to standardized testing in the classroom
Emphasis placed on test taking strategies Prior to taking proctored assessments
students required to do practice assessments A & B only once. A focused review will be done for each assessment minimum 3 hours for each total 6 hours.
Scheduled for proctored assessment, needed to have transcript with time spent on focused review as ticket to take proctored assessment.
ATI Capstone Course IntegrationCLASS OF 2014-New
During the senior semester and the Leadership and Management in Nursing Course, all 33 students were enrolled in the ATI Capstone Course.
Students were required to complete all components of the Capstone Content Review.
On 2/5/14, students were given the 2010 Comprehensive Predictor
Then they followed with nine weeks of content review. (Capstone)
On 3/26/14, the students completed the repeat of the 2010 Comprehensive Predictor
Capstone Guidelines
Successful completion of the Capstone Content Review will include:
completion of each content assessment
focused review quality post-assessment
assignments. Post-Capstone Predictor
Dr. Michelle M. Murphy-Rozanski, PhD, MSN, RN, CRNP
Name M/S Nursing 2nd M/S Post Pharmacology 2nd Pharm Post Comp Pred #12nd Predictor Post-Capstone
Increase in Proficiency
Proficeincy Level Proficeincy Level Percentage Percentage 76=97% Post CapstoneJ.A. 1 1 1 2 57.3 65.3 8S.A. 1 1 1 1 68 74.7 6.7A.B. 3 3 2 3 76 82.7 6.7M.B. 3 3 2 2 58.7 64 5.3B.C. 2 3 2 3 76.7 77.3 0.6M.C. 1 1 2 1 62 80 18B.C. 1 1 2 2 70 78 8S.E. 2 2 2 2 65.3 72.7 7.4N.E. 3 3 2 2 76.7 80.7 4B.G. 3 2 3 2 78.7 77.3 -1.4T.G. <1 1 1 2 60 58 -2A.J. 2 2 2 2 69.3 78 8.7D.J. 1 2 1 2 66.7 76 9.3
E.J. 2 3 2 2 76 70 -6D.L. 3 3 2 2 74.7 84 9.3T.M. 1 3 2 3 68 72.7 4.7E.M. 1 1 1 1 53.3 64.7 11.4D.M. 3 3 2 2 76.7 74 -2.7R.N. 2 2 2 2 66.7 62 -4.7W.N. 2 2 2 2 62.7 72 9.3A.P. 2 2 2 2 70 75.3 5.3T.R. 2 2 2 2 70 73.3 3.3S.R. 1 2 2 2 62 76 14F.R. 2 2 2 2 72.7 75.3 2.6E.R. 1 2 1 1 56.7 68 11.3D.S. 2 3 2 2 74 79.3 5.3K.S. 1 3 2 2 78 70 -8V.S 3 2 2 3 74 72.7 -1.3K.S. 2 2 2 2 68.7 76 7.3A.S 1 2 2 1 70.7 67.3 -3.4D.T. 2 2 1 1 70 73.3 3.3E.T. 1 2 1 1 62 71.3 9.3T.W. 3 3 2 2 73.3 75.3 2
STUDENT LEARNING OUTCOMES POST
CAPSTONE
Overall total N=25 179.12 points increased after Capstone on 2nd Predictor attempt
Overall mean 7.16 points per student
Range 0.6-14 points per student
6 students that failed NCLEX-RN on the First Attempt
All failed on the FIRST ATTEMPT
Medical-Surgical*** Pharmacology Comprehensive Predictor
NEHSON NCLEX-RN Pass Rates
Class of 2014 had 33 Students
27 out of 33 Passed on the first attempt
Overall Pass Rate = 81.81%
Off Provisional!!!!
https://www.youtube.com/watch?v=29fdVOqraQs&list=RDEN5zt7c7jHM&ind