Development of the Computerized Model of Performance …

CONTINUING EDUCATION2.5ANCC

CONTACT HOURS

Development of the Computerized Model ofPerformance-Based Measurement System to MeasureNurses’ Clinical CompetenceShwu-Ru Liou, PhD, Hsiu-Chen Liu, MSN, Shu-Ling Tsai, MSN, Ching-Yu Cheng, PhD, Wei-Chieh Yu, PhD, Tsui-Ping Chu, MS

Au(DGu

Thfo

Than

CoTa

Vo

Critical thinking skills and clinical competence are for pro-viding quality patient care. The purpose of this study is todevelop the Computerized Model of Performance-BasedMeasurement systembased on the Clinical ReasoningModel.The system can evaluate and identify learning needs forclinical competency and be used as a learning tool to in-crease clinical competency by using computers. The systemincludes 10 high-risk, high-volume clinical case scenarioscoupled with questions testing clinical reasoning, interper-sonal, and technical skills. Questions were sequenced toreflect patients’ changing condition and arranged by follow-ing the process of collecting and managing informa-tion, diagnosing and differentiating urgency of problems,and solving problems. The content validity and known-groups validity was established. The Kuder-RichardsonFormula 20 was 0.90 and test-retest reliability was sup-ported (r = 0.78). Nursing educators can use the systemto understand students’ needs for achieving clinical com-petence, and therefore, educational plans can be made tobetter prepare students and facilitate their smooth transi-tion to a future clinical environment. Clinical nurses can usethe system to evaluate their performance-based abilitiesand weakness in clinical reasoning. Appropriate training pro-grams can be designed and implemented to practically pro-mote nurses’ clinical competence and quality of patient care.

KEY WORDS: Computerized measurement, Clinicalcompetence, Nurse, Reliability, Validity

iring and retaining new graduate nurses is oneof the most significant and urgent solutions for theever-increasing shortage of nurses. New graduate
H nurses are expected and required to work effec-tively in current complex healthcare environments
from healthcare authorities, nurse colleagues, and health

thor Affiliations: College of Nursing, Chang Gung University of Science and Technologyr Liou, Ms Liu, Ms Tsai, Dr Cheng, and Dr Yu); and Department of Nursing, ChiaYi Changng Memorial Hospital, Chiayi, Taiwan (Ms Chu).

e authors acknowledge the Chang-Gung Medical Research Program (CMRPF6B0083) in Taiwanr funding this study.

e authors have disclosed that they have no significant relationship with, or financial interest in,y commercial companies pertaining to this article.

rresponding author: Ching-Yu Cheng, PhD, RN, 2 Chiapu Rd, West Sec, Putz, Chiayi, 61363iwan ([email protected]).

lume 34 | Number 4

Copyright © 2016 Wolters Kluwer H

professional teams to provide quality care for patients. How-ever, researchers worry and propose that new graduatenurses are not adequately prepared for performing in thereal world to meet the growing demands of today’s complexhealthcare environment, which compromises the quality ofpatient care.1–4 In addition, an online survey from morethan 5700 frontline nurse leaders showed that many nurseadministrators and nurse leaders were not fully satisfied withnew graduate nurses’ performance and imply that new grad-uates had difficulty with thinking critically on their feet or man-aging patient situations with quick and appropriate decisionmaking.4 To close the gap between education and practice,researchers in academia and clinical practice collaborated anddeveloped learning modules using patient conditions and otherlearning strategies to enhance clinical competency.2

It has been suggested that nurses need to prepare and pos-sess certain abilities or competencies to perform tasks thatwill achieve expected outcomes under various conditions inthe real world.5 The concepts of competence, competency,and performance have been widely discussed in the nursingprofession. Distinctions among these concepts were reportedindicating “competence” as a generic term, “competency” asspecific capacities, whereas “performance” as what could beobserved and measured with actual professional practice sit-uations.6,7 Benner5 asserted that nursing competence assess-ment should be grounded in real practice with situationalcontext, under pressure, and over time. Competence also in-cludes performing the necessary nursing functions safely, in-dependently, and critically. In addition, Marshburn et al8

defined performance-based measurement as evaluation ofnurses’ responses to problem management and communica-tion under given real clinical situations. All of these statementsimply that being competent in clinical nursing practice de-mands the application of integration of knowledge, criticalthinking, communication, and technical skills.9 Based on thesedefinitions, in the current study, a performance-based com-petence measurement system was developed that uses realclinical situations as cases to evaluate nurses and pregrad-uates’ abilities in responding to patients’ problems by applyingknowledge, critical thinking skills, communication skills, andtechnical skills. While designing the measurement system,

CIN: Computers, Informatics, Nursing 159

ealth, Inc. All rights reserved.

mailto:[email protected]

CONTINUING EDUCATION

emphasis was placed on how to measure skills that requireactual practice.

Clinical competence can be measured based on nurses’own perception of performance. The performance-basedmea-surement system developed by del Bueno contains videotapedscenarios and simulations and can be used to evaluate nursecompetency. Test takers are asked to write down patient prob-lems, interventions, and rationale after viewing the videos.10,11

Other than that system, nurses’ clinical competence was mostlyassessed by asking about their perceptions of their clinicalperformance using Likert-type questions.12 However, con-troversies exist in the literature regarding using nurses’ self-perception to assess competency.8 Using survey instrumentsor focus group interviews to collect data regarding clinicalcompetence, previous studies found that some nursing stu-dents or new graduate nurses appeared overconfident orfaked confidence in their clinical competence.9–13 Similarly,studies revealed that new graduate nurses self-reported thehighest values for their clinical competence,14,15 yet it has alsobeen found in clinical settings that new graduate nurses werenot adequately prepared to provide direct patient care formorethan five clients.16Del Bueno3 reported that in a performance-based evaluation, when graduate nurses’ clinical competencewas examined, only 35% of new graduate nurses could meetjob expectations related to interpersonal skills, technical skills,and critical thinking at the time of their initial employment.The problem may lie in new graduate nurses’ insufficientopportunities to perform in real situations, which limits thegrowth of their ability to perform in real situations.

The Institute of Medicine has encouraged all health-relatedprofessions to develop effective methods to ensure that theirprofessional staffs are competent in furnishing safe patientcare.17 To provide safe patient care, nurses are required to becompetent in practice. New approaches, therefore, are neededtomeasure nurses’ clinical competence in providing safe patientcare, especially in the current complex healthcare environment.These approaches must be based on a theoretical backgroundand objectively evaluated to determine their effectiveness priorto implementation, which forms the basis of this study.

PURPOSEBecause of the importance of understanding and enhancingnew graduate nurses’ competence and the lack of performance-based measures of competence, the purpose of this study isto develop a Computerized Model of Performance-BasedMeasurement (CMPBM) system and test its psychometricproperties. The CMPBM system functions in two ways.First, the CMPBM can evaluate and identify learningneeds for clinical competency. Second, the CMPBM canbe used as a learning tool to increase clinical competency.When used by nursing students, their learning needs can beidentified and they can be better prepared for clinical

160 CIN: Computers, Informatics, Nursing


practice, which facilitates a smooth transition to a futureclinical environment. Clinical nurses can use the CMPBMto identify areas for improvement.

The development and validation of the CMPBM systemoccurred in three phases: developing the system based on atheory, pilot testing the system, and psychometrically evalu-ating the CMPBM system. The study was conducted afterethical approval was obtained from a local research ethicscommittee in Taiwan.

METHODSPhase 1: Development of the Computerized Model ofPerformance-Based Measurement SystemIt is important to develop a measurement system based on atheoretical model that can help to identify specific conceptsor competencies and how they are to be measured. Failureto do so can result in a lack of validity of measuring instrumentsand a misinterpretation of collected data.18,19 In addition,applying theories to develop structured tools for assessing in-tervention operation and outcomes can assist implementersin planning and thinking through particular interventions.20

Theoretical Background for the Computerized Model ofPerformance-Based Measurement System

The Clinical Reasoning Model developed by Levett-Joneset al21 was used to construct the CMPBM. This model hadbeen applied for developing learning programs.22–24 In thistheory, clinical reasoning was characterized as “a logical pro-cess by which nurses collect cues, process the information,come to an understanding of a patient problem or situation,plan and implement interventions, evaluate outcomes, andreflect on and learn from the process.”21(p516) This proce-dure model involves higher-order thinking methods that in-tegrate critical thinking skills such as asking questions andcritiquing solutions, problem-solving that focuses on identifi-cation and resolution, and clinical decision making that con-centrates attention on making choices between options indynamic and complex contexts as to a course of action.25 Theseskills must develop concurrently to produce reasoning, clari-fication, and potential solutions.26

The Clinical Reasoning Model proposes that clinical rea-soning is a process that involves eight steps: (1) the “look” stepprovides patient information such as contextual facts to allownurses to consider the patient’s situation, (2) the “collect” stepguides nurses to collect and review patient cues with recalledknowledge, (3) the “process” step employs reasoning skills tomanage the collected information, (4) the “decide” step appliessyntheses of facts and inferences to recognize and prioritize/rationalize patient problems/issues, (5) the “plan” step involvesestablishing goal of care, (6) the “act” step concerns selecting acourse of action for patient care, (7) the “evaluate” step involvesevaluating the effectiveness of actions and outcomes, and (8) the

April 2016


“reflect” step encourages nurses to reflect on actions.21 How-ever, to be concise in this study, we modified and categorizedthese steps into three major stages, with each major stagecontaining several original steps. The three major stages andsubsections were (1) collect and manage information, whichconsiders the patient situations, collects and reviews cues/information, and processes information; (2) diagnose and dif-ferentiate urgency of problems, which includes recognizingand prioritizing/rationalizing problems; and (3) solve problems,which includes establishing goals, taking actions, evaluatethe effectiveness of actions, and reflect on actions (Figure 1).

Because the CMPBM system can also be used as a learn-ing tool, constructivism is applied for the development of thesystem. Constructivism is a learning theory that proposesthat learning results from activity and self-organization.27 Itstresses that knowledge is constructed by learners usingexisting knowledge and experiences rather than beingtransferred directly from the real world to learners’ memo-ries. Learning is controlled by the learners, and learners’capability in manipulating information is emphasized. Ac-tive learning, interpretation of information, and constructionof knowledge are other important points of constructivism as

FIGURE 1. Development of the clinical scenarios for the CMPBM sys

Volume 34 | Number 4


well.28 Case scenarios and test questions in theCMPBMweredesigned based on these conceptions.

Design and Construction of the Computerized Model ofPerformance-Based Measurement

The CMPBM consists of clinical case–based scenarios coupledwith questions to be answered. Three major dimensions ofcompetence (clinical reasoning skills, interpersonal skills, andtechnical skills; Figure 1) were embedded in the clinical sce-narios according to literature reviews. Clinical case–basedscenarios were first established and then the CMPBM wasthen computerized. Computerization of the CMPBM allowslearners to learn and test without time and space limits. In-structions about how to use the CMPBM are embedded inthe system, and therefore, no instructors are needed whileusing the system. However, experts and instructors partici-pated in the development of the system.

Clinical Case–Based Scenarios

Four senior clinical experts and instructors were invitedto develop the scenarios. These experts and educators de-signed the cases based on real situations and treatments in

tem based on the clinical reasoning model.21




clinical settings to measure test takers’ clinical performancecompetence in a real context. According to literature, 10 high-risk, high-volume, medical-surgical related cases were selectedto develop scenarios for the measurement of clinical compe-tence: acute myocardial infarction, brain stroke, chronic ob-structive pulmonary disease, diabetic ketoacidosis, end-stagerenal disease, liver cirrhosis, diabetic mellitus, cancer care,hypovolemia, and postoperation pain management. Figure 2depicts part of a sample case scenario with basic information.Each scenario includes subsituations coupled with questionsthat can stimulate the test takers to think deeply and broadlyaccording to the case’s changing conditions. Those subsit-uations and questions were sequenced to reflect a patient’schanging condition and deterioration. Test takers have toapply their experiences and existing knowledge and knowl-edge learned in case situations to answer questions. At theend, test takers solve patients’ problems and provide holistic

FIGURE 2. Screen shot of the basic information for one case scenari



care by applying clinical reasoning steps proposed by theClinical Reasoning Theory. In addition to scenarios and testquestions, participating experts and educators provided an-swers and rationale to those questions for test takers to learnappropriate skills. Educators also help correct and grade testtakers’ answers to open-ended questions.

A total of 17 questions were developed for each scenariocase: five questions reflect information about the collection andmanagement stage, six questions are for the diagnosis and dif-ferentiating urgency of problems stage, and six questions reflectthe problem-solving stage. To enhance test takers’ logicalthinking and expression abilities as well as encourage theirparticipation in the scenario cases, different question typeswere designed, includingmultiple-choice, multiple-selection,and narrative questions.

Because the healthcare system is complicated and advancesin complexity each day, interpersonal relationships and

o.

April 2016


technical skills are especially important for nurses to providesafe care for patients. Test takers’ interpersonal relationshipskills are evaluated by scenarios of nurses’ conflicts withhealthcare teams, clients, and clients’ families using open-ended questions. For example, nurses need to write their re-sponse to the problem “I don’t want to be cared for by thatnurse.”Each test question for technical skills includes problempictures, and test takers need to narrate how to safely and ef-fectively perform procedures and use of equipment to solveproblems (Figure 1). For example, nurses are asked to identifycommon intravenous (IV) infusion and blood transfusion prob-lems such as air bubbles in IV lines and infiltration and how theproblems could be solved.

Construction Principles of the Computerized Model ofPerformance-Based Measurement

To develop this interactive evaluation tool, the following eightcharacteristics were incorporated into the CMPBM system.The entire system design is shown in Figure 3. The systemincludes eight characteristics as described below.

FIGURE 3. Construction of the CMPBM system.


Copyright © 2016 Wolters Kluwer Hea

1. Web-computer-assisted test.An interactive com-puterized platform, which is a free-of-charge softwareprogram, is modified to build the CMPBM system con-sisting of scenarios. All scenarios and test items arestored in a database on a server.

2. Enable test takers to think critically.The CMPBMsystem is designed to encourage and provide an oppor-tunity for participants to construct knowledge and de-velop critical thinking skills through active engagementin interacting, thinking, and decisionmaking for the casesthat are situated, experiential, and authentic. Participantsare required to read through scenarios and answer ques-tions by “thinking like an experienced nurse.”

3. Repetitive practice. The computerized scenarios areformatted to help participants achieve competence whilethey are allowed to make mistakes. To achieve this goal,the CMPBM system is designed to allow participants toredo the scenarios and questions as many times as needed.

4. Reinforcing learning. Each subsituation of a casescenario is described first, followed by several ques-tions. Each question needs to be answered correctly


lth, Inc. All rights reserved.


before participants can proceed to the next question.This design reinforces participants to learn concur-rently while taking the test and helps participants toconstruct related knowledge and clinical reasoning ex-periences based on that scenario case.

5. Immediate explanation of feedback.All questionsin each situation are coupled with immediate explana-tion feedback on correct and incorrect answers. It hasbeen suggested that Web learning that incorporates im-mediate feedback was critical to learning as it promotedand encouraged learners’ motivation to actively partici-pate in and finish the learning.29

6. Providing the score.The questions in a situation aredesigned with four to five answer choices for participants.The CMPBM system is designed to give a score foreach answered question in which questions that areanswered correctly in different sequences are givendifferent scores. For example, when the question is an-swered correctly the first time, the computer system au-tomatically scores it with 5 points; at the second time,3 points are given; at the third time, 2 points are scored;and at the fourth time and more, 1 point is given.

7. Totaling and storing scores. The system is de-signed to automatically calculate the total score thateach student obtains for the entire test. The scores weredivided into four levels: excellent (scoring 95%-100%of total score), proficient (scoring 85%-94%), satisfac-tory (scoring 70%–84%), and unsatisfactory (below70%). All scores are stored in the database.

8. Retrieving and analyzing data. Because questionsare set up following the eight steps of clinical reasoningin the Clinical Reasoning Model, the system is able tocalculate and categorize the score earned by participantsaccording to those steps. Therefore, nurse educators ornurse administrators can analyze and understand whataspects of ability need to be reinforced in the partici-pants. Similar to the total score of the entire test, scoring95% to 100% of the total score for each step is consid-ered excellent, whereas 85% to 94% is proficient, 70%to 84% is satisfactory, and below 70% is unsatisfactory.

The characteristics of the system allow participants to workthrough the scenario cases while following the clinical reason-ing steps of the Clinical Reasoning Model at their own pace.This method allows participants to move through increasinglycomplicated operations and also asks them tomake decisionsat a critical junction in the scenarios. This enables participantsto construct their own learning while proceeding smoothlyfrom one step to the next.30

Expert Survey for Content Validity

Establishing content validity is important, as it concerns theactual relevance of the items to the content of the concept



being measured.31 Three nurse experts were invited to eval-uate the content validity of the scenarios for the CMPBM inpaper form. The definition of competence and three do-mains of competencies were provided to the experts. Theseexperts, who possessed an average of 10.2 years of clinicalexperience, were asked to read and rate all scenarios andquestions for relevance as “not relevant, somewhat rele-vant, quite relevant, or very relevant” to the defined compe-tence. They were also requested to provide their suggestionsabout the item description.2,32 The content validity for thesescenarios was confirmed. All the experts responded that eachscenario and situation designed in the CMPBM was eitherquite relevant or very relevant to the test of nurses’ perfor-mance competence needed for patient care in the real world.Results showed that both the item content validity index (CVI)and the scale CVI calculations were 1.0, which indicated evi-dence of content adequacy.33 In addition, all experts eitherresponded with strongly agree or agree with closeness of eachcase scenario to clinical patient situations, importance to clin-ical care, and meaningful learning to patient care.

Phase 2: Pilot TestThe purpose of this phase of study was to initially test thepsychometric properties of the CMPBM, whose purposewas to evaluate clinical competence, in its paper form. In-ternal consistency and test-retest stability were tested for reli-ability, whereas validity was tested using known-groups validity.The two groups for known-groups validity test included nursingstudents and clinical nurses currently working in hospitals.

Participants and Sampling

With the use of a convenience sample, nursing undergraduateswere contacted through flyers and students’ networks. Thirtystudents participated and completed the CMPBM tests. Thisnumber met the recommendation that 30 participants arerequired as a minimum sample size for the psychometric testof initial tool development.34 In addition, another group of30 experienced registered nurses who were working at adultmedical-surgical related units (ie, surgical or medical unit,emergency room, and intensive care units) were recruited totest the known-groups validity of the CMPBM. The purposeand procedures of this pilot study and participants’ rights weredescribed in a cover letter and also verbally explained to theparticipants. All participants were aware that they had the rightto stop answering any questions or participating in this studywithout any punishment. However, completing and returningthe test indicated that they agreed to participate in this study.

All undergraduate participants were women and were intheir last year of school. Their average age was 21.24 years.Ninety percent of them planned to work in the hospitals asclinical nurses and 89.65% preferred to work within 3 monthsafter graduation; 55% of them expected to have a master’sdegree as their final academic degree. The 30 experienced

April 2016


nurses had an average of 12.83 years of clinical experienceand a mean age of 35.57 years.

Data Analysis

Statistical Package for the Social Science 15.0 (SPSS Inc,Chicago, IL) was applied to manage and analyze data. De-scriptive statistics such as frequency, mean, and standarddeviation were used to analyze demographic data andeach test item. Using Pearson correlation, 2- to 4-week test-retest reliability was tested. The correlation of test and retestresults must be higher than 0.20 across a 1-month period toshow an acceptable reliability.26,35 Known-groups validity wasused and calculated using independent t test to assess the dif-ferentiation ability of the CMPBM between undergraduatesand experienced clinical nurses. According to Benner’s model,5

new graduates are regarded as novice or advanced beginner,and it takes at least 2 years in the same job for a nurse to reachthe level of competency.Therefore, clinical nurses’ score on theCMPBM was assumed to be higher than the students’ scores.

Preliminary Psychometric Test Results

The finding of the test-retest reliability demonstrated a signif-icant correlation (r = 0.70, P < .01). The significant correlationwas also found in the subsections collect and manage informa-tion (r = 0.54, P < .01), diagnose and differentiate urgency ofproblems (r = 0.73, P < .01), and solve problems (r = 0.48,P< .01). Therefore, the stability of theCMPBMwas confirmed.

In addition, known-groups validity was calculated by com-paring the competence scores of nursing students and expe-rienced clinical nurses. According to Benner’s “FromNoviceto Expert” model, the investigators logically hypothesizedthat more experienced nurses had higher scores on theCMPBM scenarios test than did less experienced nursingstudents. The results showed that undergraduates in theirfinal school year had a significantly lower score than did ex-perienced nurses (t = −4.63, P < .001). The known-groupvalidity was also shown on the subsections collect and man-age information (t=−5.27, P< .001), diagnose and differen-tiate urgency of problems (t = −4.56, P < .001), and solveproblems (t = −2.32, P = .02). This result indicated that theCMPBM scenarios test can discriminate between two groupswho differ in their present status.36 Therefore, the known-groups validity was preliminarily established.

Phase 3: Main Study

Design, Participants, and Sampling

Cross-sectional research design was employed for this phaseof the study. An independent sample from the pilot study wasrecruited using convenience sampling. Clinical nurses whowere working in hospitals during the study time frame and nurs-ing students in their final school year were recruited. Explana-tions regarding the research ethics and participant recruitment



procedures were the same as in the pilot study. A total of 180nursing undergraduates at one university were contacted, and157 of them completed the CMPBM tests, with a responserate of 87%. Sixty clinical nurses were contacted, and 52of them completed the tests, with a response rate of 86%.

Data Analysis

Descriptive statistics, such as percentage, mean, and stan-dard deviation were applied for demographic data analysis.Because all questions were coded as 0 for not answering cor-rectly and 1 for answering correctly, the Kuder-RichardsonFormula 20 (KR-20) was applied for internal consistency ofthe CMPBM tests, whereas test-retest reliability using Pearsoncorrelation was applied to examine the stability of the tests.According to Nunnally and Bernstein,37 a value of 0.70 ofKR-20 is sufficient for early stages of tool development forthe reliability coefficient. For the validity test, construct va-lidity, which was regarded as the most theoretical validitytype,38 was tested using the known-groups validity. An inde-pendent t test comparing the difference in scores on theCMPBM between nursing pregraduates and clinical experi-enced nurses was applied for the known-groups validity.

The study authors made a logical assumption that the ex-perienced clinical nurses would have higher scores on theperformance-based skills than would the nursing pregraduateswho have less experience working in real clinical settings.

RESULTS OF THE MAIN STUDYDemographic DataAs shown in Table 1, the majority of the participants werewomen. The clinical nurses were older than the undergrad-uates. The clinical nurses had a mean clinical experience of5.80 years. Most of the clinical nurses had a baccalaureatedegree in nursing, whereas more than one-third of the stu-dents expected to have a master’s degree. The majority ofthe students wished to work in medical-surgical related unitsin hospitals. The three skills that both students and clinicalnurses were the least confident in performing included elec-trocardiogram (EKG) reading, cardiopulmonary resuscita-tion (CPR), and venipuncture.

Reliability and Validity of the Computerized Model ofPerformance-Based Measurement SystemThe reliability of the CMPBMwas assessed using the KR-20and test-retest analyses. The KR-20 reliability for the entireCMPBM was 0.90 and for the three major subsections was0.76 (collect and manage information), 0.79 (diagnose anddifferentiate urgency of problems), and 0.71 (solve problems).Test-retest reliability analyses revealed a significant correla-tion between those two tests for the entire scale (r = 0.78,P< .001) and all three subsections: collect andmanage infor-mation (r = 0.73, P < .001), diagnose and differentiate



Table 1. Characteristics of the Participants in theMain Study

Variables Mean ± SD

Age, y 23.68 ± 5.22

Clinical nurses 29.75 ± 7.63

Nursing pregraduates 21.68 ± 0.97

Years of clinical experience (clinical nurses) 8.30 ± 6.58

n Valid %

Clinical nurses

Female 52 100

Education

5-Year nursing college 13 25

4-Year regular program 14 26.93

2-Year RN-BSN program 24 46.15

Master's 1 1.92

Satisfaction with clinical performance

Strongly dissatisfied 2 3.85

Dissatisfied 3 5.77

Fair 21 40.38

Satisfied 26 50

Top five less-confident skills to perform

CPR 47 90.83

EKG reading 45 86.54

Venipuncture 28 53.85

Chest tube drainage and wound care 23 44.23

Documentation 18 34.62

Nursing pregraduates

Sex

Female 148 94.27

Male 9 5.73

Expected final academic degree

Undergraduate 78 52.70

Master's 56 37.84

Doctorate 14 9.46

Missing 9 5.29

Work institution after graduation

Hospital 138 93.24

Long-term care facility 2 1.35

Outpatient clinic 1 0.68

Profession other than nursing 7 4.73

Missing 9 5.73

continues

Table 1. Characteristics of the Participants in the MainStudy, Continued

Variables n Valid %

Work unit in hospital/clinic after graduation

Intensive care unit and emergency unit 55 38.20

Medical-surgical unit 58 40.28

Obstetrics-pediatric unit 19 13.19

Others (psychiatric, operation room, oncology) 12 8.33

Missing 13 8.28

Top five less-confident skills to perform

EKG Reading 133 85.81

CPR 103 66.45

Venipuncture 90 58.06

Blood transfusion 80 51.61

Starting intravenous line 65 41.94




urgency of problems (r= 0.75, P< .001), and solve problems(r = 0.69, P < .001).

Known-groups validity was tested.As shown in Table 2, thescores from the clinical nurse group on the entireCMPBM testswere significantly higher than the scores from the nursing pre-graduates. A significant difference was also shown in all threemajor subsections: collect and manage information, diagnoseand differentiate urgency of problems, and solve problems.

DISCUSSIONTraditionally, clinical competencies have been measured bynurses’ own perceptions. However, controversies exist in theliterature regarding using nurses’ self-perception to assesscompetency because it might be prone to overestimation.8–13

Therefore, the purpose of this study was to develop a theory-driven and performance-based clinical competence measure-ment, the CMPBM, and conduct its psychometric tests. Thevalidated CMPBM provides an evidence-based performanceoutcome measure and is expected to assess the performance-based competence of patient care in the real world amongnursing undergraduates and clinical nurses. It can train test

Table 2. Known-Groups Validity of the CMPBM in theMain Study

Mean ± SD t/P

Clinical Nurse Pregraduate

Entire scale 341.44±36.35 319.92±35.59 3.02/.003

Collect and manageinformation

114.44±12.31 106.48±13.82 3.08/.003

Diagnose anddifferentiateproblem urgency

119.88±17.62 111.68±15.43 2.50/.01

Solve problems 104.29±10.54 98.94 ± 10.62 2.55/.01

April 2016


takers to use clinical reasoning skills. This computerized mea-surement system was developed based on situations of realclinical patients, interrelationships among healthcare col-leagues, and clinical technical skills.

The pilot study, using a small sample size, furnished initialevidence for the validity and stability of the CMPBM throughCVI, known-groups mean difference test, and test-retest reli-ability. In the main study, internal consistency, stability, andvalidity of the CMPBM were also confirmed with a largesample size group. Construct validity of the CMPBMwas sup-ported by known-groups validity, which indicated that theCMPBMhad the ability to distinguish individuals who differin their present status. The KR-20 coefficient for the entireCMPBM system was 0.90, which is much higher than thesuggested acceptable requirement of 0.70.36 The test-retestreliability showing a significant correlation coefficient estab-lished the stability of the CMPBM system.

The CMPBM developed and psychometrically tested inthis study could objectively measure nursing pregraduates’and clinical nurses’ performance-based competence, whichis very different from the traditional measurements of subjec-tively perceived competence. In light of the increasinglycomplex healthcare environment, clinical professionals hy-pothesized that new graduate nurses are inadequately pre-pared for practicing in the real world.4 Thus, for the nursingundergraduates, the CMPBM is conducive to evaluatingnursing students’ performance abilities before their gradua-tion from academia. After identifying what skills are neededfor students to obtain clinical competence, plans can be madeto better prepare them and facilitate a smooth transition toa future clinical environment. On the other side, healthcareadministrators found that nurses possessed good content ofknowledge and adequate procedural skills but lacked thecritical thinking skills to respond immediately and appropriatelyin critical situations.3 Healthcare administrators can use theCMPBM to evaluate nurses in every level for their performance-based abilities. The measurement results can provide informa-tion regarding individuals’ weakness points of competence.Appropriate continuing education or training programs cantherefore be designed and implemented to cultivate and pro-mote nurses’ clinical competence and quality of patient care.

CONCLUSIONThis study offered evidence to propose a more effective wayto tailor nursing courses or training strategies for upcomingand current clinical nurses to improve their clinical competence.The developed theory-based CMPBM can measure clinicalperformance-based competence from the academic andclinical sides. The findings of the study preliminarily sustainthe usefulness of and evidence about this performance-basedmeasurement for objective evaluation of nurses’ clinical com-petence, which is essential for quality patient care.



LIMITATIONS AND RECOMMENDATIONSThis study only applied to a limited number of participantsfrom limited resources of school and healthcare institutions,and this limited the generalizability of the study results. Werecommend that further stages of studies include nursing stu-dents in other nursing programs such as the regular 4-yearbachelor’s program or 5-year diploma program, as well asstudents in other schools, to increase the power of generaliz-ability of the findings. In addition, recruiting and testingmore clinical nurses in every level from diverse hospitals isnecessary to gather stronger evidence about this measure-ment and analyze nurses’ clinical performance-based compe-tence status. Proportions of critical thinking, interpersonalskills, and technical skills domains needed to form compe-tency was not defined in the literature and must be furtherexplored. Through the expansion of participants, nurse edu-cators and healthcare administrators might find that differ-ences in level of competence may exist among students indifferent nursing programs or schools, nurses in different levels,and nurses from different levels of hospitals. Therefore, differ-ent teaching and training strategiesmay be necessary to improvestudents’ and nurses’ performance-based competence. Further re-search is recommended to compare differences and similaritiesbetween self-perceived and performance-based competencemeasurements and to validate those measures using individ-uals’ actual practices in clinical settings. The CMPBM can alsobe further developed to be a form of simulation training/test.

References1. Beyea SC, von Reyn LK, Slattery MJ. A nurse residency program for competence

development usinghumanpatient simulation. J NursesStaff Dev. 2007;23(2): 77–82.

2. Burns P, Poster EC. Competency development in new registered nursegraduates: closing the gap between education and practice. J Contin EducNurs. 2008;39(2): 67–73.

3. del BuenoD. A crisis in critical thinking.NursEduc Perspect. 2005;26(5): 278–282.

4. Berkow S, Virkstis K, Stewart J, Conway L. Assessing new graduate nurseperformance. J Nurs Adm. 2008;38(11): 468–474.

5. Benner P. From Novice to Expert: Excellence and Power in Clinical NursingPractice. Upper Saddle River, NJ: Prentice Hall; 1984.

6. Rethans JJ, Norcini JJ, Barón-Maldonado M, et al.The relationship betweencompetence and performance: implications for assessing practiceperformance. Med Educ. 2002;36(10): 901–909.

7. Eraut M. Concepts of competence. J Interprof Care. 1998;12: 127–139.

8. Marshburn DM, Engelke MK, Swanson MS. Relationships of new nurses’perceptions and measured performance-based clinical competence.J Contin Educ Nurs. 2009;40(9): 426–432.

9. Liou SR, Chang CH, Tsai HM, Cheng CY. The effects of a deliberate practiceprogram on nursing students’ perception of clinical competence.Nurse Educ Today. 2013;33(4): 358–363.

10. del Bueno DJ. Experience, education, and nurses’ ability to make clinicaljudgements. Nurs Health Care. 1990;11(6): 290–294.

11. Anthony CE, Del Bueno D. A performance-based development system.Nurs Manage. 1993;24(6): 32–34.

12. Li�cen S, Plazar N. Identification of nursing competency assessment tools aspossibility of their use in nursing education in Slovenia—a systematicliterature review. Nurse Educ Today. 2015;35(4): 602–608.

13. Brown B, O’Mara L, Hunsberger M, et al. Professional confidence inbaccalaureate nursing students. Nurse Educ Pract. 2003;3(3): 163–170.




14. Hengstberger-Sims C, Cowin LS, Eagar SC, Gregory L, Andrew S, Rolley J.Relating new graduate nurse competence to frequency of use. Collegian.2008;15(2): 69–76.

15. Salonen AH, Kaunonen M, Meretoja R, Tarkka MT. Competence profiles ofrecently registered nurses working in intensive and emergency settings.J Nurs Manag. 2007;15(8): 792–800.

16. National Council of State Boards of Nursing. Transition to practice. 2006.https://www.ncsbn.org/Vol_22_web.pdf. Accessed November 31, 2011.

17. Kohn LC, Donaldson MS. To Err Is Human: Building a Safer Health System.Washington, DC: National Academy Press; 1999.

18. Tian J, Atkinson NL, Portnoy B, Lowitt NR. The development of a theory-basedinstrument to evaluate the effectiveness of continuing medical education.Acad Med. 2010;85(9): 1518–1525.

19. Jacobson S. Evaluating questionnaire for use in clinical nursing research. In:Frank-Stromborg M, Olsen SJ, eds. Questionnaires for Clinical Health-careResearch. Boston, MA: Jones & Bartlett; 2004: 3–19.

20. Finch TL,Mair FS, O’Donnell C, Murray E,May CR. From theory to ‘measurement’in complex interventions: methodological lessons from the development ofan e-health normalization instrument. BMCMed ResMethodol. 2012;12: 69.

21. Levett-Jones T, Hoffman K, Dempsey J, et al. The ‘five rights’ of clinical reasoning:an educational model to enhance nursing students’ ability to identify andmanage clinically at risk patients. Nurse Educ Today. 2010;30(6): 515–520.

22. Petit dit Dariel OJ, Raby T, Ravaut F, Rothan-Tondeur M. Developing the seriousgames potential in nursing education.Nurse Educ Today. 2013;33(12): 1569–1575.

23. Levett-Jones T, Bowen L, Morris A. Enhancing nursing students’ understandingof threshold concepts through the use of digital stories and a virtualcommunity called ‘Wiimali’. Nurse Educ Pract. 2015;15(2): 91–96.

24. Hoffman K, Dempsey J, Levett-Jones T, et al. The design and implementationof an Interactive Computerised Decision Support Framework (ICDSF) as astrategy to improve nursing students’ clinical reasoning skills. Nurse EducToday. 2011;31(6): 587–594.



25. Simpson E, Courtney M. Critical thinking in nursing education: literaturereview. Int J Nurs Pract. 2002;8(2): 89–98.

26. Fero LJ,WitsbergerCM,WesmillerSW,Zullo TG,HoffmanLA.Critical thinkingabilityof new graduate and experienced nurses. J Adv Nurs. 2009;65(1): 139–148.

27. Fosnot CT, Perry RS. Constructivism: a psychological theory of learning.In: Constructivism: Theory, Perspectives, and Practice. New York, NY:Teachers College Press; 2013:10.

28. Ertmer PA, Newby TJ. Behaviorism, cognitivism, constructivism: comparingcritical features from an instructional design perspective. Perform Improv Q.1993;6: 50–72.

29. Rush KL, Dyches CE, Waldrop S, Davis A. Critical thinking among RN to BSNdistance students participating in human patient simulation. J Nurs Educ.2008;47(11): 501–508.

30. Holmes B, Gardner J. e-Learning: Concepts and Practice. London,United Kingdom: Sage Publications; 2006.

31. Thorndike RM, Thorndike-Christ TM. Measurement and Evaluation inPsychology and Education. Boston, MA: Pearson; 2010.

32. Roberts K, Lockhart R, Sportsman S. A competency transcript to assess andpersonalize new graduate competency. J Nurs Adm. 2009;39(1): 19–25.

33. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity?Appraisal and recommendations. Res Nurs Health. 2007;30(4): 459–467.

34. Johanson GA, Brooks GP. Initial scale development: Sample size for pilotstudies. Educ Psychol Meas. 2010;70: 394–400.

35. Robinson JP, Shaver PR, Wrightsman LS.Measures of Personality and SocialPsychological Attitudes. San Diego, CA: Academic Press; 1991.

36. Polit DF, Hungler BP. Nursing Research: Principles and Methods. Philadelphia,PA: Lippincott; 1999.

37. Nunnally JC, Bernstein IH.Psychometric Theory. NewYork,NY:McGraw-Hill; 1994.

38. Aamodt MG. Industrial/Organizational Psychology: An Applied Approach.Boston, MA: Cengage Learning; 2014.

April 2016


http://www.ncsbn.org/Vol_22_web.pdf

Development of the Computerized Model of Performance …

Documents