Operationalisation of an observation instrument to explore nurse performance

Pergamon hf. J. Nurs. Stud., Vol. 33, No. 4, pp. 349-360, 1996

CopyrIght T I 1996 Elsevm Science Ltd. All rights reserved Printed in Great Britain

0020-7489196 $15 OOf0.00

0020-7489(95)00061-5

Operationalisation of an observation instrument to explore nurse performance

JOANNE M. FITZPATRICK, B.Sc., R.G.N., ALISON E. WHILE, B.Sc., M.Sc., Ph.D., R.G.N., R.H.V., Cert.Ed. JULIA D. ROBERTS, B.A., M.A., Cert.Ed., R.G.N., R.N.T. Department qf‘Nursing Studies, King’s College London, University of London, Cornwall House, Waterloo Road, London SE1 8 WA, U.K.

Abstract-This paper explores the operationalisation of an observation instru- ment to explore actual situated behaviour in the clinical setting. Challenges encountered during this process and strategies employed to address these diffi- culties are discussed. This work formed part of a comparative study of outcomes of pre-registration nurse education programmes commissioned by the English National Board for Nursing, Health Visiting and Midwifery. Copyright 0 1996 Elsevier Science Ltd.

Introduction

The aim of this study was to explore and compare the performance of nurses from three different pre-registration programmes of preparation in the practice environment (N= 99). Each education programme (integrated degree, P2000 diploma and RGN) was represented by three institutions. To fulfil the study aim, a non-participant observation method was adopted and a nurse performance scale was developed as the recording device. This instru- ment drew substantially upon an existing tool (The Slater Nursing Competencies Rating Scale, Wandelt and Stewart, 1975) and incorporated current professional, theoretical and empirical literature. The development of the observation instrument has been discussed in a separate paper (Fitzpatrick et al., 1994a). The operationalisation of the tool, difficulties

349

350 J. M. FITZPATRICK et al.

encountered, and strategies employed in an attempt to address these, are the focus of this paper.

Although actual performance in the real-life situation has been highlighted as an impor- tant outcome of an educational experience (Beare, 1985), use of observation to explore nurse performance in the clinical setting has been limited. Of further significance is the absence of a valid and reliable observation instrument (Fitzpatrick et al., 1994b). These findings may in part be owing to the many challenges associated with adopting observation as a research method. It is argued that the potential of observation in research of this nature has not been fully exploited and continued refinement and development of this method is therefore essential.

Challenges of observation as a research strategy

The methodological challenges inherent within observation research are well documented and include: potential ethical issues; potential observer influence upon participant behav- iour; potential influence of situational and personal factors on the behaviour of those observed; potential observer error; reliability of observer rating; and potential of observer drift over time (Crow, 1984; Ross et al., 1988; Carr-Hill et al., 1992; Fletcher, 1992). In this study developing an awareness of these potentially confounding variables was considered vital for several reasons including: identification of strategies to minimise potential sources of bias, enhancing effective observer performance and minimising error of measurement (Morgan and Irby, 1978; Ventura and Crosby, 1978; Bondy, 1984; Peterson et al., 1985; Stiggins, 1987; Linn et al., 1991). Addressing these challenges was incorporated into a programme of observer training, the overall aim of which was to facilitate an accurate understanding and proficient use of the instrument so that ratings were a fair representation of participant performance (Popham, 198 1; Brink, 1989).

Observer training procedure

Observer training concentrated upon developing proficiency in administering the instru- ment and addressing potential sources of bias associated with this method of data collection. This process was facilitated by the observers’ familiarity with the progressive development of the observation instrument.

Utilisation of the instrument

To facilitate effective use of the observation tool in an attempt to minimise the potential of observer error, guidelines were developed drawing upon the Slater Nursing Competencies Rating Scale manual (Wandelt and Stewart, 1975) and Qualpacs manual (Wandelt and Ager, 1974). Sessions focused upon developing an understanding of the salient issues including: the scale items and their cues; definition of a care event; use of the rating criteria in conjunction with the standard of measurement; the recording of contextual information; and other issues related to the rating process, for example, dealing with events occurring in rapid succession. Tomalin et al. (1993), with reference to the administration of Qualpacs, have noted that consideration of similar issues was important in achieving uniformity between the observers in their study.

NURSE PERFORMANCE 351

The rating process

To facilitate accurate identification of behaviour in the field setting as well as the selection and rating of appropriate items, discussion took place regarding what constituted a care event. It was agreed that single care events were those which indicated a unity and when the participant’s attention continued to be focused on a key theme, for example, discussing medication with a client. If an interruption in continuity occurred then the next care event would be treated separately. It was hoped that delineation of care events would enhance effective item selection during the rating process. As an example, items to be considered when observing drug administration as a care event might include those related to the drug administration procedure, safety, policy directives, maintaining written records and information-giving to clients. Once observers were in agreement regarding the identification of items pertinent to care events, attention was directed to the process of assigning ratings, using the standard of measurement and the rating criteria.

The standard of measurement, as detailed in the Slater Nursing Competencies Rating Scale, was:

“the quality of performance of care expected of a first level staff nurse, that is, one who is charged with the responsibility for providing nursing care that is safe, adequate, therapeutic and supportive in meeting the needs of patients” (Wandelt and Stewart, 1975, p. xiii).

In this study the standard of measurement: “the quality of performance of care expected of a first level staff nurse, that is, one who is charged with the responsibility for providing nursing care that is safe, adequate, therapeutic and supportive in meeting the needs of patients” (Wandelt and Stewart, 1975. p. xiii),

was derived from an agreement regarding sound professional practice taking into account the 13 learning outcomes of Statutory Instrument No. 1456 (1989) and the requirements of the Code of Professional Conduct (UKCC, 1992). This standard was held constant for all measurements made with the scale to facilitate comparison of programme data. Redfern et al. (1994) considered ‘newly qualified’ to be an adequate substitute for ‘first level’.

The criteria used in this study to rate the scale items, a modified version of Bondy’s (1983) criterion-referenced scale labels, have been discussed in-depth in a separate paper which details the development of the observation instrument (Fitzpatrick et al., 1994a). The rating criteria were comprised of three categories, the first of which was professional standards and procedures and encompassed the issues of safety, accuracy, effect and affect. The second category focuses upon the qualitative aspects of the performance and includes the use of time, space, equipment and energy. Demonstration of persistence under adverse circumstances was also considered within this category. The third category considered the type and degree of assistance required to perform the nursing activity effectively. These criteria were applied to four levels of performance (independent, assisted, marginal and dependent), with omissions rated in a separate column. In assigning ratings to items observers were required to judge how the performance contributed to meeting the needs of patients/others against the standard of measurement and rating criteria. In the field setting, this process was facilitated by observers receiving a patient hand-over prior to commencing observation.

Viewing of videotapes depicting direct nurse-patient interactions/interventions which comprised physical care-giving as well as communications skills was also incorporated into these sessions. Initially observers rated the videotape scenes retrospectively using the nurse performance scale until agreement was reached and then sessions were conducted where concurrent ratings were performed.

352 J. M. FITZPATRICK et al.

Addressing potential biases

Preparation prior to data collection also focused upon potential sources of bias and the challenge of identifying strategies to address these, in order to enhance data reliability and validity. Attention to such factors, it has been argued, should enable observers to:

“look for them and take precautions to counteract them as a preventive measure or to correct them when they exist” (DeMers, 1978, p. 104).

Potential biases inherent within observational research include those associated with the observer, for example, personal bias errors and the possibility of participant reactivity as a consequence of observation, as well as the potential influence of situational variables and personal factors on behaviour (Johnson and Bolstad, 1973; Messick, 1984; Feldt and Brennan, 1989).

Potential observer injluence on participant behaviour

Associated with potential observer influences on participant behaviour are reactivity effects including those related to adopting and maintaining the non-participant observer role; observer conspicuousness; interaction of participant and observer characteristics (e.g. age and personality); personal attributes of the observer (e.g. manner of dress); and the way in which the observer presents the nature and purpose of the study (Johnson and Bolstad, 1973).

The observer role

The role of non-participant observer was adopted in this study and each participant was observed for a period of 2: hours on three separate occasions. Non-participant observation, it has been argued, should enhance the potential for behaviour to proceed ‘as usual’, as well as facilitating effective observer performance (Wandelt and Ager, 1974).

Permissible interaction

For the purposes of this research it was therefore agreed that as a non-participant observer, interactions with participants, patients and others during the shadowing period would be avoided. This emerged as a challenge in the field setting and it became apparent that a certain degree of flexibility in this approach was required to facilitate successful data collection. For example, on occasions patients and others addressed the researchers during the shadowing period to greet them and to request information about the research. Adhering strictly to the rules of non-participant observation one should avoid any form of interaction otherwise one may influence the normal progress of events and furthermore risk loosing observational data. Alternatively it may be argued that the possible effect the observer may have upon events by not responding is potentially greater. For example, choosing not to respond may result in observer alienation within the field setting. As in all research of this nature this was an important consideration and specifically since each participant was observed on three occasions. Further, it was sometimes the case that several students participating in the study were allocated to the same ward. A degree of participation was thus necessitated and it was agreed that in certain situations brief observer ‘interaction’ was permissible.

Other strategies were employed in an attempt to minimise the need for observer inter- action which included distributing project information leaflets to participants’ allocated

NURSE PERFORMANCE 353

ward, wearing identity badges, and asking participants to introduce the researchers to patients, staff and others as appropriate. A decision to wear mufti, rather than a white coat, was also made so that one could not be mistaken for a member of staff and hopefully avoid requests for assistance. However, it became apparent that wearing mufti did not necessarily dissuade patients or others from trying to engage the observers in conversation and indeed on occasions making requests.

Permissible intervention

In accordance with the non-participant role, it had been decided that observer inter- vention would occur only in emergency situations, for example, a life-threatening situation involving a patient. Observer intervention was seldom necessitated, examples being to prevent a patient from falling out of bed and to inform staff of a completed intravenous infusion. However, there were occasions when generally less effective nursing care was observed, for example, lack of attention regarding infection control policies. Observer intervention was not recommended in such situations, however, in an attempt to achieve equilibrium between one’s role as a researcher and one’s role as a nurse and to enhance effective observer performance over a 14-month period it was important to reflect upon these experiences as a team.

Observer conspicuousness

Also associated with the potential of observer influence on participant behaviour is observer conspicuousness. This was particularly challenging in this study since participants were shadowed continuously throughout the observation sessions unlike other observation studies where the observer may adopt a stationary position, for example when using the Qualpacs instrument (Wandelt and Ager, 1974). Thought was therefore given as to how the presence of observers may inadvertently affect participant performance and how this might best be addressed.

It has been suggested that having a period of acclimatisation may successfully overcome any awareness of being observed (Crow, 1984). In this study all participants involved in the main data collection were observed on three occasions. Anything more than this would have demanded too much from individuals who already had made a significant contribution by their participation. Further, as Carr (1988) highlighted in relation to her role observing clients’ feeding difficulties after a stroke, repeatedly observing participants would not enable one to determine observer effect on behaviour. This is due to the fact that it would be impossible to distinguish between performance changes owing to adaptation to the observer and those arising as a result of other influences, for example, related to the working environment. Indeed, Carr (1988) concluded that the effect of the observer on a participant’s behaviour cannot be determined nor knowingly eradicated (p. 140). As well as observing participants on three occasions, observers made every effort to be discreet, for example, shadowing in a non-threatening manner. This was something which emerged as a significant challenge when observing in Accident and Emergency departments during the pilot phase due to the nature of the clinical environment. Indeed as a consequence of this experience, potential participants allocated to Accident and Emergency departments and peri-operative departments were not recruited to the main study.

354 J. M. FITZPATRICK et al.

Information given to participant and others

It has been asserted that information-giving about the nature and purpose of a study is a potential source of bias (Johnson and Bolstad, 1973). Recognising the importance of this and attempting to minimise its effect, in this study information regarding observation during practice was presented to potential participants at the first meeting between each cohort and the research team, The aim of this initial contact was to introduce the research project as a whole. A standard explanation was delivered at each of the nine participating centres thus ensuring a common frame of reference. Additionally, all potential participants were given a standard information leaflet which displayed a brief summary of the research as well as photographs of the research team. Copies of the information leaflet were also distributed to the allocated wards of those who had agreed to be observed. Standard information was given to participants prior to commencing observation sessions. Par- ticipants were informed that the focus of observation was all aspects of their nursing practice. The observer was introduced to patients/others, as appropriate, and it was ex- plained that only the participating nurse was the subject of observer attention.

Situation variables

In addition to the potential influence of the observer on participant behaviour, other extraneous variables pose a challenge to the observational method. As Feldt and Brennan (1989) have stated:

“anyone who regularly plays a game with objective scoring such as golf or bridge is acutely aware of the variability in human performance” (p. 105).

This variability, they have argued, may be explained from various perspectives, including the potential influence of situational factors within the participant’s working environment. Furthermore, Benner (1982) has suggested that when a basic skill is performed in an actual situation the characteristics of the situation have as much influence on successful performance as does knowledge of the steps for performing the activity (p. 304). Charac- teristics of the situation may include: the attitudes and expectations of superiors; the nature of the work assignment; and the quality and amount of work-related communications (Harrington and Theis, 1968) as well as the nurturing of innovation, expression of indi- viduality, and morale (While, 1994). The nature of work assignment and its influence on baccalaureate nurse performance was investigated by Christman (1971) in a pilot study (N= 42). Participants’ performance was rated using the Slater Nursing Competencies Rat- ing Scale (Slater, 1967) and results indicated a significant difference (P=O.O5) between performance score and setting. The mean score for those working in a patient centred setting (4.1) was significantly higher than those in a task centred setting (3.7).

In the current study it was not possible to control for situational variables, however, additional information was recorded regarding factors such as the organisation of nursing care, staffing levels (that is, the clinical grades of the nursing stall), and patient dependency levels (Barr, 1964) in an attempt to explore their effect upon performance. A modified version of the additional information sheet from the Qualpacs instrument was used to document this data (Wandelt and Ager, 1974). It was also anticipated that observing each participant at different times of the day (a morning, afternoon and evening period) would accommodate workload variation and therefore potentially enabled best performance to be observed. As well as the possible influence of situational variables on participant behaviour, factors specific to the field setting may also influence observer performance (McCall and

NURSE PERFORMANCE 355

Simmons, 1969). For example, how observers are received in the clinical setting by par- ticipants, patients and ward staff may impinge upon observer performance. It was important to recognise this potential source of error and aim to perform consistently as an observer regardless of such extraneous variables.

Personal tlariahles as a potential bias

Personal factors may also exert an influence upon performance and this applies to the observer as well as to those who are being observed. It has been noted, for example, that the performance process is influenced by a variety of affective, motivational, attentional and stylistic factors (Messick, 1989; Feldt and Brennan, 1989). In addition, the potential influence of circadian rhythms and other cyclical variations, such as menstrual rhythm effects, on performance have been increasingly acknowledged (Asso, 1987; Skipper et al., 1990) Direct measurement of such variables was not attempted in this study, however, an attempt was made to control for extraneous personal variables. Thus students were invited to participate in the study if they fulfilled the following criteria: female, childless, Caucasian, under 25 years of age and held two or more passes at A level. Further, it was hoped that continuous observation of participants for 2: hours on three occasions at different times of the day should accommodate the possible influence of any cyclical variations.

Observer reliabilit?

Achieving and maintaining inter-observer reliability over the data collection period also demanded consideration. In the pilot study, to facilitate observer agreement participants were initially observed by both researchers. Once agreement was attained the observers branched and conducted a series of observations separately before rejoining towards the end of this phase. Examination of the standard deviation and the mean of the differences between the two observer suggests that acceptable levels of inter-observer reliability were achieved (Table 1).

For the duration of the main study the research team held meetings at least weekly to continue training and to discuss any problems encountered during the rating process in an effort to maintain observer agreement. Notes made during observation sessions provided a valuable reference point for these discussions. The limited number ofjoint observer sessions (N=3) did not permit inter-observer reliability to be tested statistically, However, exam- ination of the standard deviation and the mean of the differences between the two observer suggests that acceptable levels of inter-observer reliability were achieved (Table 2).

Observer drift

Crow (1984) has defined observer drift as instability in recording over repeated periods of observation, this incorporates variation in the agreed acceptable levels of error both within each period of observation and over the entire study period of time for the study. In this study the main observation data collection phase spanned a 16month period and for this reason observer drift was potentially a major source of error. Johnson and Bolstad (1973) have noted that drift may arise from processes of forgetting, new learning and fatigue. While it has been documented that this source of bias is difficult to remove (Crow, 1984) it is vital, nonetheless, that preventive measures are explored and adopted as appropriate in order to minimise this threat.

A variety of approaches to address this problem were explored, for example, the use of

356 J. M. FITZPATRICK et al.

Table I. Inter-observer reliability for the pilot study

SD of differences between Mean of differences between Observer I and Observer 2

Potential variability Observer I and Observer 2 (+2xSD)

Total observation score 0.21 -0.03 0.42

Subsection 1 Physical domain 0.36 -0.10 0.72

Subsection 2 Psychosocial domain 0.71 -0.50 1.42 Subsection 3 Professional implications 0.31 -0.07 0.62

Subsection 4 Promotion of health and 0.23 -0.13 0.46 teaching skills

Subsection 5 Care management skills and 0.12 -0.07 0.24 the organisation of workload

Subsection 6 Communication skills

Subsection 7 Use of the nursing in process planning care

0.26 -0.10 0.52

0.21 0.07 0.42

Table 2. Inter-observer reliability for the main study

Total observation score

Subsection 1 Physical domain

Subsection 2 Psychosocial domain

Subsection 3 Professional implications

Subsection 4 Promotion of health and teaching skills

Subsection 5 Care management skills and the organisation of workload

SD of differences between Mean of differences between Potential variability Observer 1 and Observer 2 Observer 1 and Observer 2 (?2xSD)

0.16 -0.08 0.32

0.64 0.00 1.28

Insufficient data Insufficient data Insufficient data

0.74 -0.24 1.48

0.74 0.12 1.48

0.06 - 0.06 0.12

Subsection 6 Communication skills 0.18 -0.02 0.36

Subsection 7 Use of the nursing process in 0.23 -0.15 0.46 planning care

a ‘real’ bench mark person, such as a recent graduate nurse, who would be rated periodically and independently by the observers. This suggestion was not adopted, however, since their performance was likely to improve over time. Other suggestions included the use of an

NURSE PERFORMANCE

Table 3. Observer drift during the pilot study

351

SD of differences between Mean of differences between Independent Observer and Observer 2 Independent Observer and Observer 2 Potential variability

for the total observation score for the total observation score (k2xSD)

0.14 0.10 0.28

Table 4. Observer drift during the main study

SD of differences between Mean of differences between Independent Observer and Observer 2 Independent Observer and Observer 2 Potential variability

for the total observation score for the total observation score (k2xSD)

0.38 0.26 0.76

independent third observer to assess observer accuracy and note-making regarding the meaning and rating of items which would be referred to at observer discussions throughout data collection. Uninformed random assessment of observer accuracy or assessment on each observation session was proposed by Johnson and Bolstad (1973). However, such an approach was not feasible in this study since observation sessions were pre-arranged at the convenience of participants.

Two approaches were finally adopted: observer notes regarding the meaning and content of care events were incorporated into regular observer discussions and an independent observer was recruited. Two joint observation sessions were conducted with the independent observer during the pilot phase and four joint observation sessions were conducted during the main data collection period. Examination of the standard deviation and the mean of the differences between the two observers suggested that observer drift was minimal for both stages of the study (Tables 3 and 4).

Pilot phase

The pilot study was undertaken to:

1. Examine further the feasibility of the observation instrument and additional record sheets.

2. Examine further the feasibility of the rating criteria. 3. Conduct further procedural refinement.

Ethical approval and access were obtained for a location which was not involved in the main study and informed consent was obtained for each participant. The sample for this phase of the study consisted of final year diploma students (N=7) and nursing degree students (N=5) and field work extended over a 3-month period. Nine of the twelve participants were observed on three separate occasions during the day; two were observed twice and one was observed on one occasion only. In this study it was anticipated that three separate observation sessions for each participant at different times of the day would provide the opportunity to rate as many of the Scale items as possible. Further, it was hoped that a total observation period of 7: hours for each participant would accommodate workload variations and the potential influence of personal variables on behaviour, thereby

358 J. M. FITZPATRICK et al.

enabling typical performance to be observed. It has been argued that an individual’s average score produced by repeated ‘test’ administration on different occasions closely approximates an individual’s true score, that is, the score that one would achieve if ‘tests’ were perfectly reliable (Nunally, 1964).

Issues arising from the pilot phase

Rejinement of the instrument, rating criteria and additional recording sheets

Some minor modifications were made to scale items and cues on the basis of observer feedback, however, no item reduction occurred. Adjustments were also made to Bondy’s (1983) criterion-referenced scale, specifically the supervised level of performance was deleted. This was owing to difficulties in discriminating between Bondy’s criteria for the independent and supervised levels. Specifically, the criteria for ‘professional standards and procedures’ was identical for both the independent and supervised levels and there appeared to be minimal distinction between the levels in the ‘qualitative aspects of performance’. The only distinction lay in the type and degree of assistance required to perform the nursing activity. As a consequence the supervised level was rarely rated during pilot work. Minor adjustments were made to the wording of the criteria for the scale labels to further enhance accuracy of rating for the main phase of data collection. As noted, patient dependency levels were recorded during the observation sessions as part of the contextual data. As a result of pilot work, it was decided that patient admissions and/or discharges during the period of observation should also be documented to enhance completeness.

Procedural refinement

Several procedural refinements occurred as a result of pilot work. Regarding recruitment to the main study, it was decided that potential participants would be approached on an individual basis following completion of the first element of the research (the simulation exercises). Sensitive handling of this was essential since participants were in the final 3 months of their programmes and had a variety of issues to contend with including written and clinical assessments and the challenge of securing future employment.

The observation periods for the main study were identified as: a morning session (com- mencing observation at the beginning of the participant’s shift), an afternoon session (when the participant had been working an early shift) and an evening session, commencing observation at 1700. Further insight regarding observer procedure was also gained during the pilot phase, for example, improved shadowing technique. Of further significance was the importance of allowing adequate time to explain the study and the role of the observer to the nurse-in-charge prior to commencing observation, and allowing adequate time to receive a verbal report regarding those patients for whom the nurse would be caring.

Main study

The main data collection phase extended over a period of 14 months during which 297 observation sessions, 742.5 hours in total, were conducted. Throughout this time the research team held meetings at least weekly to discuss any difficulties encountered and to continue observer training thereby enhancing observer agreement. One difficulty to emerge during this stage concerned the ability to timetable each participant for a morning, after-

NURSE PERFORMANCE 359

noon and evening observation session. This was owing to factors such as students’ com- pletion dates for ward allocations and observer or participant sickness. It was therefore necessary on occasions to conduct observation sessions over a weekend period and on a few occasions, when it was not possible to timetable a morning, afternoon and evening observation period, a shift was duplicated, for example, observing a participant on two morning periods and an evening.

Conclusion

The trials and tribulations of observational research have been well documented in the literature (Crow, 1984; Ross et al., 1988; Carr-Hill et al., 1992; Fletcher, 1992) and this paper has highlighted some of the methodological challenges encountered when administering an observation instrument to explore nurse performance in the practice environment. Key issues to emerge included: problems associated with enhancing effective use of the instru- ment, as well as potential sources of bias and error inherent within the observational method. In this study, observer training emerged as one of the most important strategies for addressing the challenges posed by this method. Further refinement of the observation method is. however, required in order to enhance both the validity and reliability of this method as a means of exploring real-life nurse performance.

The authors were engaged in a comparative study of outcomes of pre-registration nurse education programmes funded by the English National Board for Nursing, Midwifery and Health Visiting. This article draws upon this work. Responsibility for the views expressed, issues of interpretation, questions of inclusion and omission, remain as always with the research team and do not necessarily reflect the views of the English National Board for Nursing, Midwifery and Health Visiting.

References

Ager. J. W. and Wand&, M. A. (1975). Tests of the Scale. In Sluter Nursing Competencies Rating Scale (Wandelt, M. A. and Stewart, D. S., Eds). Appleton-Century-Crofts, New York.

Asso. D. (1987). Cyclical variations. In Sex Dif@wnces in Human Perfbrmance (Baker. M. A., Ed). John Wiley, Chichester.

Barr, A. (1964) Measurement qf‘Nursing Care, Operational Research Unit Report No. 9. Oxford Regional Hospital Board, Oxford.

Beare, P. (1985). The clinical contract: an approach to competency-based clinical learning and evaluation. J. Nurs. Educ. 24(2), 75-77.

Benner, 0. (1982). From novice to expert. Am. d. Nurs. 82(3), 402407. Bondy, K. N. (1983). Criterion-referenced definitions for rating scales in clinical evaluation. J. Nurs. Educ. 22(9),

376 382. Bondy. K. N. (1984). Clinical evaluation of student performance: the effects of criteria on accuracy and reliability,

Res. Nurs. Health 7, 25-33. Brink. P. J. (1989). Issues in Reliability and Validity, In Qualitatice Nursing Research: a Contemporary Didogw,

(Morse, J. M.. Ed.). Aspen Publishers Inc. Rockville. Marvland. Carr. E. K. (1988). Feeding difficulties after stroke: their assessment and nursing management, Unpublished PhD

Thesis. Universitv of Nottineham. Carr-Hill, R.. Dixon, P.. Gibbs, I., Grithths, M., McCoughan, D. and Wright, K. (1992). Skill Mi\- and AC

Efftictirvness of’Nursin,q Care. Centre of Health Economics. Universitv of York, Christman, N. J. (1971). Clinical performance of baccalaureate graduates. Nurs. Outlook 19(l), 5456. Crow, R. (1984). Observation. In The Research Proce.u in Nursing (Cormack, D. F. S.. Ed.). Blackwell Scientific

Publications. Oxford. DeMers, J. L. (1978). Observational assessment of performance. In Evaluating Clinical Competence in the Health

Professionx (Morgan. M. K. and Irby, D.. Eds). C. V. Mosby. St Louis, Missouri.

360 J. M. FITZPATRICK et al.

Feldt, L. S. and Brennan, R. L. (1989). Reliability. In Educational Measurement, 3rd Edn, (Linn, R. L., Ed). American Council on Education, Macmillan, Phoenix.

Fitzpatrick, J. M., While, A. E. and Roberts, J. D. (1994a). Development of an observation instrument to measure nurse performance, Unpublished document.

Fitzpatrick, J. M., While, A. E. and Roberts, J. D. (1994b). The measurement of nurse performance and its differentiation by course of preparation. J. Adu. Nurs. 20, 761-768.

Fletcher, S. (1992). Competence-based Assessment Techniques. Kogan Paul, London. Gronlund, N. E. (1981). Measurement and Evaluation in Teaching. Macmillan, New York. Harrington, H. A. andTheis, E. C. (1968). Institutional factors perceived by baccalaureate graduates as influencing

their performance as staff nurses. Nurs. Res. 17(3), 228-235. Johnson, S. M. and Bolstad, 0. D. (1973). Methodological issues in naturalistic observation: some problems and

solutions for field research. In Behaviour Change: Methodology, Concepts and Practice (Hamerlynck, L. A., Handy, L. C. and Mash, E. J., Eds). Research Press, Champaign, Illinois.

Linn, R. L., Baker, E. L. and Dunbar, S. B. (1991). Complex performance-based assessment: expectations and validation criteria. Educ. Res., November 1991, 15-21.

McCall, G. J. and Simmons, J. L. (1969). Issues in Participant Observation: a Text and Reader, Addison-Wesley, Massachusetts.

Messick, S. (1984). The psychology of educational measurement. 1. Educ. Measurement 21,215-238. Messick, S. (1989). Validity. In Educational Measurement, 3rd Edn (Linn, R. L., Ed). American Council on

Education, Macmillan, Phoenix. Morgan, M. K. and Irby, D. (1978). Evaluating Clinical Competence in the Health Professions. C. V. Mosby, St

Louis, Missouri. Nunally, J. C. (1964). Psychometrir Theory. McGraw-Hill, New York. Peterson, D., Micceri, T. and Smith, 0. (1985). Measurement of teacher performance: a study in instrument

development. Teaching and Teacher Education l(l), 63-77. Popham, W. J. (ed) (1981). Modern Educational Measurement. Prentice Hall, Englewood Cliffs, NJ. Redfern, S. J., Norman, I. J. with Tomalin, D. A., Oliver, S. and Jacka, K. (1994). The Validity of Qua/if),

Assessment Instruments in Nursing: Final Report to the Department of Health. Nursing Research Unit, King’s College London.

Ross, M., Carroll, G., Knight, J., Chamberlain, M., Fothergill-Bourbonnais, F. and Linton, J. (1988). Using the OSCE to measure clinical skills performance in nursing. J. Adv. Nurs. 13, 4546.

Skipper, J., Jung, F. and Coffrey, L. (1990). Nurses and shiftwork: effects on physical health and mental depression. J. Adv. Nurs. 15, 835-842.

Slater, D. S. (1967). Slater Nursing Competencies Rating Scale. College of Nursing, Wayne State University, Detroit.

Statutory Instrument No. 1456. (1989). The Nurses, Midwives and Health Visitors (Registered Fever Nurses Amendment Rules and Training Amendment Rules) Approved Order. HMSO, London.

Stiggins, R. J. (1987). Design and development of performance assessments. Educ. Measure.: Iss. Pratt. 6(3), 33- 42.

Tomalin, D. A., Oliver, S., Redfern, S. J. and Norman, I. J. (1993). Inter-rater reliability of Monitor, Senior Monitor and Qualpacs. J. Adv. Nurs. 18, 1152-l 158.

UKCC (1992). Code of Professional Conductfor the Nurse, Midwije, and Health Visitor, 3rd Edn. UKCC, London. Ventura, M. R. and Crosby, F. (1978). Preparing the nurse observer to use the Quality Patient Care Scale: a

modular approach. J. Continuing Educ. Nurs. 9(6), 3740. Wandelt, M. A. and Ager, J. W. (1974). Quality Patient Care Scale. Appleton-Century-Crofts, New York. Wandelt, M. A. and Stewart, D. S. (1975). Slater Nursing Competencies Rating Scale. Appleton-Century-Crofts,

New York. While, A. E. (I 994). Competence versus performance: which is the more important? J. Adz;. Nurs. 20(3), 525-53 1.

(Received 5 June 1995; accepted 4 September 1995)