Research Recherche - World Health Organizationwhqlibdoc.who.int/bulletin/1993/Vol71-No1/bulletin_1993_71(1)_15... · Research Recherche Rapid evaluation ... * From the Division of

Research Recherche

Rapid evaluation methods (REM) of healthservices performance: methodologicalobservations*M. Anker,1 R.J. Guidotti,2 S. Orzeszyna,3 S.A. Sapirie,4 & M.C. Thuriaux5

The rapid evaluation method (REM) was developed by WHO in order to assess the performance andquality of health care services, identify operational problems, and assist in taking managerial action. Itwas tested in five developing countries (Botswana, Madagascar, Papua New Guinea, Uganda andZambia) between 1988 and 1991. REM consists of a set of observation- and survey-based diagnosticactivities, carried out mainly in health care facilities. The article describes the various steps of REM,methodological issues such as setting objectives and using an issue-information matrix, preparationof survey instruments, use of computer software (Epi Info), data quality control, fieldwork, and the useof data to produce useful information for decision-makers. REM aims at bringing prompt and relevantinformation to planners and decision-makers who need it for a specific purpose. In the present examples,REM provided information for preparing a programme proposal for external funding, for establishingbaseline data for a situation analysis, and for assessing staff performance after extensive training inorder to improve the curriculum.

IntroductionThe rapid evaluation method (REM) consists of a setof observation- and survey-based diagnostic activi-ties, carried out mainly in health care facilities,which provide a basis for identifying operationalproblems and taking managerial action.

The method was initially developed by WHO'sFamily Health Division and tested in five developingcountries (Botswana, Madagascar, Papua New Gui-nea, Uganda, Zambia) between 1988 and 1991. TheDivision of Epidemiological Surveillance and TrendAssessment in WHO has now extended the method

* From the Division of Epidemiological Surveillance and HealthSituation and Trend Assessment, and the Division of FamilyHealth, World Health Organization, Geneva, Switzerland.I Statistician, Epidemiological and Statistical Methodology,WHO, Geneva.2 Medical Officer, Maternal and Child Health, WHO, Geneva.3 Medical Officer, Monitoring, Evaluation and ProjectionMethodology, WHO, Geneva.4 Chief, Monitoring, Evaluation and Projection Methodology,World Health Organization, 1211 Geneva 27, Switzerland.Requests for reprints should be sent to this author.5 Medical Officer, Strengthening of Epidemiological and Statisti-cal Services, WHO, Geneva.Reprint No. 5350

to other health problems, besides mother and childhealth care or family planning. This article describesthe basic components of the method and discussessome of the methodological issues encountered whenassessing the performance and quality of health careservices, in particular those elements whichexperience has shown require improvement.

BackgroundSound management of health services requires rel-evant and timely information on the health status ofthe population and on the performance of health careinstitutions and staff. Most health services requirehealth personnel to record too many routine data ontoo many forms, and to forward these to higherlevels of the system. The data are often not analysedor used to improve the health system or the overallhealth status of the population. Because filling informs is seen as an unproductive burden by busyhealth care workers, the forms are often filled outcarelessly, are subsequently criticized for being un-reliable, and are not used for management purposes.One often proposed alternative when information isneeded for management purposes is to "conduct asurvey". Surveys, although indispensable in somecases, require much careful preparatory work, aregenerally (and often rightly) perceived as expensive,

Bulletin of the World Health Organization, 71 (1): 15-21 (1993) © World Health Organization 1993 15

M. Anker et al.

usually provide too many data, and take a long timeto process; they are thus rarely conducive to earlyand practical corrective action.

There is consequently an increasing need formethods that will accurately, quickly and economi-cally assemble the necessary information for analysisand decision-making. The available literature onsuch methods for rapid assessment of performance inhealth care is relatively scarce. A series of facility-based assessments of child health activities, wherethe results were available to decision-makers withintwo weeks after the completion of the study, weredescribed by Bryce et al. (2), while Vlassoff & Tan-ner (12) have stressed the role of rapid assessmentmethods in allowing research results to be translatedinto disease control activities. Smith (11) describedfive broad types of rapid epidemiological assessment(REA) methods, screening and individual riskassessment, community indicators of risk or healthstatus, and case-control methods for evaluation. Suchassessments of childhood disabilities have beenundertaken in Bangladesh, Jamaica and Pakistan todevelop cost-effective procedures for providinginformation for policy assessment and programmemonitoring (8). Rapid statistical and epidemiologicalmethods have recently been reviewed by Anker at a

WHO-sponsored consultation (1).Programme review techniques based on cluster

sampling have been developed from earlier experi-ence with the expanded programme on immunizationand primary health care reviews. These methodshave helped in the development of rapid assessmentprocedures for primary health care and maternal andchild health activities. Rapid assessment procedureshave several characteristics in common, such as

emphasizing the use of field observation in acquiringinformation from different levels of health care,

interviewing individuals (from ordinary members ofthe community to senior government officials), ensur-ing participation of professionals in multidisciplinaryteams, using flexible methods for the identificationand solution of problems, and providing the resultsto decision-makers in a timely fashion. Assessmentis mainly problem-oriented, using interviews withkey informants, group interviews, community meet-ings, and analysis of routine administrative and sur-

vey data; it provides findings that can be useful formid-course adjustments of projects (9). The rapidevaluation method described here applies the tech-niques of rapid assessment to a functional analysis ofthe health care system.

MethodsThe REM approach, as applied in a number of coun-

tries in recent years, has always entailed the partici-

pation of national programme managers in bothcontrol and implementation and in application of thedesign. The main role of extemal facilitators is toshare methods, formats and analytical techniques incomplementing the work done by national partici-pants. REM is designed on the premise that healthservice managers are already familiar with basic ser-

vice statistics. For example, a manager may alreadyknow the overall antenatal attendance level, but may

wish to know where and why gaps in coverage occur

or what variations exist in the quality of care provided.The first step in REM is for the national authori-

ties to outline the main objectives, to identify thoseprogrammes and services that are likely to beinvolved, and to decide on the topics and issuesthat will be addressed and those that will not. At thesame time, a "core group" is appointed by thenational authorities to take responsibility for theREM. This includes listing the main issues for REMto address, specifying the types of information tobe collected, identifying possible sources for thisinformation, indicating the schedule according towhich the results will be made available, and makingthe necessary logistical arrangements, includingstaff and budgetary matters.

The issue-information matrixAcquiring information in REM is based on a frame-work with three dimensions. The first dimensiondeals with issues reflecting specific health problemsrather than overall health care concems. Forexample, an issue concerning matemal and childhealth can be further defined as a general obstetricproblem, and subdivided into problems of obstructedlabour, hypertensive disorders, anaemia, etc. The

WHO Bulletin OMS. Vol 71 1993

Characteristics of REM* REM is planned and executed with the activeparticipation of health programme and servicemanagers, staff trainers and supervisors, and thestaff themselves.

* Information produced by REM examines thequantity, quality and client satisfaction of healthservices and, to a lesser extent, health status.

* The results of the REM are very rapidlyavailable to the decision-makers-within daysor weeks after the end of the REM field survey.

* The REM exercise is tailored for and neces-sarily followed by managerial decisions andactions ranging from improvements in trainingand supervision to new service strengtheningprojects, and overall health development plans.

Rapid evaluation methods of health services performance

Table 1: Issue-information matrixa

Information sources

Community Health workers

Issues Leaders Mothers TBAs Physicians Midwives Nurses Facilities

Antenatal risk factors FGD EXI FGD IND IND IND OBS

Obstructed labour

Emergency transportfor obstetrics

Neonatal tetanus

Growth monitoring

AIDS

Family planning

FGDIND

EXIFGDIND

FGDIND

EXIFGDIND

EXIFGDIND

FGDIND

EXIFGDIND

IND

FGDIND

FGDIND

FGDIND

FGDIND

FGDIND

FGDIND

OBS

IND

IND

IND

INDOBS

IND

IND

a TBAs, traditional birth attendants; EXI, exit interview; IND, individual interview; REC,OBS, observation of activity; CHK, checking facilities and supplies.

OBS OBS RECCHK

IND IND OBSRECCHK

IND IND OBSCHK

IND IND OBSRECCHK

INDOBS

INDOBS

OBSRECCHK

IND IND OBSCHK

IND IND OBSRECCHK

record review; FGD, focus group discussion;

level of detail at which issues are defined depends on

the objectives of each REM and on a consensusreached by the core group on the concerns of indi-vidual programmes.

Information sources (the second dimension) are

identified from community, health staff and healthcare facilities. The former are usually specified fur-ther through the identification of the individuals inthe community and the categories of health workerswho should be interviewed. Inspection of healthfacilities provides information on policy and on thetechnical and managerial aspects of the programme,while observation of equipment and supplies is usedto determine whether these necessary components ofhealth care are available and functional.

The third dimension of the matrix describes themethods used to obtain the information. An exampleof a (hypothetical) issue-information matrix is pre-sented in Table 1.

An appropriate and practical approach to datacollection is thus determined for each informationitem. Because the REM provides a picture of thefunctioning of services at different levels and fromdifferent points of view, several data collection in-struments are used. The instruments used in the five

countries under consideration included: clinic exitinterviews; health staff interviews; observation oftask performance; community and staff focus groupdiscussions; review of clinic records; checking offacilities, equipment and supplies; and householdinterviews (rarely).

A brief description and discussion of each typeof data collection instrument follows.

(1) Clinic exit interviews. In order to assess thefunctioning of the health care facility from the pointof view of the client, a random selection of patientswas interviewed immediately after the consultationor contact with the health services. The interviewprovided an opportunity to find out how the clientsperceived what happened during the visit to theclinic. Information included items on their satisfac-tion or dissatisfaction with the services rendered, andother problems encountered in seeking health care.

Through the use of questions about the patients'knowledge of health-related issues, the exit inter-views gave an opportunity to determine if clinicswere providing appropriate health education mes-

sages. In one country, for instance, women leavingthe antenatal clinic were asked to list some warning

WHO Bulletin OMS. Vol 71 1993

IND

FGDIND

FGDIND

FGDIND

FGDIND

FGDIND

FGDIND

M. Anker et al.

signs that are associated with pregnancy (e.g.,vaginal bleeding, extreme tiredness, headaches),thereby providing feedback on how well the edu-cational messages on these signs were understood bypatients. Ascertaining whether specific procedureswere carried out during the clinic session (bloodpressure measurements, abdominal palpation, tetanustoxoid injection) gave an indication of the quality ofantenatal care.

(2) Health staff interviews. Staff interviews pro-vided an opportunity to obtain information fromhealth workers about their attitudes, issues pertainingto management and supervision, job satisfaction,education and training, as well as to identify per-ceived problems and list suggestions for improve-ments. Different questionnaires were applied forvarious categories of health staff; in Zambia andUganda, the staff interview was self-administered andanonymous in order to encourage free expression.

(3) Observation of task performance. Observa-tion of the clinic staff's performance during theircontact with patients was one of the most effectiveways of leaming what actually happens duringthe encounter. Although the procedure is time-consuming and can by itself influence the findings, itwas employed in several countries. This method sup-plemented information on the quality of care obtain-ed from other instruments.

(4) Community and staff focus group discus-sions. These are in-depth discussions among a smallgroup of individuals chosen from a specific targetgroup. A trained facilitator stimulates discussion onthe basis of a prepared but flexible outline. The pur-pose of the discussion is to provide in-depth infor-mation not readily available from short structuredinterviews. Focus group discussions are especiallyuseful for eliciting information on feelings, attitudesand behaviour, or information about sensitive issueswhich cannot easily be obtained in a household inter-view. They are often used to provide informationabout community perceptions on issues related tohealth problems, health care, and service perform-ance and acceptability. Focus group discussions canalso be conducted with health staff, including tradi-tional birth attendants and public health nurses.These discussions provide better understanding ofthe work done by the staff, and the problems theyperceive in carrying out their tasks. A critical reviewof the advantages and requirements of focus groupdiscussions can be found in Khan et al. (7).

(5) Review of clinic records. Useful informationcan be collected quickly from the examination of asample of clinic records. This review served twogeneral purposes: to determine whether procedureswere recorded properly and to evaluate case manage-

ment of certain 'tracer' conditions. For instance, inreviewing antenatal care, a checklist of those items(age, blood pressure, obstetric history) that should berecorded on a clinic record card was developed. Thisreview indicated the extent to which those standardtasks were properly recorded. Specific items thatwere often missing were identified for further inves-tigation.

Case management of specific conditions wasalso evaluated. If an obstetric record indicated aspecific high-risk condition, the case managementwas evaluated according to whether the actiontaken conformed to agreed standards set by thehealth administration where such were available.For example, tests for syphilis were routinely carriedout during antenatal clinic visits in all five coun-tries. Records with positive results were examined,and the proportion of such records showing appropri-ate treatment or referral was used as an indicator ofthe quality of care. Different types of recordreviews were required for primary and higher levelsof health care facilities: while referral of a high-riskpregnancy to hospital may be proper case manage-ment for a primary facility, a secondary or tertiaryfacility would have a different management strategyand would have to be assessed accordingly.

(6) Checking of facilities, equipment and sup-plies. This was done to ensure that critical suppliesand equipment were available and functioningproperly and were adequate for the patients' healthneeds. The checklists included:- physical structure of the facility (e.g., waiting

rooms, examination rooms, storage facilities);- equipment (e.g., refrigerator, examination couch

or table, blood pressure apparatus); and- supplies (e.g., drugs, gloves, syringes, needles),

and other items important for the quality of thetype of care chosen for review. Both the avail-ability of the items in the checklist and theircondition (satisfactory or unsatisfactory) werenoted and recorded.(7) Household interviews. Household interviews

were carried out in the catchment areas of the healthfacilities selected for the REM in two countries(Botswana and Zambia). They provided a means ofsounding the knowledge and practices of people whodo not use the service. These interviews includedquestions on social and demographic status, the useof health facilities, specific problems encountered,and their knowledge of and attitudes to family plan-ning and AIDS. The use of household interviews inREM was optional because they are usually verytime-consuming and their validity is debatable, andbecause REM is essentially a health-facility-centredprocedure.

WHO Bulletin OMS. Vol 71 1993

Rapid evaluation methods of health services performance

SamplingREM is carried out within a geographical area whichusually encompasses both rural and urban condi-tions. Since REM is designed to meet the needs ofdifferent programme managers in a variety of set-tings, the details of sample design varied from studyto study. The minimum sample size required wasdetermined by the level of precision needed for deci-sions that would improve health care.

In most instances, the purpose of REM is to pro-vide answers for administrative officials at the cen-tral or provincial level. This is accomplished throughthe use of cluster samples, allowing for aggregationof data from several sampling units. Nevertheless, itmay be advisable to take large enough samples ateach level of care to allow analyses to be undertakenfor an individual clinic or health centre. Furthermore,it is usually more efficient to take a small number oflarger samples than several small samples.

Sampling follows a hierarchical pattem thatreflects the administrative structure of the health ser-vices: both primary and higher level facilities need tobe included in the sample in order to provide anoverall picture of the functioning of the healthservice system. Thus, depending on the administrativestructure, provinces (or equivalent administrativeunits) are selected at the first stage, and districts (orequivalents) within selected provinces at the secondstage. Within districts, health facilities of each typeare selected at random with a probability proportionalto the size of the population covered.

For focus group interviews and for householdsurveys, communities within the catchment area ofselected facilities are chosen at random. It is possibleto stratify on important variables: in one countrywhere a household survey investigated access tohealth facilities, communities were stratified accord-ing to their distance from the facility. Generally,sample sizes are kept relatively small, sacrificingsome precision of measurement for savings in costand staff time.

The number and availability of survey teamsweighs considerably on the number of facilities andvillages visited. The survey work for the five REMscarried out was done in 6 to 10 days, employing upto ten teams. Each team consisted of 4 to 5 members.Generally, each team was able to visit 1 provincialhealth office, 1 provincial hospital, 1 district hospi-tal, 2 to 4 health centres (or 1 health centre + othersmaller units), and 4 to 8 villages.

DiscussionSeveral methodological issues were raised during thepreparation and execution of the five REM countrystudies.

(1) Identification of issues and preparation ofinstruments. The participation of national health staffas diverse as ministry officials, physicians, mid-wives, training tutors, and nurses in charge of a healthcentre was an important aspect in conducting anational REM. This participation strengthened theidentification of national authorities and staff withthe REM; it also ensured that the instruments devel-oped addressed pertinent health issues and that thehealth care system was audited from different van-tage points. Issue identification can be a drawn-outprocess, often taking several days to complete,before consensus is reached on the questions to ask.In future REM surveys, full attention should begiven to this important initial phase, even at the priceof prolonging it. It is highly advisable that the surveyinstruments derived from the "Issue-informationmatrix" should be prepared, pre-tested, revised andagreed to by the core group during this time. Anylater changes should be avoided unless essential, andshould be considered with a high degree of care.

(2) Data quality control. Rapid surveys aresometimes thought to be less reliable than large-scaleresearch of longer duration; it is therefore essentialthat adequate care be taken to ensure the validity ofthe results. Data collection errors should be detectedand corrected in the field as early as possible. Thisshould be done by a supervisor who reads all thecompleted questionnaires and checks for consistencyand completeness. Computers can also be used tocontrol the quality of data. The data entered into alaptop computer can be quickly checked for errors,and, if needed, the necessary data collection can berepeated. Kalter (6) has discussed the validation ofdata collection instruments through the use of prob-ability statistics. Ross & Vaughan (10) suggestedthat internal consistency be checked within 24 hoursof completion of the initial interview, and all necessarycorrections should be made by another interviewer.

In the case of REM, efforts were made to checkthe staff interview sheets, record checks, and facilityobservation sheets immediately after their comple-tion, but this was not always possible. Certain itemspresented particular problems, such as the differencebetween the "Not applicable" and "Not available"category or the way to record if the treatment appliedwas appropriate. More attention should be paid infuture REMs to staff training and careful pretestingof instruments. Last-minute changes to the surveyinstruments, as mentioned earlier, are not advisablesince they may worsen inter-observer consistency.

Additional assessment of data quality can bemade through a comparison of results from differentsources. For instance, results from record reviews,exit interviews, and task observations can be com-pared to assess the consistency of results. In

WHO Bulletin OMS. Vol 71 1993

M. Anker et al.

Zambia, where this comparison was made, the con-cordance between information obtained from exitinterviews and antenatal records was high.

(3) Fieldwork. So that the survey part of REMmay proceed smoothly in the field, several aspectsshould be adequately planned for. Provincial and dis-trict health administrators should be notified early inadvance that the REM will take place in the facilitiesunder their jurisdiction. Community and villageleaders should also be alerted that their communityis to participate in the exercise.

It is useful to obtain a work schedule of the faci-lities included in the survey well in advance of thefieldwork. This will ensure efficiency in data collec-tion and avoid visiting clinics on days when staff arenot available or when the clinic session is not beingheld. Even with careful planning, however, logisticalproblems are apt to arise and should be allowed for.

A problem which occurred in some clinics wasthat there were too few patients to satisfy the samplesize. One solution was to revisit the same clinic on adifferent day; when this was not possible because oflogistical or time constraints, another nearby clinicof similar size was used.

(4) Computer support. Laptop or notebookcomputers are increasingly used in collecting andanalysing data from health surveys (4, 5). Epi Info(3) was the software used in most studies describedhere, and proved to be an effective tool for an initialanalysis of results and for the preparation of ques-tionnaires. The latter, however, is still a relativelylengthy and laborious process, even with Epi Info,for all but fairly experienced users.

(5) Use of data to produce information. Thequantitative information produced by REM wasdesigned to be tabulated and analysed quickly. Thiswas done in order to present the preliminary resultsto programme managers immediately after the field-work, as action-oriented information.

The qualitative information produced throughthe use of focus group discussions, on the otherhand, requires more time for analysis, and isemployed at a later date. Experience of focus groupdiscussion has underlined that the training of discus-sion leaders is essential both for leading the sessionsand in interpreting the results. In one country, thehigh quality of training in those areas contributed tothe understanding of community perception of thehealth services. In another country, lack of suitablepreparation of discussion leaders resulted in a vagueexpression of concem that was difficult to interpret.

(6) Timeliness of results. Rapidity is the key-word which distinguishes this method of assessmentfrom other forms of data collection for managementof health services. The REM experiences havegenerally produced results expressed in tables of

critical indicators within seven to ten days; produc-tion of a draft report usually took several weeks.Preparation of "dummy tables" early in the processassists in organizing the data into logical matricesand in writing the report.

ConclusionREM aims at bringing prompt and relevant informa-tion to planners and decision-makers who need it fora specific purpose. In the present examples the REMhas provided information for preparing a programmeproposal for extemal funding, establishing baselinedata for a situation analysis, or assessing staff per-formance after extensive training in order tomake adjustments in the curricula.

In one country the results of REM were pre-sented at a meeting of provincial health authoritiesand served as a basis for policy recommendations.The application of REM to areas other than maternaland child health has potential benefits. The output ofREM could be used in the development of a nationalhealth information system, and we should welcomethe indications that such an outcome has effectivelyoccurred.

AcknowledgementsWe are grateful to a number of national officers and WHOstaff who helped develop and implement the RapidEvaluation Method. They are too many to be listed here.However, four persons should be given special credit: DrMark Belsey, Programme Manager, Maternal and ChildHealth, Division of Family Health, World Health Organiza-tion, Geneva, who was the originator of the conceptof nationally designed rapid evaluations; Mr NorbertDreesch, Management Officer, World Health Organiza-tion, Vanuatu; Dr Anthony Radford, Professor of PrimaryHealth Care, Flinders University, South Australia; andDr Habib Rejeb, Maternal and Child Health, Division ofFamily Health, World Health Organization, Geneva.

Resume

Methodes d'6valuation rapide desperformances des services de sant6:observations methodologiquesLa m6thode d'6valuation rapide (Rapid evaluationmethod - REM), 6labor6e initialement par l'Organi-sation mondiale de la Sante, est destin6e a fournirune description rapide des aspects tant quantita-tifs que qualitatifs des services de sante, a identi-fier les probIemes operationnels et a permettre laprise de decisions en matiere de gestion sanitaire.

WHO Bulletin OMS. Vol 71 199320

Rapid evaluation methods of health services performance

La m6thode, qui a ete appliqu6e entre 1988 et1991 dans cinq pays en developpement (au Bots-wana, a Madagascar, en Ouganda, en PapouasieNouvelle-Guin6e, en Zambie), est basee sur unes6rie d'activit6s diagnostiques d'enquete etd'observation, menees principalement au sein desservices institutionnels. Elle requiert de fa,onessentielle et primordiale une participation pr6co-ce et continue des autorit6s locales a plusieursniveaux, tant dans la pr6paration que dans lamise en oeuvre des investigations, et d6bouchesur la preparation de propositions d'action auniveau central. Les principales etapes en sont:l'identification, selon une matrice tri-axiale, desproblemes a 6tudier, des sources d'information etdes m6thodes et outils a appliquer pour obtenirles informations; la r6colte des donn6es dans un6chantillon de services de sant6 et de communau-t6s; I'analyse des donn6es et leur synthese en unrapport a but de gestion. L'article discute ces dif-f6rentes 6tapes et analyse les caract6ristiques etl'application des principaux outils utilis6s: inter-views a la sortie des consultations, interviews depersonnel de sant6, observation d'activit6s, dis-cussions de groupe organisees (avec desmembres de la communaute et avec le personnelde sante), examen selectif de dossiers de soins,v6rification des locaux, de l'equipement, des res-sources, et, parfois, interviews a domicile. La dis-cussion du type de r6sultats obtenus et des diffi-cultes rencontrees couvre l'identification desproblemes a etudier, 1'echantillonnage, la pr6para-tion des instruments d'enquete, l'utilisation de logi-ciels du type Epi Info, et l'application des resultatsa la production d'information utile aux preneurs ded6cision et aux responsables des politiques desant6. Le domaine d'application de la m6thodes'est jusqu'a pr6sent limit6 au domaine de lasant6 maternelle et infantile mais peut etre 6tendua d'autres domaines, par exemple celui du contr6-le des maladies infectieuses. Dans les exemplesd6crits, la m6thode d'evaluation rapide a fournidans un d6lai relativement court l'informationn6cessaire a l'6tablissement d'une proposition de

subvention de programme, a l'elaboration desdonn6es de base pour une 6valuation de situationou pour 6valuer la performance du personnelapres formation, afin de proc6der aux ajustementsn6cessaires du curriculum.

References1. Anker, M. Epidemiological and statistical methods

for rapid health assessment: introduction. Worldhealth statistics quarterly, 44: 94-97 (1991).

2. Bryce, J. et al. Assessing the quality of facility-based child survival services. Health policy andplanning, 7: 155-163 (1992).

3. Dean, A.D. et al. Epi Info, Version 5: a word pro-cessing, database, and statistics programme for epi-demiology on microcomputers. Stone Mountain, GA,USD Incorporated, 1990.

4. Forster, D. et al. Evaluation of a computerized fielddata collection system for health surveys. Bulletin ofthe World Health Organization, 69: 107-111 (1991).

5. Frerichs, R.R. Epidemiologic surveillance indeveloping countries. Annual review of publichealth, 12: 257-280 (1991).

6. Kalter, H. The validation of interview for estimatingmorbidity. Health policy and planning, 7: 30-39(1992).

7. Khan, E.M. et al. The use of focus groups in socialand behavioural research: some methodologicalissues. World health statistics quarterly, 44:145-149 (1991).

8. Kroeger, A. Health interview surveys in developingcountries: a review of the methods and results.International journal of epidemiology, 12: 465-481(1983).

9. Pearson, R. Rapid assessment procedures arechanging the way UNICEF evaluates its projects.Hygie, 8(4): 23-25 (1989).

10. Ross, D.A. & Vaughan, J.P. Health interview sur-veys in developing countries: a methodologicalreview. Studies in family planning, 17: 78-94(1986).

11. Smith, G. Development of rapid epidemiologicassessment methods to evaluate health status anddelivery of health services. International journal ofepidemiology, 18 (Suppl. 2): S2-S15 (1989).

12. Vlassoff, C. & Tanner, M. The relevance of rapidassessment to health research and interventions.Health policy and planning, 7: 1-9 (1992).

WHO Bulletin OMS. Vol 71 1993 21