Evaluation of a surveillance system Pawel Stefanoff
Mar 27, 2015
Evaluation of a surveillance system
Pawel Stefanoff
Evaluation of a surveillance system:
a cyclic process
Gaps between needs and capacities
Implementation
Monitoring and evaluation
Planning for improvements
WHAT do we want to assess?
Technique
Disease specific surveillance
Surveillance system Complexity
The way of defining the problem methods solution.
• To assess the capacity of the system to meet its purpose and objectives
• In order to – improve its operation
– modify it to address: • priorities
• epidemiology
– optimize the available resources
Why evaluate?
• Description of the needs– Country context
– Disease patterns
– Target audience
• Description of the surveillance system– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
• Description of the needs– Country context
– Disease patterns
– Target audience
• Description of the surveillance system– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
Description of surveillance needs
• Description of the country context– Population
– Health system
– Political structure and administration
• Description of disease patterns:– Disease characteristics (epidemic potential)
– Risk factors (country-specific?)
What’s the target audience of surveillance?
• Public Health professionals?
• Clinicians / Microbiologists / Control of Infection staff?
• Environmental Health professionals?
• Health service managers?
• Health educators / teachers?
• Government / Politicians?
• Public?
• Description of the needs– Country context
– Disease patterns
– Target audience
• Description of the surveillance system– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
Description of the surveillance system
• The description of the surveillance system is a scientific process– Poor methods will generate unreliable
results
• When describing a surveillance system:– Report your methods in the “methods”
section
– Write the description in the “results” section
Methods that may be used to describe a surveillance system
1. Review of documents
2. Review of products/ outputs
3. Qualitative interviews of key informants
4. Quantitative interviews of participants
1. Reviewing documents
• Consider:– Work plans
– Guidelines
– Operation manuals
– Forms
• Generates information on the theoretical surveillance system
2. Reviewing products/ outputs
• Consider:– Records
– Data
– Feedback reports
• Generates information on the actual surveillance system
Elements of the surveillance system
to describe (1/2)
• Diseases under surveillance
• Forms used
• Case definition
• Population under surveillance– Total, sentinel or special
• Reporting mode – Passive, stimulated or active
Elements of the surveillance system
to describe (2/2)
• Data structure– Individual or aggregated
• Data transmission– Flow chart
• Data analysis procedures– Type, frequency
• Feedback – Type, frequency
• Action taken
Define objectives of a surveillance system
• Identify or help formulating the actual or desirable objective(s) of the surveillance system– Identify the goal of the programme served
– Identify what decisions may need to be taken
• Review the attributes of the system
• Review whether the system’s attributes are satisfactory to reach the objective(s)
• Description of the needs– Country context
– Disease patterns
– Target audience
• Description of the surveillance system– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
• Simplicity
• Flexibility
• Acceptability
• Data quality
• Sensitivity
• Positive predictive value
• Representativeness
• Timeliness
• Cost
System attributes
Simplicity
• Structure– information needed
– number and type of sources
– number of information users
• Ease of operation– data transmission
– system maintenance
– data analysis
– information dissemination
Flowchart for surveillance of HIV/AIDS in Norway
Referencelaboratory
Primary HIV reporting form,Blood sample for HIV test laboratory part 1
Lab report and HIV reporting form
HIV reporting form, part 2HIV infection Primary care (Prompting if necessary) National Institute
physician of Public Health
AIDS reporting formAIDS Hospital physician Semiannual check
Oral informationDeath, emigration Semiannual check
Patient
• Ability of the system to accommodate changes with little additional time, persons or allocated funds– New event to follow-up
– New case definition
– New data about an event
– New sources of information
Flexibility
Acceptability
• Knowledge: – Case definition, notification procedures
• Notification conditions– Sufficient stock of notification forms
– Working charge
– Simplicity
• Motivation– Understanding the importance
– Risk perception
– Existence of a regular feedback
Data quality
Completeness and validity of the data recorded
• Completenness of records – blanks in the surveillance forms
• Errors when computing data
• Influenced by– Simplicity of surveillance form
– Clarity of electronic surveillance forms
– Training
– Validation
Sensitivity
• Two aspects:– proportion of cases of a disease or health condition
detected by the surveillance system.
– ability to detect outbreaks, to monitor changes (out of outbreaks, how many were detected)
• Requires (in principle)– to validate the information collected and
– to collect information outside of the system to determine the frequency of the condition in the community.
Factors influencing case detection
• The probability of medical consultation– Severity of symptoms
– Access to health care
– Social and cultural values and habits
• The probability of clinical diagnosis– Pathogen causes a clinical condition also caused by
other pathogens
• The probability of lab diagnosis– Demand for lab-testing
– Existence of laboratories with the specific technique
Quantitative estimates of the sensitivity of a surveillance system
• Health care facility survey– Proportion of cases seen in the health care
facility that are reported / captured by the surveillance system
• Population survey– Proportion of cases occurring in the
population:• Seen in health care facilities• Reported / captured by the surveillance system
• Mathematical model– Cases captured /expected
Sensitivity of measles surveillance in Haora district, West Bengal,
India• Methods used
– Cluster sample with door to door case search– Review of health care facility and surveillance records
• Key results:– Of 240 cases identified in the population, 8 (3.3%)
were seen in public health care facilities and reported
• Conclusion:– Measles surveillance is only 3.3% sensitive
• Recommendation:– Promote vitamin A management in the population to
increase the use of health system in the case of measles
Positive Predictive Value
• PPV depends on– sensitivity– specificity, and the – prevalence of the condition in the population
• Consequence of a low PPV– frequent "false-positive" report– inappropriate follow-up of non-cases– incorrect identification of epidemics (artifacts)– wastage of resources– inappropriate public concern (credibility)
Measles case definitions
• Suspect case– rash and fever
• Probable case– rash, fever, and (cough, coryza or
conjunctivitis)
– epidemiological link to a confirmed case
• Laboratory confirmed– saliva/serum IgM positive
Predictive value of notified measlesEffect of change in incidence
1
10
100
1000
10000
100000
1000000
Pre-vaccine Low coverage High coverage Near elimination
Num
ber o
f cas
es
0 %
20 %
40 %
60 %
80 %
100 %
PV+
Non-measles Genuine measles
• Representative = accurately describing the distribution of a event health in the population by place and time
• Related to – data quality
– bias of data collection
– completeness of reporting
Representativeness
Time
Timeliness
• Delay / speed between steps in surveillance– onset
– diagnosis
– report
– data entry
– analysis
– interpretation
– intervention
• Chronic vs acute disease
Notification completed
Notification received
Intervention
0 2 4 6 8 10 12 14
# days since onset of symptoms
(n = 255)
(n = 264)
(n = 216)
Median delay between notification and intervention by public health
servicesShigellosis, Delaware, 1991
Cost
• Cost is the estimated economic cost of the system
• Critical evaluation:– Could the system do an identical or better
job while using less resources?
SensitivityRepresentativeness
Predictive value positive
TimelinessAcceptability
FlexibilitySimplicity
Cost
Buehler's balance of systems attributes
• Description of the needs– Country context
– Disease patterns
– Target audience
• Description of the surveillance system– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
Gaps between needs and capacities
• Is the system able to– adapt to changing needs
– detect cases and outbreaks
– investigate and intervene
– disseminate and exchange information
identification of weaknesses
• Description of the needs– Country context
– Disease patterns
– Target audience
• Description of the surveillance system– Objectives
– Process
– Structure: resources, system
• Analysis of attributes
• Analysis of capacities /needs
• Implementation of prioritized modifications
Plan of evaluation
Weaknesses and their causes - - - - - - -
Prioritisation - - - - - -
Objectives for improvement - - - -
Prioritisation of weaknesses to be addressed
Example 1
• Surveillance of HIV/AIDS in Spain– notification on AIDS cases
– aggregate reporting by laboratories on new HIV diagnoses
– sentinel clinics report on HIV testing
• Which attributes are relevant?
Example 2
• Surveillance for surgical site infections in a hospital– each patient after surgery is observed
– complications are recorded on the observation card
• Which attributes are relevant?
Example 3
• Surveillance collecting information on viral meningitis in Poland– detection of enteroviral outbreaks
– investigation for poliovirus
– monitoring tick-borne encephalitis foci
– identification of new neurotropic viruses
• Which atributes are relevant?