Task Force framework for assessment of surveillance data using examples from Tanzania, Brazil and Russia Ana Bierrenbach Geneva, March 2010
Task Force framework for assessment of surveillance data
using examples from Tanzania, Brazil and Russia
Ana Bierrenbach
Geneva, March 2010
March 26, 20102 |
Task Force frameworkTask Force framework
March 26, 20103 |
Standards or benchmarks for direct measure of TB incidence and mortality
Standards or benchmarks for direct measure of TB incidence and mortality
� To be attained by TB surveillance system
� To be attained by TB surveillance data
� To be attained for direct measurement of trends
� To be attained for surveillance systems to be
considered as missing negligible number of cases and
deaths
March 26, 20104 |
Progress in applying framework (2009)Progress in applying framework (2009)
4 regional workshops (red) with > 50 countries,
3 country missions (blue)
March 26, 20105 |
What do we do in the workshops
and country visits?
What do we do in the workshops
and country visits?
WORKBOOK
March 26, 20106 |
1. Data quality1. Data quality
March 26, 20107 |
Data qualityData quality
1. Completeness of notification data and other quality checks
• Are all reports complete and compiled?
• Are there duplications and misclassifications?
2. Internal consistency
• Is there more sub-national variability in notification rates than
expected?
• Is there more variability over time than expected?
• Is laboratory diagnosis of documented quality?
3. External consistency
• Are proportions and rates consistent with current knowledge on TB
epidemiology?
March 26, 20108 |
Removing duplicates in Brazil (2006)Removing duplicates in Brazil (2006)
+4.465.162.3-6.337.439.969,80274,492
afterbeforeafterbeforeafterbefore
Change(%)
Cured
(%)
Change(%)
Notification
rate
New cases
Source: Bierrenbach A et al. IJTLD 2010 (in press)
Duplicates/misclassifications inflate TB notification rates
March 26, 20109 |
Trends in TB notification rates
by case type in Tanzania
Trends in TB notification rates
by case type in Tanzania
05,000
10,00015,00020,00025,00030,00035,00040,00045,00050,00055,00060,00065,00070,000
Smear + Smear - Extra-P Relapse Return Failure Other
Smooth curve in line with HIV epidemic
Consistency in the distribution of case types over time
Reporting of retreatment case initiated 2001
March 26, 201010 |
Notification rates by region in Tanzania - 2007Notification rates by region in Tanzania - 2007
0
200
400
600
800
1,00 0
1,20 0
1,40 0
1,60 0
1,80 0
2,00 0
Ne
w c
ase
s pe
r 100
K p
op
Aru
sha
Dar
es s
ala
am
Dod
om
aIr
inga
Ka
ge
raK
aska
zin
i P
em
ba
Kaska
zin
i U
ng
uja
Kig
om
aK
ilim
an
jaro
Ku
sin
i P
em
ba
Ku
sin
i U
ng
uja
Lin
di
Ma
gh
ari
bi U
ng
uja
Man
yara
Mara
Mb
eya
Moro
go
roM
twara
Mw
an
za
Pw
an
i
Ru
kw
aR
uvum
aS
hin
ya
nga
Sin
gid
a
Ta
bo
raT
an
ga
Overdispersion of TB notification rates, only partially explained by HIV prevalence
March 26, 201011 |
Indicators for internal and external consistency
by region in Tanzania - 2007
Indicators for internal and external consistency
by region in Tanzania - 2007
0
20
40
60
80
100
% n
ew
/all
Aru
sha
Dar
es s
ala
am
Dodom
aIr
inga
Kagera
Kaskazin
i P
em
ba
Kaskazin
i U
nguja
Kig
om
aK
ilim
anja
roK
usin
i P
em
ba
Kusin
i U
nguja
Lin
di
Maghari
bi U
nguja
Manyara
Mara
Mbeya
Moro
goro
Mtw
ara
Mw
anza
Pw
ani
Rukw
aR
uvum
aS
hin
yanga
Sin
gid
aT
abora
Tanga
0
20
40
60
80
100
% p
ulm
/new
Aru
sha
Dar
es s
ala
am
Dodom
aIr
inga
Kagera
Kaskazin
i P
em
ba
Kaskazin
i U
nguja
Kig
om
aK
ilim
anja
roK
usin
i P
em
ba
Kusin
i U
nguja
Lin
di
Maghari
bi U
nguja
Manyara
Mara
Mbeya
Moro
goro
Mtw
ara
Mw
anza
Pw
ani
Rukw
aR
uvum
aS
hin
yanga
Sin
gid
aT
abora
Tanga
% new / all % pulm / new
% new / all : consistent, but above expected value
% pulm / new : consistent, variations in line HIV prev.
March 26, 201012 |
Standards/benchmarks forTB surveillance system
Standards/benchmarks forTB surveillance system
� NTPs routinely assess completeness of reporting of TB notification data
� Clear documentation of changes to case definitions and reporting of cases for at least the last 10 years
� Data on case finding efforts and main determinants of TB are available for several years
� NTPs have a case-based electronic TB database
� NTPs check and correct for duplications and misclassifications in notifications: linkage within the last 3 years covering a period of at least 5 years
� NTPs perform inventory studies using routinely collected data and through special studies where appropriate (large number of providers TB care not linked to NTP).
� NTPs analyse TB mortality data from vital registration systems
March 26, 201013 |
Standards/benchmarks forTB surveillance data
Standards/benchmarks forTB surveillance data
� Limited missing data for main variables, for at least 10
years
� Limited variability over time and space, for at least 10
years, or notification data show considerable variability,
which can be explained
� Data consistent with expected values based on TB
epidemiology knowledge
� Limited duplications and misclassifications in full audits
and spot checks, for at least 5 years
March 26, 201014 |
2. Trends2. Trends
March 26, 201015 |
Do surveillance data reflect trends in incidence and mortality?
Do surveillance data reflect trends in incidence and mortality?
1. Have notifications been increasing, decreasing or stable over
time?
2. Were there any changes in case-finding efforts and/or recording
and reporting that might have affected notifications over time?
3. How have factors that may influence TB incidence changed over
time, and have they had any impact on underlying TB
incidence?
March 26, 201016 |
0.00
0.02
0.04
0.06
0.08
1970 1980 1990 2000 2010
HIV
pre
vale
nce
Tanzania: Adults 15 years old or moreUNAIDS estimates
0
200
400
600
800
1000
Full-
tim
e T
B n
urs
es
1995 2000 2005 2010
Factors affecting
TB notifications in Tanzania
Factors affecting
TB notifications in Tanzania
0
200
400
600
800
Sm
ear
labs
1995 2000 2005 2010
↑ notification mirrors ↑ case finding and HIV, but difficult to disentangle given lack of data disaggregated by HIV and case type, for ≠ years
March 26, 201017 |
0
20000
40000
60000
80000
100000
120000
140000
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Nu
mb
er
of
ne
w T
B c
as
es
0
20
40
60
80
100
120
Ne
w c
as
es
pe
r 1
00
,00
0 p
op
ula
tio
n
Reported new civilian cases Reported new non-civilian cases
TB notification rate (civilian) TB notification rate (non-civilian)
Civilian TB
Non-civilian TB cases
Source of referral
of TB notifications in Russia
Source of referral
of TB notifications in Russia
Impact of changes in notification policy
March 26, 201018 |
Standards/benchmarks for direct measurement of TB trends (examples)
Standards/benchmarks for direct measurement of TB trends (examples)
� Trends in TB notifications assessed to mirror trends in incidence:
– Trends in TB notifications mirror trends in incidence after corrections for changes in case definitions, policies in reporting and case finding
• Need notifications disaggregated on HIV status by case type, at least in high HIV burden countries
– Trends in TB notifications mirror trends in TB mortality, and no evidence in changes in case fatality rates
� Trends in TB deaths assessed to mirror trends in mortality:
– No evidence of changes in consistency in coding, policies in reporting and coverage of the VR system over 10 years
March 26, 201019 |
3. Are we capturing all cases and deaths?
3. Are we capturing all cases and deaths?
March 26, 201020 |
Are we capturing all TB cases and deaths?Are we capturing all TB cases and deaths?
1. What proportion of incident cases are missing from
routine notifications and why?
2. What proportion of TB deaths are missing from the
vital registration systems and why?
March 26, 201021 |
The Onion Model
All TB cases
Undiagnosed cases
Diagnosed but not notified cases
Notified casesRecorded in
notification data
Diagnosed by NTP or collaborating
providers
Diagnosed by public or private providers, but
not notified
Access to health facilities, but don't go
No access to health care
Presenting to health facilities, but undiagnosed
March 26, 201022 |
Estimates in Tanzania
before and after discussions
Estimates in Tanzania
before and after discussions
-7586.2Country's CDR (2007)
-2413.8Sum of % of missing cases:
layers 2 to 6
1.51.52. Diagnosed by NTP but not notified
- Exclusive distribution of TB
drugs by NTP
0.90.93. Diagnosed by public non-NTP
92.44. Presenting but not diagnosed
-↑ diagnosis TB following
introduction recent
interventions
-↑ diagnostic delay
52.75. Access but do not go
-93% pop within 10 km basic
health care unit6.36.36. No access to health care
Source of evidenceAfter
discussions
Before
discussions
Onion layers (% total new cases
missed in each layer)
March 26, 201023 |
Can we avoid having to
depend on expert opinion?
Can we avoid having to
depend on expert opinion?
� Inventory studies
� Vital registration data
� Capture-recapture studies
� Prevalence of TB disease surveys (health care seeking behaviour)
� Innovative operational research
March 26, 201024 |
Substantiating expert opinionSubstantiating expert opinion
� Access to health from demographic and health surveys data (Layer 6)
� Overall performance of health systems as measured by: (Layer 5, 6)– Infant mortality ratio
– Number of primary heath care units or doctors per population
– % of assisted births
� Performance of TB diagnostic systems (Layer 4, 5)– % people who died from TB (Vital registration data) and never accessed
TB diagnosis and treatment
– EQA of labs
– KAP studies (health seeking behaviour), delay studies
� Contribution of different TB care providers (Layer 3)– Health expenditure in the private or nongovernmental organization sectors,
out-of-pocket expenditure
� TB drug distribution (Layer 2)
March 26, 201025 |
Standards/benchmarks to be attained for surveillance systems to be considered as missing
negligible number of cases and deaths
Standards/benchmarks to be attained for surveillance systems to be considered as missing
negligible number of cases and deaths� VR systems meets minimum standards of quality and coverage
� Excellent geographical and financial access to health care facilities where TB diagnosis is available
� Staff have proven competency to diagnose TB
� Bacteriology laboratories are quality assured
� Limited or negligible undernotification from the private sector as evidenced by:
– Limited health care expenditures in the private sector (e.g. ≤ 5%)
� Limited or negligible undernotification from the public sector as evidenced by:
– All public providers only have access to anti-TB drugs following notification of a TB case
March 26, 201026 |
Framework for TanzaniaFramework for Tanzania
Relied on "Onion" model based on (mostly) expert opinion combined with some evidence about health system coverage/access and some TB-specific KAP study data
DATA QUALITY
TRENDS
Do surveillance data
reflect trends in TB incidence and
mortality?
ARE ALL TB CASES AND TB
DEATHS CAPTURED IN
SURVEILLANCE DATA?
IMPROVE
surveillance
Data do not yet provide direct measurement. Incidence estimates revised downwards (CDR up). Prevalence survey will provide important new data
UPDATE estimates of TB incidence and mortality
Completeness – appears to be verified, BUT analysis not available at national level Cannot assess duplications/ misclassifications since data are aggregated (not case-based); Data mostly consistent, within ranges expected, but extreme values in a few regions
EVALUATE
trends and impact of TB control
notifications ≈ TB incidence
VR TB mortality ≈ TB deaths
HIV and case-finding have affected trends in notification BUT difficult to disentangle effects - notifications disaggregated by HIV status not available and data on case-finding only available at national level
Need data disaggregated by HIV status at district level to be compiled at national level
4. Implement updated R&R recommendations
1. Roll-out case-based ERR
3. Strengthen M&E supervision
2. Routinely assess data quality esp. in "outliers"
March 26, 201027 |
Major lessons learnedMajor lessons learned
� Vital importance of case-based electronic R&R systems
� Data on notifications by case type disaggregated by HIV status need to be routinely collected, especially in high HIV-prevalence setting
� Need to strengthen analytical capacity
� Need for improvement for supervision of M&E
� Need to progressively increase reliance on data to assess cases missed by surveillance and progressively reduce use of expert opinion
� Need for standards and benchmarks
� Need for better availability and use of vital registration data, as well as more inventory studies and operational research
March 26, 201028 |
Major next stepsMajor next steps
� Further workshops and country missions with Global Fund
� Train group of consultants to participate in country missions
� Develop and reach agreement on standards / benchmarks
– Working group to further develop standards/benchmarks
� Help countries transition to TB electronic R&R
– Minimum standards for a TB electronic R&R system + software specifications
needed to achieve them
� Develop training material (SOPs, guidelines):
– Operational research to assess data quality and M&E systems
– NTP supervision of M&E activities
– Programme review missions
� Jointly review M&E GF grant applications
March 26, 201029 |
Questions to the Task ForceQuestions to the Task Force
1. What are your general comments on progress to date in applying the Task Force framework?
2. Do you agree with the proposed next steps? Is there
anything important that is missing or should be deleted?
3. Which of the standards listed in the annex do you agree or disagree with, and do you have any suggestions for
additional standards that could be used?
4. Would you be willing to be part of a small working groupresponsible for further development of standards and
benchmarks that would need to be met for surveillance data to be considered a direct measurement of disease burden?