DATA QUALITY ASSESSMENT TOOL - WHO · The data quality assessment (DQA) tool provides a stepwise method to assess the quality of health facility data for some key coverage indicators.

1

DATA QUALITY ASSESSMENT TOOL INSTRUCTIONS FOR DATA PREPARATION

2

The data quality assessment (DQA) tool provides a stepwise method to assess the quality of health facility data for some key coverage indicators. Users are asked to prepare data from their routine facility reporting, also called the health management information system (HMIS) in a specific format. Each of the worksheets in the tool and their data requirements will be described in detail below. This DQA tool requires users to enter or paste the data and it will result in a Data Quality Report Card being created. There are a total of 19 worksheets in this tool, including the cover page, the report card and graph worksheets. The user has to input (enter/paste) information in 12 of these worksheets. The following instructions will describe each of the worksheets and the data that needs to be entered in the individual worksheets. The main indicators selected for this tool and for the Report Card are ANC1, deliveries, DTP1, DTP3 and OPD. General information about the tool

1. In the data entry sheets which are tabbed blue , the following information applies:

a. Cells in yellow are total cells and are write-protected b. Only cells in green allow data entry c. Cells in white are also write-protected as they are linked to other cells

in the same or different worksheets d. When you copy and paste data, please use the paste special function and

only paste "Values" e. The icon indicates a cell with a drop-down list

2. The red tabbed are write-protected. The user will not be able to change any information on these sheets.

When you open the tool When you open the tool some dialog boxes might open. The type of dialog boxes are shown below. Please follow instructions as given when you see a dialog box.

If you get the warning as shown above, please click on “Enable content” to proceed.

Please note: Data in your HMIS database are kept in different tables. Sometimes your administrative units are ordered differently in the different tables. To make sure you are copying and pasting the correct information for each administrative unit, please ensure that the data is for the appropriate administrative unit.

3

If the dialog box as shown above open, please click on “Don’t Update” button.

The preceding dialog box explains what the tool does. Please read the information and click on the “Ok” button.

This box might appear multiple times. Please click the “Ok” button until this box disappears. Worksheet # 1: “Title” The first sheet is the “Title” page. It has the version number and the date of last update made to the tool.

4

Worksheet # 2: “Input_basic_info” There are 7 items/questions that need to be filled in this sheet. Instructions are given below for each of the items/questions separately.

1. Please use the drop-down list and select your country

2. This tool can analyse data quality for one year at time. Please select the year for

which you are analysing data quality. If your reporting year stars at some other month besides January, e.g. June, please select the year in which the last month of reporting falls. If you want to look at data quality for another year, you have to open a copy of the blank file and input data for the new year.

3. Please use the drop-down list to select the reporting levels in your country. The

examples, below a couple of different scenarios that might apply to your situation.

5

4. This question requires the administrative level for which data quality analysis is being conducted. Please ensure that the administrative level at which you are analysing data is one of the reporting levels specified in item/question 3. If the administrative level of analysis is a district, please select “District” from the drop-down list. The one main criterion for selecting the reporting level of analysis is that it is essential to have population estimates for the level of analysis.

Example 1: The reporting levels in your country are Facilities District Province National

1. The first level is the facility and cannot be changed 2. In the 1st green drop-down box, please select “District” 3. In the 2nd green drop-down box, please select “Province” 4. Since provinces report to the national level, which is the highest reporting level, please

leave the last green drop-down box blank

Example 2: With many countries instituting web-based HMIS, facilities are now directly reporting to the national level. In this case, please input the level that has administrative responsibilities for facilities. For example, if a province is responsible for all the facilities in within its border, then the reporting level would be Facilities Province National.

1. The first level is the facility and cannot be changed 2. In the 1st green drop-down box, please select “Province” 3. Since provinces report to the national level, which is the highest reporting level, please

leave the last two green drop-down boxes blank

Example 1: The reporting levels in your country are Facilities District Province National. However, your facility catchment area population estimates are not very robust. You believe your district level estimates are better, as these are projections from the census/statistics bureau.

1. You will select “District” from the green drop-down box Example 2: In the example where there is a web-based HMIS system and facilities are directly reporting to the national level, examine the reporting levels and determine the levels for which you have population estimates. If the administrative levels in your country are Facility District Province National. However, you do not have population estimates for the facility or district level. You do have population estimates for the provincial level from census projections.

1. You will select “Province” from the green drop-down box.

6

5. In this green drop-down box, please select the level for which are inputting service output data. Service output data, here, means data ANC1, deliveries, DTP1, DTP3 and OPD. You can input service output data starting from the facility level (based on what information you have available to you).

6. Due to large sample sizes and costs, surveys often only provide population estimates at either the national, state/province/regional level. Please provide the aggregation level for the most recent population-based survey. If you have multiple surveys conducted within a short time frame from each other, please select the aggregation level used in the survey you are most interested in.

Example 1:

In your country you have a web-based system where facilities directly report to the national level. You have facility level data and you would like to analyze with facility level data.

1. You will select “Facility” from the green drop-down box Example 2: The reporting levels in your country is Facility District Province National. However, as facilities submit paper forms to the district, and the districts submit only aggregate data to the province, the starting reporting level that you have information for is the district.

1. You will select “District” form the green drop-down box

7

7. Please type in the names of the months starting with the first month of the reporting year. For example, if the reporting year starts in July and goes till June of the following year, the first row will have July and the last row will have June.

8

Worksheet # 3: ““Input_admin_units”” In this worksheet, the user will be entering details on the names of the administrative units for the level which they are inputting facility data and the corresponding reporting levels. Two examples are presented below. Col. 1 title “No” is the same in all worksheets. This column is used for numbering the administrative units that will appear in Column 2. When there are names of 100 administrative units in Col 2, please type 1-100 in Column 1. Please DO NOT type any additional numbers beyond the total number of administrative units that you are analysing. Columns 2, 3, 4, and 5 are filled based on selections that were made for the different items/questions in “Input_basic_info” worksheet. Example 1:

The country reporting levels from item/question 3 in “Input_basic_info”_worksheet is Facility District Province (since province reports to the national level, the national levl is not entered in the drop-down box). Column 2 says “District” and is the administrative level for which you are inputting data. Even though facilities are the first administrative level, the user selected the level of data entry at the district level (answer to item/question “5” in the “Input_basic_info” worksheet). Hence, the administrative unit level that appears in Column 2 is “District.” Column 3 says “Province” and is the level to which Column 2 reports to (if it exists). Explanation of example above:

1. Enter 1-n for the total number of administrative units appearing in Column 2. In this example, 1-12 has been entered for the total number of districts in Column 2.

2. Enter names of all the administrative units in the administrative level for which you are inputting data. In example 1, please paste or type in the name of all the districts in your country.

9

3. Enter the names of the administrative units to which the administrative units in Column 2 report to. If there is no reporting level for Column 2, Column 3 will not have an administrative unit level title. In this example, Districts report to Provinces.

Columns 4 and 5 are not populated as Province is the last reporting level before the national level. Example 2:

The country reporting levels from item/question 3 in “Input_basic_info”_worksheet is Facility District Province. Your country has a web-based HMIS where facilities report directly to the national level. However, administratively, your facilities are managed by districts which in-turn report to provinces . These columns were populated based on the following selections made in the “Input_basic_info” worksheet.

1. Column 1: Since data is being inputted at the facility level (as shown by Column 2) and there are 30 facilities, please number 1-20 in Column 1.

2. Column 2: Enter the names of all the facilities in your country. 3. Column 3: Enter the names of the districts to which the facilities report to. 4. Column 4: please enter the names of provinces to which the districts report to.

Column 5 is not populated as Province is the last reporting level before the national level. Please use "Paste special" and paste ONLY VALUES when you are copying and pasting information.

10

Worksheet # 4: “Survey_mapping” This worksheet will map the administrative unit level that is being analysed to aggregation level that is used in the most recent population-based surveys. For example, in a country the reporting levels for facility data is Facilities Districts Provinces National. However, it is rare for a survey to provide district level estimates for key indicators, especially in a country where there are many districts. Typically, survey estimates are for higher aggregation levels, such as province level. There are two examples given below that will explain a couple of different scenarios. Example 1

The country reporting levels from item/question 3 in “Input_basic_info”_worksheet is Facility District Province. However, the administrative unit of analysis (as selected in item/question 4) is “District”.

1. Column 1 title, “District”, is automatically populated by the response choice in item/question 4 in “Input_basic_info” worksheet.

2. The administrative unit names in Column 1, in this case the district names, are automatically populated as well from the information entered in “Input_admin_units” worksheet. [Columns that are white are write-protected].

3. Column 2 title, “Province”, is automatically populated based on the selection made on item/question 6 in “Input_basic_info” worksheet. In the example above, the user

Please note: Data should only be entered/pasted in the green colored cells. The cells in white and purple/mauve are column headings and are write-protected. User will not be able to enter data in the white or purple cells.

11

needs to map the districts to the provinces as seen above. Districts A, B and C are in Provine M, Districts D and E are in Province O and so on.

Example 2: Sometimes the survey aggregation levels are not similar to existing administrative levels. Example 2 below gives an illustration of this situation. In example 2, the survey aggregation level selected in item/question 6 in “Input_basic_info” worksheet was “Region”. The country’s administrative levels are Facilities Districts Provinces National. “Region” is not an adminsitrative level. However, the survey aggregation was done at a regional level. In this case, you would map the main unit of analysis (the “District” was selected in item/question 4 in “Input_basic_info”_worksheet) as best as possible to the “Regions” that were selected for the survey. In the figure below, Districts (the main unit of analysis) A, B, C, D, E, F, G, H, I have been mapped to regions.

In example 2, the survey aggregation level selected in item/question 6 in “Input_basic_info” worksheet was “Region”. The country’s administrative levels are Facilities Districts Provinces National. “Region” is not an adminsitrative level.

1. Column 1 title, “District,” is automatically populated by the response choice in item/question 4 in “Input_basic_info” worksheet.

2. The administrative unit names in Column 1, in this case the district names, are automatically populated as well from the informatin entered in “Input_admin_units” worksheet. [Columns that are white are write-protected].

3. Column 2 title, “Region”, is automatically populated based on the selection made on item/question 6 in “Input_basic_info” worksheet. In this case, you would map the main unit of analysis, “Districts” to “Regions” that were selected for the survey. In the figure below, Districts (the main unit of analysis) A, B, C, D, E, F, G, H, I have been mapped to Regions 1 and 2.

12

Worksheets # 5 – 9: “Input_ANC1”, “Input_Deliveries”, “Input_DTP1”, “Input_DTP3” and “Input_OPD” Tables 3-7 are found in the worksheets “Input_ANC1”, “Input_Deliveries”, “Input_DTP1”, “Input_DTP3” and “Input_OPD”. Please enter the monthly service outputs for ANC1, deliveries, DTP1, DTP3 and OPD in the appropriate worksheet. Only 1 example table (for ANC1) is shown below as the table format is exactly the same in the above worksheets.

1. Columns 1 –5 are automatically populated based on information entered in the “Input_admin_units”

2. In Columns 6 – 17 please fill in the service outputs for ANC1, deliveries, DTP1, DTP3 and OPD in the respective worksheets.

3. Each cell in Column 18 is a sum of the service outputs entered in columns 6-17 for a specific row.

4. Column 19 counts if there are any missing values in the data entered. If the user only enters data for 6 months, the missing values will only be counted amongst the months for which data has been entered. For example, if the user has entered data for Jan-June and the remaining 6 columns are blank, the formula in this column will only search for missing values in the months Jan-June.

5. Column 20 counts all the zero values in the entered data.

Please note: Data should only be entered/pasted in the green colored cells. The cells in light yellow are protected and contain totals. The cells in white and purple/mauve are column headings and are also write-protected. The cells in white are administrative unit numbers and names. User will not be able to enter data in the white, purple or yellow cells.

13

Example for Worksheets # 5 – 9

Example for Worksheet # 10

14

Worksheet # 10: “Input_trend” In this worksheet, Table 7, please input (enter/paste) annual service outputs for ANC1, Deliveries, DTP3 and OPD for up to three years preceding the year of analysis for each admin unit. If you do not have data for all three years please enter the data for the years you have complete information.

1. Columns 1 to 5 are automatically filled based on the information filled in the “Input_basic_info” and “Input_admin_units” worksheets.

2. ANC1, Deliveries, DTP3 and OPD each have four columns. The user can input historical service output data for up to three years. The last column for each indicator is the total service output for the data analysis year. This column in write-protected and the totals are carried over from the information inputted in “Input_ANC1”, “Input_Deliveries”, “Input_DTP3” and “Input_OPD”.

3. The user does not have to input data for all three past years. They can input data for available years.

4. In the example above, the data for one of the years for deliveries is missing. This is allowed.

5. The historical indicator data has to be for consecutive years. This information is automatically determined by the tool. For example, if the year of a data analysis is 2011, the required historical indicator information would be for 2008, 2009 and 2010.

Please note: Data should only be entered/pasted in the green colored cells. The cells in light yellow are protected and contain totals. The cells in white and purple/mauve are column headings and are also write-protected. The cells in white are administrative unit numbers and names. User will not be able to enter data in the white, purple or yellow cells.

15

Worksheet # 11: “Input_population” In this worksheet, Table 1, you will be entering information about the administrative unit of analysis such as their names, denominators for the indicators analysed in this tool and reporting completeness information by the administrative level that is being analysed.

The following information is required for each of the Columns specified.

1. Col 1 is the number for the administrative unit of analysis. The numbering will be in ascending order from 1 to the total number of administrative units in your country that you are analysing in this exercise (based on answer to question 4 in “Input_basic_info”). If your unit of analysis is district (from question 4 in (“Input_basic_info” worksheet) and you have hundred districts in your country, then Col 1 will be numbered 1 to 100.

2. Col 2 lists the names of the administrative units in the administrative level that is being analysed. The administrative unit names are filled based on the answer to question 4 in “Input_basic_info” (on the administrative level of analysis) and names of administrative units that were entered in “Input_admin_units” worksheets.

3. Col 3 requires the user to enter the population (or population projection) for each administrative unit for the year of analysis.

Please note: Data should only be entered/pasted in the green colored cells. The cells in light yellow are protected and contain totals. The cells in white and purple/mauve are column headings and are also write-protected. The cells in white are administrative unit numbers and names. User will not be able to enter data in the white, purple or yellow cells. .

16

4. Col 4 to Col 7 – Please enter the total number of expected pregnancies, expected births, surviving infants and total population for each administrative unit. These totals are denominator information that will be used to calculate rates. Expected pregnancies will be used to calculate denominators for ANC1, expected births for deliveries, surviving infants for DTP3, and total population for outpatient visits (OPD). Please input the denominator information that is normally used to calculate coverage rates for these indicators.

Worksheet # 12: “Input_reports_received” The information entered in this worksheet will be used to calculate reporting completeness.

1. Column 1 is the number for the administrative unit of analysis. The numbering will be pre-filled based on the total numbers of administrative units in the administrative level selected for anlaysis. For example, Column 1 has been pre-filled 1-12 as there are 12 districts in the country. District was the selected administrative level of analysis (from question 4 in “Input_basic_info” worksheet.

2. Column 2 lists the names of the administrative units in the administrative level that is being analysed and is pre-filled based on the information in “Input_admin_units” worksheet.

3. In Column 3 please fill in the total number of reports received from the administrative level selected for the data quality analysis. For example, if District A is supposed to submit a monthly report to the province and they submitted 12

17

reports in 2011, you should enter 12 in the cell. It is possible, especially in countries that have web-based reporting at the facility level, there might not be an reports received from an intermediate level.

4. In Column 4 please enter the total number of health facilities providing services in each administrative unit for the year of analysis (please enter the total number that is used to normally calculate completeness rate).

5. In Column 5 please enter the total number of reports received from health facilities in the administrative unit for the year of analysis. District A has 110 reporting facilities and a total of 1209 reports received in 2011. If all expected facilities are supposed to report every month, the total number of expected reports for Admin unit 1 would have been 204 reports. However, only 190 reports were received. This indicates that not all facilities submitted a report every month.

At the top right corner of the worksheet, there are two other items that need to be filled. The first item asks for the reporting cycle for facilities. Please select the answer from the drop-down list. The second item asks for the reporting cycle at the administrative unit level. Please select from the drop-down list.

18

Worksheet # 13: External_data_sources In Table 11 below, the user will be entering coverage estimates from population-based surveys to compare with facility-based coverage rates.

1. In the row referred to as (1) please select your data source from the drop-down list. 2. In row referred to as (2) please select the year of the estimate from the drop-down

list. 3. In Columns 3a and 3b, please enter the coverage estimates and the standard errors

for ANC1. In row that has “Overall estimates”, for the aforementioned columns, please enter the overall (most often national) coverage estimate and standard error for ANC1. In subsequent rows please enter the survey coverage estimates for the administrative level or aggregation level specified in “Survey_mapping” worksheet.

4. Please do the same for Deliveries (Columns 4a and 4b), DTP1 (Columns 5a and 5b) and DTP3 (Columns 6a and 6b).

5. The columns for the survey estimates and the SE are all in "%" format. Please do not change this format. For example, if your survey report says the ANC estimate is 0.89, please enter 89 in the column and not 0.89. MICS and DHS reports have standard errors (SE) for a number of indicators in the annex of the survey report. Please see you can find the SE for the indicators listed below. If you find that the SE for DTP3 is 0.04, please enter 4 and not 0.04. If you do not find SE for the indicators listed

1

2

19

below, please enter 0 (zero) in the SE column. Please DO NOT leave the SE column blank.

6. Above Table 11, there is an item that says “Total # of live births for the year of analysis” and a white cell with a number. This number of live births is pre-filled from the World Prospects database (2010 version) from the United Nations Population Division. This number is pre-filled as soon as you select your country of analysis in the “Input_basic_info” worksheet. This number will be used to compare to the denominators for live births used for calculating some of the indicators. If you are doing sub-national analysis, the information in this cell will not be applicable to you.

Worksheet # 14: Report_card In this worksheet, the Data Quality Report Card (DQRC) indicators will be calculated for the user based on the data that has been inputted. The DQRC examines and presents a data quality “score” for indicators spanning across four dimensions of data quality. These dimensions are: 1) completeness of reporting; 2) internal consistency of reported data; 3) external consistency of population data; 4) external consistency of coverage rates. The indicators are described in detail in the document titled “Guide to the Health Facility Data Quality Report Card,” which can be found at: (add link)

1

20

The “Report Card” worksheet provides results for the indicator at the national level and sub-national level. At the sub-national level, it also identifies administrative units that are performing poorly. For example, please see (1) in the figure above. The right most column says “District C.” Calculations are automatically done in the tool and sub-national units with less than 80% facility reporting rates are listed. In this example, District C has facility reporting rate of less than 80%. Worksheets # 15-19: Graphs Worksheets 15-19 provide graphs for some of the indicators.

1. Worksheet 15 “Graphs_indicator_2b” presents graphs for indicator 2b which looks at the consistency of the indicators over time.

2. Worksheet 16 “Graphs_indicator_2c” shows the consistency between ANC1 and DTP1.

3. Worksheet 17 “Graphs_indicator_2d” shows the consistency between DTP3 and DTP1.

4. Worksheet 18 “Graphs_indicator_3b1_3b2” compares different population denominators

5. Worksheet 18 “Graphs_indicator_4a-4c” compare coverage estimates from facility data with coverage estimates from surveys for ANC1, deliveries and DTP3.