Using Commcare Data in MVP Detailed Reference. OUTLINE BACKGROUND The Importance of Data (Slide 3) The CommCare Software (Slide 4) Summary of Data.

Using Commcare Data in MVPDetailed Reference

OUTLINEBACKGROUND The Importance of Data (Slide 3) The CommCare Software (Slide 4) Summary of Data Flow in the CommCare Platform (Slide 5) Key CommCareHQ Terminology (Slide 6)

ACCESSING COMMCARE DATA Permissions for Data Access (Slide 7) How to Manage User Access (Slide 8)

TYPES OF COMMCARE REPORTS Methods to Use CommCare Data (9) Examples of Custom Reports (Slides 11-15) Built-In Worker Monitoring Reports (Slide 16)

Worker Activity (Slides 17-18) Daily Form Activity (Slide 19) Submissions by Form (Slide 20) Form Completion (Slide 21) Case Activity (Slide 22) Form Completion vs. Submission Trends (Slide 23) Worker Activity Times (Slide 24)

Setting Up Email Delivery of CommCareHQ Reports (Slide 25) Additional Monitoring Tools (Slides 26-27)

RAW DATA Why Export Raw Data? (Slide 28) Exporting Raw Data: Cases (Slide 29) Exporting Raw Data: Forms (Slides 30-31) Why Set Up a Custom Export (Slide 32) How to Set Up a Custom Export (Slides 33-35) Where to Find Your Custom Export (Slide 36) Excel Tips for Calculating Indicators (Slide 37) Calculating Indicators from Raw Data (Slides 38-40)

CASE STUDY Using Maternal & Child Health Application to Identify Problem &

Inform Response (Slides 41-46) Using Morbidity Data to Identify Problem & Inform Response

(Slides 47-49)

SUMMARY (Slide 50)

BACKGROUND

The Importance of Data

Data collected through mobile health technologies such as CommCare can help improve health service delivery by: Providing real-time, immediate feedback on:

The performance of community health workers (CHWs) The health status of the community

Helping to identify challenges in service delivery and inform solutions Providing evidence-based support for operational decisions and health

sector programming

BACKGROUND

The CommCare Software Millennium Villages Project (MVP) uses CommCare as its primary mobile health

platform for monitoring its CHW program CommCare is a mobile phone application designed to serve as a decision guide

during household visits, a counseling aid, and a data collection tool CommCare’s web-based interface allows for real-time monitoring of CHWs’ activities

and community health status by health managers CommCare includes:

Easy to follow questionnaires with point-of-care decision making algorithms to ensure quality visits and reduce data entry errors

Longitudinal monitoring of thousands of children, mothers, and households over long periods of time SMS-based visit reminders for CHWs Mapping of cases via GPS Multimedia for strengthened counseling (video, audio, image) Capacity to produce real-time reports on a number of worker activity and health status reports

BACKGROUND

Summary of Data Flow in CommCare Platform

Cloud server

De-identified static reports

Web user

Web user

Mobile user

BACKGROUNDKey CommCareHQ TerminologyTerminology Description

Application The complete CommCare executable that gets downloaded and installed on a phone. This is the application that will be used by the mobile user to submit data to CommCareHQ. Example MVP applications include: Maternal and Child Health (MCH), Verbal Autopsy (VA), CHW Supervision

FormThe questionnaire that is filled out by the CHW. A form often corresponds to an action. In the MVP MCH application, there are 4 main form types – register a new case, visit a case, close a case, and edit the registration information of a case. In the MVP VA application, there are 3 main forms - Verbal Autopsy Neonatal, Verbal Autopsy Child, and Verbal Autopsy Adult

Case Anything that is being tracked by the CommCare system over time, often an individual or a household

Submission A form that has been sent to the CommCare server, which is dependent on internet access

Completion Time Time at which a form is actually completed, as indicated when reaching the end of the form questionnaire and pressing “save and exit”

Received Time Time at which a completed form has been received by CommCareHQ, which is dependent on internet access

Mobile User Someone who completes the forms in CommCare on a mobile device - typically a CHW, VA Specialist, or CHW Supervisor

Web User Someone who uses CommCareHQ to manage the project or view reports – typically a CHW Manager, Health Manager or eHealth Manager

Group A collection of users that can be manually classified into “groups” by geographic area (i.e. parish)

ACCESSING COMMCARE DATA

Permissions for Data Access Ask yourself: Should this user have access to

patient-level health data? If not, grant access only to aggregate-level or de-identified

reports Access to the various reports can and should be

customized based on whether this user should have access to patient-level health data

Appropriate permissions can be set when creating new web user accounts on CommCareHQ

ACCESSING COMMCARE DATA

How to Manage User Access Select “Users” in the CommCare toolbar

and “Web Users & Roles”

To assign permissions to a user: Add a user by clicking the “+Invite New

user” button Enter the e-mail address of the user you

want to invite Assign the permissions/level of access for

the user using the “Role” dropdown

To edit permissions of a role: Scroll down on the “Web Users & Roles”

page to the “Roles” section Click on “Edit Role.” You can rename the

project role if you wish Check off the permissions you want to

give to the role. Click “Save”https://confluence.dimagi.com/display/commcarepublic/CommCareHQ+Web+Users

TYPES OF COMMCARE REPORTS

Methods to Use CommCare Data CommCare mobile data submissions can be used to generate a variety

of accessible, user-friendly reports to monitor CHW performance and community health status, including: Worker Activity reports available on CommCareHQ for all CommCare projects Commissioned custom reports programmed to be available on CommCareHQ Other custom collections of indicators or visualizations generated from data

submissions in Excel or by uploading submissions to a third-party analytics tool

TYPES OF COMMCARE REPORTS

Data Tools Definition Pros Cons Examples

Custom Reports

(General)

Reports generated by manipulating CommCare raw data exports in Excel or by uploading CommCare submissions into a third-party analytics tool

• Provides the most flexibility for defining indicators and generating visualizations

• Can be easily integrated with qualitative observations• Can be e-mailed, which facilitates data consumption in

areas were internet connectivity is too unreliable to load CommCare reports regularly

• Static report• Aggregate level data only• Requires staff with strong

knowledge in Excel or other data analytics tool

• Can be time consuming to develop

• Quarterly Health Program Status Report (Microsoft Excel)

• Monthly CHW Report Card (Microsoft Word)

Custom Reports(CommCareHQ)

Data aggregated at the CHW-level for performance analysis and to protect patient data privacy. These were custom designed by the MVP team with Dimagi at the outset of the project specifically for the MVP Maternal & Child Health application

• Updated in real-time and generated automatically• Can be e-mailed, which facilitates data consumption in

areas where internet connectivity is too unreliable to load CommCare reports regularly

• Health Coordinator Report recommended for overall indicator tracking; CHW Manager Report recommended for individual performance tracking

• Can have data errors if there is an issue with how report is programmed

• Health Coordinator Report• CHW Manager Report• Verbal Autopsy Report

Worker Activity Reports

CommCareHQ report that comes with each CommCare project. This is useful for monitoring worker activity – i.e. number of CommCare submissions per day by CHW

• Updated in real-time and generated automatically• Can be e-mailed, which facilitates data consumption in

areas where internet connectivity is too unreliable to load CommCare reports regularly

• Limited to worker activity indicators only

• Worker Activity• Daily Form Activity• Submissions by Form• Form Completion• Case Activity• Form Completion vs.

Submission Trends• Worker Activity Times

Raw Data Exports

CSV or Excel files that can be exported from any CommCare project. For MVP, these raw exports contain all submissions entered by CHWs into CommCare

• Most flexibility for defining indicators and creating visualizations

• Useful for looking at case-specific information (i.e. Who are the specific U5s who did not receive a CHW visit last week?)

• Most accurate and detailed because viewing CHW submissions directly

• Large spreadsheets can be cumbersome to use and time-consuming to analyze

• Large files can take a long time to download

N/A

MVP Data Reports

TYPES OF COMMCARE REPORTS

Examples of Custom Reports in MVPHealth Coordinator Report on CommCareHQ

Aggregated health indicators. Available in real-time on CommCareHQ

Note: These reports are not automatically available on CommCareHQ. Dimagi, the company that owns CommCare, programmed them for MVP at additional cost.

TYPES OF COMMCARE REPORTS

Examples of Custom Reports in MVP CHW Manager Report on CommCareHQ

Health indicators organized by CHW. Available in real-time on CommCareHQ

Note: These reports are not automatically available on CommCareHQ. Dimagi, the company that owns CommCare, programmed them for MVP at additional cost.

TYPES OF COMMCARE REPORTS

Examples of Custom Reports in MVP CommCare Verbal Autopsy Report on CommCareHQ

Indicators on types of deaths that are occurring, including social and medical factors contributing to deaths. Available in real-time on CommCareHQ

Note: This report is not automatically available on CommCareHQ. It was developed by the MVP eHealth team.

TYPES OF COMMCARE REPORTS

Examples of Custom Reports in MVP Health Sector Report

Static cross-site health indicator performance, generated monthly in Microsoft Excel

Note: These reports are not automatically available on CommCareHQ. MVP staff designed these reports.

TYPES OF COMMCARE REPORTS

Examples of Custom Reports in MVP CHW Report Card

Selection of CHW performance indicators, generated monthly in Microsoft Word

Note: These reports are not automatically available on CommCareHQ. MVP staff designed these reports.

TYPES OF COMMCARE REPORTS

Built-In Worker Monitoring Reports on CommCareHQ 1. Select “Reports” in the toolbar

https://confluence.dimagi.com/display/commcarepublic/Worker+Monitoring+Reports

For quick performance assessment of CHWs:

• Worker Activity• Daily Form Activity• Submissions by Form• Form Completion Time• Case Activity• Form Completion vs. Submission Trends• Worker Activity Times

TYPES OF COMMCARE REPORTS

(1) Worker Activity, Pt 1Worker Activity Report shows a summary of form and case activity by user or group. Can be used to identify: Users not using the phone (no forms sent in) Amount of work done (number of forms sent in) Follow-up rates with beneficiaries

Number of forms sent in time period

Forms sent on average in time period of same length

Date of last form sent (or “None” if no forms sent)

TYPES OF COMMCARE REPORTS

(1) Worker Activity, Pt 2

New contacts registered

Child, pregnant woman, or household closed (child now over 5 yrs, woman no longer

pregnant, death, moved, etc.)

Number of contacts being visited in date range

Total number of cases per user

% of contacts visited*

*% of contacts visited could be greater than 100% if there were contacts who were closed, but also visited in the time frame

TYPES OF COMMCARE REPORTS

(2) Daily Form ActivityThis report displays daily form submissions organized by date and by user. This report allows the manager to see exactly when CHWs are working (submitting forms), and when they are able to submit the forms to CommCareHQ.

Filter by completion time or submission time

Daily number of forms completed or submitted per CHW

TYPES OF COMMCARE REPORTS

(3) Submissions by FormThis report displays daily form submissions organized by date and by user. This report allows the manager to see exactly when CHWs are working (based on when they are completing forms), and when they are able to submit the forms to CommCareHQ.

Daily number of forms submitted per CHW, per form

TYPES OF COMMCARE REPORTS

(4) Form CompletionThis report displays the average completion times for a specific form, per user, over a given time period. This allows an understanding of how long a CHW is taking to complete a form on their phone. Completion time can be an important red flag to investigate performance; if CHWs are completing a form very quickly, then they may not be using the form correctly.

No. of forms: the total number of forms submitted in the

selected time period

Average: the average completion time for the selected form

Standard Deviation: the standard deviation of this users' completion times

Shortest: the shortest completion time for the selected form

Longest: the longest completion time for the selected form

TYPES OF COMMCARE REPORTS

(5) Case ActivityThis report displays the number and percentage of open cases (i.e. child, household, pregnant women) that have been modified over a given time period. This helps identify whether the CHWs are routinely visiting their clients up to your expectation.

TYPES OF COMMCARE REPORTS

(6) Form Completion vs. Submission TrendsThis report displays a log of all submissions, comparing submission time to completion time. Long delays between completion and submission times could indicate connectivity issues or errors with a mobile user phone’s date configuration.

Filter by module and by the type of form

For each of the selected form that the CHW submitted, one can see the completion time, submission time, and time difference between the two

TYPES OF COMMCARE REPORTS

(7) Worker Activity TimesThis report is a graphical representation of the form completion or submission trends over the entire duration of the project. Y axis: displays each day of the week. X axis: displays the hours of the day. This can also be filtered by specific CHWs, as well as by a specific date range.

Filter by module and by the type of form

TYPES OF COMMCARE REPORTS

Setting Up Email Delivery of CommCareHQ Reports Click on “Reports” “My Scheduled Reports” Click the large green button “Create a New Scheduled Report” Select the saved report(s) you would like to schedule as e-mails by clicking on the button next to the report name Select the interval, day, time, recipient e-mail addresses, and whether you want an Excel report attached Click on “Submit”

https://confluence.dimagi.com/display/commcarepublic/Managing+Saved+and+Scheduled+Email+Reports

Tip: You can also generate an email report at any time by applying the desired filters in CommCareHQ and then selecting the “Email Report” button.

TYPES OF COMMCARE REPORTS

Additional Monitoring Tools “Manage Deployments” – Use these reports to support troubleshooting

Track which CHWs have updated to most recent version of CommCare Click on “Reports” in CommCare Header”Application Status” under “Manage Deployments”

Investigate if form submissions are not making it into the automated reports because of an error Click on “Reports” in CommCare Header ”Raw forms, errors, and duplicates” under “Manage

Deployments”

https://confluence.dimagi.com/display/commcarepublic/Manage+Deployment+Reports

TYPES OF COMMCARE REPORTS

Additional Monitoring Tools “Inspect Data” – Use these reports to access specific

details on CHW submissions without having to export the raw data in Excel View complete patient history

Click on “Reports” in CommCare Header”Case List” under “Inspect Data”

To find a specific user’s or group’s cases, type the name into the “Groups or Users” box in the filter

Review forms submitted by CHWs within a designated time period

Click on “Reports” in CommCare Header”Submit History” under “Inspect Data”

Review by group or individual CHW Worker Activity Reports will link to these tools too

https://confluence.dimagi.com/display/commcarepublic/Data+Inspection

RAW DATA

Why Export Raw Data? Raw data provides case-specific information, categorizing submitted

data by patient instead of aggregated information by CHW Raw data is the first place to look if any issues with custom or

CommCareHQ reports are suspected Remember raw data is organized in the export as follows:

For form exports: Each row is a visit/individual form submission by a CHW NOT unique because an export may include multiple visits/records for the same person

For case exports: Each row is a case (i.e. a unique household, child, or pregnant woman)

https://confluence.dimagi.com/display/commcarepublic/Data+Export+Overview

RAW DATA

Exporting Raw Data: Cases1. Select “Data” in the heading

3. If desired, filter for a specific group (i.e. geographic area, type of user)

4. Select “all open” (only active contacts or “all” (includes closed

contacts)

2. Select export cases 5. Download Export

RAW DATA

Exporting Raw Data: Forms1. Select “Data” in the heading

2. Select export forms

3. If desired, filter by group (i.e. geographic area, CHW supervision groups, type of user)

RAW DATA

Exporting Raw Data: Forms4. Download the form data you want

Tip: To avoid a separate export for each form type, on the right-hand side of the screen, check the box to include the desired forms in a bulk export. This is not recommend when network connectivity is poor.

RAW DATA

Why Set Up a Custom Export?Sometimes, there is too much data! Downloading full raw data exports can take longer than desired and can lead to possible errors when analyzing. If you know what information you need, you can specify for CommCare to only output those fields.

Forms: https://confluence.dimagi.com/display/commcarepublic/Form+Data+ExportCase: https://confluence.dimagi.com/display/commcarepublic/Case+Data+Export

RAW DATA

How to Set Up a Custom Export

1. Select “Data” in the heading

2. Select export forms

RAW DATA

How to Set Up a Custom Export

3. Create Custom Export

RAW DATA

How to Set Up a Custom Export4. Check which data fields you would like to include in your export. You can reorder fields by dragging and dropping fields. You can also rename fields

5. At bottom of page, press “Create”

RAW DATA

Where to Find Your Custom ExportDownload your custom export to Excel:

1. Return to “Data””Export Forms”

2. Find the name of the Custom Export

3. Download your Custom Export

1

2

3

RAW DATA

Excel Tips for Calculating Indicators from Raw Data ExportsHOW TO: Filter

When to use: to narrow down the data in your spreadsheet, allowing you to view only the information you need

Additional resources on filtering in Excel Pivot Table

When to use: to quickly summarize and analyze large amounts of data in lists and tables—independent of the original data layout in the spreadsheet—by dragging and dropping columns to different rows, columns, or summary positions

Additional resources on creating a pivot table in Excel V-lookup

When to use: to centrally locate data from multiple Excel sheets or hard-to-see location; need to match associated data (i.e. when comparing # of children visited by a CHW this month with the # children the CHW is responsible for, use the V-lookup function with CHW name as the “Lookup Value”)

Command: =VLOOKUP() Additional resources on using V-lookup in Excel

RAW DATA

Calculating Indicators from Raw Data ToolsExample 1: Number of Rapid Diagnostic Tests (RDTs) positive for malaria per CHW:

1. Export FormsChild List ModuleVisit: download as CSV file

2. In Excel, remove duplicates for form|case|@case_id This is using Excel 2008 for Mac; other Excel versions may slightly

differ. Column names may differ slightly if full export vs. custom export is used

3. Data tabPivot Table Report

This is what exports to Excel from CommCare.

Note: This example is based on the structure of the MVP MCH application.

RAW DATA

Calculating Indicators from Raw Data4. Creating an Excel Pivot Table, place form|meta|username in rows. Place form|case|@case_id in values and then place

form|patient_available|referral_follow_on|rdt_result in values as well5. Right click form|patient_available|referral_follow_on|rdt_result headerField SettingsSubtotals Custom: Count. Hide

all items but Positive6. Copy and paste the CHWs who have found +RDTs and their number of +RDTs in another Excel sheet (titled “Working”)

CHWs with 0 + RDTs will not be included in Pivot Table

RAW DATA

Calculating Indicators from Raw Data7. In another table with all CHW names, write a VLOOKUP formula to search in “Working” sheet how many +RDT cases

each CHW has. Replace N/A with 0

*Note: The Excel file pictured here is created from what CommCare exports. In a new sheet of the exported CommCare Excel file, create columns for the health indicators of interest for each CHW and use the Pivot Table function to find the corresponding values for each CHW.

CASE STUDY

Using Maternal & Child Health Application to Identify Problem & Inform Response Case Study – Appropriate Treatment of Malaria:

More severe malaria cases were coming to the health center than during similar times of the year. After

using the MCH application to access and data by CHW, it was noted that some CHWs found children with

positive Rapid Diagnostic Tests for malaria (+RDTs), but not all the CHWs were treating children with +RDTs

with coartem as per protocol. Upon further questioning of the CHWs, the local health team learned that

CHWs in several parishes had run out of coartem stock, prompting a replenishment of supplies.

Additionally, CHWs were given refresher training on the importance of timely referral of malaria cases to

health centers and reporting of stockouts.

CASE STUDY

Using Maternal & Child Health Application to Identify Problem & Inform Response Why the MCH Application was Important:

The MCH application documented that some CHWs were diagnosing malaria but not treating malaria, which

was leading to more severe cases of malaria coming to the health center. Data collected by the application

also allowed the health management team to see which CHWs were not providing appropriate treatment to

child malaria cases and to follow-up with these CHWs. Upon further investigation, the health management

team realized that the government stocks of coartem needed to be replenished. Although CHWs are

supposed to report low stock levels, they do not always do so in a timely manner. Without the MCH

application, the reason for an increase in severe child malaria cases may not have been well-understood

and the problem may not have been resolved so quickly.

CASE STUDY

Using Maternal & Child Health Application to Identify Problem & Inform Response How readily accessible reports can be used to quickly identify a problem:

Health Coordinator Report can be used to identify general issues (i.e. performance on malaria

treatment indicator is low)

CHW Manager Report can be used to identify which specific CHWs have not provided treatment for

their U-5 cases during the time period

Finally, if further investigation is necessary, raw data can be used to identify specific cases that did not

receive treatment.

CASE STUDY

Using Maternal & Child Health Application to Identify Problem & Inform Response (Continuing from Calculating Indicators from Raw Data example)

8) In Excel Pivot Table, remove form|patient_available|referral_follow-on|rdt_result and place form|

patient_available|medication_type in column

9) Right click form|patient_available|medication_type headerField SettingsSubtotals Custom: Count.

Hide all medications that do not include coartem

CASE STUDY

Using Maternal & Child Health Application to Identify Problem & Inform Response 10) Copy and paste the total count of coartem medications administered by each CHW in another Excel

sheet (titled “Working.” if CHW has not given any coartem, the CHW’s name will not be included in

the table produced by Pivot Table)

11) In the column next to the existing table with CHWs who have found +RDTs, write VLOOKUP formula

to search how much coartem each CHW has. Replace N/A with 0

12) Divide the # +RDTs by # coartem for each CHW

CASE STUDY

Using Maternal & Child Health Application to Identify Problem & Inform Response 13) In table with all CHW names, write VLOOKUP formula to search the percentage of +RDTs treated with

coartem. Replace N/A with --

14) Select this column’s cells, right click on cells, and “Format Cell”NumberCategory: Percentage to

show percentage

15) Highlight any CHW that did not treat 100% +RDT cases with coartem

CASE STUDY

Using Morbidity Data to Identify Problem & Inform ResponseCase Study – Midwife Performance:

In one MVP site, several neonatal deaths were occurring around one health center. After using the

CommCare Verbal Autopsy application to access and analyze Verbal Autopsy (VA) data on these deaths, it

was noted that the same midwife was responsible for these mothers. Upon interviewing these mothers

further, it was found that that the mothers were afraid of this particular midwife, and therefore avoiding the

health center until too late or opting to go to a traditional healer instead. The midwife was found to be

aggressive, unresponsive and intimidating to the mothers, leading mothers to delay seeking care and

ultimately leading to neonatal mortalities.

CASE STUDY

Using Morbidity Data to Identify Problem & Inform ResponseHow the VA Data Contributed to the Solution:

Trends are typically hard to notice through paper forms, which are tedious to aggregate. Electronic VA data

made it possible to see that the same midwife played a role in each of the neonatal deaths, permitting the

management team to take appropriate action. Without the VA data, factors contributing to the deaths

may have been incorrectly attributed to medical or other reasons rather than a preventable social cause.

CASE STUDY

Using Morbidity Data to Identify Problem & Inform Response1. “Data” in headingExport Forms

2. Export Full FormsVA Specialist ApplicationVerbal AutopsyDownload form of choice

If desired, filter by group

Basic mortality information can also be found by exporting the Child Close forms (Export VisitsChild ListClose Form, download CSV). This data is also included in a vital statistics section in the MVP custom reports created by Dimagi.

SUMMARY

By facilitating real-time data monitoring, CommCare can help management teams identify trends in CHW performance or community health status, and inform a response

CHWs enter data into the CommCare application on their mobile phones, which is uploaded to a cloud server. A CommCareHQ user can then access and download the data for analysis

It is important to consider what type of reporting tool will help your team reach their program goals, as well as who should have access to different levels of data

Data collected with CommCare can be accessed and analyzed through a variety of tools:

Raw data: full records of case-specific information in CSV or Excel CommCareHQ custom reports: automated reports with data aggregated by CHW or for the entire

project Other custom reports: manual, static reports generated from CommCare submissions CommCareHQ Worker Activity Reports: automated reports to monitor CHW frequency of CHW

submissions