Group 1: Cheryl Tang, Divya Balasubramanian, Hsu Yee Wint, Lai Yi Feng, Valerie Koh CO5222 Program Evaluation Assignment #3 ACTION Case Study Process Evaluation Plan
Group 1: Cheryl Tang, Divya Balasubramanian, Hsu Yee Wint, Lai Yi Feng, Valerie Koh
CO5222 Program Evaluation Assignment #3
ACTION Case Study Process Evaluation Plan
Outline
1. Purpose of Evaluationa. Engaging stakeholdersb. Roles and responsibilities of evaluation team members
2. Description of Program Aspects to be Evaluateda. Logic modelb. Key process objectives
3. Process Evaluation Design4. Data Collection and Analysis5. Recommendations and Dissemination Plan
Framework for Program Evaluation in Public Health
Engaging Stakeholders
Stakeholder Name Interest or Perspective
MOH ● Interventions leading to improved health outcomes● “Cost - effectiveness” of the interventions
AIC – Program Owner ● Capabilities of ACTION teams ● Effective partnership with step-down care facilities● Development and sustainability of the program and
resources involved
CMB & Hospital Management
● Burden of the workload on involved staff ● Reduction in hospital’s readmission rates / medical
errors upon discharge
Engaging Stakeholders
Stakeholder Name Interest or Perspective
Hospital Care Transition teams (clinicians, nurses and
case managers)
● Accuracy of decision making tool ● Adequacy of recruited patients ● Clear referral mechanism● Extent of patient’s needs addressed throughout
the program
Patients & Caregivers ● Clarity of information and communication● Satisfaction level towards care coordinators
Community Providers (GPs,Polyclinics, FMCs, Social
Support Groups)
● Clear referral mechanism in place to enable smooth transition
Members Roles and responsibilitiesPrincipal evaluator (Program Director)
● Overall coordinator, ensure evaluation carried out as planned.
● Data interpretation and final report vetting.● Communicate and Disseminate findings to key stakeholders
Co-evaluators (Program Managers)
● Study design.● Pilot testing of survey questionnaire and topic guides.● Implementation of evaluation plan.● Data collection, analyze and interpret findings● Drafting of report
Statistician (external evaluator)
● Quantitative Analysis
Sociologist/Psychologist (external evaluator)
● Draft FGD & IDI structure ● Perform thematic analysis
Roles and Responsibilities of Evaluation Team Members
Logic Model
Key Process ObjectivesStages of Program Process Objectives
At Admission Increased coverage of target population for enrolment into ACTION
Admission to Discharge Early initiation of discharge planning for patients according to their individual needs
Post-discharge Increased utilisation of post-discharge interventions
● Follow-up telephone calls● Home visits● Referrals to appropriate community service
providers
Evaluation Design
Time frame of evaluation: 4 years over entire implementation of the program.
Evaluation questions will focus on key process objectives and mapped onto the following areas of evaluation:
1. Reach/Utilization: Coverage analysis from hospital/program records2. Fidelity/Dose: Adherence to procedures from program records review3. Satisfaction: Patient/caregivers and staff feedback from surveys, interviews,
focus group discussions.
A combination of qualitative and quantitative methods will be employed for specific evaluation questions.
Timeline for Process Evaluation
Activities Month
Year 1Y2 Y3 Y4
1 2 3 4 5 6 7 8 9 10 11 12
Coverage analysis - Participation by target group
Quarterly reports
Review of program records- Fidelity and dose of services delivered
Quarterly reports
Engagement of participants and relevant stakeholders for satisfaction ratings and feedback on the program
Yearly reports
Evaluation variables
● Dependent variables = outcome variables from evaluation questions● Independent variables = Factors to stratify evaluation results by
○ Clinical disciplines○ Individual care coordinators○ Geographical regions of patients/clients
Data Collection and Data Analysis
Process Evaluation: Reach /UtilizationEvaluation Question
Data Collection
MethodSource(s) of Data Process Indicator(s) and
Standard/Criteria (if applicable) Data Analysis
How many patients were enrolled in the program?
Quantitative Program records Number of participants enrolled per hospital per quarter.
Total number of patients enrolled in ACTION
What proportion of the target population participated in the program?
Quantitative Hospital admissions and program records
Not more than 20% of patients who are assessed as eligible by the care coordinator decline to participate in the program per quarter.
Numerator: # of Patients enrolled in ACTION program in the quarterDenominator: # of patients assessed as eligible for ACTION in the same quarter
Has the program served the intended clients?
Quantitative Program records- Patients’
demographic characteristics
- Past admissions and medical history
- Social needs assessment
At least 80% of participants enrolled per quarter fulfill AIC’s selection criteria.
Numerator: # of patients who satisfy the eligibility criteria of ACTION
Denominator: # of patients enrolled in ACTION
What was the program dropout rate and characteristics of those who dropped out?
Quantitative Program records- Exit records
Not more than 10% of participants drop out prior to being discharged from the program by the care coordinator, at the end of the first quarter of the program.
Numerator: # of patients who discontinue service before planned closure/ exit assessmentDenominator: # of patients enrolled in ACTION
Variable(Patients enrolled in ACTION)
Patients discharged from ACTION Patients dropped out from ACTION
N (%),Mean P-value N(%), Mean P-value
Age
Gender
Ethnicity Chinese Indian Malay
Socio-economic status (Ward Class)
Geographic area
Number of Outpatient visits in previous year
Social Needs Assessment score
Average Length of stay in Hospital
Care Coordinator Incharge
Department Admitted to
Table: Patient Dropout Analysis
Process Evaluation: FidelityEvaluation Question
Data Collection
MethodSource of Data
Process Indicator(s) andStandard/Criteria (if
applicable)Data Analysis
Was there standardized training provided for the CCs?
Quantitative Staff training attendance records
All CCs completed the prescribed trainings on the ACTION program.
Numerator: # of recruited CCs who completed the prescribed trainings
Denominator: # of CCs recruited per year
Did the CCs follow the protocol in screening for high-risk patients?
Quantitative Hospital admissions and program records
At least 80% of patients admitted per quarter are screened for eligibility by the CCs within 24 hours of admission.
Numerator: # of inpatients who were screened within 24 hours of admission
Denominator: # of patients admitted per quarter
Was there a proper needs assessment conducted for identified high-risk patients?
Quantitative Program records At least 80% of patients who are identified as high-risk are referred to a CC for a needs assessment using the InterRai assessment suite per quarter.
Numerator: # of high-risk patients who were referred to a CC for a needs assessment using the InterRai assessment suite
Denominator: # of patients admitted per quarter who are eligible for enrolment based on AIC’s selection criteria
Process Evaluation: DoseEvaluation Question
Data Collection
MethodSource(s) of Data Process Indicator(s) and
Standard/Criteria (if applicable) Data Analysis
What interventions, how much and how often did the participants receive the interventions?
Quantitative Patients’ case notes and program records
All participants have a documented, personalized care plan, prior or upon their discharge from the hospital.
Numerator: # of enrolled patients who have a documented, personalized care plan, prior or upon their discharge from the hospital
Denominator: # of patients enrolled in ACTION per quarter
Program records- Activities log of telephone calls and home visits conducted by the CC
At least 75% of participants receive a follow-up telephone call from their assigned CC within 7 days after discharge.
Numerator: # of enrolled patients who receive a follow-up telephone call from their assigned CC within 7 days after discharge
Denominator: # of enrolled patients discharged per quarter
At least 75% of participants receive a home visit by their assigned CC within 1 month after discharge.
Numerator: # of enrolled patients who receive a home visit from their assigned CC within 1 month after discharge
Denominator: # of enrolled patients discharged per quarter
Process Evaluation: Dose (cont’d)Evaluation Question
Data Collection
Method
Source(s) of Data
Process Indicator(s) andStandard/Criteria (if
applicable)Data Analysis
What interventions, how much and how often did the participants receive the interventions?
Quantitative Program records- Date of referrals- Date patient contact by community service providers- Description of services provided
At least 75% of participants with referrals to community service providers are contacted by their first service provider within 7 days afterdischarge.
Numerator: # of enrolled patients with referrals to community service providers who are contacted by their first service provider within 7 days after discharge
Denominator: # of enrolled patients discharged per quarter with referrals to community service providers
Patient Data Fields
Hospital, Program records
Hospital records
Program records
Program records
Patients’ case notes,
Program records
Hospital records
Program records
Program records
Program records
Patient identifiers
Date and time of
admission
Date and time
screened by CC
Date of needs
assessment by CC
Care plan documented?
Date of discharge
Date of follow-up telephone
call
Date of home
visit by CC
Referral to community
service providers
SXXXXXX 11.10.15 (09:00)
11.10.15 (14:00)
11.10.15 Present / absent
13.10.15 18.10.15 1.11.15 Contacted by service provider on
15.10.15
Process Evaluation: Quality of ImplementationEvaluation Question
Data Collection
MethodSource of Data
Process Indicator(s) andStandard/Criteria (if
applicable)Data Analysis
What were the strengths, barriers and challenges in implementation?
Qualitativemethod
Sampling method - Purposive sampling (all the healthcare workers from hospital and care coordinators)Sample size - 2 FGD with 4-8 per each FGD
Instrument - open ended questions
Mode of administration - 2 FGD (1) FGD with healthcare workers who communicate with the care coordinators(2) all care coordinators
Feedback from care-coordinators on the strengths, barriers or challenges in implementation and communication with healthcare workers.
Feedback from healthcare workers on the strengths, barriers or challenges in implementation and communication with care coordinators.
Thematic analysis to find out the strengths, barriers and challenges during implementation.
Process Evaluation: Quality of Implementation
Evaluation Question
Data Collection
MethodSource of Data Process
Indicator Data Analysis
What were participants’ perceptions of the program and care coordinators?
Mixed method(Sequential design)
Qualitative methodSampling method - Quota sampling (who complete the program and who drop the program )Sample size - about 15 participants (until reach the data situation)Instrument - Open-ended questions.Mode of administration - IDIs with patients and their caregivers
Qualitative feedback from patients and their caregivers regarding their experience with the program and care-coordinators
Thematic analysis to find out the patient satisfaction towards program and care coordinators during the program regards 1)Discharge process2)Experience with care coordinator3)Experience with Program
Quota sampling of participants for IDI
participant complete the program
participant drop out from the program
care coordinator 1 3 1
care coordinator 2 3 1
care coordinator 3 3 1
care coordinator 4 3 1
Sample questions for the interview;How do you feel about the program before, during and discharge from the program?How do you feel about the care coordinators?
Process Evaluation: Quality of Implementation
Evaluation Question
Data Collection
MethodSource of Data
Process Indicator(s) and
Standard/Criteria (if applicable)
Data Analysis
What were participants’ perceptions of the program and care coordinators?
Mixed method(Sequential design)
Quantitative methodSampling method - convenience sampling (who complete the program)Sample size - all the participants who enrolled in the program and completed the programInstrument - Patient satisfaction surveyMode of administration - Self administered questionnaire upon discharge from the program
At least 60% of participants have good perception towards care coordinators and program.
Calculate the % of participants who rate the program and care coordinators as good at the end of the first year of the program
5 point likert scale - 1:strongly disagree, 2:disagree, 3:neutral, 4:agree, 5:strongly agreeSample questions:
● CC explain details about information on medication.
● I understand the information provided by CC.
● I have confidence in CC.
Recommendations and Dissemination Plan
Proposing evaluation recommendations● At the end of process evaluation, findings will be mapped into conclusions
○ Connecting evaluation questions to key process objectives in admission, discharge and post-discharge activities
○ Considering strengths, value and meaning of the findings
● From the conclusions, recommendations for future directions can be made○ Developing the next steps for adjustments to ongoing or future programs
● Conclusion and recommendations should be discussed during review meeting among key program staff and stakeholders (program owner, implementation site’s management)
○ Deciding and prioritizing recommendations○ Set target dates for implementation○ Depending on the nature of the adjustments, different stakeholders may be required to act
Linking evaluation questions to decisionsIndependent
variables Purpose Potential Recommendations
Clinical disciplines
Identify difficulties and challenges faced by each department
Review allocation of resources across disciplines
Geographical regions of patients/clients
Evaluate performance by patients/clients’ location post-discharge
Identify reason(s) for poor performance
Specific recommendations for program owner and community partners (e.g. modifications to administration of outreach activities, suggestion to expand geographical coverage by community providers)
Individual care coordinators
Evaluate performance by each care coordinator and correlate quantitative results from coverage analysis with qualitative data
Identify reasons for poor performance
Specific recommendation for individuals or group of individuals (e.g. counselling, refresher training for staff)
Implementation of evaluation recommendationsAspects of findings/recommendations Action by
Reach of the intended target audience ( no. of patients recruited, no. of referrals - physician referral to ACTION and patients’ acceptance)
Program owner and key program staff to enhance buy-in from frontline care providers such as physicians, and patients/caregivers
Utilization of services ( no.of home visits; telephone follow-ups)
Program owner and key program staff for re-allocation or resources and re-prioritization of objectives
Fidelity of services provided ( proportion of cases screened within 24 hours after admission & the proportion of cases screened accordingly to criteria/screening tools)
Key program staff and institution management. May include key opinion leaders of front-line staff it recommendation requires adjustments to existing workflows
Satisfaction - feedbacks collected and barriers identified via survey, focus groups or interviews
Frontline care providers if recommendation is care-specific. Institution/partner management if recommendation is system-specific
Target audience● Purpose of the final report on process-evaluation:
○ To understand, report and discuss learning points from ACTion implementation in a hospital over 4 years.■ Coverage and Utilization■ Fidelity and Dose■ Quality of implementation- Satisfaction and Barriers
● The following groups of people will form the target audience for the report○ Funder-MOH/MOF○ Program owner: AIC○ Stakeholders in implementation site: Management and front-line staff○ Clients/patients and caregivers○ Community providers and partner institutions
Target AudienceTarget audience Interest Purpose
Funder- MOH/MOF Coverage and UtilizationFidelity and DoseQuality of implementationAppropriatenessSatisfactionBarriers
Determine future funding potential, impact to population health or system/resource utilization
Program owner- AIC Design program adjustments, react to identified problems and challenges. Determine program success/failure, potential for expansion or need to scale down
Hospital Management Determine program success/failure, learning points for other hospital programs/initiatives
Hospital Front-line Staff SatisfactionBarriers
Determine gaps and problems, providing feedback for program refinement. Understand changes in care seeking/delivery experience by providers and end-users.Patients and caregivers
Community partners Quality of implementationAppropriatenessSatisfactionBarriers
Determine program success/failure, potential for further collaboration. Assist in workload projection and future resource allocation.
Dissemination methodsTarget
audienceInterest Dissemination methods Frequency of
report
Funder- MOH/MOF
Coverage and doseQuality of implementationAppropriatenessSatisfactionBarriers
Formal detailed report +/- presentation Annual report at the end of every year
Program owner- AIC
Formal detailed report +/- presentation Quarterly reports at the end of every quarter
Hospital Management
Formal detailed report +/- presentation Annual report at the end of every year
Hospital Front-line Staff
SatisfactionBarriers
Presentation of key findings during department meeting, and feedback gathering
Annual report at the end of every year
Patients and caregivers
Key findings and message via roadshows, article in hospital publication, short video to be screened on TVs within hospital campus
Annual report at the end of every year
Community partners
Quality of implementationAppropriatenessSatisfactionBarriers
Summary report, newsletters +/- short presentation Annual report at the end of every year
Implications for stakeholdersTarget audience Purpose Implications
Funder- MOH/MOF Determine future funding potential, impact to population health or system/resource utilization
Adjustment to funding model, structure and amount; Adjustment to KPIs or objectives.
Program owner- AIC Determine program success/failure, potential for expansion or need to scale down
Refinement of program details and ongoing/ future implementation plans, mapping findings into conclusion and recommendations
Hospital Management
Determine program success/failure, learning points for other hospital programs/initiatives
Refinement of program implementation in consultation with program owner
Hospital Front-line Staff
Determine gaps and problems, providing feedback for program refinement. Understand changes in care seeking/delivery experience by providers and end-users.
Better understanding of the program; Assisting in decision on program enrolment
Patients and caregivers
Community partners Determine program success/failure, potential for further collaboration. Assist in workload projection and future resource allocation.
Assisting in decision on continuation in partnership/collaboration, strengthening or scaling back of efforts.
Thank You!
Questions?