Healthdirect Australia Research and Evaluation Strategy 2014-18 October 2014
Healthdirect Australia
Research and Evaluation Strategy
2014-18
October 2014
Healthdirect Australia Research and Evaluation Strategy 2014-2018 2
Contents Contents ......................................................................................................................... 2
Introduction .................................................................................................................... 4
Building the research and evaluation agenda ............................................................... 4
Development of the strategy ......................................................................................... 5
Part 1: Aligning to Healthdirect Australia Strategic directions ................................... 5
Research and evaluation as an integral part of Healthdirect’s work program ................ 8
Aims of the research and evaluation strategy ............................................................... 9
Part 2: Research and evaluation governance ............................................................ 10
Purpose ...................................................................................................................... 10
Aim ............................................................................................................................. 10
What we will do .......................................................................................................... 10
Oversight .................................................................................................................... 10
Funding ...................................................................................................................... 11
Research governance ................................................................................................ 11
Part 3: Setting research and evaluation priorities ..................................................... 13
Purpose ...................................................................................................................... 13
Aim ............................................................................................................................. 13
What we will do .......................................................................................................... 13
Healthdirect research and evaluation priorities for 2014-15 ........................................ 15
Research, evaluation and monitoring priorities ....................................................... 15
Part 4: Develop and implement an evaluation framework ........................................ 16
Purpose ...................................................................................................................... 16
Aim ............................................................................................................................. 16
What we will do .......................................................................................................... 17
Routine collection of data ........................................................................................... 20
Using the results of evaluation studies ....................................................................... 20
Part 5: A framework for responsive research ............................................................ 22
Why research is important for Healthdirect ................................................................. 22
Aim ............................................................................................................................. 22
What we will do .......................................................................................................... 22
Partnering in research ................................................................................................ 22
Commissioning researchers ....................................................................................... 23
Applying Criteria for selecting research partners ........................................................ 23
Healthdirect Australia Research and Evaluation Strategy 2014-2018 3
Managing requests for data and research collaborations ........................................... 23
Commissioning research or reviews of research ........................................................ 23
Part 6: Communications .............................................................................................. 25
Purpose ...................................................................................................................... 25
Aim ............................................................................................................................. 25
What we will do .......................................................................................................... 25
Appendix 1 ................................................................................................................... 26
Levels of Monitoring and Evaluation ........................................................................... 26
Appendix 2 ................................................................................................................... 27
Evaluation and monitoring framework - Healthdirect services and programs .............. 27
Appendix 3 ................................................................................................................... 29
Key drivers for evaluation ........................................................................................... 29
Introduction
The National Health Call Centre Network (NHCCN) trading as Healthdirect Australia
(Healthdirect) is a company established by the Council of Australian Governments
(COAG) which strategically procures publically funded telehealth services.
The following services are currently provided inter alia through Healthdirect:
Healthdirect – nurse triage line
After hours GP Helpline
Pregnancy, Birth and Baby, Counselling service and website
mindhealthconnect
National Health Services Directory
My Aged Care Gateway (website and contact centre)
Healthdirect (website)
Get healthy Service
As and when new services come on board they will be incorporated into the research
and evaluation strategy.
Building the research and evaluation agenda
As Healthdirect continues to embed its services into the Australian health system there is
now a recognised need for a long-term strategic plan that addresses Healthdirect’s
research and evaluation agenda.
This involves ensuring that Healthdirect participates in relevant and timely research and
evaluations to continue to create services that are innovative and evidence-based,
technologically sound, address the needs of both consumers and the Australian
healthcare system, are appropriately targeted, and provide value for the resources
invested.
As Healthdirect’s core business is providing health services rather than being a research
entity, the priorities for research will be driven by business and service imperatives. This
Research and Evaluation strategy aligns to Healthdirect Australia’s organisational
strategy (2014-2018). As it seeks to strengthen its reputation and expertise in order to
be:
An evidence-based service provider of quality programs that improve health
outcomes
A contributor to the body of knowledge about the effectiveness of telehealth and
online health services and how to deliver these to reach all the Australian
population
An innovative leader in telehealth and online health services
An authoritative voice on future directions and investment priorities in telehealth
and online health services.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 4
Healthdirect Australia Research and Evaluation Strategy 2014-2018 5
Development of the strategy
The research and evaluation strategy (the Strategy) was developed on the basis of
consultations and discussions with Healthdirect executives, the Clinical Governance
Advisory Committee, the Joint Customer Advisory Committee and the Digital Services
Advisory Committee. In addition a workshop was held with key staff, shareholders and
stakeholders to consolidate summarise and synthesise current activities and directions,
set new priorities for the future, and agree on the infrastructure and processes that are
needed to set these in place.
Part 1: Aligning to Healthdirect Australia Strategic directions
Healthdirect has recently moved into a longer term funding cycle and is accordingly
developing new strategies to advance work into the future. The Research and Evaluation
Strategy 2014-2018 aligns with the Healthdirect strategic plan (Figure 1) and follows the
current NHCCN Research and Evaluation Strategy 2012-2014.
The process of ensuring this strategy has aligned to the Organisational strategic plan has
resulted in the development of six strategic initiatives and a range of actions described in
Figure 1. These six strategic initiatives, along with the actions, have been developed as
resources to provide guidance to Healthdirect staff when planning and managing
research and evaluation projects. Further details about these actions are described
throughout the document.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 6
To be a key part of quality Australian
health system by helping consumers
manage their own health through
leveraging technology to enable timely
access to health and related services
To provide a set of principles and activities
that can be used to support the delivery of
high quality research and evaluation which
is relevant and responsive to the
Healthdirect Australia strategic directions.
Align to Healthdirect
Strategy
Research and evaluation
governance
Setting research and
evaluation priorities
Develop and implement
robust evaluation framework
Develop and implement
framework for responsive research
Commun
ication
Identify, develop solutions, aggregate,
procure and evaluate publically funded
health and related services delivered through
innovative technologies
Healthdirect
Australia Vision
Strategic
initiatives
Ensure evidence-based
policy and services
Apply best practice
principles
Enable continuing
improvements in service
delivery and targeting
Deliver products and services
that are competitive and world leading
Ensure clinical appropriateness
of services
Measure cost effectiveness /
value for investment
Justify future funding for
research and evaluation
Purpose
Research and
evaluation
strategy aim
Figure 1. The Framework
1 2 3 4 5 6
Healthdirect Australia Research and Evaluation Strategy 2014-2018 7
Develop and enhance multi/cross-platform initiatives for relevant services to new channels and technologies
services
Deliver evidence-based, best practice clinical go services
Enhance current portfolio of service
Develop new services Facilitate health relationships Enhance organisationalcapability
• Develop and enhance multi/cross-platform initiatives for relevant services
• Expand relevant services to new channels and technologies
• Develop and enhance call services • Deliver and enhance online
services• Deliver evidence-based, best
practice clinical governance of all services
• Provide strategic leadership for new initiatives
• Investigate identified health related services opportunities
• Enhanced medication information services
• Improve health literacy• Integrate video consulting
capability into new services
Promote and initiate consumer initiatives
Build evaluation capacity across the organisation
Part 2: Research and evaluation governance: Implement a research governance framework
Part 2: Research and evaluation governance: Implement a research governance framework
Part 5: Develop and implement a framework for responsive research: Partner with the most suitable research organisations, with high quality researchers to undertake exploratory research
Part 2: Research and evaluation governance: Establish and maintain an organisation-wide steering group.Seek endorsement for research and evaluation concepts and projects through CGAG and PROWAC
Part 3: Setting research and evaluation priorities: Establish andfacilitate a process for identifying and updating Healthdirect research priorities. Through the use of a triage system.
Part 3: Setting research and evaluation priorities: Ensure funded research funded aligns to Healthdirectpriorities . Determined through the Advisory Group and approved by CGAG and PROWAC
Part 6: Communications: Promote research and evaluation to the wider health and research community through a range of mediums Every research project will have a company communication plan
Part 4: Develop and implement a functional evaluation framework: Encourage a culture that supports the use of research and evaluation findingsDisseminates findings through Company committees and PROWAC
Part 4: Develop and implement a functional evaluation framework: Ensure evidence, evaluation and monitoring data is used to enhance program effectiveness. The Advisory group will be used to facilitate this process.
Part 4: Develop and implement a functional evaluation framework: Strengthen organisational evaluation capacity and processes
Ensure research and evaluation promotes equity and health literacy.
Part 5: Develop and implement a framework for responsive research: Commission high-quality evidence checks to ensure technology enhancements are based upon the evidence available.
Part 5: Develop and implement a framework for responsive research: Partner with the most suitable research organisations, with the most suitable expertise to undertake exploratory research
Organisational Strategic
pillars
Initiatives that
require a research
and evaluation
function
Research and
evaluation actions –what we will do
Figure 2. Strategic alignment to organisational goals
Healthdirect Australia Research and Evaluation Strategy 2014-2018 8
Research and evaluation as an integral part of Healthdirect’s work program
This agenda is critical to the ongoing integration of
Healthdirect into Australian health system. Aligned to
the Healthdirect Clinical Governance framework this
Strategy drives:
1. Research that informs management decision
processes about new services, service gaps or
changed circumstances that can ultimately
influence Healthdirect services.
2. Evaluation to manage the process of
continuous quality improvement (CQI). To
ultimately understand impact and improve the
efficiency and effectiveness of existing services
The following are key reasons for Healthdirect Australia
to prioritise research and evaluation:
Ensure evidence-based policy and services
Apply best practice principles
Enable continuing improvements in service delivery and targeting
Deliver products and services that are competitive and world leading
Ensure clinical appropriateness of services
Measure cost effectiveness / value for investment1
Justify future funding for research and evaluation
Be seen as a leader in tele-health.
Healthdirect is committed to:
Embedding research and evaluation as an
integral part of the organisation’s approach to
program and service development and delivery;
and
Ensuring that the results are considered,
disseminated and utilised.
1 Note that there are several opportunities for measuring value, with cost–effectiveness being just one. Assessments of cost utility, benefit,
efficiency or even appropriate cost-shifting may be relevant in different instances. 2 http://www.dpc.nsw.gov.au/__data/assets/pdf_file/0009/155844/NSW_Government_Evaluation_Framework_Aug_2013.pdf 3 http://www.innovation.gov.au/Research/ResearchBlockGrants/Pages/HigherEducationResearchDataCollection.aspx
Monitoring2 is a management
process to periodically report against
planned targets (or KPIs). For the
most part, monitoring is not
concerned with questions about the
purpose, merit, or relevance of
services. Monitoring is frequently
based on outputs as opposed to
outcomes.
Evaluation2 is the systematic
collection of information about the
activities, characteristics, value and
outcomes of services to make
judgements about the services’’ merit
or worth, usually in relation to their
effectiveness, efficiency and
appropriateness. The information
gained is used to improve program
effectiveness and /or inform
decisions about future programming.
Research3 is the creation of new
knowledge and/or the use of existing
knowledge in a new and creative way
so as to generate new concepts,
methods and understandings.
The primary difference between
research and evaluation is that
research is conducted with the intent
to generalise the findings from a
sample to a larger population. In
contrast, evaluation particularises,
focusing on an internal situation with
no intent to generalise the results to
other settings and situations.
Research informs management
decision processes about new or
enhanced services to meet service
gaps or changed circumstances.
Evaluation guides management in the
process of CQI to improve existing
services in terms of efficiency and
effectiveness.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 9
Aims of the research and evaluation strategy
The overarching aim of the strategy is to provide a set of principles and activities that can be
used to support the delivery of high quality research and evaluation that is relevant and
responsive to the Healthdirect Australia strategic directions and support a consistent
approach to enhancing research and evaluation. The following sub aims have been
developed to support this.
To develop and implement research processes that can:
- Identify gaps in knowledge and enable problem identification and the
development of evidence based solutions to those problems
- Recognise and respond to health and healthcare priorities, rapidly changing
technologies, and patterns of consumer behaviour with respect to these
technologies
- Support priority setting and future directions for research
- Enable the translation of research findings into actionable performance
improvement measures.
To determine criteria and guidelines that can:
- Enable the identification and evaluation of possible research partnerships
- Position Healthdirect as a recognised and valuable research partner.
To develop and implement evaluation measures that can:
- Assess impacts of Healthdirect services on individual health, population health
and the healthcare system
- Recognise local area and specific sub-population needs
- Include measures of cost-efficiency, cost-effectiveness and cost-shifting
- Enable Healthdirect to state with confidence the effectiveness, efficiency and
safety of its services
- Be utilised to improve services.
To ensure that Healthdirect services meet the needs of all Australians and are
culturally sensitive to those needs.
To measure the demand for current Healthdirect services and assess how this could
be improved, particularly with respect to equity and access of healthcare.
To identify and describe the infrastructure, governance and resources that will assist
Healthdirect’s work in research and evaluation and ensure this work is of high quality
and recognised.
To investigate similar international services to identify strategies that can be
translated into the Australian model.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 10
Part 2: Research and evaluation governance
Purpose
The following section sets out the purpose and principles for the governance of the research
and evaluation strategy.
Good governance firstly ensures accountability and transparency of decision-making. This
will include introducing a standardised and streamlined approach to the development of
research and evaluation briefs to ensure they align to organisational needs. Secondly, it
ensures the implementation of good practice principles. This includes ensuring that NHMRC
ethical principles of research and privacy principles are adhered to so participants/users of
Healthdirect services are not put at risk.
Overseeing the delivery of this strategy will involve building evaluation capacity across the
organisation. This will be achieved by actively encouraging engagement, knowledge transfer
and collaboration across divisions and external agencies. Additionally, this supports the
production of new knowledge from research/evaluation which is based upon clearly defined
questions relevant to Healthdirect services, underpinned by rigorous and valid research
methods and of publishable quality.
Aim
To ensure all research and evaluation studies adhere to best practice guidelines in accordance with national research bodies and Healthdirect quality assurance and governance processes.
What we will do
Oversight
The strategic oversight of the research and evaluation strategy will be undertaken by the:
Clinical Governance Advisory Group (CGAG): It will be the role of this
Committee to endorse research and evaluation projects, and provide clinically
relevant advice as and when required.
Project Review and WHS Advisory Committee (PROWAC) In accordance with
PROWAC scope of responsibilities any research and evaluation project which
meets the PROWAC criteria will be reviewed by this committee.
Portfolio Management Office: If projects are of a substantial size and link to
business case development the PMO will be informed and involved in accordance
with their Terms of Reference
To manage the implementation of the strategy, we will establish an internal steering group
led by the Clinical Governance Division. The role of this group is to:
Triage and approve proposals against the criteria for identifying suitable Healthdirect
research
Prepare research and evaluation proposals for review and acceptance by CGAG
Ensure research governance principles are adhered to
Assess the risk of the proposals
Monitor the execution of research or evaluation studies
Provide guidance on the use of research and evaluation results
Come together to resolve issues and share best practice
Healthdirect Australia Research and Evaluation Strategy 2014-2018 11
This group will meet bi-monthly to oversee the implementation of this strategy and will
provide monitoring for existing research.
Funding
Some evaluations or small research/ need assessment projects will be undertaken as
Business as Usual (BAU). Any other research and large scale evaluations will require
business case development and funding from Commonwealth, jurisdictional Government or
another source. It will be the responsibility of the steering group to approve these business
cases before they are reviewed by other appropriate channels.
Research governance
A research governance framework will be developed and implemented across the
organisation to develop a standard process for procuring and monitoring research projects.
This will include a process which defines and communicates good practice principles and
requirements. In summary, the framework will provide a structure for:
Sound management of routinely-collected data that are secure and quality-assured
Guidelines and IT infrastructure for making routinely-collected data available to
researchers upon receiving approvals
Process for gaining ethics approvals
Resources, training and capacity for managing grants
Resources training and capacity for managing research relationships and
partnerships
Access to ethics committees and resources for managing ethics approvals
Policies to guide decisions on publication and intellectual property and moral rights
Procedures for managing instances of research misconduct.
The implementation of these actions will be discussed at the Research and Evaluation
Steering Group.
As a Joint responsibility between the divisions of Clinical Governance and Marketing and
Strategy a database will be developed and record all research and evaluation studies that
have been led by Healthdirect or in which Healthdirect has participated.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 12
Table 3 identifies the principles that will be included in the Research Governance Framework
to inform Healthdirect’s decisions regarding research and evaluation partnerships and similar
arrangements2.
2 Adapted from the principles for research governance developed by the Sax Institute 2014
Table 3. Key Principles
For any research and evaluation partnership or similar arrangement, the following principles may apply:
a) A financial arrangement should not influence, impose or imply conditions that would limit, or appear to limit, Healthdirect’s ability to carry out its functions fully and impartially, with integrity and in accordance with high professional standards.
Healthdirect will not engage in relationships where third parties seek to inappropriately influence the content of research outputs or the means by which evidence from research is provided to policy-makers, in order to further their own private interests.
We will maintain the highest academic standards and abide by all applicable research codes of conduct and ethics.
We will not enter into any relationships which hinder our ability to act with integrity in accordance with such standards and codes of conduct.
There should be no real or apparent conflict of interest between the mission and objectives of Healthdirect and those of a funder or partner.
Healthdirect will not enter into arrangements with organisations whose core business is in any way contrary to the mission, or that are deemed not to operate in the best interests of improving health.
b) Healthdirect will seek to avoid any potential risks to protect public confidence in the company and its reputation.
Healthdirect will not engage in any form of private sector arrangement which would seek to utilise our resources for purposes that are not in the public interest.
c) All private sector arrangements will be documented in writing and/or set out in a formal contract.
The terms and conditions of all arrangements will be documented. This may include emails, letters of agreement, and/or formal contracts, depending on the nature and value of the arrangement. Verbal agreements must be noted in writing for the record.
The elements which must be documented are the agreed fees, conditions and benefits, Company and sponsor brand management protocols and the life time of the agreement.
All funding arrangements are to be endorsed by the relevant program head and approved by the Chief Executive Officer. Board advice will be sought as required.
As Healthdirect seeks to improve health, and health services and programs, organisations that are involved in activities and/or products that present a direct hazard to community health and wellbeing are regarded as being inappropriate for any form of sponsorship. Some examples include companies involved in the production and sale of tobacco and alcohol products.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 13
Part 3: Setting research and evaluation priorities
Purpose
To ensure that research and evaluation are driven by Healthdirect business and service
imperatives, a “themes” approach to setting research and evaluation priorities will be taken.
Researching health priorities may not always be relevant to Healthdirect consumers,
therefore key topics and priorities will be based upon the Company, shareholder and
consumer needs. These “themes” will be reviewed annually in conjunction with the review of
divisional business plans. Development of the themes will be based on:
National health needs
Emerging issues in telehealth and e-health
Driving best practice
Evaluation data and analyses
Consultations with shareholders and stakeholders
Information from pilot and demonstration programs
Healthdirect’s Strategic Plan.
Healthdirect’s current themes for its research and evaluation priorities are:
Ensuring equity of access to appropriate health advice and services provided
by Healthdirect –including reaching populations with difficulties accessing services
because of language, physical access and cultural needs.
Empowering users - enabling users to make the most effective use of the services
and information provided by Healthdirect in accordance with their needs and
expectations. This would include improving health literacy.
Coordination and integration – helping people with chronic disease and disabilities
(and their healthcare providers and carers) to better manage their conditions.
Improving technology literacy - utilising new technologies effectively where they
are available and supporting people to use accessible technologies most effectively.
Economic analyses and impacts – assessing cost benefits, cost-effectiveness, cost
shifting and value in terms of outcomes both for consumers and for Healthdirect.
Aim
To optimise research conducted through Healthdirect by ensuring it aligns to organisational
goals and commitments.
What we will do
Establish and facilitate a process for identifying and updating Healthdirect priorities and
assessing research and evaluation studies against the identified key themes.
This process will involve proposed research studies being triaged against criteria (Table 2)
for identifying suitable Healthdirect research and ensuring adequate probity for partnerships
(Table 3). These will be used to ensure:
Healthdirect Australia Research and Evaluation Strategy 2014-2018 14
All research and evaluation studies are relevant and structured around Company
priorities as described in the Healthdirect Australia Strategic plan.
Collaboration with external shareholders and agencies to translate and integrate
research findings into new services or enhance existing services.
Research and evaluation outcomes are communicated in the most suitable format.
Table 4. Criteria for identifying and approving Healthdirect research proposals:
The proposed research or evaluation aligns to the key themes described above
The proposed research is relevant and aligns to Healthdirect and shareholders’
strategic objectives
The research addresses a gap in knowledge, adds value and is timely
The research findings can contribute towards growing the evidence base
The proposal includes sound methods that are relevant and responsive
Offers a return on investment through the use of results evidence to grow new
services or enhance existing Healthdirect services
The proposed research addresses health inequities and improving health
literacy
Provides opportunities for partnerships between (researchers, Healthdirect and
policy-makers)
Healthdirect Australia Research and Evaluation Strategy 2014-2018 15
Healthdirect research and evaluation priorities for 2014-15
These priorities were developed on the basis of current research being undertaken,
consultations with senior management, a review of literature on current issues and through a
questionnaire to workshop participants. This list is not exhaustive, additional research and
evaluations will be undertaken as and when new initiatives or opportunities arise.
Research, evaluation and monitoring priorities
Research Conduct concept testing for new telehealth care delivery models for mental
health in nurse triage
Increase understanding, measure and improve health literacy through
Healthdirect services
Test the effectiveness and cost effectiveness of remote monitoring for chronic
conditions
Concept testing new digital delivery models for disability services
Evaluation Develop a clinical scenario testing tool for the After Hours GP Helpline and
nurse triage service
Conduct periodic clinical audits of Healthdirect nurse triage service and AGPH
Evaluate the acceptability, appropriateness and use of the Healthdirect
website
Conduct a longitudinal study on the effectiveness of video consultation
Track cohorts of users through the Primary care linkage program – assess
usability and outcomes
Assess the impact of the Symptom Checker on consumer satisfaction and
outcomes
Monitoring Monitor the usability of Symptom Checker and outcomes
Use Healthdirect data as an early warning surveillance system for infectious
diseases
Develop a framework to measure access and effectiveness of digital services
part of an ongoing monitoring
In addition to research, evaluation and ongoing monitoring a range of feasibility and needs
assessment studies will be conducted to collect and analyse information to inform future
services or business proposals. These include:
E prescribing for general practitioners on the Afterhours GP Helpline
Evaluation of current services for providing medication information – telephone and
digital services,
Primary care alternatives to emergency departments.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 16
Part 4: Develop and implement an evaluation framework
Purpose
When an evaluation is planned and conducted well, it can provide timely evidence on service
effectiveness and efficacy. This evidence can be used to improve services and provide
accountability and transparency to shareholders and consumers.
Evaluation can play a key role in testing the impact of new services and the ongoing
effectiveness of mainstream Healthdirect services. To achieve this, we aim to ensure that an
evaluation plan is developed concurrently with service business plans and funding to conduct
evaluations is built into the business planning process. This will facilitate an evaluation
process that can be used throughout the lifecycle of a program.
To help gather evidence that meets its intended goal, evaluations can be classified as
formative, process or summative on the basis of:
1. The operative stage of the service (this can include development, recently
implemented, long-term)
2. The purpose of the evaluation (e.g. can the program be improved, is it well-targeted;
is it cost effective)
3. How the information obtained will be used (e.g. to support the implementation of a
new program or the expansion of an existing one; to justify continued funding for a
program).
Monitoring and routine data collection can also be included as a preliminary step.
Aim
To introduce a clear program of change that can be used to improve evaluation outcomes.
To ensure each evaluation adds value, the following functional framework will be introduced
across the organisation as a structured approach to enhance:
a) The planning phase of an evaluation to ensure that an evaluation achieves its
intended goal, and
b) Building the capacity of Healthdirect staff when participating in the design and
implementation of service evaluations3.
This is important because it:
Ensures that evaluations are appropriately funded
Evaluation objectives and outcomes are properly identified
Guides consistent, transparent and timely evaluations
Ensures that evaluations of individual programs
- work toward a common purpose
- are complementary
- address questions important to the organisation as a whole
Identifies the level of inquiry
Identifies high-level outcomes.
3 Healthdirect Australia, Strategic Plan 2014-2018. Pillar 4. Promoting Organisational capability, Business plan initiative No 16:
Enhancement of services though continuous development of research and evaluation capacity and strategy.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 17
What we will do
To strengthen the evaluation capacity conducted through Healthdirect, in this section we
describe seven steps involved in developing an evaluation brief. These steps, which have
been adapted from the NSW Government Evaluation Framework aim to support an
organisational approach to achieving this:
STEP 1: Build an evaluation framework into service design
Planning an evaluation before a service is implemented provides better outcomes. This will
be achieved by:
Building evaluation objectives into the business planning phase
Developing a program logic model that explains how the intended outcomes relate to
the activity, together with measurable performance indicators and outcomes and
interim measures
Articulating evaluation questions relevant to the evaluation purpose
Identifying data needs and key stakeholders, if necessary comparison data
Engaging to most appropriate stakeholders.
Figure 4 provides a flowchart for managing the execution of an evaluation study
STEP 2: Apply a Framework to streamline the planning of an evaluation
The evaluation framework described in Appendix 2 will be used, as and when appropriate, to
streamline the evaluation planning process. The purpose of the framework is to provide a
menu for assessing, monitoring priorities, determining what is feasible and ensures
consistency across evaluations. Key drivers for evaluation are shown in Appendix 3.
Using a key drivers approach, the framework identifies nine areas of interest
specifically relating to the implementation and anticipated outcomes of Healthdirect
services. Depending on the program or service to be evaluated, individual drivers will
have greater or lesser prominence.
Using level of monitoring and evaluation approach, the framework identifies different
types of evaluation that may be applied to each of the nine key areas. As a general
rule:
o All programs and services should be subject to input/output monitoring
o Most to all services should be subject to process evaluation
o Emerging services should be subject to formative evaluation
o High impact, new, and important services and programs should be subject to
summative evaluation.
The framework can be used as a guide in setting key priorities and key questions for each
evaluation and each field in the framework should be considered for its relevance and
appropriateness for the service or program being evaluated.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 18
STEP 3: Ensure evaluation methods are rigorous with appropriate scale and
design
This will be achieved by:
Employing data collection methods that are most appropriate to the research
objectives and program operations.
Using best practice methods and relevant data sets which are reliable and can be
accessed for repeated evaluation.
Since resources and time are required to conduct evaluations, they should be designed and
scaled to each program in accordance with the program’s size, risk, and significance.
STEP 4: Use the right mix of expertise and independence
Healthdirect may commission a suitable agency to conduct an evaluation on behalf of the
company, judgements about the most suitable agency will be based upon the purpose of the
evaluation, questions that need answering and resourcing. This will be undertaken through
the Healthdirect Procurement process.
To add to the body of knowledge relating to each service area program managers, other staff
and stakeholders involved in service delivery will be encouraged to participate in the planning
and review of recommendations of the evaluation.
STEP 5: Ensure the planning of evaluation includes an appropriate timeframe
Evaluations should conclude before decisions are made, with consideration given to the
realistic amount of time needed to conduct them to answer the questions being asked.
Summative evaluations should not be undertaken too early, in recognition of the time it can
take to accrue sufficient evidence and produce measurable outcomes.
STEP 6: Recognise the context in which the activity is conducted
Evaluations should understand how the political, social, economic, technological and cultural
contexts have affected the program being evaluated and the people who are using it.
STEP 7: Apply good governance
Comprehensive information on all aspects of the evaluation should be systematically
recorded, including choice of methods, analyses and conclusions. The extent to which
evaluation findings, methods and data are shared within and external to Healthdirect should
be determined during planning and be transparent and open to scrutiny (as described in Part
2).
Healthdirect Australia Research and Evaluation Strategy 2014-2018 19
Figure 4. A flow chart for managing evaluation studies
Describe of what needs to be evaluated
• Program description
• Program logic model
• Consultation with evaluation framework
Engage and participate
• Need involvement of program managers, experts, evaluators (internal and external), stakeholders and those who need / will use the findings
Develop evaluation brief
• Purpose, scope, key evaluation questions, timelines, available resources
• Review questions against evaluation framework
Develop of evaluation design
• How data will be collected, analysed and reported to answer key evaluation questions
• Ensure statistical significance / meaningful results
• Match design and results to significance and risk of program to be evaluated
Develop of work plan
• Timeline for deliverables, responsibilities
• Agree on communication, publication issues and risk management of findings
Manage implementation of
work plan, and report/s
• Timeline for deliverables, responsibilities
Disseminate report/s and use
of findings
• Recommend for policy and /or action
Healthdirect Australia Research and Evaluation Strategy 2014-2018 20
Routine collection of data
Part of the planning of evaluation is to ensure adequate comparisons can be made. For this
reason, processes for ensuring the necessary data are collected, in a way that is efficient
and not burdensome. In driving the decisions around data collection, the following will be
considered:
What is currently measured
What needs to be measured
What evaluation studies have been completed / are currently underway
What can be evaluated through quantitative data and what needs to be assessed
through qualitative data
What aspects should be regularly evaluated (to give longitudinal data)
How to cross link information from the various evaluation studies
Usefulness of the data.
It is necessary to ensure that sufficient data are collected to deliver statistical significance to
the final results and this factor might affect how some low frequency or highly targeted
services are best measured.
Decisions about routine data collection would be the responsibility of the internal research
and evaluation steering group.
Using the results of evaluation studies
To ensure the results of an evaluation are fed back into the organisation in an efficient way,
staff and stakeholders will be regularly provided with relevant and useful information that can
be utilised in service changes, foresight discussions and communications and public
awareness initiatives.
In the first instance this will be through the use of Company intranet “Pulse” and through
existing Healthdirect Committees.
This will support a continuous evaluation process (sometimes referred to as a ‘benefit cycle’
or a ‘virtuous cycle’) whereby evaluation data are regularly assessed and findings are fed
back into programs to improve functioning, targeting, efficiency and outcomes.
Figure 5 below represents the dynamic process of evaluation built around the support and
effective response of the organisation in undertaking an evaluation as part of its core
business activities and responding effectively to evaluation findings. The process is
described in circular fashion as it is a constant process of review – analysis - implementation.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 21
Organisational
support and response
Figure 5 The continuous evaluation cycle
Engage stakeholders
Ensure evaluation design fits the program
Gather and analyse evidence
Justify conclusions in
light of program goals
Ensure use and share lessons
learned
Program adjustments and / or research to
address information
needs
Healthdirect Australia Research and Evaluation Strategy 2014-2018 22
Part 5: A framework for responsive research
Why research is important for Healthdirect
Effective health research can contribute to identifying business development opportunities,
service enhancements and professional development opportunities for call centre and digital
staff. This can result from understanding the trends in customer needs and service
opportunities that will improve Healthdirect’s performance over time. Investment (and
disinvestment) decisions should be guided by the outcomes of the research service.
A key aim of this Strategy is to ensure a sound research structure that will have the capacity
to:
Identify gaps in knowledge
Recognise and respond to health and
healthcare priorities, rapidly changing
technologies, and patterns of consumer
behaviour with respect to these technologies
Facilitate priority setting and future directions
for research
Enable the selection and evaluation of
possible research partnerships
Ensure the success of research
collaborations and partnerships.
Aim
Optimise the use of research to inform existing and new services
What we will do
Ensure that all funded research provides outcomes that can be used to enhance existing
services or provide the evidence to develop new services. This will be achieved through:
Partnering in research
Partnering in research provides Healthdirect with access to existing capacity, knowledge and
resources across different organisations and settings and brings these together for high
quality, resource-efficient research. This approach provides opportunities to work
collaboratively with academic partners to answer questions important to the delivery of
Healthdirect’s services. Working collaboratively also provides research partners with the
ability to use large data sets to answer important questions on the delivery of health services
to Australians. This may include research on emerging technologies that may be of
importance to the delivery of Healthdirect’s services, or on topic areas that have been
determined to be priorities for Healthdirect.
Partnerships may be developed for individual research projects, or may be established
through a memorandum of understanding or similar for collaboration over a longer period of
time and may include several or more research projects. Partnerships may be initiated by
Healthdirect, or by another organisation including a university, research institute, or other
agency.
What is responsive research?
Research that: 1. Responds to community
needs, shareholders’ priorities and knowledge gaps, and;
2. Can help drive and improve Healthdirect’s services.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 23
Building research partnerships may involve a proactive search for research partner/s or a
reactive response to a request for Healthdirect to be a research partner. In either case this
will involve:
Determining and refining research question/s
Determining study design
Gaining ethics approvals
Applying for funds, holding funds
Agreeing on intellectual property, moral rights and publication of results.
Commissioning researchers
Through the development of the research triage system (Part 3) and the establishment of a
research database, we will endeavour to commission research organisations that are most
appropriate for research initiatives. Commissioned research will generally be carried out for:
Reviews and analysis of research areas
When answers are needed to a specific question
When an independent process is desirable or required.
Where Healthdirect is interested in knowing what the existing evidence for a particular
question is, commissioning a review of existing literature will be undertaken.
Applying Criteria for selecting research partners
All applicants to conduct commissioned research and all potential research partners will be
assessed against Healthdirect’s criteria for selecting research partners (see the principles
established in Part 2).
Managing requests for data and research collaborations
As and where applicable support external agencies to access and use Healthdirect data and
provide responses to data requests in a timely and transparent fashion. This includes:
A clear process for submitting requests
An agreed timeframe and procedure for dealing with such submissions
Designated responsibilities for managing this procedure
Consideration of all requests on the basis of Healthdirect’s research priorities and
guidelines for research partnerships.
See Figure 6 for an outline of how such requests will be managed.
Commissioning research or reviews of research
There will be occasions when Healthdirect’s research needs can only be met by
commissioning research or reviews of research from external experts.
Procedures for commissioning research will require development of appropriate mechanisms
to manage research commissioning processes. This will include agreement on roles and
responsibilities of the business units and the research and evaluation advisory board,
including decisions on:
- Funding and resources approvals
- Appropriate procurement methods
- Selection of research teams
Healthdirect Australia Research and Evaluation Strategy 2014-2018 24
Figure 6: A flow chart for managing research requests
1. Submission portal
•Clear point or points of entry for requests for data and research partnerships
•Contact details for Research Manager to assist discussion
2. Initial consideration
• Regularly scheduled meeting of research and evaluation advisory board
• Consider request in light of criteria for research partners, established priorities, existing work and resources
• Decision to proceed or deny
• Response back to party making the request within a stated timeframe
3. Consultation
May involve external experts
• Decision on research design, timeframes, deliverables
• Agreement on responsibilities, additional partners if needed
• Agreement on funding and resources
• Agreements on intellectual property, moral rights , publication of results as appropriate
• Legal consultation
4. Application for funding (if applicable)
•Joint agreement on development of funding application
•Response to reviews as necessary
5. Ethics approval (if required)
•Joint agreement on ethics submission
•Response to reviews as necessary
6. Conduct of research
•Regular reports back to research and evaluation advisory board, as previously agreed
7. Reporting research
• Writing of reports and papers, as previously agreed
• Publication of results, as previously agreed
• Communication of results - to staff, shareholders and stakeholders, policy makers, media, professional groups as appropriate
8. Using the research
•Analysis of implication of research results for Healthdirect programs
• Incorporation of findings into program development and operations as appropriate
Healthdirect Australia Research and Evaluation Strategy 2014-2018 25
Part 6: Communications
Purpose
To improve communications across the organisation and to external agencies, an important
part of this Strategy is to adopt processes that ensure research and evaluation results are
made available in an effective and timely manner that will:
Drive knowledge transfer inside and outside Healthdirect
Strengthen Healthdirect’s profile as an authoritative voice on future directions and
investment priorities in telehealth and online health services
Boost public confidence in the effectiveness, safety and value of Healthdirect’s
services.
Aim
Increase awareness of Healthdirect research across the Company, stakeholders and
shareholders and to the wider community.
What we will do
Every research project and evaluation program will have an accompanying communications
plan. This will consider when and how results will be provided to the following groups:
Healthdirect board, program managers and staff
Shareholders and stakeholders
Research partners and potential research partners
Policy makers
Users of Healthdirect services and general public.
This will include:
Developing a page on Healthdirect intranet “Pulse” to provide updates for staff
Providing updates for Committees part of the Healthdirect governance structures
Refine the process for publishing and presenting research and evaluation findings to
a range of audiences. Providing opportunities for all Healthdirect staff to participate in
this process.
Healthdirect Australia Research and Evaluation Strategy 2014-2018 26
Appendix 1
Levels of Monitoring and Evaluation
Type Description Example Questions to be answered
Monitoring
/ Routine
data
collection
Provides quantitative information about
services. Needs further analysis and
interpretation.
How many people access a service?
Wait times for service access?
How much money was spent on the
service?
Formative Formative evaluation provides information
on:
How new services might be developed
How existing services might be improved
What is the problem to be solved?
Where is the problem and how big or
serious is it?
What are the characteristics and needs of
the target population?
What is the most appropriate solution?
What is the expected outcome and how to
measure if this is achieved?
Process Process evaluation looks at how a service
is delivered. It can help differentiate
ineffective services from failures of
implementation.
For ongoing services it can be used for
continual improvement.
How is the service being implemented?
Are the activities being delivered as
intended?
Are participants being reached as
intended?
Summative Summative evaluation generally reports
when a service has been running long
enough to produce results. It may also be
known as:
Outcome evaluation – did the service
cause demonstrable effects on the agreed
outcomes?
Cost effectiveness – addresses the
questions of efficiency.
What are the net effects?
To what extent can changes be attributed
to the service?
Is the service the best use of the
resources?
Healthdirect Australia Research and Evaluation Strategy 2014-2018 27
Appendix 2
Evaluation and monitoring framework - Healthdirect services and programs
Key drivers
Levels of
monitoring
and evaluation
Demand Targeting Effectiveness Quality and
safety
Awareness Equity Cost-
effectiveness /
value
Impacts on
health
services
Client
satisfaction
Investigating the needs and expectations of target populations for Healthdirect services.
Investigating whether Healthdirect services are reaching the right people and meeting their needs.
Asking about whether Healthdirect is achieving its intended goals.
Benchmarking Healthdirect services against standards and expectations of quality and safety.
Ensuring Healthdirect services are known in the community and used appropriately
Ensuring vulnerable populations have equality of access to and care from Healthdirect’s services.
Measuring the value of Healthdirect services.
Investigating wider impacts of Healthdirect services and programs.
Evaluating the values and opinions of Healthdirect service users.
Input/output
monitoring
Data- driven
monitoring
(usually done
routinely) of key
inputs and
outputs of a
program.
Who is using the service/program?
How are they using it?
What is the frequency of use of the program/service? What are the main reasons for use? What would the user otherwise have done?
Are services provided in accordance with agreed guidelines and standards?
Have there been adverse events?
Which users of services are from populations groups of interest? e.g. Aboriginal and Torres Strait Islander people; CALD; people with disabilities (especially those who are blind and/or deaf); people with low educational levels
What are the direct costs of implementing the service?
Did the user of the service intend to use a different health provider?
Client satisfaction surveys.
Process
Evaluation
Examines how a
program is
delivered seeking
potential
improvements.
Has the program been implemented in a way intended? Has technology been utilised optimally?
What factors determine whether the advice given is followed?
What are the barriers and facilitators to delivering the service as intended?
What caused any adverse events?
How does the public find out about services? How are services marketed to the public?
How well is the program or service designed to accommodate the needs of at-need population groups? Barriers and facilitators to implementing these design elements?
What are the real and hidden costs of service implementation? What costs are higher or lower than expected?
What aspects of Healthdirect services determine the use (appropriate and otherwise) of other health services?
Do front line service deliverers follow courtesy guidelines? Are these culturally sensitive?
What are the barriers and facilitators to this?
Formative Examines how a What is the unmet What are the What should be How can the What is the level What barriers What outcomes What potential What is an
Healthdirect Australia Research and Evaluation Strategy 2014-2018 28
Evaluation program can be
developed and
optimised in a
specific context.
need for
services? How
does this differ in
population
subgroups?
appropriate population cohorts for each service?
the expected outcomes for users of Healthdirect services?
quality and safety of the service be improved?
Are adverse events handled appropriately?
of awareness of Healthdirect services among key population groups?
hinder access by various population sub-groups?
are direct relevance to cost-effectiveness? Where can costs be reduced?
impacts on other health services are of interest?
acceptable level of client satisfaction? How is this best measured?
Summative
Evaluation
Examines net
effects of a
program.
What is the
demand for the
service being
evaluated? Is that
demand being
met?
What are the barriers to effective use of current services? eg. English language, health literacy, technology access, other variables.
What are the actual outcomes for users of services and how do they differ from those of non-users?
To what extent do people who use services follow the advice given?
Do services ease after-hours pressures on GPs?
Do services improve health literacy?
Do people who use services interpret the advice they are given accurately?
Are services responsive to disabilities, cultural and language needs? Is this demonstrated in the use of data?
What is the public opinion of services, amongst key population groups?
What is their level of satisfaction with services?
What is likelihood of using / recommending services?
Ares services seen as culturally safe?
Do all population subgroups have adequate access to services?
What are the costs of services benchmarked against those for similar services in Australia and internationally?
What return on investment does the program or service provide?
What value do Australians / users place on services? Is this commensurate with actual operational costs?
Is there evidence that use of services changes the use of other health / healthcare services and what are the cost implications of this?
What are the net effects of the service or program on other health services?
To what extent can changes be attributed to Healthdirect services?
What is the level of satisfaction with services?
Healthdirect Australia Research and Evaluation Strategy 2014-2018 29
Appendix 3
Key drivers for evaluation
1. Demand
What is the demand for Healthdirect services?
Demographics of this demand
What is the unmet need for Healthdirect services?
Demographics of this.
2. Targeting
Are Healthdirect services targeting the right population cohorts?
Does each Healthdirect service reach the designated population cohort/s?
What are the barriers to effective use of current Healthdirect services?
What impact does English language literacy have on use of Healthdirect services?
To what extent does technology influence targeting of Healthdirect services?
What impact does technology literacy have on use of Healthdirect services?
3. Effectiveness
What are the expected outcomes for users of Healthdirect services?
What are the actual outcomes for users of Healthdirect services and how do they differ from
those of non-users?
To what extent do people who use Healthdirect services follow the advice given?
What factors determine whether advice given is followed?
Are users of Healthdirect services more or less likely to:
o Receive after-hours treatment / care?
o Visit an Emergency Department for treatment?
Do Healthdirect services ease after-hours pressures on GPs?
Do Healthdirect services improve health literacy?
4. Safety and appropriateness
Are Healthdirect services provided in accordance with agreed guidelines and standards?
Do people who use Healthdirect services interpret the advice they are given accurately?
What are the barriers to compliance with Healthdirect advice?
Are Healthdirect services responsive to disabilities, cultural and language needs? Is this
demonstrated in the use data?
Is there evidence for adverse events for people who use Healthdirect services?
Are adverse events handled appropriately?
5. Awareness
What is the level of awareness of Healthdirect services among:
o General population
o Young mothers
o People with chronic illness and their carers
o Staff in residential care facilities
o Community nurses
o GPs
o Pharmacists
What is their opinion of Healthdirect services?
What is their level of satisfaction with Healthdirect services?
What is likelihood of using / recommending Healthdirect services?
Healthdirect Australia Research and Evaluation Strategy 2014-2018 30
How do people learn about Healthdirect services?
6. Equity
What do the data say about use of Healthdirect services by:
o Aboriginal and Torres Strait Islander people
o People whose first language is not English
o People with disabilities (especially those who are blind and / or deaf)
o People with low educational levels
What barriers hinder access by these groups?
Are Healthdirect services seen as culturally safe by Aboriginal and Torres Strait Islander
people?
7. Cost effectiveness / value / cost-shifting
Costs of Healthdirect services can be benchmarked against those for similar services in
Australia and internationally.
What value do Australians / users place on Healthdirect services? Is this commensurate with
actual operational costs?
Is there evidence that use of Healthdirect services changes the use of other health /
healthcare services and what are the cost implications of this?
8. Impacts on health services
What health service/s does this service supplement or substitute for?
What services are likely to be affected by this service (positively and negatively)?
Are there impacts on health workforce?
How will unintended consequences be assessed?
9. Client satisfaction
Did people get what they wanted / needed from the service or service?
Were their expectations reasonable?
If someone else used the service on behalf of a patient, was the patient satisfied?
Did client understand and act appropriately on the advice given?
Would clients use the service again?
Would clients recommend the service to others?
Are procedures to address clients’ grievances effective? How is this effectiveness measured?