Mixed Methods: Myths, Traditions & Strategies Canada Research Chair Seminar Series, Aging and Community Health Research Unit, School of Nursing, McMaster University May 5 2015 By: Kathryn Fisher, PhD
Mixed Methods:
Myths, Traditions &
Strategies
Canada Research Chair Seminar Series, Aging and Community Health Research Unit, School of Nursing, McMaster University
May 5 2015
By: Kathryn Fisher, PhD
Topics
1. Defining features of mixed methods research
2. Current techniques to integrate qualitative and
quantitative research
3. Value of using qualitative approaches in intervention
studies
4. Key challenges in using qualitative research alongside
RCTs
1. Defining Features of Mixed
Methods Research
Defining Feature #1: Controversy
Creswell’s 11 Key Controversies1
1. How should mixed methods be defined?
2. Does the use of “quantitative” and “qualitative” in mixed methods create an artificial and unnecessary binary distinction?
3. Should there be a bilingual nomenclature for mixed methods?
4. Have we arrived at an understanding of the types of research designs in mixed methods?
5. Can we “mix” paradigms in mixed methods?
6. What is the value added by mixed methods beyond conducting a quantitative or a qualitative study?
7. What is driving the interest in mixed methods?
8. Is mixed methods a “new” approach?
9. Does mixed methods privilege post-positivism?
10. Is there a dominant, meta-narrative discourse emerging in mixed methods?
11. Is mixed methods claiming other designs as “their own” ?
Controversy - Continued
fundamental questions continue to be debated:
• What is mixed methods?
• Can we do it?
• How do we do it?
• Why do it?
• Does it really exist?
Despite these debates, we observe…
Controversy - Continued
long tradition of mixed methods in sociological research (since1800’s)2,3
qualitative methods now common in4,5 :
clinical trials
surveys of attitudes/beliefs
epidemiological studies
Guidelines and operating procedures now exist – examples:
Best Practice Guidelines for Mixed Methods Research in the Health Sciences (U.S., 2011)6
Standard Operating Procedures for Qualitative Methods in Clinical Trials (U.K., 2013)7
Defining Feature #2: Qualitative + Quantitative
Most researchers use the term mixed methods to refer to
the use of qualitative and quantitative research methods
to investigate a topic8-11
Some researchers consider multiple methods, even within
the same tradition, as mixed methods, particularly if they
have a core & supplementary component12,13
Defining Feature #3: Integration
Most researchers claim that mixed methods requires9:
1. Deliberate integration.
2. Sum greater than the parts.
Some researchers claim that mixed methods can be10:
1. Mere use of qualitative and quantitative methods in same study.
2. Comparing qualitative and quantitative results.
3. Actual integration/merging of the two data types.
Defining Feature #4: Pragmatism
“Paradigm Wars” of 80’s & 90’s have ended and methodological
focus is now on integration2
Pragmatism now the dominant epistemology8,10
At philosophical level: accept that scientific truths are both
constructed and grounded in the world
At practical level: choose design assemblages (methods) that
best address research aim(s)
Defining Feature #5: Complementarity
main goal of mixed methods:
to gain a more complete picture (complementarity)
not “triangulating” in traditional sense (one method validatingthe other, ˃1 method addressing same research question)
Why validation not the goal:
Defy direct comparison: qualitative and quantitative methods
address different questions & differ in what they look at, how
they see it, & how they describe it42-44
Similar findings: comforting/reassuring but not validating
Defining Feature #6: Prototypical Designs
15+ mixed methods design typologies published14
Current view:
the research question(s) determine the method(s)15
use existing typologies as guide to tailor-make design that
answers your research question(s)16
Design typologies differ on 2 dimensions:
Timing
Emphasis
Prototypical Designs - Continued
most typologies recognize 2 major designs originally proposed
by Cresswell & Plano Clark9,16 :
1. Concurrent Designs:
Timing: qualitative and quantitative components
undertaken simultaneously
Emphasis: varies (equal if done in parallel, if embedded
less emphasis on nested method)
2. Sequential Designs:
Timing: qualitative completed before quantitative
component or vice versa
Emphasis: on method that comes first
2. Integrating Qualitative and
Quantitative Findings
Integration
Options for integration span 2 dimensions:
How to integrate
Where to integrate
Sampling
Data collection
Data analysis
Interpretation
Mixed Methods Matrix , Triangulation Protocols:
Qualitative & quantitative findings displayed on same page
Identify patterns, cross-cutting (“meta”) themes, paradoxes
Following a thread:
Theme from one component followed up with the other
Example: hypothesis from qualitative findings tested by survey
Data transformation:
“Quantitizing” qualitative data
“Qualitizing” quantitative data
Example: cross-tab qualitative themes against quantitative data
How to Integrate- Specific Approaches10,42
Where to Integrate
often conceptualized by linking methods of data collection
and analysis (methods level integration)14
Methods level integration relates to the study design14,32
We will discuss:
Methods level integration paths for main study designs
Examples from intervention studies (or study protocols)
17
QUAN
Data &
Results Interpretation
QUAL
Data &
Results
1. Convergent Parallel Design
Concurrent Mixed Methods Designs
• Most common design used in healthcare research29
• Timing:
• concurrent data collection
• data may or may not be collected from same study sample
• Emphasis: equal for qualitative and quantitative components
• Integration (“Merging”, “Triangulating”):• two databases merged for analysis &/or interpretation• identify convergence, divergence or complementarity
Example 1: Group Cohesion in Patients Participating in Physical Exercise Intervention41
Purpose: explore cohesion and quality of life (QoL) in cancer patients participating in
a 6 week physical exercise intervention.
Methods:
• QUAN: validated survey administered at baseline and 6 weeks used to determine
changes in QoL and health status
• QUAL: focus groups conducted post intervention to explore group cohesion.
Results:
• QUAN: showed significant improvements in emotional functioning, social functioning and mental health
• QUAL: showed that group setting motivated patients by developing a sense of
obligation to train and do their best, thereby improving their social and emotional
functioning and mental healthDiscussion: the results converge to support the theory that group cohesion and sense
of belonging facilitates achievement of social and emotional functioning.
Concurrent Mixed Methods Designs
1. Convergent Parallel Design (Example – Convergent Results)
Example 1: Caregiver Outcomes & Experiences of a Respite Care Intervention40
Purpose: explore caregivers’ stress and perceptions of respite services provided by an
independent hospice.
Methods:
• QUAN: validated Relative Stress Scale Inventory (RSSI) was used to measure stress & was completed (pre & post) by 12 caregivers
• QUAL: 12 caregivers also completed interviews about views of respite care
Results:
• QUAN: showed no significance difference between pre & post stress levels
• QUAL: findings supportive of the intervention, showing that most caregivers saw
respite care as important as it gave them a break/rest from caregiving
Discussion: researchers concluded that divergent results indicated an inadequate
questionnaire (RSSI), others29 suggest that theory underpinning the research should be changed to suggest that respite care may relieve instrumental caregiving
responsibilities, but that other support is needed to relieve caregiving distress.
Concurrent Mixed Methods Designs
1. Convergent Parallel Design (Example – Divergent Results)
Example 2: Palliative Care Practices & Meanings in End of Life Nursing Home Residents31
Purpose: explore palliative care practices in nursing home residents with dementia at
end of life (EOL) and perceptions (of residents, family, nursing home staff) of EOL care.
Methods:
• QUAN: independent strand (n=30) with its own questions, samples, data collection,
and analysis techniques. Used a retrospective chart review to assess (a) symptoms
(b) interventions, (c) decisions to limit curative care, and (d) death cause/location.
• QUAL: independent strand (n=30) that used an ethnographic field study to examine
views about what was good/not good and what could be done differently.
Results:
• QUAN: life-prolonging measures rare, symptom management less conservative and
symptoms increased dramatically in the last month of life
• QUAL: challenges in gauging decline, determining when to intervene & how, & how
to maintain normalcy & dignity.
Discussion: QUAL and QUAN results were complementary, indicating that palliative care
can be appropriate in nursing homes, but should begin earlier to help residents/families understand the condition & treatment options, and adjust to changes over time.
Concurrent Mixed Methods Designs
1. Convergent Parallel Design (Example – Complementary Results)
21
QUAN
Pre-test
Data &
Results
QUAN
Post-test
Data &
Results
Intervention
qual
Process
(before, during or after)
Interpretation
2. Embedded Design (AKA “Basic Intervention Mixed Methods Design”9,32)
Concurrent Mixed Methods Designs
• Qualitative data collected mainly to support development of intervention,
understand contextual factors, and/or explain results5,9
• Timing: two data collection methods, one embedded in the other, embedded
one can occur before, during or after intervention
• Emphasis: priority given to method addressing primary question (QUAN)9
• Integration (“Embedding”): results often embedded at multiple points, data integrated for analysis and interpretation
22
2. Embedded Design (Example)
Concurrent Mixed Methods Designs
Example: Study Protocol for Pilot RCT of Stepped Care Treatment of Depression
(STEPS)33
Purpose: determine feasibility and acceptability of STEPS intervention; results to inform
design of a fully-powered RCT on the effectiveness & efficiency of stepped care.Methods:
• QUAN: obtain pilot data on recruitment, retention and the pathway of patients
through treatment to assess feasibility. Outcome data (depressive symptoms, worry,
anxiety, QoL) collected at baseline and 6 months on the effects of stepped care
compared with high-intensity therapy. A minimum of 60 patients with Major
Depressive Disorder will be recruited from an Improving Access to Psychological
Therapies service and randomly allocated to each group.
• QUAL: interviews will obtain data on acceptability. Pilot trial and interviews will be undertaken concurrently. Quantitative and qualitative data will be analysed
separately and then integrated.
23
Sequential Designs Mixed Methods Designs
QUAN
Data &
ResultsInterpretation
qual
Data &
ResultsConnecting to (following up)
1. Explanatory Design
• Usually used when quantitative results are unexpected, or measures
known to be insufficient to address research question,
• Timing: collection and analysis of quantitative data followed by the
collection and analysis of qualitative data.
• Emphasis: priority may be with quantitative method, or can be equal
• Integration (“Connecting”): qualitative database links to the
quantitative database through sampling (e.g., subset of survey respondents selected based on survey scores)
24
Sequential Designs Mixed Methods Designs
1. Explanatory Design (Example)
Example: Prevalence and predictors of premature discontinuation of antiplatelet
drug therapy after stent placement36,37
Purpose: determine the prevalence and predictors of discontinuing antiplatelet
drugs before the recommended duration.Methods:
• QUAN: patients with acute myocardial infarction (AMI) (n=500) followed to see if
they took the recommended antiplatelet drug for the recommended duration.
• Qual: AMI patients who discontinued either clopidogrel (n=11) or cholesterol-
lowering therapy (n=29) interviewed to determine reasons for discontinuation.
Results:
• QUAN: 14% of patients with AFI discontinued drugs despite potentially fatal
consequences for early termination
• Qual: patients cited a number of reasons for discontinuing the drugs, many
related to poor communication between physician and patient.
Discussion: the reasons for discontinuing the drugs informed the development of a
guide to support patient-clinician communication about heart medications.38
25
Sequential Designs Mixed Methods Designs
QUAL
Data &
Results
quan
Data &
ResultsInterpretation
Building to
2. Exploratory Design
• Often used in instrument development
• Timing: collection and analysis of qualitative data followed by the
collection and analysis of quantitative data
• Emphasis: priority may be with qualitative method or equal
• Integration (“Building”): qualitative results inform data collection approach used in quantitative component (e.g., items in survey built from previously collected qualitative data)
26
Sequential Designs Mixed Methods Designs
2. Exploratory Design (Example)
Example: Use and Perceptions of Electronic Dietary Assessment (e-DA) Tools by Health
Care Professionals39
Purpose: explore provider use & perspectives on e-DA tools in mobile apps & websites.
Methods:
• QUAL: 11 interdisciplinary focus groups with 50 providers to obtain perspectives on
use of e-DA. Focus group transcripts used to develop web-survey, interpretive themes added depth/context.
• Quan: web-based survey sent to Family Health Teams throughout Ontario,
descriptive and bivariate analyses completed.
Results:
• QUAL: suggested barriers to using e-DA include: patients’ lack of comfort with using
technology, patient misinterpretation of e-DA results, time and education for
providers to interpret results & train/educate patients.• QUAN: indicated e-DA used to improve: 1) patients’ eating habits; 2) quality of
dietary assessment; and, 3) care process. Dietitians used e-DA more than other
providers. Strong interest across disciplines in using e-DA tools for managing
obesity, diabetes and heart disease, especially for patient self-monitoring.
3. Value of Using Qualitative
Approaches in Intervention
Studies
Value of Qualitative Approaches in Intervention Studies
we focus on:
1. how qualitative research is used in RCTs
2. the value, or potential value, of qualitative results in generating evidence for the effectiveness of interventions.
Key reviews: O’Cathain et al. (2013)45, Lewin et al. (2011)5
Q1: How is Qualitative Research Actually Used?
Before a trial:
Explore research question, health condition & context
Identify ethical issues
Explore issues relating to recruitment, retention, diversity
Identify models, mechanisms, theory & hypotheses for RCT
Develop or refine intervention content & delivery
Explore feasibility & acceptability of intervention & trial
Develop or select outcome measures
During a trial:
Determine Intervention actually delivered & fidelity
“Unpack” processes of implementation and change
Explore feasibility & acceptability of intervention & trialAfter a trial:
Explain findings
Explain variation in effectiveness of intervention
Assess appropriateness of theory, modify accordinglyGenerate further questions, hypotheses, future studies
Q2: What is the Value of Qualitative Research in RCTs?
Item Potential Value Examples
1. Bias Avoids measurement
bias
-Helps test face and content validity of
instruments with patients/providers
2. Efficiency Increases recruitment
rate
-Use of observation and interviews to
identify problems with recruitment
Saves money -stops attempts to undertake full trials of
poor/unacceptable interventions or
weak trial designs
-ensures full trials are only undertaken on
promising/optimized interventions
3. Ethics Determine sensitivities
of providers & patients
-design recruitment and communication
strategies to ensure positive experience
Improves informed
consent
-communication valued as much as
information (“gold standard” of informed
consent)
Q2: What is the Value of Qualitative Research in RCTs?
Item Potential Value Examples
4. Implementation Replication in real
world
-describes components and essential
elements of intervention to enable
replication
Transferability -identifies mechanisms or action or
contextual issues needed for success
5. Interpretation Explain findings -explains null findings
-contextualizes successful interventions to
disseminate & transfer to real world
-explains variation in outcomes
6. Relevance Ensures intervention
meets needs of
providers and patients
-identifies benefits/values as seen by
providers & patients
-ensures appropriate (culture, context)
7. Success Makes trial viable,
feasible, successful
-engenders stakeholder support
-ensures trial appropriate (culture, context)
Q2: What is the Value of Qualitative Research in RCTs?
Item Potential Value Examples
8. Validity Internal Validity - Ensures right outcomes & measures used
External Validity - Identifies and addresses recruitment issues
to obtain hard to reach groups
Q2: What is the Value of Qualitative Research in RCTs?
O’Cathain et al. (2013) & Lewin et al. (2011) both found little evidence of integration
and few discussions of how qualitative methods were used to explain trial findings
this may be why the previous list of values understates a significant contribution that
qualitative methods can bring to RCTs:
AN ENRICHED UNDERSTANDING OF THE RESEARCH PROBLEM RESULTING FROM
COMPARING QUALITATIVE AND QUANTITATIVE FINDINGS
this contribution may be significant particularly where findings conflict:
Next step: reciprocal interrogation of both sets of findings, which may
reveal deficiencies in quantitative instruments/measures29,47
highlight methodological or design weaknesses47
uncover a deviant or “off-quandrant” dimension of problem3, 46
modify existing theories or create new ones3,29
4. Key Challenges in Using
Qualitative Research Alongside
RCTs
Key Challenges - Continued
Challenges arise related to:
Different mixed methods designs
Broader Issues that impact all designs
Challenges (or Strengths) Relating to Design Types16
Issue Sequential Designs Concurrent Designs
Personnel Can be done by a single or small group of researchers
- requires team of researchers (multiple activities occur in parallel)- team dynamics, power issues (can
threaten quality of each component)
Skills Difficult for single or small group to
acquire different skills to do both qualitative and quantitative research
-team members can be skilled in
qualitative and quantitative research-may need mixed methods specialist
Duration Longer than concurrent designs Quicker but intensive in short term
Funding,Ethics
-Challenging to secure upfront if cannot specify subsequent component(s) before completing 1st one-Amendments required if cannot specify upfront
Easier to specify all components upfront in sufficient detail to secure funding & ethics approval
Publication Lends itself to separate publications Lends itself to joint publication (one component may make little sense without the other)
Challenges (or Strengths) Relating to Design Types - Continued17
Issue Sequential Designs Concurrent Designs
Feedback of Interim or Final Qualitative Findings
-feedback not problematic as one component is intended to inform the other
-knowledge of dissatisfaction with, or acceptability of, the intervention or research process may result in changes
to intervention or recruitment/follow-up/outcome procedures, which can threaten scientific integrity-knowledge that adherence to, or acceptability of, the intervention differs by sub-group may bias recruitment in favour or particular groups-feedback of participant views of the intervention can result in unblinding
.
need guidelines & measures for evaluating complex interventions18,19
Pragmatic RCTs:
CONSORT extension focuses on effects, not process or mixed
methods20
Complex interventions:
CONSORT extension for complex interventions is being developed21
UK’s MRC 2008 update22: called for evaluation of both effects and
process but few details on how to evaluate process
Process evaluation – useful frameworks:
Normalization Process Theory (NPT)23
Implementation Fidelity24
Program Theory25
UK’s Standard Operating Procedures for Qualitative Research in Trials7
Challenges Relating to all Designs
Integrating qualitative and quantitative findings remains a
challenge in RCTs despite the many textbooks, journal
articles and best practice guidelines now available
Lewin et al.’s (2011) systematic review of RCTs of complex
interventions5 found only 30% used qualitative approaches,
and in these:
little discussion of integration
few discussed contributions of both methods to
overall study interpretations
Other studies support Lewin et al.’s (2011) findings34,35,45
Challenges Relating to all Designs
integration challenges in RCTs may persist because the
legacy of the epistemological divide lingers in16,28 :
training researchers
biases favouring one approach over the other
imbalances in status and power between groups of
researchers
Challenges Relating to all Designs
the role of theory should be considered from the beginning
theory emphasized in UK’s MRC 2008 guidelines for complex
interventions and design of standard operating procedures
for qualitative methods used in UK clinical trials7
theory may be important for integration of qualitative results:
Lewin et al.(2011) found that twice as many RCTs that
included qualitative work had a clearly stated theoretical
basis compared with RCTs without any qualitative work
Theory increases likelihood that the role of qualitative work
is planned ahead of time and funded appropriately
Challenges Relating to all Designs
References1. Creswell, J. W. (2011). Controversies in mixed methods. In N. K. Denzin and Y. S. Lincoln (Eds.) SAGE handbook of qualitative
research (4th ed.). Thousand Oaks, CA: Sage.2. Erzberger, C. & Prein, G. (1977). Triangulation: Validity and empirically-based hypothesis construction. Quality and Quantity 31,
141-154.3. Jick, T. (1979). Mixing Qualitative and Quantitative Methods: Triangulation in Action. Administrative Science Quarterly. 24(4),
602-611. Published by Sage Publicatons Inc. on behalf of Johnson Graduate School of Management, Cornell University. URL: http://www.jstor.org/stable/2392366.
4. Plano Clark, V. (2010). The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qualitative Inquiry. 16(6), 428-440.
5. Lewin, S., Glenton, C., Oxman, A. (2009). Use of qualitative methods alongside randomized controlled trials of complex healthcare interventions: methodological study. BMJ. 339:b3496. doi: 10.1136/bmjb3496.
6. Creswell, J., Klassen, A, Plano Clark, V, Clegg Smith, K. (2011). Best Practices for Mixed Methods Research in the Health Sciences. Office of Behavioural and Social Sciences Research (OBSSR). U.S.
7. Rapport, F., Porter, A., Snooks, H., Jones, K., Peconi, J., Sanchex, A. et al. (2013). Qualitative research within trials: developing a standard operating procedure for a clinical trials unit. Trials. 14(54), http://www.trialsjournal.com/content/14/1/54.
8. Yardley, L. & Bishop, F. (2015). Editorial: Using mixed methods in health research: Benefits and challenges. British Journal of Health Psychology. 20, 1-4.
9. Curry, L., Nunez-Smith, M. (2015). Mixed Methods in Health Sciences Research - A Practical Primer. Vicki Plano Clark and Nataliya V. Ivankova (Series Editors). Sage Publications.
10. Sandelowski, M. (2014). Guest Editorial: Unmixing Mixed-Methods Research. Research in Nursing & Health. 37, 3-8.11. Johnson, Onwuegbuzie, and Turner. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods
Research. 1(2), 112-133 (pg. 123).12. Morse, J.M. (2010). Simultaneous and sequential qualitative mixed method designs. Qualitative inquiry. 16, 483-491. 13. Morse, J.M. & Niehaus, L. (2009). Mixed method design. Principles and procedures. Walnut Creek, CA: Left Coast Press Inc.14. Creswell, J. & Plano Clark, V. (2011). Designing and conducting mixed methods research. 2nd Edition. Thousand Oaks, CA.15. Tashakkori, A. & Teddlie, C. (2010). Handbook of mixed methods in social and behavioural research. 2nd Editiion. Thousand
Oaks, CA: Sage.16. Bishop, F. (2015). Using mixed methods research designs in health psychology: An illustrated discussion from a pragmatist
perspective. British Journal of Health Psychology. 20, 5-20.
17. Cooper, C., O’Cathain, A., Hind, D, Adamson, J., Lawton, J., Baird, W. (2014). Conducting qualitative research within Clinical Trial Units: Avoiding potential pitfalls. Contemporary Clinical Trials. 38, 338-343.
18. Bakker, F., van Munster, B., Hulscher, M. (2013). Evidence from Multicomponent Interventions: Value of Process Evaluations. JAGS. 61(5), 844-845.
19. Moore, G. et al. (2014). Process evaluation in complex public health intervention studies: the need for guidance. J Epi CommHealth.68(2),101-102.
20. Zwarenstein et al. (2008). Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 337:a2390.
21. CONSORT extension for complex interventions. (2013 on-line). Website: http://www.spi.ox.ac.uk/research/centre-forevidence-based-intervention/consort-study.html. Accessed April 12 2015.
References22. Craig, P. & Petticrew, M. (2013). Developing and evaluating complex interventions: Reflections on the 2008 MRC
guidance. International Journal of Nursing Studies, 50, 585-592.
23. May, C., Finch, T et al. NPT Users’ Manual and Toolkit. (2010 on-line). Website: www.normalizationprocess.org. Accessed
April 23 2015.
24. Carroll, C. et al. (2007). A conceptual framework for implementation fidelity. Implementation Science. 2:40, doi:
10.1186/1748-5908-2-40.
25. Sidani, S. & Sechrest C. (1999). Putting Program Theory into Operation. American Journal of Evaluation. 20(2), 227-238.
26. Curry, L, Krumholz, H., O’Cathain, A., Plano Clark, V., Cherlin, E., Bradley, E. (2013). Mixed Methods in Biomedical and
Health Services Research. Circ Cardiovasc Qual Outcomes. 6, 119-123. Website: http://circoutcomes.aha.journals.org.
Accessed April 14 2015.
27. O’Cathain, A., Murphy, E., Nicholl, J. (2007). Integration and publications as indicators of ‘yield’ from mixed methods
studies. Journal of Mixed Methods. 1, 147-163.
28. Lunde, E, Heggen, K., Strand, R. (2013). Knowledge and power: Exploring unproductive interplay between quantitative
and qualitative researchers. Journal of Mixed Methods Research. 7, 197-210.
29. Östlund et al. (2011). Combining qualitative and quantitative research within mixed method research designs: A
methodological review. International Journal of Nursing Studies. 48, 369-383.
30. Fredenberg & Silverman. (2014). Effects of cognitive-behavioural music therapy on fatigue in patients in a blood and
marrow transplantation unit: A mixed-method pilot study. The Arts in Psychotherapy. 41: 433-444.
31. Powers, B.A. & Watson, N.M. (2008). Meaning and Practice of Palliative Care for Nursing Home Residents with Dementia at
End of Life. American Journal of Alzheimer’s Disease and Other Dementia’s. 23(4): 319-325.\
32. Fetters, M.D., Curry, L.A., Creswell, J. (2013). Achieving Integration in Mixed Methods Designs – Principles and Practices.
Health Services Research. 2134-2156. doi: 10.1111/1475-6773.12117.
33. Hill, J., Kuyken, W., & Richards, D. (2014). Developing stepped care treatment for depression (STEPS): study protocol for a pilot randomised controlled trial. Trials. 15:452. http://www.trialsjournal.com/content/15/1/452.
34. Kinn, S. & Curzio, J. (2005). Integrating qualitative and quantitative research methods,” Journal of Research in Nursing, 10, 317–336.
35. Vuckovic, N. (2002). Integrating qualitative methods in randomized controlled trials: the experience of the Oregon Center for Complementary and Alternative Medicine,” Journal of Alternative and Complementary Medicine, 8(3), 225–227
References
36. Spertus, J., Kettelkamp, R, Vance, C., Decker, C., Jones, P., Rumsfeld, J. et al. (2006). Prevalence, predictors, and outcomes
of premature discontinuation of thienopyridine therapy after drug-eluting stent placement: results from the PREMIER
registry. Circulation. 113, 2803-2809.
37. Garavalia, L. Garavalia, B., Spertus, J., Decker, C. (2009) Exploring patients’ reasons for discontinuance of heart
medications. Journal of Cardiovascular Nursing. 24, 371-379.
38. Garavalia, L., Garavalia, B., Spertus, J., Decker, C. (2011). Medication Discussion Questions (MedDQ): developing a guide
to facilitate patient-clinician communication about heart medications. Journal of Cardiovascular Nursing. 26: E12-E19.
39. Bonilla et al. (2015). Use of electronic dietary assessment tools in primary care: an interdisciplinary perspective. BMC
Medical Informatics and Decision Making. 15:14
40. Skilbeck, J.K. et a. (2005). An exploration of family carers’ experience of respite services in one specialist palliative care unit.
Palliative Medicine. 19(8), 610-618.41. Midtgarrd, J. et al. (2006). The group matters: an explorative study of group cohesion and quality of life in cancer patients
participating in physical exercise intervention during treatment. European Journal of Cancer Care. 15(1), 25-33.
42. O’Cathain, A., Murphy, E., Nicholl, J. (2010). Three techniques for integrating data in mixed methods studies. BMJ.
341:c4587.
43. Giacomini, M. (2001). The rocky road: qualitative research as evidence. EBM notebook. 6(5), 4-6.
44. Sale, J., Lohfeld, L., Brazil, K. 2002. Revisiting the Quantitative-Qualitative Debate: Implications for Mixed Methods Research.
Quality & Quantity. 36, 43-53
45. O’Cathain, A., Thomas, K., Drabble, S., Rudolph, A., Hewison, J. (2013). What can qualitative research do for randomised
controlled trials? A systematic mapping review. BMJ. 3:e002889. doi.10.1136/bmjopen-2013-002889.
46. Barbour, R. (2001). Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 322,
1115-1117.
47. Moffatt, S., White, M., Mackintosh, J., Howel, D. (2006). Using quantitative and qualitative data in health services research –
what happens when mixed methods findings conflit? [SRCTN61522618]. BMC Health Services Research. 6:28, doi:
10.1186/1472-6963-6-28.
48. Lee, C., & Rowlands, I. (2015). When mixed methods produce mixed results: Integrating disparate findings about
miscarriage and women’s well-being. British Journal of Health Psychology. 20, 36-44.