Rhetorical moves and audience considerations in the discussion sections of randomized controlled trials of health interventions--ECA--2017-06-22

Rhetorical Moves and Audience

Considerations in the Discussion

Sections of Randomized Controlled Trials

of Health InterventionsJodi Schneider (University of Illinois at Urbana Champaign, USA)

[email protected]

Graciela Rosemblat (National Library of Medicine, USA)

Shabnam Tafreshi (The George Washington University, USA)

Halil Kilicoglu (National Library of Medicine, USA)

European Conference on Argumentation, Fribourg, Switzerland

2017-06-22

Disclaimer

The views and opinions expressed do not

necessarily state or reflect those of the U.S.

Government, and they may not be used for

advertising or product endorsement

purposes.

Why study scientific papers?

• “The interpretive face of science is

characterized by controversy,

disagreement, dissensus.” (Prelli, “A

Rhetoric of Science”, 1989, p4)

• “Scientific publications are documentary

representations of defeasible arguments,

supported by data and repeatable

methods.” (Clark, Ciccarese, Goble 2014)

• Prelli, Lawrence J. "A rhetoric of science: Inventing scientific discourse." (1991).• Clark, Tim, Paolo N. Ciccarese, and Carole A. Goble. "Micropublications: a

semantic model for claims, evidence, arguments and annotations in biomedical communications." Journal of biomedical semantics 5.1 (2014): 28.

Why study medical papers?

• Large potential impact on human health

• Input to evidence-based practice

• Well-elaborated study of evidence

Figure credit: Duke University Medical Center Library. Introduction to

Evidence-based Practice. What is Evidence-Based Practice (EBP)? http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036021

Hierachy of Evidence

Hierachy of Evidence

Randomized Controlled Trial

(RCT)

• Medical RCTs test the efficacy of medical treatments.

• Features:

– Control group (placebo or standard treatment)

– Random assignment

– Neither doctors nor patients know which group is which (“double blind”)

Image credit: Ancker, Jessica S & Quynh Pham. "Beyond the RCT

Evaluating innovations in the Learning Health System” AMIA 2016 Tutorial

Corpus

• 37 research articles reporting on medical

trials, mainly RCTs

• Full-text available from PubMed Central

• Chosen for an ongoing citation annotation

study using Xu et al.’s corpus

Xu, Jun, Yaoyun Zhang, Yonghui Wu, Jingqi Wang, Xiao Dong, and Hua Xu. "Citation sentiment analysis in clinical trial papers." In AMIA Annual Symposium Proceedings, vol. 2015, p. 1334. American Medical Informatics Association, 2015.

Other ongoing study on this corpus

• Understand citation structure of Discussion sections of Randomized Clinical Trials

• Annotate two aspects of citations:

– Citation scope

– Citation matter

• Began summer 2016 at U.S. National Library of Medicine, led by Halil Kilicoglu, with Graciela Rosemblat leading annotation.

• Ongoing study in biomedical informatics

Corpus characteristics

• Various conditions, e.g.– Brain injury

– Degenerative eye disease

– Malaria

– Postpartum back pain

• Various settings, e.g. – Critical care

– Community/Ambulatory care

– Internet-based treatment

– Laboratory treatment

– School-based treatment

Corpus characteristics

• Various locales, e.g. – Madagascar

– Malawi

– Netherlands

– Uganda

– United Kingdom

– United States

• Single site and Multisite

• Industry-sponsored and Researcher-focused

• Various stages of clinical trial

Focus: 37 discussion sections

http://europepmc.org/articles/PMC1420299

Rhetorical Moves in RCT

Discussion Sections

Rhetorical Moves in RCT

Discussion Sections• Novelty

• Gaps/opportunities

• Contextualization

• Comparability to previous results

• Validity of methods

• Anticipating objections

• Pointers to ongoing work

• Acknowledging limitations

• Future work

• Summaries

Novelty

• “We report the first safety data collected in the setting of a randomized, controlled study for patients sustaining TBI with polytrauma.” (2206502)

• “This is the first double blind prospective study designed to evaluate the effects of fruit juice feedings during diarrheal disease in young children.” (1217327)

Gaps/opportunities

• "At the start of this study, little was known about the specific effect of psychosocial determinants in SCs, although interventions aiming to modify these determinants, such as cognitive behavioural programs were considered to be promising for musculoskeletal pain in general [4-8]. We developed the EAP to fill a gap in UC, which focuses mainly on biomedical determinants." (2211478)

• "We have already studied the stimulation of periodontal tissue regeneration by FGF-2 in animal models and believe that the protein represents a major candidate for a periodontal tissue-regenerating agent." (2432040)

Contextualization

• “This study is part of a series to explore effective xylitol delivery vehicles that can be used in school programs in the U.S. Results from the xylitol dose study [3] showed...” (2527560)

• “In general we did not deviate from the original study design, which was described independently of the study results [15].” (1420299)

• "The first report on this study showed that the intra-operative use of TA significantly decreased the incidence of intra-operative complications associated with PPV, and no adverse events related to surgery were found over a 3-month observation period [13]." (2413124)

Comparability to previous results

• "However, none of these 11 studies were performed on pediatric populations, making further inferences or comparisons problematic" (1876598)

• "This is in line with previous studies of internet-driven CBT for posttraumatic stress reactions [9], complicated grief [30] and CBT interventions in face-to-face studies [10,31]." (1885249)

Validity of methods

"Algometry is used to measure the sensitivity of pain or pressure [4].

Algometer instrumentation can include manual and electric models.

The use of any experimental instrument including the PGA must

be tested for validity and reliability between examiners and

between performances of the same examiner. The PGA used in

this study has been tested against itself, palpation, pressure

plates with reliable results [17,18]. However, the above is based on

the assumption that the tester is trained in the application of the PGA

otherwise issues associated with the rate of pressure application [45-

47] the determination of an end point based on a verbal patient

response [45,46] and the possible sensitisation of a selected landmark

based on repeated measures may all alter the accuracy of the

outcome." (2427032)

Validity of methods

"The technique of using the consumption of morphine during PCA treatment of postoperative pain, as a measure of the effect of the analgesic regime under study, has been used in several other studies of this kind [5,6]." (1637100)

Anticipating objections

"It might be argued that these observations of blood stage immunity (reduction in growth rates, presence of crisis forms etc.) may rather be related to the potential pre-erythrocytic action of PEV3A. However, in previous challenge studies, the number of parasites emerging from the liver of unvaccinated control volunteers has been shown to vary as much as five fold [7]. Despite this variation, rates of parasite growth in these volunteers were similar. Equally, crisis forms have never been observed historically in our studies of pre-erythrocytic vaccine candidates. Indeed, following their observation in this study, the same slide reader went back to examine a selection of blood films from vaccinated and control volunteers in two previous studies where some evidence of pre-erythrocytic efficacy has been observed (VAC021 [28] and VAC023 [29]). The slide reader was blinded to the group allocation of the volunteers. In total 72 slides were selected for re-examination (6 slides each from 6 volunteers from each study, 3 vaccinees plus 3 controls) and

no crisis forms were observed. It seems likely therefore, that the differences observed here are indeed related to vaccine induced blood stage immunity." (2204057)

Pointers to ongoing work

• "An ongoing phase III study is likely to provide a better evaluation of safety in this patient population." (2206502)

• Further analysis of the 18 months follow-up data and the examination of other potentially relevant moderators such as posttraumatic growth [Maercker & Knaevelsrud, in preparation] will hopefully enhance our understanding of online therapeutic processes." (188529)

Acknowledging limitations

• "Among the limitations of this study is the screening strategy for the recruitment of the patients. We deliberately handled strict exclusion criteria for participation in this study. We excluded 72% (n = 253) of the patients who wanted treatment but did not meet the inclusion criteria. This might limit the generalizability of our results.” (1885249)

Future work

• "New studies should be carried out, with longer follow-up times and larger samples, to evaluate the effects of NPPV on the quality of life of patients on weaning ventilation support and to understand how the cause of ARF could affect the results of different weaning ventilation

methods. Cost evaluation should also be included in these studies." (2447605)

• "Future studies directed on possible central mechanisms of this complicated chronic pain syndrome are warranted."(1815612)

Summaries

• "Based on present evidence BTX cannot be recommended as treatment for neck pain in chronic whiplash patients."(1815612)

• "The evidence from this study is that Hall PMCs allow a reliable, low-maintenance seal to be achieved by GDPs." (2265270)

• "Therefore, the mouthwash clearly demonstrated an anti-malodor effect on morning breath potential without any measurable side effects in healthy subjects." (2637235)

Rhetorical Moves in RCT

Discussion Sections• Novelty

• Gaps/opportunities

• Contextualization

• Comparability to previous results

• Validity of methods

• Anticipating objections

• Pointers to ongoing work

• Acknowledging limitations

• Future work

• Summaries

Audience

Multiple audiences

“Audiences of specialists…do not comprise

the only audiences who respond to scientific

or other professional discourse. There may

be a number of interested audiences for any

particular discourse.” (Prelli,1989, p26)

Interested audiences include:

• Specialists responsible for “authorizing the

claims to knowledge” (Prelli,1989, p26)

• Non-specialists who might “judge the work

as interested witnesses” especially if there

are “broad social implications” (Prelli,1989,

p27)

Fulfilling audience expectations

“To succeed rhetorically, any rhetor must ground a

position in what the addressed audience considers

reasonable”(Prelli,1989, p27-8)

“…[draw] on standard or conventional patterns of

thinking that will be employed by one’s specific

audience”(Prelli,1989, p37)

“…choose purposes and contents likely to be

deemed pertinent and conventionally appropriate

by the specifically situated thought groups that

comprise the audience.”(Prelli,1989, p38)

Audience-specific sections from

discussion sections of reports of

Randomized Controlled Trials

Researchers are the expected

audience for RCTs

Mellis, Craig. "Evidence-based Medicine: What Has Happened in the Past 50 Years?" J Paediatr Child Health 51, no. 1 (2015): doi:10.1111/jpc.12800.

Audiences we find

• Specialist researchers in the same area

• Clinicians

• Clinical practice managers

• Grant funders

• Health systems

Specialist researchers in the same

area

• “The most plausible explanation for the

similar trends in BV reduction in both arms

of this study is the inherent activity of the

placebo formulation.” (1851729)

Clinicians

• “caution should be exercised in applying these findings

to the general management of dental caries without

being aware of potential pit falls.”(2265270)

• “none of the results of this particular clinical trial suggest

any clinical problems concerning the safety of

administering FGF-2 to patients with periodontitis. One

more piece of evidence supporting the high safety of

FGF-2 applied locally to periodontal tissue is that this

therapy has already been used for more than 5 years as

a remedy for intractable ulcers (Fiblast spray).”

(2432040)

Clinical practice managers

• “Our findings are particularly pertinent

to smaller clinical practices that might

not be aligned with larger organizations,

and who will have to choose EMR systems

and DSSs on their own.”(1876598)

Grant funders

• “Additional research is needed to determine whether these improvements translate into even longer-term reductions of health-related costs. Although the sample was heterogeneous on demographic factors such as income, the group was relatively ethnically and racially homogenous, consisting primarily of white women. Future replications with diverse populations are indicated to further evaluate the promise and translation potential of the MLP.” (1783667)

Grant funders

"Admittedly, overall complexity of the issue addressed by the present study clearly merits pursuit of further studies of a closely similar, multi-factorial intervention design, though preferably conducted over much longer time span, as well as best targeting the over 60s as the population standing by far the best chance of a successful outcome, in order to verify whether the present findings, clearly encouraging as they appear, might actually have sufficient potential to develop into a trend of indisputable clinical significance, especially in terms of possible application in a comprehensively designed, nationwide programme specifically aimed at addressing the problem of a steadily growing proportion of the frail seniors dependent for their activities of daily living." (2637855)

Health systems

• “Given the rising costs and growing burden of diabetes on the health-care system [57], strategies that aim to prevent or delay the onset of this disease might be beneficial [58]” (1865086)

• “AQ+AS (0.51 US$) [28] is currently four times more expensive than AQ+SP (0.13 US$). In Madagascar, with a reported 2,114,400 cases of suspected malaria (2003), the use of AQ+SP instead of AQ+AS would reduce the annual antimalarial treatments costs by 800,000 US$. In view of the recent report by Tagbor and colleagues [29] there is hope that AQ+SP could serve as a safe and effective alternative for malaria treatment in pregnancy and for intermittent preventive treatment until the safety of ACTs for pregnant women has been established.” (1887535)

Audiences we find

• Specialist researchers in the same area

• Clinicians

• Clinical practice managers

• Grant funders

• Health systems

Rhetorical Moves in RCT

Discussion Sections• Novelty

• Gaps/opportunities

• Contextualization

• Comparability to previous results

• Validity of methods

• Anticipating objections

• Pointers to ongoing work

• Acknowledging limitations

• Future work

• Summaries

Future work

• Systematically code rhetorical moves and audiences – likely to lead to revisions

• Compare to previous work in English for Specific Purposes for medical and general scientific work

• Identify key suggestions for writing effective RCT reports, to reach their intended audiences.

• Develop Argumentation Mining algorithms to automatically detect rhetorical moves.– Could be used to automatically extract discussion

topics for evidence synthesis of RCTs.

– Could be used to give authors fast, automatic feedback about readability.

Citation examples from

discussion sections of reports of

Randomized Controlled Trials

“This” work agrees with…

• “This is in accordance with earlier studies

in the ambulatory surgical setting [3]” -

PMC1637100

Definitions and background info

• “Self-efficacy, which may relate to

motivation, is the perceived confidence in

one's ability to accomplish a specific task

[19].” - PMC2194735

Presenting a range of evidence

• “Except in one study [20], short-term

administration of GH transiently worsened

insulin resistance [19,53] and increased

fasting glucose levels [53].” - PMC1865086

Clause-level changes in meaning

• “Two of four randomised clinical trials

…have found a difference in admission

rate [12,19] and two have not [22,23].” -

PMC1142326

A single citation can support a

whole paragraph• Dutton and colleagues [8] described a series

of 81 coagulopathic trauma patients treated with rFVIIa. Of these, 20 received rFVIIa for treatment of coagulopathy related to TBI. Six of these patients had additional polytrauma. The outcome of these patients was poor and 15 of 20 patients died. The authors attributed this high mortality rate to the severity of brain injury. None of the 81 trauma patients in this series had any clinical indication of TE events.”

Discussing treatments, outcomes,

other authors’ conclusions• Dutton and colleagues [8] described a series

of 81 coagulopathic trauma patients treated with rFVIIa. Of these, 20 received rFVIIa for treatment of coagulopathy related to TBI. Six of these patients had additional polytrauma. The outcome of these patients was poor and 15 of 20 patients died. The authors attributed this high mortality rate to the severity of brain injury. None of the 81 trauma patients in this series had any clinical indication of TE events.”

Sometimes several parallel

paragraphs.

• Dutton and colleagues [8] described a series of 81 …patients treated with rFVIIa”

• “Zaaroor and Bar-Lavie [23] reported the first series of five patients …”

• “Morenski and colleagues [24] described …three pediatric … cases”

Multiple citations in a paragraph

• “Berger et al. [42] compared the efficacy of hypertonic saline and mannitol to reduce ICP after a combination of two different neuronal injuries. Initially, ….The authors demonstrated that …After …. It is remarkable that … An accumulation …These different effects … [42]. Furthermore, Prough et al. observed a higher regional cerebral blood flow in dogs with induced intracerebral hemorrhage after hypertonic saline without any increase of the CPP [43].” - PMC1297608

Avoiding a 1-sentence paragraph?

• “Berger et al. [42] compared the efficacy of hypertonic saline and mannitol to reduce ICP after a combination of two different neuronal injuries. Initially, ….The authors demonstrated that …After …. It is remarkable that … An accumulation …These different effects … [42]. Furthermore, Prough et al. observed a higher regional cerebral blood flow in dogs with induced intracerebral hemorrhage after hypertonic saline without any increase of the CPP [43].” - PMC1297608