DE-CODE: a coding scheme for assessing debriefing interactionsdevelopment of coding scheme The coding scheme was developed both deductively and induc-tively (figure 1). A subteam of

51Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

DE-CODE: a coding scheme for assessing debriefing interactionsJulia C Seelandt,1,2 Bastian Grande,2,3 Sarah Kriech,2 Michaela Kolbe2,4

Original research

To cite: Seelandt JC, Grande B, Kriech S, et al. BMJ Stel 2018;4:51–58.

► Additional material is published online only. To view please visit the journal online (http:// dx. doi. org/ 10. 1136/ bmjstel- 2017- 000233).

1Quality Management and Patient Safety, University Hospital Zurich, Zurich, Switzerland2Simulation Center, University Hospital Zurich, Zurich, Switzerland3Institute of Anesthesiology, University Hospital Zurich, Zurich, Switzerland4ETH Zurich, Zurich, Switzerland

Correspondence toDr Michaela Kolbe; mkolbe@ ethz. ch

Accepted 20 September 2017Published Online First 8 November 2017

AbsTrACTDebriefings are crucial for learning during simulation-based training (SBT). Although the quality of debriefings is very important for SBT, few studies have examined actual debriefing conversations. Investigating debriefing conversations is important for identifying typical debriefer–learner interaction patterns, obtaining insights into associations between debriefers’ communication and learners’ reflection and comparing different debriefing approaches. We aim at contributing to the science of debriefings by developing DE-CODE, a valid and reliable coding scheme for assessing debriefers’ and learners’ communication in debriefings. It is applicable for both direct, on-site observations and video-based coding.Methods The coding scheme was developed both deductively and inductively from literature on team learning and debriefing and observing debriefings during SBT, respectively. Inter-rater reliability was calculated using Cohen’s kappa. DE-CODE was tested for both live and video-based coding. results DE-CODE consists of 32 codes for debriefers’ communication and 15 codes for learners’ communication. For live coding, coders achieved good inter-rater reliabilities with the exception of four codes for debriefers’ communication and two codes for learners’ communication. For video-based coding, coders achieved substantial inter-rater reliabilities with the exception of five codes for debriefers’ communication and three codes for learners’ communication. Conclusion DE-CODE is designed as micro-level measurement tool for coding debriefing conversations applicable to any debriefing of SBT in any field (except for the code medical input). It is reliable for direct, on-site observations as well as for video-based coding. DE-CODE is intended to allow for obtaining insights into what works and what does not work during debriefings and contribute to the science of debriefing.

InTroduCTIonDebriefing is a core element of team learning and simulation-based training (SBT).1–3 It is an instruc-tor-guided conversation among trainees that aims to explore and understand the relationships among events, actions, thought and feeling processes and performance outcomes of the simulation.1 2 4 5 In effective debriefings, learners are encouraged to transfer learning from the simulated setting to the patient care context through reflection.6 7

There are various debriefing approaches available providing advice on how to promote learners’ reflec-tion, for example, the Debriefing with Good Judg-ment,1 PEARLS,8 The Diamond9 and TeamGAINS.10

In addition, there are techniques available for creating a psychologically safe and engaging setting,11 codebriefing7 and debriefer communication such as advocacy inquiry1 and circular questions.12 Though evidence on the effectiveness of debriefings is growing,5 13–15 empirical research evaluating debrief-ings during SBT is rare, as are studies comparing different debriefing approaches in SBT.6 Even more, a recent meta-analysis on team training in healthcare concluded that training programmes that involved feedback were less effective than programmes without feedback.16 Although debriefing includes much more than giving feedback, this finding is unsettling and calls for further and more detailed research.

Tools have been developed to assess the quality of debriefings, for example, the Debriefing Assess-ment for Simulation in Healthcare (DASH)17 and the Objective Structured Assessment of Debriefing (OSAD).18 19 These are behavioural marker systems. When using behavioural marker methodology, users rate the overall quality of different behavioural classes (eg, teamwork and communication) rather than single behaviours.20 Both DASH and OSAD have good psychometric qualities17 18 and are extremely useful for developing simulation instructors’ debriefing competencies. However, in a recent study investi-gating the value of a 360° OSAD-based evaluation of debriefings by examining expert debriefing eval-uators, debriefers and learners, significant differ-ences between these groups were found: debriefers

What this paper adds

What is already known on this subject ► Debriefings are crucial for learning during simulation-based training.

► Although the quality of debriefings is very important for SBT, few studies have examined actual debriefing conversations.

► More knowledge on debriefing interactions is important for addressing research gaps and targeting faculty development.

What this study adds ► This study aims to contribute to debriefing science by providing DE-CODE, a coding scheme for assessing debriefers’ and learners’ communication in debriefings.

► DE-CODE may be used in full version (47 codes) for research purpose and in reduced version (selected codes) for faculty development and other purposes.

► DE-CODE is reliable for direct, on-site observations as well as for video-based coding.

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

52 Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

perceived the quality of their debriefings more favourably than expert debriefing evaluators.21 Also, weak agreement between learner and expert evaluators’ perceptions as well as debriefers’ perceptions were found.21 Thus, measuring debriefing quality and processes seems challenging.

Typically, debriefings are conversations within teams.4 12 22 Applying methods of team interaction analysis may thus be a fruitful debriefing measurement avenue. Though behaviourally anchored rating scales are very useful for providing immediate feedback after rating,23 they are less suitable for assessing team dynamics or team interactions, mainly because of their static nature that cannot capture dynamic processes.24 25 For studying team processes and interactions, behaviour coding is the method of choice.26–29 In contrast to behaviour rating, in which the quality, quantity or degree of a behaviour is assessed, in behaviour coding, the occur-rence and timing and, mostly, duration are coded.29 Behaviour coding allows for a more descriptive assessment of behaviour as it occurs and for uncovering team patterns and dynamics that become apparent over time. Research following that approach has been able to show via pattern and lag sequential analysis that patterns of behaviours among team members—rather than individual actions of single team members—are what discrim-inates higher from lower performing teams.30 31 For example, a behavioural observational study on teamwork and communication within surgical teams has shown that more patient-irrelevant and case-irrelevant communication during wound closure is related to worse patient outcomes.32 Such findings were of paramount importance for understanding which factors contribute to effec-tive teamwork; similar analysis of debriefing conversations would provide empirical evidence of what works and what does not work during debriefings.

So far, empirical insights into debriefing interaction patterns are scarce. Few studies have examined actual debriefing conver-sations and how differences in debriefers’ communication influ-ence learners’ outcomes.5 33–36 More knowledge on debriefing

interactions is important for addressing research gaps that have been identified in the debriefing literature37–40 such as (A) identi-fying typical debriefer–learner interaction patterns, (B) obtaining insights into associations between debriefer communication and learners’ reflection, (C) targeting faculty development providing feedback based on identified debriefer–learner communica-tion patterns, as well as for (D) comparing different debriefing approaches.

We aim at contributing to debriefing science by devel-oping DE-CODE, a coding scheme for assessing debriefers’ and learners’ communication in debriefings. We will describe the development of DE-CODE and its reliability and content validity for both video-based coding41 and direct, on-site obser-vations.41 Moreover, DE-CODE is intended to be applicable to any debriefing of SBT in any field.

MeThodsdevelopment of coding schemeThe coding scheme was developed both deductively and induc-tively (figure 1). A subteam of the authors (JCS and MK), who have extensive experience in researching team dynamics and behaviour coding,12 15 30 32 40 42–48 first reviewed the literature on team learning and team debriefings in SBT.1–6 8–10 17 18 33 34 49–58 They also watched five videotaped debriefings during SBT and took notes in free-text form about the observed communication. This SBT took place in a large university hospital in Switzer-land; debriefers were familiar with the following approaches: Debriefing with Good Judgment,59 TeamGAINS,44 guided team self-correction22 as well as circular questions12; participants were anaesthesia care providers; debriefers and participants knew each other from working together in the clinical setting. Based on the findings from literature review and their notes, JCS and MK extracted possible codes and developed a first version of the coding scheme.

Figure 1 Process for developing the coding scheme.

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

53Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

This version was subsequently discussed with subject matter experts for assuring content validity. These subject matter experts were chosen based on their expertise with SBT, their familiarity with the debriefing approaches mentioned above, their extensive debriefing experience in different medical disciplines and profes-sions as well as different status levels to avoid bias. Three senior consultants (two anaesthetists and one trauma surgeon) and one emergency nurse reviewed the coding scheme and suggested revisions. Based on their feedback, small changes were made (eg, overlapping or inappropriate codes were excluded). We also refined the first version in repeated, iterative cycles by applying it to coding five videotaped debriefings that took place in the same hospital during another 2-week SBT session. Debriefers were familiar with the approaches mentioned above; partici-pants were anaesthesia care providers. Two psychologists (JCS and MK) discussed remaining difficulties and decided on the final version of the coding scheme. Code definitions and exam-ples are described in online supplementary table 1 (debriefer communication) and online supplementary table 2 (learner communication).

We developed specific, mutually exclusive codes that are used when communication may be assigned to only one code.60 This requires less cognitive load because it provides coders with clear rules for code assignment.23

Training phase for observersThe five observers underwent an extensive training procedure (between 30 hours and 34 hours for the full coding scheme with all codes) prior to analysing debriefings. All observers held at least a bachelor’s degree in psychology. First, observers were provided with literature on team debriefings, SBT and behaviour coding to familiarise themselves with the context, setting and method.23 24 33 61 62 Second, after studying the literature, observers participated in an observing role in SBT, including the debriefings. Third, observers were introduced to DE-CODE and the coding software. Fourth, with a debriefing expert (JCS), they watched and discussed one videotaped debriefing. Fifth, observers independently coded three to five videotaped debrief-ings. Experts familiar with coding and conducting debriefings (JCS and MK) reviewed these codings and provided extensive, written feedback. Sixth, discrepancies were resolved by discus-sions and further explanations. Observers performed live coding after having coded at least 30 videotaped debriefings.

ParticipantsParticipants were 168 anaesthesia care providers (82 men and 86 women) from a large teaching hospital in Switzerland including 25 attending anaesthetists, 74 resident anaesthetists, 57 anaes-thesia nurses and 12 participants having another educational background. The mean age was 36.40 years (SD=8.75), and participants mean work experience was 7.4 years (SD=7.98) ranging between 0 and 35 years.

ProcedureTo test whether DE-CODE is applicable and reliable for both video-based and on-site coding, we applied it to 50 videotaped and 12 live debriefings performed during SBT. The videotaped debriefings were collected during two 2-week SBT sessions in a large university hospital in Switzerland.

All simulation scenarios used in this study included crit-ical situations during common patient treatment. They were designed based on critical incident reports and were specially written for SBT. They contained the management of medication

(over) doses, anaesthesia inductions in critically ill patients and respiratory problems during anaesthesia induction; leadership and re-evaluation were important components of learning objec-tives. The participants acted in their usual roles.

Videotaped debriefings were coded 2 months after they had been recorded. During live coding, observers sat in close prox-imity to the debriefing table. They were able to watch and listen to the debriefing but did not interfere with it at all. Observers were fully blinded to each other’s scoring during live coding.

Observations for both video and live coding started once all learners and debriefers were seated around the table in the debriefing room. They ended with the announcement of a questionnaire for evaluating the debriefing. Participants were informed about coding before SBT and before debriefings started; participation in the SBT was voluntary. Observers did not know debriefers and participants further than from their observation role.

data codingTo code debriefing data in a way that would allow for further frequency, duration, co-occurrence, sequential and pattern infor-mation to be derived, we applied timed event-based coding.63 Events, also called coding units, were defined as units (ie, mostly sentences) to which a respective code of (online supplemen-tary tables 1 and 2) could be assigned. That is, these predefined communication behaviours were coded each time they occurred involving logging the onset and offset of the event and assigning the respective DE-CODE code. Events were coded based on a similar procedure for video-based and direct, on-site coding. Interact coding software (for video-based coding, figure 2) or the corresponding iOS app (for direct, on-site coding, figure 3) was used for this process.64 To test for inter-rater reliability, every fourth video-taped debriefing was coded independently by two observers, and all direct, on-site codings were independently performed by two observers.

statistical analysisInter-rater reliability was calculated using Cohen’s kappa (κ). κ is a coefficient of inter-rater agreement for nominal scales and ranges from −1.00 to +1.00 with values lower than 0.41 consid-ered as fair, between 0.60 and 0.80 as substantial and above 0.81 as almost perfect agreement.65 We calculated Cohen’s κ for the occurrence versus non-occurrence of each code for every 1 min segment of the coded period.23 Statistical analyses were performed using SPSS V.22.0 software.66

resulTsde-Code coding schemeThe final DE-CODE coding schemes consists of 32 codes for coding debriefers’ communication and 15 codes for coding learners’ communication. Codes are organised in five catego-ries that are based on Tobert and Taylor’s four types of speech (ie, framing, advocating, illustrating and inquiring)58 and an additional category other. Codes, definitions and examples are provided in online supplementary tables 1 and 2, respectively. The complete Coding Manual is available as online supplemen-tary file 1.

Inter-rater reliabilityKappa values for debriefer codes for both video-based and live coding are shown in online supplementary table 1.

For live coding, all Cohen’s κ values were above 0.66 repre-senting substantial to very good agreement67 except for irony and

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

54 Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

humour (κ=0.30), laughing (κ=0.33), normalisation (κ=0.46) and repeating (κ=0.56). The codes pseudo-observations (κ=1), knowledge (κ=1), circular (κ=1), guess-what-I-am-thinking (κ=1) and roleplay (κ=1) received the highest reliability values for live coding.

For video-based coding, all Cohen’s κ values were above 0.60 representing substantial agreement67 except for opinion (κ=0.59), input simulation (κ=0.59), laughing (κ=0.58), structuring (κ=0.59) and psychological input (κ=0.55). The codes guess-what-I-am-thinking (κ=1) and

Figure 2 Screenshot using Interact coding software with codes for debriefers’ and learners’ communication. Since the debriefings and the study were conducted in a German-speaking setting, the coding software was set up in German as well.

Figure 3 Screenshot using Interact iOS app with codes for debriefers’ and learners’ communication.

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

55Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

inquiry (κ=1) received the highest reliability values for video-based coding.

Kappa values for learner codes for both video-based and live coding are shown in online supplementary table 2.

For live coding, all Cohen’s κ values were above 0.68 repre-senting substantial to very good agreement67 except for evalua-tion of learners’ own actions (κ=0.49) and evaluation of team members’ actions (κ=0.50). The codes feelings (κ=1), positive relevance (κ=1), negative relevance (κ=1) and negative evalu-ation of simulation (κ=1) received the highest reliabilities for live coding.

For video-based coding, all Cohen’s κ values are above 0.62 representing substantial agreement67 except for evaluation of learners’ own actions (κ=0.38), negative relevance (κ=0.57) and expressions of humour (κ=0.59). The code action plan (κ=1) received the highest reliability for video-based coding.

dIsCussIonDebriefings are crucial elements of SBT.1–3 They help learners to derive meaning from the simulated learning opportunity.7 Promoting reflection and learning, and ultimately performance and patient care, are goals of debriefings during SBT.17 While evidence on the effectiveness of debriefings5 13–15 and the number of debriefing approaches1 7–12 are growing, empirical research evaluating debriefings during SBT is rare, as are studies comparing different debriefing approaches in SBT.

With DE-CODE, we contribute to the science of debrief-ings by providing a micro-level measurement tool for coding debriefing conversations. DE-CODE provides 32 and 15 codes for debriefer and learner communication, respectively. It is intended to be a comprehensive coding scheme for use in debriefing research in any field. Its codes can be used separately, or combined into larger categories, for selected, direct feedback on debriefings and targeted faculty development. If DE-CODE is used for faculty development, selecting a reduced number of codes (eg, 5 or 6) for the behaviours of interest is recommended and would in turn reduce time for training coders and coding. The use of DE-CODE is not limited to a particular debriefing method, findings from literature on team learning and various team debriefing approaches in SBT were included. DE-CODE complements existing debriefing assessment tools that are based on behaviourally anchored rating scales such as the DASH17 and the OSAD18 19 by allowing for measuring the debriefing communication process more descriptively as it occurs and for empirically identifying debriefing communication dynamics. In analogy with teamwork coding schemes such as the act4teams coding scheme68 and Co-ACT,24 using DE-CODE may provide the database for performing statistical analyses that can identify interaction patterns that occur above change and relate them with outcomes such as performance or learning indicators.30 68 It also may allow for empirically exploring common debriefing issues, for example, how particular debriefer questions may typi-cally trigger certain learner reactions or what debriefer commu-nications may typically be followed by learners verbalising their mental models. These findings would provide important empir-ical insights into debriefing effectiveness.

With respect to DE-CODE’s psychometric qualities, we tested its inter-rater reliability for coding both video-taped debriefing as well as on-site, live debriefings. For live coding, coders achieved good inter-rater reliabilities with the exception of four codes for debriefers’ communication (ie, irony and humour, laughing, normalisation and repeating for debriefers’ commu-nication) as well as for two codes for learners’ communication

(ie, evaluation of learners’ own actions and evaluation of team members’ actions). We will now discuss the potential challenges of applying these six codes.

First, it seems difficult to code irony and humour as well as laughing reliably. One explanation could be that irony, humour and laughing are mostly expressed in groups, and several people are involved, which makes it challenging to distinguish and code all group members that are laughing at the same time and to distinguish debriefers from learners. We recommend using these codes for video-based data, which allows coders to watch an interaction multiple times, and when humour in debriefing is of particular interest. Second, the code normalisation achieved only fair reliability. Since this code contains the debriefer’s subjective evaluation, one might speculate that coders had difficulty distin-guishing it from opinions and suggest emphasising the difference among these codes during observer training. Third, the code repeating seemed challenging, too. It could be difficult for coders to distinguish whether the debriefer repeats completely what the learner had articulated from whether he or she repeats it in his or her words, which would require the code paraphrasing. Again, emphasising the difference among these codes during observer training seems important. Finally, coding evaluation of learners’ own actions and evaluation of team members’ actions seems difficult as well. To correctly assign learners’ communication to these two codes, coders must remember who of the participants had been involved in the scenario, resulting in additional cogni-tive load during coding. Making respective notes prior to the debriefing might provide a remedy.

For video-based coding, coders also achieved good inter-rater reliabilities with the exception of five codes for debriefers’ communication (ie, opinion, input simulation, laughing, struc-turing and psychological input) and three codes for learners’ communication (ie, evaluation of learners’ own actions, negative relevance and expressions of humour).

We have discussed the challenges involved in applying these codes for learners’ communication (ie, evaluation of learners’ own actions and expressions of humour) above. Regarding input simula-tion and psychological input, it seems difficult to distinguish these two codes. One explanation could be that all simulation scenarios were based on critical situation containing human factor aspects and psychological phenomena making it difficult for coders to distinguish which communication referred to the scenario design (input simulation) and which communication adds information on psychological research and phenomena (psychological input). We recommend briefing coders about the scenario design and learning objectives during observer training. In addition, it seemed chal-lenging to reliably code opinion and structuring. Structuring contains debriefers’ communication about what they are now going to talk about, which is part of the debriefers point of view, making it diffi-cult to distinguish opinion and structuring. Providing more precise coding instructions in the coding manual might provide a remedy.

Overall, the results of reliability testing indicate a promising reli-ability of DE-CODE, particularly given the coders’ high workload when applying a range of 47 different codes. The results show that live coding leads to similar reliabilities compared with video-based coding. Live coding is less time-consuming because the time needed for coding corresponds to the duration of the debriefing. In contrast, video-based coding is typically more time-consuming; using videos allows for watching sequences several times by going back and forth, which prolongs coding time. As it is reliable, we recommend coding debriefings on-site because it reduces other potential problems of video-taping (eg, costs for cameras in the debriefing room, legal and ethical issues related to filming and data storage).

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

56 Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

With respect to validity, support for DE-CODE’s content validity is grounded in its meticulous design process involving extensive literature review, debriefing expert reviews and iterative tests. Of course, more evidence should be sought from its application to a broader range of debriefing settings. Further research is required to test DE-CODE’s predictive validity and determine in how far DE-CODE data will be able to discriminate debriefing styles and outcomes.

With respect to feasibility, we are aware that DE-CODE has an extensive number of codes. Compared with other debriefing assessment tools such as DASH and OSAD, which require rater training as well, it may seem resource-intense, daunting and effortful. DE-CODE’s intention is neither to substitute these tools nor to be feasible at the expense of its capability to assess debriefing dynamics. In fact, many of the most relevant findings on team communication required the use of extensive behaviour coding schemes,69–72 especially in healthcare73–83 and also in other high-risk settings.31 84 85 Debriefing is a team process that is by definition dynamic86; the lack of research addressing this dynamic has been criticised repeatedly.87–90 We think it is important that DE-CODE allows for assessing team debriefings in a way that enables statis-tically analysing debriefings because recent group research has shown that patterns of behaviours among group members—rather than frequencies of individual group members’ actions—are what discriminate high-performing from low-performing groups.78 91–94 In line with similar approaches,69–71 we recom-mend to use DE-CODE flexibly in accordance with the respec-tive research question. For example, if only selected debriefing behaviours are of interest, only they may be coded.

This study has limitations. DE-CODE’s reliability has so far been exclusively tested during anaesthesia SBT within a single institu-tion that may limit the generalisability of the respective reliability findings. Broader reliability tests in other debriefing settings—even outside of SBT—are necessary to provide more robust reliability results. Selected DE-CODE codes did not yet reach sufficient reliability values (eg, normalisation and repeating). More double observations are required to obtain more data to explore and, ulti-mately, improve these values. We particularly encourage observa-tions from a variety of observers to reduce observer bias. Similar to other behaviour coding schemes, the application of DE-CODE is limited to trained observers. Observer training (between 30 hours and 34 hours for the complete coding scheme with all codes) will be required prior to its use, especially for live coding. The required observer training will vary according to the desired use of DECODE: more time will be required if DE-CODE is used in full version for research purpose and less time will be required if only selected codes of DE-CODE are used for faculty development. It is necessary to consider in advance which device could be applied for coding (eg, tablet apps and software) and how feasible the respec-tive data analysis would be.

DE-CODE has practical and research implications. It may be used to obtain insights into debriefing interaction patterns, for identifying effective debriefing methods and for comparing different debriefing approaches and methods in different work and cultural contexts (eg, interprofessional vs intraprofessional teams, surgical teams vs emergency teams, single debriefer vs code briefing setting and so on). Particularly, knowledge on debriefing inter-action patterns is needed for understanding the process of how debriefer communication impacts learning. This feedback can help debriefers to develop and improve their competence in conducting debriefings. In that respect, data obtained with DE-CODE may be valuable for faculty development because it provides insights into what specifically debriefers can do and say during debriefings that helps learners to reflect on their mental models. This may

also contribute to defining best practices for debriefings during SBT and further enhance their impact on learning, performance, and patient safety. In addition, the DE-CODE-based empirical evidence about what debriefer communications are most effective to improve participants’ learning during debriefing conversations could be used to design effective debriefing tools for the clinical setting helping teams to debrief themselves.

Acknowledgements The authors would like to thank Hubert Heckel, Adrian Marty, Valentin Neuhaus, Niels Buse and Michael Hanusch for their help in collecting data, Lynn Häsler and Rebecca Hasler for their help in data coding and Alfons Scherrer and Andrea Nef for their operational support.

Contributors JCS and MK were involved in the planning, conduct and reporting of this study and developed the coding scheme. JCS and MK prepared the study proposal and created the manuscript. JCS collected data and performed the statistics. BG served as a scientific advisor, helped with data collection and revised the final manuscript. SK collected data and revised the final manuscript.

Funding The research was supported by a grant from the Swiss National Science Foundation (Grant No. 100014_152822).

Competing interests None declared.

ethics approval The study was approved by the local ethics committee (Kantonale Ethikkommission Zürich (KEK-ZH-No. 2013–0592)).

Provenance and peer review Not commissioned; externally peer reviewed.

© Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

RefeRences 1 Rudolph JW, Simon R, Rivard P, et al. Debriefing with good judgment: Combining

rigorous feedback with genuine inquiry. Anesthesiol Clin 2007;25:361–76. 2 Fanning RM, Gaba DM. The role of debriefing in simulation-based learning. Simul

Healthc 2007;2:115–25. 3 Raemer D, Anderson M, Cheng A, et al. Research regarding debriefing as part of the

learning process. Simul Healthc 2011;6:S52–S57. 4 Salas E, Klein C, King H, et al. Debriefing medical teams: 12 evidence-based best

practices and tips. Jt Comm J Qual Patient Saf 2008;34:518–27. 5 Eddy ER, Tannenbaum SI, Mathieu JE. Helping teams to help themselves: Comparing

two team-led debriefing methods. Pers Psychol 2013;66:975–1008. 6 Cheng A, Eppich W, Grant V, et al. Debriefing for technology-enhanced simulation: a

systematic review and meta-analysis. Med Educ 2014;48:657–66. 7 Cheng A, Palaganas J, Eppich W, et al. Co-debriefing for simulation-based education:

a primer for facilitators. Simul Healthc 2015;10:69–75. 8 Eppich W, Cheng A. Promoting Excellence and Reflective Learning in Simulation

(PEARLS): development and rationale for a blended approach to health care simulation debriefing. Simul Healthc 2015;10:106–15.

9 Jaye P, Thomas L, Reedy G. ’The Diamond’: a structure for simulation debrief. Clin Teach 2015;12:171–5.

10 Kolbe M, Weiss M, Grote G, et al. TeamGAINS: a tool for structured debriefings for simulation-based team trainings. BMJ Qual Saf 2013;22:541–53.

11 Rudolph JW, Raemer DB, Simon R. Establishing a safe container for learning in simulation: the role of the presimulation briefing. Simul Healthc 2014;9:339–49.

12 Kolbe M, Marty A, Seelandt J, et al. How to debrief teamwork interactions: using circular questions to explore and change team interaction patterns. Adv Simul 2016;1:29.

13 Vashdi DR, Bamberger PA, Erez M. Can surgical teams ever learn? The role of coordination, complexity, and transitivity in action team learning. Acad Manage J 2013;56:945–71.

14 Tannenbaum SI, Cerasoli CP. Do team and individual debriefs enhance performance? A meta-analysis. Hum Factors 2013;55:231–45.

15 Weiss M, Kolbe M, Grote G, et al. Why didn’t you say something? Effects of after-event reviews on voice behaviour and hierarchy beliefs in multi-professional action teams. Eur J Work Organ Psychol 2017;26:66–80.

16 Hughes AM, Gregory ME, Joseph DL, et al. Saving lives: A meta-analysis of team training in healthcare. J Appl Psychol 2016;101:1266–304.

17 Brett-Fleegler M, Rudolph J, Eppich W, et al. Debriefing assessment for simulation in healthcare: development and psychometric properties. Simul Healthc 2012;7:288–94.

18 Arora S, Ahmed M, Paige J, et al. Objective structured assessment of debriefing: bringing science to the art of debriefing in surgery. Ann Surg 2012;256:982–8.

19 Runnacles J, Thomas L, Sevdalis N, et al. Development of a tool to improve performance debriefing and learning: the paediatric Objective Structured Assessment of Debriefing (OSAD) tool. Postgrad Med J 2014;90:613–21.

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

57Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

20 Yule S, Flin R, Maran N, et al. Development and evaluation of the NOTTS behavior rating system for intraoperative surgery. In: Flin R, Mitchell L, eds. Safer Surgery Analysing Behaviour in the Operating Theatre. London: Ashgate, 2009.

21 Hull L, Russ S, Ahmed M, et al. Quality of interdisciplinary postsimulation debriefing: 360° evaluation. BMJ Simulation and Technology Enhanced Learning 2017;3:9–16.

22 Smith-Jentsch KA, Cannon-Bowers JA, Tannenbaum S, et al. Guided team self-correction: Impacts on team mental models, processes, and effectiveness. Small Group Research 2008;39:303–29.

23 Seelandt JC, Tschan F, Keller S, et al. Assessing distractors and teamwork during surgery: developing an event-based method for direct observation. BMJ Qual Saf 2014;23:918–29.

24 Kolbe M, Burtscher MJ, Manser T. Co-ACT—a framework for observing coordination behaviour in acute care teams. BMJ Qual Saf Health Care 2013:1–10.

25 Dietz AS, Pronovost PJ, Benson KN, et al. A systematic review of behavioural marker systems in healthcare: what do we know about their attributes, validity and application? BMJ Qual Saf 2014;23:1031–9.

26 In: Brauner E, Boos M, Kolbe M, eds. The Cambridge Handbook of group interaction analysis. Cambridge: Cambridge University Pressin press.

27 Weingart LR. How did they do that? The ways and means of studying group process. Res Organ Behav 1997;19:189–239.

28 Bakeman R. Behavioral observation and coding. In: Reis HT, Judd CM, eds. Handbook of research methods in social and personality psychology. New York: Cambridge University Press, 2000.

29 Bakeman R, Quera V. Sequential analysis and observational methods for the behavioral sciences. New York, NY: Cambridge University Press, 2011.

30 Kolbe M, Grote G, Waller MJ, et al. Monitoring and talking to the room: Autochthonous coordination patterns in team interaction and performance. J Appl Psychol 2014;99:1254–67.

31 Stachowski AA, Kaplan SA, Waller MJ. The benefits of flexible team interaction during crises. J Appl Psychol 2009;94:1536–43.

32 Tschan F, Seelandt JC, Keller S, et al. Impact of case-relevant and case-irrelevant communication within the surgical team on surgical-site infection. Br J Surg 2015;102:1718–25.

33 Husebø SE, Dieckmann P, Rystedt H, et al. The relationship between facilitators’ questions and the level of reflection in postsimulation debriefing. Simul Healthc 2013;8:135–42.

34 Kihlgren P, Spanager L, Dieckmann P. Investigating novice doctors’ reflections in debriefings after simulation scenarios. Med Teach 2015;37:437–43.

35 Ahmed M, Arora S, Russ S, et al. Operation debrief: a SHARP improvement in performance feedback in the operating room. Ann Surg 2013;258:958–63.

36 Papaspyros SC, Javangula KC, Adluri RK, et al. Briefing and debriefing in the cardiac operating room. Analysis of impact on theatre team attitude and patient safety. Interact Cardiovasc Thorac Surg 2010;10:43–7.

37 Cheng A, Eppich W, Grant V, et al. Debriefing for technology-enhanced simulation: a systematic review and meta-analysis. Med Educ 2014;48:657–66.

38 Cheng A, Grant V, Dieckmann P, et al. Faculty development for simulation programs: Five issues for the future of debriefing training. Simul Healthc 2015;10:217–22.

39 Cheng A, Grant V, Huffman J, et al. Coaching the debriefer: Peer coaching to improve debriefing quality in simulation programs. Simul Healthc 2017.

40 Kolbe M, Grande B, Spahn DR. Briefing and debriefing during simulation-based training and beyond: Content, structure, attitude and setting. Best Pract Res Clin Anaesthesiol 2015;29:87–96.

41 Yoder P, Symons F. Observational measurement of behavior: Springer Publishing Company. 2010.

42 Seelandt JC, Tschan F, Keller S, et al. Assessing distractors and teamwork during surgery: developing an event-based method for direct observation. BMJ Qual Saf 2014;23:918–29.

43 Kolbe M, Grote G, Waller MJ, et al. Monitoring and talking to the room: Autochthonous coordination patterns in team interaction and performance. J Appl Psychol 2014;99:1254–67.

44 Kolbe M, Weiss M, Grote G, et al. TeamGAINS: a tool for structured debriefings for simulation-based team trainings. BMJ Qual Saf 2013;22:541–53.

45 Kolbe M, Burtscher MJ, Manser T. Co-ACT--a framework for observing coordination behaviour in acute care teams. BMJ Qual Saf 2013;22:596–605.

46 Kolbe M, Burtscher MJ, Wacker J, et al. Speaking up is related to better team performance in simulated anesthesia inductions: an observational study. Anesth Analg 2012;115:1099–108.

47 Kolbe M, Künzle B, Zala-Mezö E, et al. Measuring coordination behaviour in anaesthesia teams during induction of general anaesthetics. In: Flin R, Mitchell L, eds. Safer surgery Analysing behaviour in the operating theatre. Aldershot: Ashgate, 2009:203–21.

48 Kolbe M, Strack M, Stein A, et al. Effective coordination in human group decision making: MICRO-CO. A micro-analytical taxonomy for analysing explicit coordination mechanisms in decision-making groups. In: Boos M, Kolbe M, Kappeler P, eds. Springer. Heidelberg: Coordination in human and primate groups, 2011:199–-219.

49 Butler RE. LOFT: Full-mission simulation as crew resource management training. In: Wiener EL, Kanki BG, Helmreich RL, eds. Cockpit resource mangement. San Diego, CA: Academic Press, 1993:231–59.

50 Rudolph JW, Foldy EG, Robinson T, et al. Helping without harming: the instructor’s feedback dilemma in debriefing--a case study. Simul Healthc 2013;8:304–16.

51 Dieckmann P, Molin Friis S, Lippert A, et al. The art and science of debriefing in simulation: Ideal and practice. Med Teach 2009;31:e287–e294.

52 Dismukes RK, Gaba DM, Howard SK. So many roads: facilitated debriefing in healthcare. Simul Healthc 2006;1:23–5.

53 Flanagan B. Debriefing: Theory and techniques. Riley RH, ed. Manual of simulation in healthcare. Oxford: Oxford University Press, 2008:155–70.

54 Kolbe M, Grande B, Spahn DR. Briefing and debriefing during simulation-based training and beyond: Content, structure, attitude and setting. Best Pract Res Clin Anaesthesiol 2015;29:87–96.

55 Krogh K, Bearman M, Nestel D. Expert practice of video-assisted debriefing: An australian qualitative study. Clin Simul Nurs 2015;11:180–7.

56 Villado AJ, Arthur W. The comparative effect of subjective and objective after-action reviews on team performance on a complex task. J Appl Psychol 2013;98:514–28.

57 Gibbs G. Learning by doing: A guide to teaching and learning methods. London, UK: FEU, 1988.

58 Torbert WR, Taylor SS. Action inquiry: Interweaving multiple qualities of attention for timely action. In: Reason P, Bradbury H, eds. The Sage handbook of action research. London: Sage, 2008:239–51.

59 Rudolph JW, Simon R, Rivard P, et al. Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiol Clin 2007;25:361–76.

60 Little TD. The Oxford handbook of quantitative methods volume 1: Foundations: Oxford University Press, 2013.

61 Gardner R, debriefing I. Seminars in perinatology. 2013;37:166–74. 62 Rudolph JW, Simon R, Dufresne RL, et al. There’s no such thing as "nonjudgmental"

debriefing: a theory and method for debriefing with good judgment. Simul Healthc 2006;1:49–55.

63 Bakeman R, Quera V, Gnisci A. Observer agreement for timed-event sequential data: a comparison of time-based and event-based algorithms. Behav Res Methods 2009;41:137–47.

64 INTERACT Benutzerhandbuch. Mangold International GmbH (Hrsg.) [program]. 2014. 65 Cohen J. A Coefficient of agreement for nominal scales. Educational Psychol Meas

1960;20:37–46. 66 IBM SPSS Statistics for Windows, Version 22.0 [program]: IBM Corp Armonk, NY,

2013. 67 Landis JR, Koch GG. The measurement of observer agreement for categorical data.

Biometrics 1977;33:159–74. 68 Kauffeld S, Lehmann-Willenbrock N. Meetings Matter. Small Group Research.

2012;43:58–130. 69 Lehmann-Willenbrock N, Chiu MM, Lei Z, et al. Understanding positivity within

dynamic team interactions: A statistical discourse analysis. Group & Organization Management 2016.

70 Lehmann-Willenbrock N, Allen JA, Kauffeld S. A sequential analysis of procedural meeting communication: How teams facilitate their meetings. J Appl Commun Res 2013;41:365–88.

71 Lehmann-Willenbrock N, Meyers RA, Kauffeld S, et al. Verbal interaction sequences and group mood: Exploring the role of team planning communication. Small Group Res 2011;42:639–68.

72 Kauffeld S, Meyers RA. Complaint and solution-oriented circles in work groups. Eur J Work Organ Psychol 2009;18:267–94.

73 Schmutz J, Hoffmann F, Heimberg E, et al. Effective coordination in medical emergency teams: The moderating role of task type. Eur J Work Organ Psychol 2015;24:761–76.

74 Burtscher MJ, Kolbe M, Wacker J, et al. Interactions of team mental models and monitoring behaviors predict team performance in simulated anesthesia inductions. J Exp Psychol Appl 2011;17:257–69.

75 Burtscher MJ, Manser T, Kolbe M, et al. Adaptation in anaesthesia team coordination in response to a simulated critical event and its relationship to clinical performance. Br J Anaesth 2011;106:801–6.

76 Burtscher MJ, Wacker J, Grote G, et al. Managing nonroutine events in anesthesia: the role of adaptive coordination. Hum Factors 2010;52:282–94.

77 Manser T, Harrison TK, Gaba DM, et al. Coordination patterns related to high clinical performance in a simulated anesthetic crisis. Anesth Analg 2009;108:1606–15.

78 Su L, Kaplan S, Burd R, et al. Trauma resuscitation: can team behaviours in the prearrival period predict resuscitation performance? BMJ STEL 2017;3:106–10.

79 Künzle B, Zala-Mezö E, Kolbe M, et al. Substitutes for leadership in anaesthesia teams and their impact on leadership effectiveness. Eur J Work Organ Psychol 2010;19:505–31.

80 Tschan F, Seelandt JC, Keller S, et al. Impact of case-relevant and case-irrelevant communication within the surgical team on surgical-site infection. Br J Surg 2015;102:1718–25.

81 Tschan F, Semmer NK, Gurtner A, et al. Explicit reasoning, confirmation bias, and illusory transactive memory. A simulation study of group medical decision making. Small Group Research 2009;40:271–300.

82 Bogenstätter Y, Tschan F, Semmer NK, et al. How accurate is information transmitted to medical professionals joining a medical emergency? A simulator study. Hum Factors 2009;51:115–25.

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from

58 Seelandt JC, et al. BMJ Stel 2018;4:51–58. doi:10.1136/bmjstel-2017-000233

original research

83 Tschan F, Semmer NK, Gautschi D, et al. Leading to recovery: Group performance and coordinative activities in medical emergency driven groups. Hum Perform 2006;19:277–304.

84 Lei Z, Waller MJ, Hagen J, et al. Team adaptiveness in dynamic contexts: Contextualizing the roles of interaction patterns and in-process planning. Group & Organization Management 2015.

85 Grote G, Kolbe M, Zala-Mezö E, et al. Adaptive coordination and heedfulness make better cockpit crews. Ergonomics 2010;53:211–28.

86 Marks MA, Mathieu JE, Zaccaro SJ. A temporally based framework and taxonomy of team processes. Acad Manage Rev 2001;26:356–76.

87 Kozlowski SW. Advancing research on team process dynamics Theoretical, methodological, and measurement considerations. Organ Psychol Rev 2015;5:270–99.

88 Mathieu JE, Tannenbaum SI, Donsbach JS, et al. A review and integration of team composition models moving toward a dynamic and temporal framework. J  Manag 2014;40:130–60.

89 Cronin MA, Weingart LR, Todorova G. Dynamics in groups: Are we there yet? Acad Manag Ann 2011;5:571–612.

90 Roe RA. Time in applied psychology: The study of “what happens” rather than “what is”. Eur Psychol 2008;13:37–52.

91 Kolbe M, Grote G, Waller MJ, et al. Monitoring and talking to the room: Autochthonous coordination patterns in team interaction and performance. J Appl Psychol 2014;99:1254–67.

92 Kim T, McFee E, Olguin Olguin D, et al. Sociometric badges: Using sensor technology to capture new forms of collaboration. J Organ Behav 2012;33:412–27.

93 Zijlstra FRH, Waller MJ, Phillips SI. Setting the tone: Early interaction patterns in swift-starting teams as a predictor of effectiveness. Eur J Work Organ Psychol 2012;21:749–77.

94 Lei Z, Waller MJ, Hagen J, et al. Team adaptiveness in dynamic contexts: Contextualizing the roles of interaction patterns and in-process planning. Group & Organization Management 2016;41:491–525.

on August 11, 2021 by guest. P

rotected by copyright.http://stel.bm

j.com/

BM

J ST

EL: first published as 10.1136/bm

jstel-2017-000233 on 8 Novem

ber 2017. Dow

nloaded from