Two make a network: using graphs to assess the quality of collaboration of dyads Irene-Angelica Chounta 1 , Tobias Hecking 1 , H. Ulrich Hoppe 1 , Nikolaos Avouris 2 1 Collide, University of Duisburg-Essen, Germany 2 HCI Group, University of Patras, Greece {chounta, giemza, hoppe}@collide.info [email protected]
21
Embed
Two make a network: using network graphs to assess the quality of collaboration of dyads
CRIWG2014 - In this paper we explore the application of network analysis techniques in order to analyze synchronous collaborative activities of dyads. The collaborative activi-ties are represented and visualized as networks. We argue that the characteristics and properties of the networks reflect the quality of collaboration and therefore can support the analysis of collaborative activities in an automated way. To sup-port this argument we studied the collaborative practice of 228 dyads based on network graphs. The properties of each graph were evaluated in comparison to ratings of collaboration quality as assessed by human experts. The activities were also examined with respect to the solution quality. The paper presents the method and the findings of the study.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Two make a network:
using graphs to assess the quality of collaboration of dyads
Learning Analytics: „the measurement, collection, analysis and reporting of data about learners and their contexts” (LAK 2011)
Collaborative Activities: Analysis and Evaluation
Methods &Tools:
� Logfile Analysisactivity metrics retrieved from logfiles
� Interaction Analysismetrics of interaction among users while working together
� Social Network Analysis and Graph Theoryuser activity and interaction is represented through graph representations
Objectives of the study
� To represent the collaborative activities of dyads using networks
� To use network metrics in order to assess qualitative aspects of collaboration
Research hypothesis:
The properties and characteristics of networks that represent collaborative activities, reflect the quality of collaboration
Method of the study
� Network generation from logfiles of previously evaluated collaborative activities
� Visual inspection of networks representing good vs. bad collaboration quality
� Study of the relation of human ratings and network properties in a systematic way (correlation analysis)
Collaborative Activities of Dyads
� Team: two students working over a shared-spaces (Synergo)
� Task: the construction of an algorithmic flowchart
� Time: synchronous communication for 90 minutes
• Dataset of 228 Collaborative Sessions
• Pre-evaluated with qualitative and quantitative methods with respect to collaboration quality
(a) Two human experts evaluated the dataset [1]
� Quality of Collaboration (CQA) was evaluated on a 5-point Likert scale [-2, +2]
� Well-established results for inter-rater reliability and consistency
(b) The dataset was used to validate an automatic rater (time-series classification) [2]
� Meaningful Activity takes place in time frames of
15 – 30 seconds
� Time series depict the quality of collaboration
Related Work
[1] Kahrimanis, G., Meier, A., Chounta, I.-A., Voyiatzaki, E., Spada, H., Rummel, N. et al.: Assessing collaboration quality in synchronous CSCL problem-solving activities: Adaptation and empirical evaluation of a rating scheme (2009)[2] Chounta, I.-A.,Avouris, N.: Time Series Analysis of Collaborative Activities. CRIWG2012 (2012)
Quality of Collaboration
Rating scheme[3] for the assessment of the quality of collaboration (CQA):
General aspects of collaboration
Collaborative Dimensions
CommunicationCollaboration flow (CF)
Sustaining mutual understanding (SMU)
Joint information processingKnowledge exchange (KE)
Argumentation (Ar)
CoordinationStructuring the problem solving process
CQA = average(CF, SMU, KE, Ar, SPSP, CO)[3] Kahrimanis, G., Meier, A., Chounta, I.-A., Voyiatzaki, E., Spada, H., Rummel, N. et al.: Assessing collaboration quality in synchronous CSCL problem-solving activities: Adaptation and empirical evaluation of a rating scheme (2009)
Network generation from logfiles
� Network maps generated from logfiles of collaborative activities:
- Nodes represent user actions
- Edges represent dependencies among actions
• To track dependencies:
(a) The time distance between actions ranges from
10 to 30 seconds
(b) Relevance on temporal and spatial proximity
(c) The identity of the actor should differ
Logfile example:
Resulting Network:
Network generation from logfiles
Visual inspection of networks
� The SiSOB Workbench[4] was used for the visualization and analysis of the network graphs
Good Collaboration Quality Bad Collaboration Quality
[4] Göhnert, T., Harrer, A., Hecking, T., Hoppe, H. U.: A workbench to construct and re-use network analysis workflows: concept, implementation, and example case. (2013)
Network metrics
• The number of nodes (N) and the number of edges (E)
• The diameter of the network (d)
• The average path length (APL)
• The density of the network (D) = (��
�������
• The Power Law Fit (PLF)
k
P(k)
Hubs
Results
� Comparison of Network metrics and Ratings of human experts for Quality of Collaboration (CQA)
� Most metrics correlate (p<.05) to CQA and to individual collaborative dimensions
#Nodes #Edges Density Diameter (PLF) (APL)
Quality of Collaboration (CQA)
0.446 0.18 -0.394 0.294 -0.233 0.243
Results
� #Nodes correlates highly with collaboration quality
� Sessions with intense activity point to bettercollaboration
� ...In particular to successful argumentation and knowledge exchange � efficient communication
� Good collaboration produces to larger event networks
CF SMU KE Ar SPSP CO CQA
#Nodes (N) 0.358 0.351 0.4 0.416 0.339 0.41 0.446
#Edges (E) 0.179 0.19 0.169 0.156 0.195 0.18
Results
� Diameter and Average path length correlate positively with collaboration quality