Connecting Levels and Methods of Analysis in Networked Learning Communities

Workshop on

Connecting Levels and Methods of Analysis in

Networked CommunitiesLearning Analytics and Knowledge Conference 2012

Vancouver

Dan Suthers, Ulrich HoppeMaarten de Laat, Simon Buckingham-Shum

Version edited for SlideShare

Motivations Multiple levels of learning agency in social settings

Individual - social setting as stimulus Small Group - “maintaining a joint conception of a

problem”; “group cognition” Community - “knowledge building” Networked - “networked individualism”

Learners (could) participate in multiple simultaneous forms of learning in contextual and constitutive relationships

Analytic Challenges Ontology mismatches between data record and desired level

of analysis Summary representations needed to see emergent patterns

but fail to capture how learning is actually accomplished Distributed nature of (interaction in) socio-technical networks And many others …

Questions: Networked LearningHow does learning take place through the interplay

between individual and collective agency? How does local (individual and small group) activity

aggregate to Create resources that are then available network-wide for

others' individual and small group learning? Drive advances in community knowledge building?

(How) does the connectivity afforded by ICTs facilitate learners’ participation at multiple levels?

What theoretical perspectives are relevant to bridging levels of analysis? Do the different levels of analysis need different theories,

and how can they be articulated? Are there theories and methods that bridge the levels of

analysis?

Questions: MethodHow can a plurality of methods help us make sense of

levels of learning and their interplay? How can aggregate levels of analysis inform where

to “dive in” for local analysis, for example to make sense of results at the aggregate level? find local sources of innovation?

How can local levels of analysis generate hypotheses to be tested at the network level? identify events to be traced to other times and places?

What practical techniques such as different types of triangulation or visualization can help to connect different levels and approaches of analysis?

What are the prospects of technical integration of analysis tools through open architecture, adequate representations and user interface metaphors?

Questions: Multilevel Analysis in Context How can analyses be made available to participants

of all types to support their awareness and reflection on learning processes?

Dealing with the complexity of living practices in which people learn …

and the potential added value of learning analytics to

raise awareness, help reflect on (social) learning behavior and connect learners in networks and communities where

value is being created

How should progress on these issues and appropriate applications be promoted in the context of the emerging Learning Analytics community?

Condensed History & Future of WorkshopsOrigins in workshops by Koschmann, Suthers, StahlProductive Multivocality Series

ICLS 2008: "A Common Framework for CSCL Interaction Analysis”

CSCL 2009: "Common Objects for Productive Multivocality in Analysis”

ARV 2009: "Pinpointing Pivotal Moments in Collaboration”ICLS 2010: “Productive Multivocality in the Analysis of

Collaborative Learning”ARV 2011: ibid

Levels of Analysis Series CSCL 2011: “Connecting Levels of Learning in Networked

Communities”LAK 2012: “Connecting Levels and Methods of Analysis in

Networked Learning Communities”ICLS 2012 (July): “Analyzing Collaborative Learning at

Multiple Levels”

Today’s Workshop Activities Framing presentations Extended examples of concepts and

tools applied to two data corpora Briefer presentations on other

approaches Small group and full group discussion

Schedule09:00-10:30: Introductory Session

10m: Introduction to the workshop - Dan Suthers20m: Introductions to each other30m: Conceptual Framing Presentations - Organizers20m: Visualizing Informal Networked Learning Activities (Issues and Tool Proposals) -

Bieke Schreurs, Chris Teplovs & Maarten de Laat 10:30-11:00: Coffee/Tea 11:10-12:00: Part II: Issues & Tools

45m: Conceptual and Computational Tools (with examples from Wikiversity) - Ulrich Hoppe

05m: Form small groups to discuss topics after the break. (Suggested topics: Your examples of analytic questions; Key theoretical issues; Needs for computational support)

12:00-13:00: Lunch (on site)13:00-14:30: Part III: Thinking about Levels

20m: Small Group Discussion: What are the key theoretical issues and needs for computational support?

25m: Small Groups report 15m: Multimodality in Levels of Analysis - Sharon Oviatt 20m: Multi-level Microanalysis - Alyssa Wise15m: Full group open discussion

14:30-15:00 Coffee/Tea 15:00-17:00 Part IV: Outlook

45m: An Analytic Hierarchy (with examples from Tapped In) - Dan Suthers 45m: Discussion

IntroductionsOrganizers Dan Suthers, University of Hawai‘i Maarten de Laat, Open Universiteit Nederland Simon Buckingham-Shum, Open University UK Ulrich Hoppe, University of Duisburg-EssenOther Presenters Alyssa Wise, Simon Fraser University Bieke Schreurs, Open Universiteit Nederland Chris Teplovs, University of Windsor Sharon Oviatt, Incaa Designs

… and you Liaqat Ali Bill Anderson Michael Atkisson George Bradford Al Byers Darren Cambridge Cathleen Galas Rabbi Zidnii Ilman Murray Logan Piotr Mitros

Shanta Rohse Toshiyuki Takeda Ravi Vatrapu Sue Whale John Whitmer Phil Winne Robert Yerex Gary Williams Vladimir Stoyak Caitlin Martin

Framing Comments

Dan Suthers(with comments

from others)

Learning in Socio-Technical Networks

AgencyWho or what is the agent

that learns? Individual Small groups Networks (communities,

cultures, societies)

Epistemologies What is the process of

learning? Acquisition Intersubjective

meaning-making Participatory

The correspondence is not strict, and analysis can be applied at local or network levels

How do social settings foster learning?

Based on Suthers (ijCSCL 2006)

Levels of Agency and Epistemologies Individual Epistemologies

Learning as acquisition of information, knowledge or skills

Local: contribution theory, given/new contract, explanation, conceptual change, role practice, etc.

Network: weak ties, diffusion theories (contagion theory, diffusion of innovations)

Intersubjective epistemologiesLearning as intersubjective meaning-making Local: argumentation, co-construction, group cognition Network: Knowledge building, communities of scientists

Participatory epistemologies (most bridge both levels)Learning as changes in social participation and

identity Local: apprenticeship, mentoring ... Network: apprenticeship as LPP, CoP

Let’s not get stuck at one level!Claim: individuals participate in the foregoing

forms of learning simultaneously This leads to a fundamental question:

How does learning take place through the interplay between individual and collective agency in socio-technical networks?

Requires coordinated multi-level analysis Requires coordinated multi-level theorizing

(I address analysis and theory after some examples)

Examples of Connecting Levels Questions I've got a huge amount of data. Where to start?

Where are productive interactions that I should look at? Who is playing important roles?

I found some interesting patterns of participation and knowledge building in a network: how did these come about? What are participants actually doing? Can I account for network-level phenomena in terms of

how persons follow STN 'affordances'? I analyzed an interesting and productive session.

Does it have any significance beyond the session? Are ideas taken up by others who express them

elsewhere? Do personal encounters led to new participation

elsewhere? Can I account for individual learning in terms of

their participation in social phenomena?

Analytic Challenges Logs may record activity in the wrong ontology for

analysis (e.g., media-level events rather than interaction or ties)

Activity is distributed across time, sites, media Traces of activity may be fragmented across multiple logs,

breaking up participants’ singular experience Thus, distributed activity may be analytically cloaked

People draw on the resources of the setting in diverse ways Referencing and modifying available media Echoing notational and semantic elements Sensitive to temporal and spatial setting

The phenomenological situation is extended The “situation” for participants need not be limited to our

selection of a transcript and may be non-local in time and space

Comments on TheoriesTheories that make contributions, but are limited in

scope: Socio-constructivism

All knowledge is constructed by the individual (possibly in social interaction)

Psychological; limited at social or more aggregate levels of analysis

Distributed Cognition Transformation of representations in an STN effects a

“cognitive” computation “Cognition” at multiple network levels? Better for functional explanations than generativity

Communities of Practice Engagement, alignment, imagination; local/global duality Limited to specific kinds of communities (shared domain,

interacting to sustain a shared practice)

More Comments on Theories

QuickTime™ and a decompressor

are needed to see this picture.

Promising: CHAT (Activity Theory)

Vygotsky, Leontiv, ... Cole & Engestrom (1993)

Intended to encompass all human activity Explicit consideration of how artifacts bridge levels

Actor-Network Theory Latour (2005): Localizing the Global Redistributing the Local

(Action is Overtaken, Connecting Sites)

Artifacts (“actants”) again key for constructing the network Critiqued for devaluing human agency Methodological strategy that prioritizes the data

Incoherence due to Incommensurability?Methodological determinism:

“Methods rest on philosophical presuppositions. These remain embedded in them, even if they are not taught or discussed or attended to explicitly.” (Yanow & Schwartz-Shea, 2006)

In what ways (if any) do we risk incoherence by mixing methods from different theoretical traditions? There are potentially intractable differences

between objectives and between explanatory theories

Are these forced on us by methods? How? What “carries” the presupposition? Do researchers have agency in overcoming this?

Example: Social Network AnalysisMarin & Wellman (2010): claim that social network

analysis is not just a method; it is a perspective “Causation / explanatory power is located in

relations (social structure), not attributes” Intrinsic to the analysis method?

One could use network analysis under other theoretical commitments

Example: homophily of attributes due to selection, not diffusion

We have some agency in applying the method But the network representation implies

commitments Actors or social entities can be discretely identified Binary relationships are relevant and primary

Example: The Percentage or Ratio Schegloff (1993): “Reflections on Quantification in

the Study of Conversation” comments on a study of sociability that used “laughter per

minute” and “backchannels per minute” as measures

Numerator Laughs

Denominator Minute Denominator: Laughter is responsive, relevant at some points

and not others Numerator and denominator are mutually constitutive

Ratio commits one to quantify events relative to some set of potential events, but this can be refined

One analytic tradition can inform (fill in the blind spots) of another

Opportunities taken

Opportunities available

Unpacking Methodological Determinism Beyond “guilt by association”

Often methods are taught within a setting with a viewpoint But a user of a method does not necessarily adopt the

entire tradition that the method comes from Methods consist of notations and practices

Notations (inscriptions) and available operations carry largely inescapable biases Suthers (2008) Knowledge Cartography

The associated practices by which inscriptions become representations (Medina et al, 2009) also carry biases

Biases of practices can be confronted and modified in reflective practice

Reflective use of methods requires stepping out of the viewpoint provided by the method A goal of the Productive Multivocality project

Suthers et (16) al., (CSCL 2011) and book in progress

Strategies for Productive Multivocality Shared data repositories, standards, metadata Dialogue about same data

Agree on what data is worth considering? Have shared analytic objective (e.g., pivotal

moments) Interpretable by each tradition (objective as boundary

object) Probably vague at first (projective stimulus)

Pair up diverse methods Different individuals representing those methods!

Push methods outside of their comfort zone Potential problems in appropriateness of data

Align data and analytic representations Iteration is required.

Eliminate inconsequential differences in the first pass Repeat to focus on more essential differences and

convergences (conceptual, epistemological)

Simon Says …We know that learning is a multifaceted, extraordinarily complex phenomenon,

certainly happening in the individual's mind, but shaped by interaction with the others and the environment. So it makes sense to build as rich a picture as possible of what is going on at many levels in a learning ecosystem, prior to, during and after critical incidents. 

The intriguing promise of learning analytics goes beyond detecting patterns more efficiently than has been possible, as more digital data becomes available (= simply doing what we've always done, but faster/better).

More radically, could the learning sciences be entering a new era (following genomics and other big data fields), in which there is more data than we know what to do with, and far greater complexity in the relationships that may be embedded in it?

This requires the deployment of exploratory techniques seeking robust patterns for which there may be no motivating hypothesis, or explanatory theory. New patterns between different levels in the ecosystem might then become the catalysts for revising theory to find plausible explanations.

This is not Chris Anderson's 'death of theory', but analytics as scientifically valid probes that 'report back' to us as analysts, from the deep space that big data opens up.

Discussion Topics Examples of ways in which you have

connected or would like to connect levels of analysis?

What analytic methods or tools are needed at different levels (e.g., content analysis, sequential/process analysis, SNA) and how do we coordinate them?

What approaches would you take to theoretical articulation or integration?

Dan [email protected]

lilt.ics.hawaii.edu