UT-Austin Guest Lecture, ""Patterns and Outcomes of Youth Engagement in Collaborative Information Seeking with Varied Resources During Guided Discovery-Based Learning."

Patterns and Outcomes of Youth Engagement in

Collaborative Information Seeking with Varied

Resources During Guided Discovery-Based

Learning.

UT-Austin Guest LectureRebecca Reynolds, Assistant Professor

School of Communication & InformationLibrary and Information Science

Rutgers University

• Theoretical debates in the learning and information

sciences around learning, structure and agency

• Current landscape of educational technology at this

present historical moment (map technological terrain;

proliferation into K-12, without research evidence).

• Research findings from Globaloria, a game design learning

innovation that includes pilot locations in East Austin, TX

• Intersections of learning sciences, information science

theory in light of findings, and questions invited.

Agenda for Today’s Talk

Socio-technical systems / social informatics perspectives and

technology determinism (understanding reciprocal interrelationships

between humans and machines in both design and application; shaping

both the technical and the social conditions of work, such that efficiency

and humanity do not contradict each other)

Learning sciences, social constructivism, Constructionism,

design-based research (LS developed as a reaction against the

random control trial as the gold standard research methodology in

evaluation and ed policymaking, and the standardized testing paradigm

as the deterministic driver for curriculum development instead of vice

versa)

Both share roots in social-constructivist theory; both foster

more human / community-centered design of innovation

Parallels Between Socio-technical Systems Perspectives and the Goals of the “Learning Sciences”

To the extent that “constructivist” teaching pedagogies

facilitate exploration and “discovery-based” learning…

Along come Kirschner, Sweller & Clark (2006): Why Minimal Guidance During Instruction Does Not Work: An Analysis of

the Failure of Constructivist, Discovery, Problem-Based, Experiential, and

Inquiry-Based Teaching (Cited by 2759)

• Staunch objection to constructivist learning on basis of

cognitive load research

• If educators / designers have a learning objective in a

knowledge domain they are trying to meet, asking learners

to search for their own resources and make sense of the

information they find is de-motivating and frustrating;

detracts from learning, rather than contributing.

Setting Up a Straw Man Argument Against Constructivism in Education . . . . The Issue of Cognitive Load

• Objection criticized for over-simplifying “constructivism”

and minimizing role of intervention design, scaffolding and

guidance present in such interventions (e.g., Hmelo-Silver,

Duncan, Chinn, 2007, cited by 808)

• Rich research evidence for effectiveness of PBL, IPjBL &

IBL…(e.g., scholars such as Hmelo-Silver, Martin, Kapur & Kinzer,

Blumenfeld, Eccles, Kuhlthau, Eisenberg, Chu)

• We also have a history of positive results in Constructionist

contexts regarding student computational thinking,

engagement, affect, meta-cognition outcomes (e.g., Harel &

Papert, 1991; Harel, 1991; Kafai, 1995; Bruckman & Resnick, 1995; Kafai &

Resnick, 1996; Urrea, 2001, 2002; Cavallo, 2004; Kafai & Ching, 2004; Kafai,

2006; Peppler, Kafai & Chiu, 2007; Klopfer, 2008; Reynolds, 2008).

“Constructivist” learning interventions come in many shapes

and sizes. . .

Counter-arguments to Kirschner, Sweller, Clark

What About Learning in the Context of Information Seeking?

Inquiry-based learning “in the wild” tends to be less-structured, occurring more so in informal, learner-driven information search contexts without a human guide (unless one is lucky enough to have a personal librarian!).

While learning during information seeking is often the underlying goal of search (explicit or implicit), if we are to believe Kirschner, Sweller & Clark,

• Open ended, unstructured exploration and resource use without an expert guide is not optimal way to learn things

• If someone really wants to learn material, better suited to do so in guided, mentored, scaffolded, problem-based scenarios designed by experts. . . (schools/teachers)

• Informal/Formal . . . Extent of structure are factors

Any information scientist will advocate there is an appropriate role for independent research in the knowledge creation process

Key Factor in Success: Learner Expertise (Novices vs. Experts)

Educational Technologies as Information Systems. . . Mapping the Terrain

• While information search systems tend to be more generic, archival, multi-use, targeted towards experts….

• Educational technologies as information environments tend to be more specific to formal learning settings involving lessons and discrete learning objectives, and users with lesser extents of expertise/domain knowledge (e.g., youth).

What comes to mind?

Hardware (Computers/laptops/tablets/mobiles; Smartboards; robotics; other devices)

Software and web services:

MS Office and related multimedia productivity tools

Learning management systems

Games

Simulations

Mobile apps

Game design platforms (Unity; Flash; HTML5, etc.)

Educational Technologies as Information Systems. . .

Some software and web services (LMS’s, games, sims) have been designed specifically to teach a particular knowledge domain. . . The learning objective drives the design; design is emergent from the learning objective.

Others are more generic . . . Multi-purpose . . . Educator populates the content [e.g., course shells]

Still others provide “canned” curriculum . . . Publishing company products distributing existing print material e.g., text book content, in electronic format (not much of a departure in pedagogy per se; transformation of print to digital content, with multiple choice tests added on to the end).

Contexts of E-Learning Implementation: Varying… informal (e.g., after-school) vs. formal (during school time, for credit and a grade); includes line distance education, blended e-learning, fusion environments in which instructor leads in FTF classes, leveraging online environments/experiences/content. . . Role of educator varies across.

A few quick examples…

Games designed around key learning objectives (e.g., GLS games)• .

Simulations in the school subject domains. . .

• .

Learning Management System Course Shells for K-12

Generic Course Management

Generic Course Resource Sharing

Pre-Packaged, Standards-Driven Curriculum Content from Existing Publishers, Delivered in Increasingly Complex Modalities and Online Distribution Channel Models for K-12

Research Paradigms, Challenges

IES: Effects

data on research effectiveness

WhatWorks:

“Blended learning”

12 results

Clearinghouses: So useful in theory,so challenging toSUSTAIN

Research Paradigms, Challenges

Learning sciences scholarship: Smaller-N studies

• While yielding many necessary innovations, SO FAR most of these smaller N programs featuring “ideal” features don’t scale.

More generically designed LMS Solutions:

• Scaling, but relatively little research evidence base for effectiveness

• Schools are diving in, making do, based on sales claims.

The diffusion of K–12 blended learning is far outpacing the availability of useful research

So Much Variation. . .

How do we know what is effective?

• Considering Kirschner et al’s critiques Design innovations for specific knowledge domains that are refined closely to meet explicit learning objectives would be higher quality.

• And yet. . . LMS shell systems proliferate, and perhaps… a lowest common denominator of innovation.

• To what extent do less-guided systems support learning?

• Role of research . . . .

My Research Focus:

Issues of structure and agency in the context of educational technology design

and learner engagement.

Globaloria has served as a rich and complex test case in which to investigate these questions. . .

Where does Globaloria fit in the theoretical / design features landscape?

Design-based research-driven learning innovation developed by PhD researchers with many iterations of R & D. “Guided Discovery” . . .

Learning objectives: Introductory CS principles, digital literacy, 6-CLAs, through game design for MS and HS students [not in US core curriculum]

Pragmatic, semi-structured: Many teachers are not yet domain experts in CS/digital literacy, but the need for teaching children in this NOW is clear. . .[industry job market; live/livelihood skills Scholarship on CS Education, computational thinking]

Globaloria LMS has pre-populated game design tutorials, innovations in content delivery, scope & sequencing (rather than mere digitization of existing print textbook materials). Theoretically driven by Constructionism.

Guide AND autonomy-supportive: Electives; teachers w varying levels of expertise; does require self-driven learning and discovery but within a closed system that offers many rich resources in an organized sequence

INTERVENTION: Guided discovery-based game design program and curriculum offered by the World Wide Workshop. MS, HS

teachers and students gain experience and expertise in a range of agentive digital practices.

Domains of Learning and Expertise

• Constructionist digital literacy (skills needed in knowledge economy => 6-CLAs)

• Computational thinking through game design in Flash and programming in Actionscript

• Core curricular subject matter:

o When game subjects are linked to core curriculum and students deepen knowledge about topic through online research and design

• STEM career interests: Technology & Engineering; Computer Science

• Motivation, Affect, Attitudes, Life Choices, New Possibilities and Horizons

E-Learning Environment as Information System

Inquiry, Collaboration, Creation during a Game Design Course

My research investigates MS and HS student engagement in collaborative information-seeking behavior, within a pilot game design learning program involving a wiki-based LMS as a “coordinating representation” and productive social media platform.

Design affordances / constraints of the environment

Student inquiry and collaborative processes

Learning outcomes: successes, struggles / challenges

• This testbed environment and program is richly and deeply integrated into the schools in which it is being piloted (students engage daily, for credit and a grade, for a full year +)

• Findings may be generalizable to wider phenomena of shell LMS proliferation.... Which are much *less* structured, technologically.

Broad, Top-Level Research Question

Is Globaloria the best way of teaching these domains of knowledge to middle schoolers and high schoolers?

Not many existing programs yet. . . Against what can we compare?

Flexibility/Adaptability of curriculum = Some non-uniformity

Learn what we can about this implementation, share, compare when possible

Existing Effects findings

Globaloria participation increases science, social studies, and reading WESTEST standardized test scores, compared with matched case non-participants in West Virginia

Quasi-experimental research with match-case controls [Chadwick & Gore (2010) , Chadwick & Gore (2011) , Ho, Gore & Chadwick (2012) , Ho, Gore & Chadwick (2013)

Globaloria participation increases student engagement and self-efficacy in the “6 Contemporary Learning Abilities” (a framework of 6 dimensions of digital expertise specified in Reynolds & Harel Caperton (2009), Harel Caperton (2010), and tested in non-experimental pre/post design by Reynolds (2011, 2013).

Globaloria attenuates known Digital Divide effects including gender, socio-economic status, and some race categories given sample (Reynolds & Chiu,

forthcoming, JASIST)

Existing Effects findings

Conditions among factors that influence the learning, as measured quantitatively, thus far:

Intrinsic motivational disposition among students predicts successful knowledge outcomes in Globaloria (Reynolds & Chiu, 2013, ICLS)

Self-reported uses of the learning management system features in a survey (Reynolds & Baik, 2013, ASIST)

Current Study: By What Mechanisms? Globaloria

Student Learning Processes and Information Uses

DBR research agenda with multiple strands of data collection

underway (survey, observational video cases, screen capture, trace data,

content analysis of student artifacts and communications, LMS features and

natural experiments).

THIS STUDY’S RQs:(1) What tasks are central to students’ participation in this social

constructivist context?

(2) How are students engaging in inquiry and using resources to address

emergent, in situ questions and solve problems?

(3) Which resources appear to help most?

(4) In what ways might these findings contribute to learning sciences

debates on social constructivism/critiques around structure?

(5) In what ways might these findings contribute to information science

theory-building on guided inquiry, Task, CIS, learning outcomes and

sense-making?

Methods, Dataset 1: (See Slides 46-47 for details)

Google Analytics page read data for a sample of students who

participated in the Globaloria Game Design Program, during the

2012/2013 school year. NOTE: Google Analytics data available at SCHOOL level only. [Limitations]

Methods, Dataset 2:

During Spring semester 2013 (a week in March and a week in

May), we followed 2 teams per grade in grades 6-8 during site

visits at one school in AUSTIN, TX, collected observational

video data, for a total of 6 team case studies. Filmed,

qualitatively coded, categorized, analyzed, cross-tabbed the

frequencies.

Methods, Dataset 3:

Content analysis of student game quality, ranked by team,

school (inter-coder reliable analysis across 4 dimensions)

Profile

Project

Team

Google Analytics LMS Pages Reported in Findings: Curriculum (Intro, Wiki Tools, Units 1-4, Actionscript Tutorials); Social Media Pages (Profile,Project, Team)

Results, Dataset 1: Descriptive data on resource uses

[in the aggregate – all 21 schools]

Social media pages:

• Students appear to engage more so with their profile pages

than the more productivity-oriented project pages or team

pages.

• Schools varied substantially in their extent of use of these

pages.

Curriculum unit pages:

• Students appear to engage more so with earlier curriculum

topics (Intro, Wiki Tools, Unit 1, Unit 2) than later (Units 3, 4,

Actionscript Tutorials).

• A small number of schools spike in their pvs to latter curricular

topics later in the school year

Standardized PVs for Schools with Higher Avg. Game

Evaluation Scores

Standardized PVs for Schools with Lower Avg. Game

Evaluation Scores

Standardized PVs for Schools with a 0 Game Evaluation

Score (No final games to evaluate)

Results, Dataset 1: Apparent relationships between

resource uses and outcomes

• Schools with lower game design quality averages and

schools creating 0 games appear to visit profile pages more

frequently than project and team pages, whereas,

• Schools with higher game design quality averages appear to

visit the team pages to a greater extent, and also more so as

the program proceeds over time.

• Curriculum PVs: Higher performing schools also appear to

view Unit 2, Unit 3, Unit 4 and Actionscript curriculum more so

than those with lower game evaluation scores, and utilize

latter units in the sequence increasingly as the school year

progresses. Engaging in teamwork with the wiki and using its resources seem to

contribute to learning. But how are students using these resources? [lots of

hidden phenomena in school-level aggregates]

Results, Dataset 2: Variables of interest

• Task: What was student engaged in?

• Collaborative Information Seeking Modality:

Permutations of collaborative modality and information

source

• Inquiry incident outcomes: Was each incident complete

and resolved, or did it remain in limbo?

Results,

Dataset 2: Descriptive

Findings, Task, CIS

Modality, Outcome

TABLE1.CodingschemeandNofincidentsforemergentcollaborativeinformationseeking(CIS)variablesTaskDrivingInquiry CISModality:Solo,team,tchr,byresource Outcome

Graphicdesignanddrawingofdigitalimages

93

Teammateshelpingeachotherwiththeirownexpertise

109

Successfulsolutionfound

151

Advancedprogrammingfunctions

62 SoloinquiryusingwiderInternet 38 Taskstillinprogress 103

Basicprogrammingfunctions

52 Teacherhelpingstudentwiththeirownexpertise

36 Nosolutionfound,problemleftinlimbo

33

Gamesubjectdevelopment

39 Teammateshelpingeachotherwithwiderinternet

34 Notsure/researcherunabletodiscern

14

Versioncontrolandfilemgmt

15 SoloinquiryusingwikiLMS 31 Nosolutionfound,taskabandoned

8

Howtosearch 12 Teammateshelpingeachotherusingwiki 25 Problemsolutionfoundandtestedbutfailed

6

Characteranimation 10 Helpseekingfromclassmateswiththeirownexpertise

14 Problemsolutionfoundbutrejected

5

Music 10 Teacherinitiateshelpforstudent 13

Notsure 10 Teacherhelpingstudentusingwiki 11

Sharingofformalfeedback

8 Teacherhelpingstudentwithwiderinternet

4

Reviewofothergames 5 Helpseekingfromclassmatesusingwiki 2

Basiccomputerfunctions 4 Classmateshelpingeachotherwithwiderinternet

2

Results, Dataset 2: Cross-tabs, Frequency Distributions

across 2 Variables…

Task X CIS Modality:

• Appears that for easier graphic design tasks, the students

consulted each other and the internet more so, whereas for

more complex programming, students made attempts to

utilize informational wiki resources to a greater extent.

CIS Modality x Inquiry Incident Outcome

• When students relied on their peers’ expertise (most

prevalent), the results had more failed inquiry attempts

than wiki-based LMS resource uses.

Wiki can be a good source, for those who actually can/do use

it…

Nested Cross-tabs: Which Tasks and CIS Modalities

Reflect Higher Unresolved Rates? Those Highlighted N Row Labels Resolved Unresolved In Progress

186 Graphdesdraw

57% 10% 32%

2 alonewiki 0% 0% 0%

54 aloneint 59% 7% 33%

78 teamself 56% 10% 33%

6 teamwik 0% 0% 100%

38 teamint 58% 16% 26%

4 nteamself 100% 0% 0%

4 teachself 100% 0% 0%

156 Progbasic 38% 8% 46%

30 alonewiki 10% 0% 60%

69 teamself 52% 9% 39%

21 teamwik 43% 0% 43%

3 teamint 0% 0% 100%

6 nteamself 0% 50% 50%

3 nteamwik 100% 0% 0%

18 teachself 50% 17% 33%

6 teacherinitiated

0% 0% 100%

N Row Labels Resolved Unresolved In Progress

248 Progadv 26% 32% 39%

56 alonewiki 7% 21% 64%

40 teamself 60% 30% 10%

28 teamwik 14% 14% 71%

16 nteamself 0% 75% 25%

4 nteamwik 0% 0% 100%

4 nteamint 0% 100% 0%

44 teachself 36% 27% 36%

40 teachwik 30% 40% 20%

8 teachint 50% 50% 0%

8 teacherinitiated

0% 50% 50%

195 Gamesubdev 67% 10% 23%

35 aloneint 100% 0% 0%

75 teamself 67% 7% 27%

40 teamint 75% 25% 0%

15 teachself 33% 33% 33%

10 teachint 100% 0% 0%

20 teacherinitiated

0% 0% 100%

Results, Dataset 2: Cross-tabs, Frequency Distributions

across 2 Variables…

For the most complex task of Advanced Programming. . . • Higher proportions of unresolved incidents than for other tasks, across

several CIS modalities

• When you separate out information-seeking by Task, CIS Modality and

Outcomes, we see that these 3 variables appear related, such that in

guided discovery based settings like this: • For the more complex task of advanced programming, the more likely

students are to seek online information resources in the LMS than for

other tasks

• For the more complex task of advanced programming, the more likely the

inquiry incidents are to be unresolved.

• Role of educators’ expertise needs enhancement. . .

• And/or role of students’ information literacy (thus, more IL instruction) . . .

Implications

Important to note:

• Tasks and game quality variables are local to this Constructionist learning

intervention, but may be applicable to others teaching game design / digital

literacy

• CIS modality categories that emerged (collaboration type, melded with

information resource) may be applicable to many other informational and

educational settings . . .

• Inquiry outcome categories that emerged (successful / failed inquiry

attempts) also may be applicable to many other informational and

educational settings . . .

These are in-progress results!

Currently reviewing literature on Task, CIS and “sense-making” and

considering theoretical linkages.

Broad Summary of Findings

Is this an “effective” intervention?

• Digital literacy and digital divide positive effects (Reynolds & Chiu, forthcoming)

• School standardized tests positive effects (Edvantia research reports)

• Globaloria is autonomy-supportive and appears more conducive for

intrinsically motivated students (Reynolds & Chiu, 2012)

Information resource use research on Globaloria indicates:

To the extent that more advanced game programming is a program learning

objective, then it appears the program needs more structure for this. . .

Teachers, System, Information Literacy

Next Steps

Need to deploy and test improvements in the following areas:

• Teacher expertise (professional development)

• LMS system design refinements in the more advanced game

programming topics o More research needed on what programming functions pose greatest

challenges, how to design better

• Offer more information literacy instruction o Student question formulation, resource review and search, synthesis,

problem-solving, application

Future Questions invited

• What “levels of analysis” exert greater effects on learning

outcomes in complex, social constructivist “social learning

systems”?

• Design of information system/curricular features, scope,

sequencing?

• Teacher qualities / pedagogical practices;

• Student qualities / collaborative information seeking

practices (Individual; Team)

• Multi-level analysis modeling with colleague, Dr. Ming Ming Chiu

[email protected]

Rutgers University website

http://comminfo.rutgers.edu/directory/rbreynol/index.html

Thanks to IMLS!

Thanks to my partners!

Globaloria.org

Worldwideworkshop.org

Thank you!

Extra Slides from Slidebank….

Methods, Dataset 1, additional information

Google Analytics page read data for a sample of students who

participated in the Globaloria Game Design Program, during the

2012/2013 school year.

• 708 students in grades 6-12

• 31 rural West Virginia public schools participated for a full

school year, in Globaloria.

• Omitting single semester and mixed group schools, and thus

the findings below represent the 21 full year schools only, for

a total of 455 students

• These students created a total of 340 individual hidden object

games, and 93 team games.

• NOTE: Google Analytics data available at SCHOOL level

only.

Methods, Dataset 1 (additional information)

Part 1. Descriptive Page Views

Aggregate page view findings by school, observe variation

Part 2. Relationship of Resource Uses (PVs) to Learning

Outcomes

Comparative analysis by school based on descriptive data on

resource uses across time

• Sort process data by outcomes:

• High, Mid, Low performing groups

• Visually observe patterns

• Build rationale for new hypotheses based on apparent

observable differences

Prevalence of E-Learning in the U.S. 1Evergreen Education Group (2013); 2NCES (2011)

25 states have state virtual schools operating in 2013-2014.1

29 states and Washington, DC have statewide full-time online schools operating in 2013-14 1

An estimated 1,816,400 enrollments seen in distance-education courses in K-12 school districts in 2009-2010, almost all of which were online courses. 74% of these enrollments were in high schools. Online courses with the highest level of enrollment fall under the categories of credit recovery (62%), dual enrollment (47%), and advanced placement (29%).2

This enrollment estimate does not include students attending most full-time online schools — approximately 200,000 full-time students in 2009-2010. As of 2012-2013, the number of students has grown to 310,000. 1

Single and multi-district blended and online programs are the largest and fastest-growing segment of online and blended learning. 1

Top reasons school districts make online learning opportunities available to their students are to offer courses not otherwise available, and provide opportunities for students to recover course credits. Credit recovery is especially important in urban environments where 81% of schools indicate this is an issue.2

Blended Learning Models

1. Traditional Schools, But With Online Options

• Online Lab Model: Instruction delivered by online teachers through a digital learning platform but within a brick-and-mortar lab environment.

• Self-Blend Model: Students take one or multiple online courses outside the school facility to supplement schooling.

2. Blended Schools

• Rotation Model: Fixed schedule rotating students between online learning and traditional classroom learning. The face-to-face teacher accountable for both the online and in-classroom work.

• Flex Model: Most of the instruction is delivered by an online platform with face-to-face teachers available for on-site support. Teachers provide tutoring sessions and small group sessions.

Blended Learning Models

3. More Virtual Than Traditional

• Online Driver Model: Students receive all primary instruction online (through an online platform and from an online teacher) with occasional face-to-face check-ins.

• On/Off-Site Rotations: Students receive most primary instruction online, but also come on-site on a scheduled part-time basis.

4. Lab Programs

• Full-time programs at a site attached to a school, including on-site teachers or paraprofessionals, online instructors, subject-area

5. Supplemental Programs:

• Serve students in a blended setting for one to several courses, while they take the rest of their curriculum in a traditional face-to-face environment.

1

2

3

4

Contemporary Learning Ability

(CLA)

Practices Representing Each CLA & How They Are Articulated/Integrated in Globaloria

Invention, creation and completion

of a digital project stemming from

an original idea

Brainstorming and developing game and simulation ideas and storylines using Web2.0 tools

Writing an original game narrative and a proposal to explain it

Generating creative ideas for designs to express the subject of the game and the user experience

Planning/programming/completing a game demo that illustrates the original game design and

functionality Developing knowledge of the game's domain or topic through game invention/creation/research

Project planning, project

management, teamwork (e.g., role-taking,

task delegation),

problem-solving

Coordinating and managing the design/creation/programming of game elements

Managing the project’s execution by creating/organizing a wiki and by sharing project assets and progress updates

Managing team work by defining and assigning team roles/coordinating tasks/executing roles

Project troubleshooting for self and others

Gaining leadership experience through the project management of all game production elements

Publishing and distribution of

self-created digital media artifacts

to an audience and/or community

of peers

Creating a wiki profile page and project pages

Integrating and publishing text/video/photos/audio/programming code/animations/digital designs

on wiki pages

Posting completed assignments/game design iteration and assets/notes and reflections about

projects to wiki Developing a blog

Giving and getting feedback

about project through social interaction,

participation, exchange

Collaborating by using Web2.0 tools such as posting to wikis/blogs/open source help

forums/instant messaging Exchanging/sharing feedback and resources by posting information/links/source code questions/

answers

Reading and commenting on others’ blogs and wiki pages

Presenting final digital projects for others both virtually in game galleries and in person in live

game demonstrations

5

6 Figure1.Globalorialearningobjectives:Promotedevelopmentofsixcontemporarylearningabilities(CLAs)

Inquiry, information-seeking,

agentive

use of resources (human and text/digital content), to support

the artifact’s topic/message,

and design/execution

Searching the Web for answers and help on specific issues related to programming games

Searching and finding resources on MyGLife.org network, website, and wiki

Searching the Web for new Flash design, animation and programming resources Searching for information in support of the game’s educational subject and storyline

Surfing, experimentation and play with existing Web

applications and tools

Surfing to MyGLife.org starter kit site and other game sites and playing games online Keeping track of and bookmarking surfing results that are relevant to projects

Browsing Web2.0 content sites such as YouTube, Flickr, blogs, Google tools

Constructionist, knowledge-building conditions supporting development

of 6-CLAs (Reynolds & Harel Caperton 2009; Reynolds & Hmelo-Silver,

2013)

Learning Management System as Information System

Learning Management System as Information System