Ndn pd day yarnall 2013

Louise Yarnall, Ph.D.SRI International

Notre Dame de Namur Faculty Professional Development

January 6, 2013

Background

Center for Technology in Learning

Introduction Next Generation Learning

Challenge, Gates Foundation

Domain-Specific Assessment, U.S. Department of Education

Community College Partnerships’ Instructional Impacts, National Science Foundation

Destination Scenario-Based Learning, National Science Foundation

Key Topics What are some innovative higher ed uses of

technology in and out of the classroom?

What technology supports faculty creation and use of formative and summative assessments both in and out of the classroom?

How can I use technology to teach, hone, and/or assess critical thinking skills and information literacy?

Technology at inflection point Good bandwidth, wireless access

Greater access to technology for most students

Broader range of types of learning technologies

Greater portability via mobile phones

More faculty comfortable with technology

Higher ed technology use trends 20% of all undergraduates & 22% postbaccalaureate

students took at least 1 distance learning course (NCES, 2011)

4% undergraduates and 9% postbaccalaureate students took entire program online

95% 4-year undergraduates, 93% graduate students, and 78% 2-year students have home broadband (Pew Internet, 2011)

>94% college students have cell phones; 70% have laptop computer

4-9% college students have tablet or e-books

Innovative higher ed uses of technology “Disruptive” or “Old Wine, New Bottles”?

E-Books

Open Education Resources

E-Books Lower cost

Highlighting & annotation capability

Enhanced features:

Social networking

Quizzes of student comprehension

Shared student notes on chapters

BioBook Navigate content TOC

flexibly

Required comprehension quizzes, prof can review

Interact with other students around text, suggest text revisions

Inquire iPad-compatible, Campbell Biology

Enhanced highlighting, running record of notes

Artificial intelligence system poses “deeper learning” questions

E-Book Status Quo Study of students in 5 universities

(Internet2 eTextbook, 2012)

Appeal Lower cost

Highlighting capability

Down sides Faculty not using enhanced features: sharing notes, tracking

students, question/answer, additional links

No faculty use => No student use

Some problems with text enlargement in some platforms

College bookstores resist because undercuts their business model

Open Education Resources Began a decade ago with

MIT OpenCourseWareand Open University (England)

Image:

Colleges making their most popular lecturers’ talks available online

Disruptive technology to reinvent higher ed

Open Education Resources Also, a way to share text

material with open licenses

Datebase: http://www.oercommons.org/

Materials:

http://www.oercommons.org/landing/oer-training

Open Education Resources Status Quo Fact: Mostly online text

materials

Only 9% of users are educators

42% are independent learners (MIT; NYT, 2012)

42% students at other colleges (MIT; NYT, 2012)

Costly to put a course online ($30K-$40K)

Few offer credentials

Open Education Resources Status Quo Faculty (n = 4,678)

reported they use web to obtain (McMartin et al., 2008): Graphics for instruction

Primary texts for students

Very few seek animations or simulations or teaching/learning support

Barriers are: Lack of time to vet online

materials

Open Education Resources v2. Carnegie Mellon’s Open

Learning Initiative $250K per course Focus on low-income

students, first-time college students

Virtual labs, simulations, continuous feedback

Peer-to-Peer University, University of the People Social support around

materials Some provision of

credential

Open Education Resources v2. Next step for faculty:

Social recommendation system is encouraged: “Learning Registry”

Use them, provide feedback/rating

Open Education Resources Gooru:

http://www.goorulearning.org/gooru/index.g#!/home

Sample of materials that faculty can curate in online datebase

Technology and Assessment What technology supports faculty creation and use of

formative and summative assessments both in and out of the classroom?

Adaptive assessment Identify the learning

experiences that will benefit a particular learner (U.S. Dept. of Education, Draft National Education Technology Plan, 2010)

Knowledge-check quizzes, low-stakes, after lecture/reading

“Muddiest point” checks, after lecture/reading

Short essay prompts asking students to connect concepts after learning activity

Interactive simulations that students manipulate; probes check understanding

National Training and Education Resource (NTER)

Open Source LMS

Tools to create own 3-D simulations

Seek to instrument for learner analytics

Online Assessment DesignTools Prototyped online tool to

improve assessment design and production

Simplified version of PADI system used by professional test builders

Useful for describing hard-to-test knowledge and skills

Identifying range of ways to measure

Technology and Deeper Learning How can I use technology to teach, hone, and/or

assess critical thinking skills and information literacy?

Rubrics for complex tasks and performances Technology for rubrics

Complex tasks and research reports Grade with rubrics

Rubric tool online

http://rubistar.4teachers.org/index.php?screen=NewRubric

Technology and Deeper Learning Online threaded

discussions around student group projects Instructor who “lurks” to

track/coach team decisions

Online concept maps to check students’ planning during complex projects

Design Principles: http://www.edu-design-principles.org/dp/designHome.php

Technology and Deeper Learning Mobile phone reminders

to keep students focused on time management

Remind101: https://www.remind101.com/

The non-technological side of assessing deeper learning All these technological tools depend on a sound use

model

Close: A few points from my R&D work with higher edfaculty around deeper learning

Study their practice:

How standards are defined

How variations in student prep affects coverage of standards

How learning objectives are articulated

How student learning is measured

What faculty most want students to learn

Tradeoffs faculty make in designing assessments

Problems with higher-order assessment Not consistently applied

Every teacher applies rubrics differently: Weak reliability

Global vs. fine-grained

Solution: Group calibration on student work “Knowledge progressions”

Future technology Instructor crowdsourcing

Problems with higher-order assessment Tradeoffs between what learning

faculty value and what learning their assessments emphasize

OK content validity BUT

Poor construct validity

Solution: Group calibration on learning objectives

Balance among lower-order knowledge and higher-order application

Problems with higher-order assessment

Easy

Hard

Improving the construct validity of critical thinking learning standards Research indicates there

are some commonalities of good reasoning across domains

But many important differences

You improve your capacity to meet critical thinking standards by understanding those differences

Evidence that critical thinking is elusive in college

Omnibus Test Results:DSA: R2 = 0.15, F(2, 145) = 12.54, p< .0001Test B: R2 = 0.16, F(2, 129) = 12.43, p< .0001Test C: non-significant

Pairwise Contrast (Tukey’s)B1 vs. B2, B2 vs. B3, B1 vs. B3DSA effect size range: 0.53 to 1.31DSA t statistics: 3.06 – 4.90DSA p values: <0.001 – 0.007

More evidence that critical thinking is elusive in college

Omnibus Test Results:DSA: R2 = 0.29, F(2, 144) = 29.64, p< .0001Test B: R2 = 0.13, F(2, 134) = 9.78, p< .0001Test C: non-significant

Pairwise Contrast (Tukey’s)B1 vs. B2, B2 vs. B3, B1 vs. B3DSA effect size range: 0.62 to 1.60DSA t statistics: 4.11 – 7.52DSA p values: all <0.001

Is there a difference for learning?Between using this test question…

And this one?

Cost description Amount

a. Sleeping bag she bought for camping last year $90

b. Bus ride to the store $10

c. Lost income from a part-time night shift job $80

d. Cost of the console $500

Benjamin wanted to go to a baseball game when his favorite team was playing at home. He was quite upset when he found out that all the tickets were sold out.

So, the game night, he approached scalpers who were selling tickets on the sidewalk. Benjamin asked a few scalpers how much they wanted for a ticket.

The first one wanted much more than the original price, so Benjamin asked another one who wanted even more. He kept on asking around and got different prices.

After an hour, it was getting closer to the game time and he saw that the prices were not lower. He went back only to discover that the first seller had doubled the price.

Using economic terms and reasoning, explain why the prices of tickets went up.

Which of the cost items (a ~ d) are sunk costs?

A girl decides to wait in line all night to be the first to get a new game console.

Solution: Calibrate critical thinking for your subject better

Know your tradeoffs!

Schematic

Declarative Procedural

Strategic

Shavelson et al.

What a college sophomore should know about energy flow and matter cycling…

What a college sophomore should know about supply and demand….

A way to think about assessment to avoid making bad tradeoffs

)(

)(

1 ii

ii

bDa

bDa

e

e

A way to think about assessment to avoid making bad tradeoffs

)(

)(

1 ii

ii

bDa

bDa

e

e

Documentation to help you avoid bad tradeoffs

Attribute Description Model

Focal Knowledge The primary knowledge/skill/abilities targeted bythis design pattern

Student

Additional Knowledge

Other knowledge that may be required by this design pattern

Student

PotentialObservations

Some possible things one could see students doing that would give evidence about student knowledge

Evidence

Potential Work Products

Modes, like a written product or a spoken answer, in which students might produce evidence about student knowledge

Task

CharacteristicTask Features

Aspects of assessment situations that are likely to evoke the desired evidence

Task

Variable Task Features

Aspects of assessment situations that can be varied in order to shift difficulty or focus.

Task

Critical thinking varies by domain Talk to experts who work in industry and use the key

knowledge and skills you teach

Talk to academic researchers and innovators who use the key knowledge and skills you teach

Ask them: What knowledge and what skills do you use most of all?

Think about: What do I want to make sure my students know and use from my class 10 years after they graduate?

Thank you!

[email protected]