Tom Lewis, Director, Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin, Director Program for Educational Transformation.

Tom Lewis, Director, Ed-Tech Development GroupEducational Partnerships & Learning Technologies

Scott Macklin, DirectorProgram for Educational Transformation through Technology

Copyright Tom Lewis and Scott Macklin, 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

The Catalyst E-Portfolio: Using Assessment to Shape

Enterprise-Wide, Learner-Centered Technologies

How do you scale beyond the early adopters to

reach wary adopters and encourage innovation?

PETTT Educational Technology Surveys

Faculty, Spring 2001 2000 respondents

Students, Fall 2002 1000 respondents

•20% of college students began using a computer between 5 and 8 years old.

•By the time they are 16-18, all had used a computer.

•80% of UW undergraduates use a computer daily.

Technology Adoption: Student Driven

•72% of students think all courses should have a Web site

•Students particularly want online: course syllabi (81%)grading criteria (79%)problem sets or exercises (78.6%)links to electronic course reserves (72%)

•7% of students stated that educational technology is used consistently through UW

Students…

“I’m graduating now and I’m pretty upset with how little I know about technology, I really wish that I was more confident and knew more after four years of my education at a university. A lot of it is probably independent work that you can do to learn it but we’re busy enough with jobs and school that I just really haven’t felt like I had the time to actually independently learn it…it would have been really helpful to have not been expected to know it but to have actually been taught it.”

-Undergraduate Student

Which of the following concern faculty when using information technology to do university work?

0

10

20

30

40

50

60

Percent

Lack o

f Tim

e

Lack o

f Skil

ls

Lack o

f Ince

ntive

Student

learn

ing

Hardw

are

Scaling Innovation with the 4-Tiered Catalyst Initiative

Lesson Learned:

Scale support through partnerships, making disparate resources and innovative practices visible and available to all.

http://catalyst.washington.edu

On an average day in Autumn 2002, instructors viewed 5000 pages ofCatalyst content every day.

8557 instructors have created 18,592 implementations of Catalyst Web Tools.

Catalyst Web Tools were used for 66,000 learning activities during Autumn 2002.

The Catalyst Initiative

Catalyst Web Site

Participatory Design & Catalyst Web Tools

       Lesson Learned: Create tools that meet pedagogical needs expressed by faculty and constantly engage with faculty to make sure the tools remain current and easy to use.

Collaborative Development

Assessing what the educator wants is central:• Get feedback at every step of design.• Process is more important than

product.• Direct communication between

developers and educator.• Make feedback and interaction easy.• Value educator’s ideas and time.

SimpleSite

• Create Course Web pages without HTML or an editor

• Publish Web pages without FTP

• Keeps usage statistics for you

VirtualCase

• For case studies and problem-based learning activities

• Contains a message board and collaborative file sharing

• Allows for group decision making

Partnerships as a support strategyto bring easy to use, flexible &campus-wide support.

       Lesson Learned: Make sure that all the units that should be shaping technology innovation are shaping innovation.

Portfolio NEW!

Resources

• Catalyst group (under Vice-Provost of Educational Partnerships & Learning Technologies) supplied main funding

• Student Technology Fee committee gave financial support.

• Program for Educational Transformation through Technology provided financial and assessment support.

• Computing & Communications furnished server space and maintenance

• Office of Undergraduate Education offered the Freshman Interest Group program as a test-bed

Project Assumptions

• Tool cannot be built without input from students, instructors, and academic and career advisers

• Tool should be flexible—students should be able to use it for classes, extra-curricular activities, job searches, or personal reflection

• Tool should be useful for students in all colleges and schools

How did we build it?• Asked everyone on campus who has ever

uttered, or is likely to utter the word “Portfolio”: 200 +

• Looked at other universities’ tools and commercial products

• Read scholarly articles on the uses of portfolios in education

• Conducted focus groups and usability tests with UW instructors and students

Basic Decisions:

• Students should be able to collect anything they want to and have exclusive access to those collections

• Students should be able to publish many different portfolios of their work on the Web

• Instructors and advisors should be able to easily lead students through the process of building certain types of portfolios, but students still own these portfolios

What is a portfolio tool?

• There is a distinction between collecting artifacts, reflecting upon them, and thoughtfully presenting them in a portfolio

• Students will be collecting artifacts throughout their academic careers and should have exclusive access to their collections

• Using portfolios for reflection is a process that is important in all phases of an academic career, not just at the end of senior year

What is a portfolio tool (cont.)?

• Instructors, advisers, and peers should collaborate with students to help them present artifacts in a reflective way

• Students should be able to present a variety of portfolios for different objectives: completing coursework or demonstrating academic, personal, or career development

Lesson learned: Need research and assessment to inform the development of learning technologies and teaching strategies.

Instructors need help in effectively integrating technologies into teaching.

1. Perform studies in real settings to investigate what is learned, who learns, and how it happens,

2. which in turn inform the design of appropriate tools.

3. These new tools are integrated into educational settings,

4. in which new studies are conducted to understand how these new tools have affected learning and learners.

The process

The FIG Program and Portfolios:

• Pilot project in Autumn 2001 with three Freshman Interest Group (FIG) sections (~60 students).

• Development of the Catalyst Portfolio Tool in 2001-2002.

• Introduction of the tool into the FIG program in Autumn 2002, with a study by PETTT researchers.

Our Goals for Autumn 2002:

• Introduce the concept of portfolios to 3,200 freshman students.

• Within the goals of the existing University Community class, encourage reflective learning.

• Learn lessons that can guide future portfolio projects and inform future revisions of the tool.

What was our experiment?

Collect data on the use of the tool by FIG leaders and students through:• Surveys• Focus groups• Classroom observation• Examination of student portfolios

Study design: • 162 Freshman Interest Groups• 162 FIG leaders, each with a class of 10-

25 students• Special group: 20 FIG leaders, paid to

give regular, individualized feedback to students

Study questions:

1) What method(s) of instruction help students feel most competent using Portfolio?

2) How can Portfolio be best integrated into the GENST 199 curriculum?

3) What instructional supports (in-class discussion, feedback, prompts within tool) might best support reflective thinking?

Data collection:

1) First day classroom observations2) Student surveys3) FIG leader surveys4) Focus groups with students and FIG

leaders5) Classroom observations & student

interviews6) Review of student portfolios

Three themes from preliminary analysis:

• Students found Portfolio easy and convenient to use. However, they largely perceived answering the questions as busywork.

• FIG leaders' attitude toward the tool and the work affected students' response to the tool and to class.

• Feedback was important--Students wanted to know that someone was reading what they wrote.

These are issues to consider as we head towards a redesign and make recommendation for how to use Portfolio. NOT reasons to abandon the project.

Quotes…

“I feel it’s really fun to me, I mean maybe it’s weird but I think it’s almost like a penpal letter kind of to our FIG leader and we can answer the questions and sometimes they’re really in-depth and we actually have to think about what we think about life and on these certain subjects, we have to think about what our beliefs are, and I think that’s helped me a lot to really figure out what I believe and why I believe it, and apply that to the discussion that we’ve had in class and stuff.”

-FIG Student

“It’s nice when they return it and they add comments so you know that they’re actually looking at it.” (student)

-FIG Student “If the FIG leaders did give feedback… it

does kind of make it more interesting and you do feel like your FIG leader is paying a little bit of attention to you. It might even create discussions that you could have in your FIG class”.

-FIG Student

“[Portfolio and the feedback function] helps you know more about your students, share and get ideas”.

-FIG leader

“Because I comment so much, they [students] see me as a friend and come talk to me. I learned a lot from them.”

-FIG leader

Building Catalyst through Co-Branding

       Lesson Learned: Renew and refresh anytime-anywhere technology support resources by co- branding and co-developing them with campus teaching practitioners.

Students learn through portfolios

when they…• assemble a collection of relevant

artifacts • select items from that collection for a

specific purpose• reflect on the selected items and/or

process of selection • design the presentation of their portfolio

for a particular audience/purpose• receive feedback on the quality of their

selection, reflection, and/or design

Considerations for teaching with portfolios…

• specific purpose/audience • assignments to produce relevant artifacts• guidelines for item selection• guidelines for and examples of “good”

reflection• guidelines for and examples of “good”

design• time for feedback on quality of selection,

reflection, and/or design

More considerations for teaching…

Evaluation • individual items? • whole? • reflection? • establish clear criteria & SHARE with

students ahead of time. • will the evaluation count toward the grade? • if students don’t have opportunity to learn

from feedback, a grade isn’t a measure of what they learned—only innate ability.

http://catalyst.washington.edu

Tom Lewis <<[email protected]>>Scott Macklin <<[email protected]>>

http://depts.washington.edu/pettt