Assessment for LI sessions

Information Literacy AssessmentChallenges & Opportunities for One-shot Library Instruction

Presented by Diana Myers HyattOctober 23, 2012

Introduction

Examine assessment process andbest practices for assessing LI sessionsPlanning for learning and assessment, from the program to the session levelSpecific assessment tools suitable for sessions, their strengths and weaknessesSpecific examples of incorporating assessmentHow can we leverage limited resources for maximum learning impact?

ChallengesChallenges of assessing in one-shot instruction sessionsLimited class time. How do you effectively measure

learning in such a short time? Librarian staffing and workloadLimited librarian training in teaching and

assessmentLimited support from faculty/college administrationUCR trends? Budget cuts?

Why assess? Improve instructional programs and teaching

methods to better meet learning goals. Improve student learning. Gives them a chance to

review learning goals, see their progress and what they still need to learn.

Part of accountability, meeting department and institutional learning goals.

Demonstrate value of library with evidence.

What is assessment? Assessment to improve learning:

How well are students learning what we intend them to learn? Measurement of student learning based on established learning outcomes.

Continuous, cyclical process of incremental improvement and change

Assessment Primer: How to do assessment, University of Connecticut www.assessment.uconn.edu/primer/how1.html

Assessment Best Practices (ACRL)Program evaluation: Develops a process for program planning, evaluation, and revision. Measures the progress of meeting the program’s goals and objectives Integrates with course and curriculum assessment, institutional evaluations

and regional/professional accreditation initiatives. Uses appropriate assessment/evaluation method for relevant purposes, for

example formative and summative and/or short-term and longitudinal.

Student outcomes: Acknowledge differences in learning and teaching styles in the outcome

measures. Multiple methods: needs assessment, pre-tests, post-tests, portfolio

assessment, oral defense, quizzes, direct observation, peer and self review, and experience.

Focus on learner performance, knowledge acquisition, and attitude appraisal.

Assess the learners’ process and product. Include learner-, peer-, and self-evaluation.

Characteristics of Programs of Information Literacy that Illustrate Best Practices: A Guidelinehttp://www.ala.org/acrl/standards/characteristics

Assessment Best Practices

Assessment program should: be continuous: drive change and innovation in an

ongoing, iterative process, pinpointing areas for improvement and development

be learner-centered: focus on student learning rather than on us

measure what students are learning as accurately and directly as possible

learning goals should be flexible enough to be adaptable to specific courses, integrating current program goals (implicit and explicit) with ACRL standards

When possible, allow for immediate teaching improvement by informing teaching librarian of current student skill levels through assessment opportunities prior to the workshop

Strategic plan for learning and assessment Moving from the general level to the specific (session level) Library develops main learning goals integrating university

and library missions, ACRL standards and input from librarians and faculty

From these goals, articulate specific learning outcomes which can be assessed and measured

◦ target groups, scaffolding (e.g. first-year English, graduate art students)

◦ Delivery vehicles (LI sessions, online tutorials, embedded teaching)

Determine which specific learning outcomes are best addressed in LI sessions.

Mapping learning outcomes to specific courses

Oakleaf, M. (2009). WRITING INFORMATION LITERACY ASSESSMENT PLANS: A guide to best practice. Communications in Information Literacy, 3(2), 80-90.

Learning plan for LI sessionsLearning outcomes for specific LI sessions based on:Library-level goals for information literacyLearning goals of the specific course and

professor. Instructor collaboration, course syllabus and library curriculum map

Students’ current course assignments and IL mastery level. Assignment description, pre-session assessment (assignment or pre-test)

Assessment component of learning plan

A good plan gives teaching librarians time to focus on teaching and engaging students, because learning objectives and assessment tools for each workshop are clear.

Articulate what specific results will show that students have mastered a learning outcome.

How will assessment measures be gathered and reported? How will the gathered data be reported and shared?

Assessment plan is reviewed on a regular basis. New assessment methods should be tested before

using on a widespread basis.

Assessment planning for LI sessions Rubric-based assessment plan combining learning outcomes paired

with appropriate assessment tools and scoring. Done collaboratively within the library.

Learning outcome Assessment instrument

Acceptable performance criteria

Explores general information sources to increase familiarity with the topic

Learning activity: Find and cite one appropriate article from online subject encyclopedia

Appropriate source, reason given, proper citation format

Assessment resources developed (test banks, learning activities with worksheets, etc.)

How to measure success. What scores will be considered adequate?

Teaching librarian now has appropriate learning outcomes and ready-to-use assessment tools that can be easily adapted for specific LI sessions.

Types of assessment instruments

Technique Potential Strength Potential Weakness

Locally developed multiple choice tests

Can provide direct evidence of student mastery of learning objectives

Appropriate mixes of items allow faculty to address various types of learning objectives

Can be centrally developed and then adapted for specific workshops

Easily scored and tabulated

Reliability and validity generally are unknown

Creating effective exams requires time and skill

Traditional testing methods may not provide authentic measurement

Embedded assignment and activities

Can provide direct evidence of student mastery of learning objectives

Involves active learning, possibly more retention

Can provide authentic assessment of learning objectives

Data collection is unobtrusive to students

Requires time to develop and coordinate and score, need simple grading rubrics

Requires faculty trust that the program will be assessed, not individual teachers

Reliability and validity generally are unknown

Norms generally are not available

Summary of assessment techniques

University of Connecticut Assessment Primer, Assessment Primer, http://assessment.uconn.edu/primer/how1.html

Summary of assessment techniques

Technique Potential Strength Potential Weakness

Surveys, one-minute papers, 1-2-3 mode, Survey monkey, clicker technology, Poll Anywhere

Are flexible in format and can include questions about many issues

Can be administered to large groups of respondents

Can easily assess the views of various stakeholders

Questions generally have a clear relationship to the objectives being assessed

Can be conducted relatively quickly and easily

Responses to closed-ended questions are easy to tabulate and to report in tables or graphs

Open-ended questions allow faculty to uncover unanticipated results

Can be used to track opinions across time to explore trends

Polls can be used for quick pre-tests with instant feedback, encourage active participation

Provide indirect evidence about student learning

Validity depends on the quality of the questions and response options (avoid guessing)

What people say they do or know may be inconsistent with what they actually do or know

Open-ended responses can be difficult and time-consuming to analyze

Embedded performance-based assessment Our assessment methods should support problem-based

and other active learning scenarios rather than straight lectures.

Combining active learning activities with assessment Could include collaborative learning exercises, creating

3-item bibliography (encyclopedia article, book, journal article), other focused library activities with worksheets, concept maps or other concrete product that can be assessed -- better learning using constructivist methods.

Rubric for quantitative measurements. Or multiple choice post-test afterwards for easier assessment.

Trade-off: more complex learning, but harder to measure in class.

LI session evaluation by students3-2-1 model using Google

forms. More structured than one-minute paper.

Indirect measure, but can reveal important information. How well are responses aligned with our teaching goals?

Gives students a chance to reflect on and articulate what they’ve learned for better retention

Although open-ended questions are harder to compile for statistical purposes, they give better feedback to librarians to improve instruction.

Provides an opportunity for personal follow-up with students who want it by providing space for questions and email info.

Nardine, J. and Meier, C. (2010), "Assessing the one-shot instruction session: Leveraging technology for optimum results", Virginia Libraries, Vol. 56 No. 3, pp. 25-8

Assessment outside workshops

Library/Faculty collaboration on class assignments integrating IL learning outcomes.

Target key classes and assignments such as English composition, introductory classes. ◦ bibliographies, annotated bibliographies◦ research blogs/journals on their research

experience - very qualitative but could be eye-opening.

◦ shared assignments with linked learning outcomes◦ students complete online IL module with tutorials,

activities and assessments.

Evaluating our assessment methods

Is the method accurately measuring the learning outcomes it’s supposed to?

Is it valuable? Does it improve our teaching?

Does it put reasonable demands on our staff and resources for the value it brings?

Comparison of multiple forms of evidence for the same learning outcome (including class assignments)

Assessment method examplesACRL standard 1.1: The information literate student defines

and articulates the need for information. Small group activity: hypothetical information need situation

(buying a car, taking a trip), articulate the information need and specifics of what information is needed, also brainstorm possible sources (friends, consumer reports, review or travel websites, travel guides). Assessment: state general information need, 3 questions that need to be answered, 3 possible sources in Google Docs

Multiple choice post-test using Google form: what is the most important first step in library research? ◦ What is the most important first step in library research?

a) identify key authors on research problem.b) know research problemc) Locate key articles on research problemd) Locate key books on research problem

Give them a broad topic such as “commuters and Internet crime”. Have them come up with a good research question and identify keywords for first stage of research. Could use Google spreadsheet or Google form. Could use concept map or Wordle for keyword brainstorming.

Sample use of Google Docs for learning activity

Shannon R. Simpson, Google Spreadsheets and real-time assessment: Instant feedback for library instruction, Coll. res. libr. news October 2012 73:528-549.

Standard 1.1c Explores general information sources to increase familiarity with the topic.Performance based exercise: Given a topic by librarian, find an appropriate encyclopedia article that gives background information on aspect of the topic and create citation. Answers cut and pasted into Google spreadsheet or fill out Google Docs form.

Reliability and Validity of Assessment Methods

Reliability Consistent responses over time. Variation in scores is actual difference

in what is being measured, such as knowledge Students have adequate time to take assessment, guessing is

minimized Consistent methods for all sessions (e.g. same post-test questions).

Importance of collaboration and buy-in in developing assessment methods.

Must have consistent scoring or rating scale, so easy to score and guidelines for scoring are clear

Validity The assessment tool measures what we want to measure? Do our results correlate with other proven measures that assess the

same learning outcome? Does our method accurately predict future student performance?

Conclusion

No one best way to assess student learning. Use a variety of methods, direct and indirect.

Trade-offs between more thorough assessments and time needed to do them. Have to prioritize learning outcomes and make sure they are realistic given time and staff constraints.

Developing a “learning bank” of active learning activities and specific assessment tools, such as a test bank, can give teaching librarians more time to focus on improving their teaching methods.

Program-level learning plan helps us see the big picture. Learning objectives in one LI session are scaffolded with other sessions. Complement other learning vehicles such as class assignments, tours.

Use assessment methods that involve a learning activity. Learning and assessment together.

“Good teaching is inseparable from good assessing. The question, therefore, is not whether to evaluate students, but how to measure performance in ways that will enrich learning, rather than restrict it.”

Ernest L. Boyer, The Basic School: A Community for Learning (New York: The Carnegie Foundation for the Advancement of Teaching, 1995), 104.