Quants, Quals, and Pals Library Assessment in the 21 st Century Steve Hiller University of Washington Libraries CUNY Assessment Conference June 6, 2014.

Quants, Quals, and Pals Library Assessment in the 21st Century

Steve HillerUniversity of Washington Libraries

CUNY Assessment Conference June 6, 2014

Four Useful Assessment Assumptions

• Your problem/issue is not as unique as you think• You have more data/information than you think• You need less data/information than you think• There are useful methods that are much simpler than

you think

Adapted from Douglas Hubbard, “How to Measure Anything” (2010)

What Will We Assess and Measure

• Institutional assessment efforts should not be concerned about valuing what can be measured, but instead about measuring what is valued. A. Astin

• Not everything that can be counted counts, and not everything that counts can be counted. Attributed to Albert Einstein

Some Questions Before You StartWhat Do You Need to Know and Why?

• What do you need to know• Why do you need to know it• How will you collect data (is it already available?)• When do you need the data• How will you analyze the data• How will you use data• What outcomes are expected

The answers will determine methods used

Assess What’s Important

• Contribution to student and faculty success• Contribution to institutional mission/visibility

And, perhaps (depending on library and institution type)

• Accountability/Efficiency/Effectiveness• Use• Revenue generation (including fund raising) • Benchmarking and comparisons with others

Not how good is this library? Rather, 'How much good does it do?' (R.H Orr, 1973)

Recent Trends in Library Assessment and Performance Measurement

• Greater reliance on external measures and user impacts; aligned with planning

• Customer-centered library concept • Outcomes-based assessment and metrics that made

use of multiple methods, including qualitative • Collaboration with institutional and other partners • Demonstrating library impact and value on

individuals and communities

Changing the Focus: Quants, Quals, Pals

• Quantitative– Focus changes from counting to analysis and action

• Qualitative– Focus changes from research study to practical methods

which emphasize understanding

• Pals (Partners)– Focus changes from using our own data and internal

assessments to working with other on broader collaborative assessments

Quants

• First mentioned in science fiction for those with exceptional quantitative skills and analysis

• SF Quants were often androids, or enhanced with implants or through breeding

• SF Quants, especially androids, had a high termination rate

• Quants became human (?) in the late 20th century as a name for those with high analytical skills, especially in business

Quants in the 20th Century: We Measure What We Can Count

• Inputs - Size– Collections– Budget– Staff– Facilities– Discovery tools

• Size used as indicator of excellence

• Outputs - Usage– Collections– Reference services– Instruction sessions– Facilities usage– Web sessions

• May indicate if inputs used but don’t tell us what users were able to accomplish

Data Accuracy

Many Library Statistics Are Self-Reported and Lack Independent Verification

The Challenge for Libraries

• Traditional statistics/measures are no longer sufficient– Emphasize inputs/outputs – how big and how many– Do not tell the library’s or customers’ story– Usually not aligned with organizational goals and plans or

support library strategic directions and objectives– Do not capture the impact and value of the library

• Need to demonstrate the difference the library makes– To the individual, community and the organization

Why your kids will want to be data scientists

Big money, lots of jobs, challenging and they’ll take care of you in old age!

Quants in the 21st Century:Data Analysis and Action

Finding meaning and application in:• Big data sets• Data mining• Data exchange • Actionable data• Data formats• Metadata

Supported by sophisticated and robust data analysis tools

Some Quant Examples

• Infosphere use and application• Relationship between library use and student, faculty

and research success• Data trends• Data comparability and integration• Analytics

Quals - Definition

An abbreviation for the word Quality. Quals is used when the item being described surpasses the quality of everything else, by far, leaving only one word left - Quals.

Only to be used by the super-cool and often used more frequently when drunk and rowdy.

Source: Urban Dictionary

Classic Qual(itative)s

• Social Science research methodology using time-consuming protocols

• Interviews and observations (ethnographic) • Focus groups• Relatively few library qualitative studies until 25

years ago

Quals: The Past 25 Years

• Widely employed in libraries and for outcomes-based efforts

• Shorter, more focused and actionable studies• Usability• User-centered design• Ethnographic• Advocacy (narratives)• Qualitative work integral part of market research

FutureQUAL®

• Merging quantitative and qualitative as data collection and analysis tools incorporate digital technology

• Understand individual and group behavior beyond information, libraries and organizations

• Narratives and personas closely linked to customized services, marketing and advocacy

Pals (Partners)

Partners and Collaborators

• Partners– Close, recognized

relationships– Work towards similar

goals and outcomes– Value each others

contributions– Watch each others backs– Ride off in the sunset

together

• Collaborators– Brought together for

specific efforts– May have different goals– Bring special expertise– Watch your back!– Often shot at the end of

spy/war movies

Partners and Collaborative Assessment:

• Collaboration with other institutional programs/units• Collaboration with other libraries/consortia/

organizations• Linked to strategic planning, accountability, advocacy• Fostering assessment & performance measurement that

focuses on community success

What Libraries Bring to Academic Partnerships

• Big picture understanding of higher education, scholarly communication, the institution and its programs

• Close links to academic programs through liaisons• Service to entire community• Student-centered approach and research collaboration• Diverse skills and expertise

From Library-Centered to External Partnerships

Research Traditional: Collection Development• Collections most valued library

research support. Remains important

Now: Information Management• Embed library experts within research

programs

• Support research cycle from discovery through communication and archiving

• Collaborate with a variety of partners

Instruction Traditional: Bib Instruction• Library and tool focused• Not always tied to course work or

outcomes• Evaluation by pre- and post-tests

Now: Teaching & Learning• Focus on student learning

outcomes and authentic assessment

• Partnerships & contribution to teaching/program outcomes

• Broad use of educational technology

Catalyst for Partnerships Was Often External

• Association and information standards• Integrated approach to accreditation (professional and

institutional)• Consortia• Value and impact studies, advocacy

Library Value & Impact

• Lib-Value Project (ARL, Tennessee)• Library Impact Data Project (Huddersfield/JISC)• Methods and procedures for assessing the impact of

libraries (ISO 16439)• Discovering the Impact of Library Use and Student

Performance (Wollongong)• ACRL Assessment-in-Action project• University of Minnesota Library Data & Student

Success

• Value of Academic Libraries (Oakleaf, 2010)

Value & Impact: Key Themes

• Aligning library’s role with core mission & values of institution: “allow institutional missions to guide library assessment” (Oakleaf 2010, p, 30).

• Communicating value to stakeholders

• Connecting institutional & library data

U.S. Regional Accreditation Standards for Institutions of Higher Education NWCCU 2010

http://www.nwccu.org/index.htm

• Institutional mission, core themes and expectations• Resources and capacity

– Governance– Educational resources– Library and information resources– Financial resources– Leadership and management– Infrastructure

• Planning and implementation• Assessment• Mission fulfillment, adaptation and sustainability

U.S. Regional Higher Education Accreditation: Library Standards (NWCCU)

The institution holds or provides access to library and information resources with an appropriate level of currency, depth, and breadth to support the institution’s mission, core themes, programs, and services

Planning for library and information resources is guided by data that include feedback from affected users and appropriate library and information resources faculty, staff, and administrators.

The institution provides appropriate instruction and support to enhance efficiency and effectiveness in obtaining, evaluating, and using library and information resources that support its programs

The institution regularly and systematically evaluates the quality, adequacy, and utilization of library and information resources and services, wherever offered and however delivered.

ACRL Standards for Libraries in Higher Education http://www.ala.org/acrl/standards/standardslibraries

The Standards assume that libraries:

•Develop user-centered, measurable outcomes that articulate specifically what the user is able to do as an outcome

•Conduct assessments that may be quantitative and/or qualitative;

•Collect evidence from assessments that demonstrate degree of success

•Use assessment data for continuous improvement of library operations.

ACRL Standards for Libraries in Higher Education (2010)

Institutional Effectiveness• The library defines, develops and measures outcomes in the

context of institutional mission and aligned with institutional outcomes

• The library develops and maintains a body of evidence that demonstrates its impact

• The library articulates how it contributes to student learning,

ACRL Standards: Libraries Contribution to Institutional Effectiveness

The library contributes to student recruitment, retention, time to degree, and academic success.

Students improve their academic performance over their college experience through their contact with the library.

Students who have more contact with the library show higher levels of improvement in performance on standardized tests.

Students are able to describe the role of the library in improving their academic performance from freshman to senior.

Partnerships in the 21st Century

• Institutional teams working on institutional level issues with functional specialists

• Multi-institutional level networking and expertise sharing

• Defining institutional and programmatic goals, priorities, and outcomes

• Blurring of administrative and academic boundaries• Computational savvy staff

Lorcan Dempsey’s 3 Challenges (2013)• Engagement

– Libraries work to create distinctive value in the research, learning and teaching workflows of their users in ways which go beyond the provision of collections

• Rightscaling– Libraries moving from “institution scale” to collaborative, shared

systems – especially through functional and regional consortia. The need for local infrastructure declines. As libraries want to emphasize impact and engagement, and de-emphasize activities which do not create distinctive local value, rightscaling becomes a key question.

• Institutional innovation– Library as an organization which reconfigures to map changes in the

user environment and expectations. Libraries have to develop new and routine ways of collaborating to achieve their goals, which involves evolution of organizational, cultural and communication approaches.

– The learning that flows from institutional innovation

Four Library Assessment Questions

• What do we need to know about our communities and customers to make them successful?

• Who are our partners in collaborative assessment?

• How do we measure the effectiveness of our services, programs and resources and how they contribute to institutional and user success.

• What do our stakeholders need to know in order to provide the resources needed for a successful library?

2014 Library Assessment ConferenceAugust 4-6, University of Washington

http://libraryassessment.org/