ARL Statistics and Measurement Program Effective, Sustainable and Practical Assessment Martha Kyrillidou Director, ARL Statistics and Service Quality Programs.

ARL Statistics and Measurement ProgramEffective, Sustainable and Practical Assessment

Martha Kyrillidou Director, ARL Statistics and Service Quality Programs

Association of Research Libraries

Steve Hiller Director, Assessment & Planning, University of Washington Libraries

ARL Visiting Program Officer

Jim SelfDirector, Management Information Services, University of Virginia Library

ARL Visiting Program Officer

University of Cape Town Libraries16 August 2007

ARL www.arl.org

ARL Mission

• Non-profit organization of the libraries of research institutions in North America

• Forum for exchange of ideas

• Agent for collective action

Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication.

Members: 123 major research libraries in North America.

Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources.

Users: Three million students and faculty served.

Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 37 percent is invested in access to electronic resources.

www.arl.org

What makes a research Library?

• Breadth and quality of collections and services• Sustained institutional commitment to the library• Distinctive resources in a variety of media• Services to the scholarly community• Preservation of research resources• Contributions of staff to the profession• Effective and innovative use of technology• Engagement of the library in academic planning --from ARL ‘Principles of Membership’

The Beauty and the Beast

• What makes a quality research library?

“Quality much like beauty is in the eye of the beholder”

The Beauty and the Beast

• What makes a quality research library?

“Quality much like beauty is in the eye of the beholder”

The Beauty and the Beast

•What makes a quality research library?

“Quality much like beauty is in the eye of the beholder”

A gateway to assessment tools: ARL StatsQUAL™:

ARL Statistics -- E-Metrics

LibQUAL+®

DigiQUAL™

MINES for Libraries™

•Library Assessment Conferences

•Service Quality Evaluation Academy

•Library Assessment blog

•Making Library Assessment Work

•ESP Assessment

– Effective, Sustainable, Practical

Assessment at ARL

www.arl.org

.

StatsQUAL™

Duane Webster, Executive Director www.statsqual.org

StatsHome

LibQUAL+DigiQUAL(/digiqual)

MINES(/mines)

SAILS(/sails)

E-Metrics(/emetrics)

ARL Statistics(/arlstats)

Interactive Statistics

(/interactive)?

Login

UserProfile

InstitutionProfile

www.libqual.org

SurveyManagement

StatsQUAL™

www.statsqual.org

Updating the Traditional ARL Statistics

• E-Metrics = ARL Supplementary Statistics– On going efforts to update and refine core data.

– Exploring feasibility of collecting e-metrics.

• ARL Task Force on New Ways of Measuring Collections :– Growing concern with utility of membership index.

– Study ARL statistics to determine relevance.

– Develop Profile of Emerging Research Libraries.

www.statsqual.org

Tradition

• ARL membership criteria index:– Volumes held– Volumes Added Gross– Current Serials– Total Expenditures– Professional plus support staff

Task Force Recommendations

• Reserve use of the current membership criteria index to those occasions when it is needed for consideration of membership issues

• Implement an expenditure-focused index

• Use the new expenditure-focused index for any public reports, such as the Chronicle of Higher Education

• Begin to develop a services-based index that combines the following three factors: collections, services, and collaborative relationships

• Revise definitions for collections-related data categories, such as serials, and experiment with a variety of new measures, including usage data, strength of collections, and service quality measures to develop a richer set of variables for potential inclusion in the three factor alternative index

• Collect qualitative data to develop a profile of ARL member libraries

The power of narrative … (Peter Brophy, Yvonna Lincoln, Colleen Cook …..)

The focus on economics

• ARL Expenditures focused index:– Total library expenditures– Expenditures for Professional staff– Expenditures for Library materials– Professional plus support staff

Linear Regression

0 25 50 75 100

Rank of index03

0

25

50

75

100

Ran

k o

f ex

pin

d03

Rank of expind03 = 3.24 + 0.94 * rindx03R-Square = 0.89

Linear Regression

0 25 50 75 100

Rank of index05

0

25

50

75

100

Ran

k o

f ex

pin

d05

Rank of expind05 = 2.50 + 0.94 * rindx05R-Square = 0.89

Thinking Strategically About Library Futures

• What is the central work of the library and how can we do more, differently, and at less cost?

• What important set of services does the library provide that others can’t? What new roles are needed?

• What advantages does the research library possess?

• What will be the most needed by our community of users in the next decade? How is user behavior changing?

• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?

• What are the essential factors responsible for the success of the library?

www.arl.org

Defining Success in a Digital Environment

• Crafting new measures of success.

• Moving from measuring inputs to outputs.

• Understanding impact of library roles and services.

• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals.

• Reallocating and managing capabilities to focus on new definitions of success.

www.arl.org

• "You're going to be able to go back and forth from Google Book Search into Harvard, or from Harvard's catalog into Google,“ reminding users that it was important to keep the HOLLIS catalog the "starting point for their research," since only HOLLIS can give a full picture of the 15.5 million books in Harvard's collection, the nation's largest …Harvard would look to scan all of its-out-of copyright books, about one million, over time but that

ultimately "the ideal" is one big digital ultimately "the ideal" is one big digital library of books held in libraries around the library of books held in libraries around the world. world. ”It shouldn't matter whether it came from Harvard or Michigan"

- Sidney Verba, Harvard library director, - Library Journal Academic Newswire – April 24, 2007

old.libqual.org

A LibQUAL+® Update

The LibQUAL+® premise, dimensions, and methodology

LibQUAL+® resultsLibQUAL+® in action

old.libqual.org

Dimensions ofLibrary Service Quality

Information Control

LibraryServiceQuality

Model 3

Self-Reliance

Equipment

Timeliness

Ease of Navigation

Convenience

Scope of Content

Affect of Service

Library as Place

Reliability

Assurance

Responsiveness

Empathy

Refuge

Symbol

Utilitarian Space

old.libqual.org

Survey Instrument – “22 items…

old.libqual.org

…and a Box”

Why the Box is so Important:• About 40% of participants provide open-

ended comments, and these are linked to demographics and quantitative data

• Users elaborate the details of their concerns• Users feel the need to be constructive in their

criticisms, and offer specific suggestions for action

Background: ServQUAL LibQUAL+® DigiQUAL™

LibQUAL+® Dimensions of Service Quality:• Affect of Service• Information Control• Library as Place

Developing DigiQUAL™ Survey Items

DigiQUAL™12 themes of service quality:• Accessibility• Navigability• Interoperability• Collection building• Resource Use• Evaluating collections• DL as community for users• DL as community for developers• DL as community for reviewers• Copyright• Role of Federations• DL Sustainability

www.digiqual.org

Assessing the Value of Networked Electronic Services:

Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™

www.arl.org/stats/newmeas/mines.html

The MINES survey

• A research methodology consisting of a web-based survey form and a sampling plan.

• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.

• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.

• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E-Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™).

What is MINES?

• How extensively do sponsored researchers use the new digital information environment?

• Are researchers more likely to use networked electronic resources from inside or outside the library?

• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?

• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?

• Are particular network configurations more conducive to studies of digital libraries patron use?

Questions Addressed

Where are the most critical assessment needs and opportunities?

• Complementing LibQUAL+® with additional measures.

• Developing impact studies on user success, economic value, and community return on investment.

• Moving target: what is a digital library?

• E-Resources: understanding usage.

• Gaining acceptance and use of standard measures for e-resources.

• Building a climate of assessment throughout library.

What are the lessons learned?

• Understanding changes in users approach to information resources.

• Service quality improvement is a key factor.

• Understanding the impact of e-resources on library services - TRL.

• Learning how to compete and/or collaborate with Google.

• Upfront investment in design and development.

• Making the assessment service affordable, practical, & effective.

• Assessment needs to be satisfying and fun.

In Closing

• As higher education is challenged on accountability and effectiveness issues so will libraries.

• A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work.

• Basic questions of role, vision, and impact must be answered by library community.

What’s in a “Library”

• A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. – --Justice Oliver Wendell Holmes

Selected References

• Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. [Available at http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf]

• Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf]

• Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at http://InformationR.net/ir/9-4/paper187.html]

• Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): 20-21. [Available at http://www.arl.org/newsltr/230/usage.html].

• Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, 2003. [Available at http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf]

• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. [Available at http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf]

• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, 2005. [Available at http://www.libqual.org/documents/admin/ISKO.PDF]

Selected References