Measuring the Impact of Networked Electronic Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure Services: Developing Assessment Infrastructure for Libraries, State, and Other Types of for Libraries, State, and Other Types of Consortia Consortia Presented By: Terry Plum, Simmons GSLIS Brinley Franklin, University of Connecticut Martha Kyrillidou, ARL Gary Roebuck, ARL Raynna Bowlby, ARL Consultant & Simmons GSLIS MaShana Davis, ARL Kristina Justh, ARL Library Assessment Conference 2008 University of Washington Seattle, WA
49
Embed
Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for Libraries, State, and Other Types of Consortia Presented.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Measuring the Impact of Networked Electronic Services: Measuring the Impact of Networked Electronic Services: Developing Assessment Infrastructure for Libraries, Developing Assessment Infrastructure for Libraries,
State, and Other Types of ConsortiaState, and Other Types of Consortia
Presented By:Terry Plum, Simmons GSLIS
Brinley Franklin, University of ConnecticutMartha Kyrillidou, ARL
Gary Roebuck, ARLRaynna Bowlby, ARL Consultant & Simmons GSLIS
MaShana Davis, ARL Kristina Justh, ARL
Library Assessment Conference 2008University of Washington
Seattle, WA
http://www.minesforlibraries.org/
ARL New Measures Toolkit: ARL New Measures Toolkit: StatsQUALStatsQUAL®®
LibQUAL+® is a rigorously tested Web-
based survey that libraries use to solicit, track, understand, and
act upon users‘ opinions of service
quality.
LibQUAL+®
DigiQUAL®
The DigiQUAL® online survey designed for
users of digital libraries that measures reliability and trustworthiness of
Web sites. DigiQUAL® is an adaptation of
LibQUAL+® in the digital environment.
MINES for Libraries®
Measuring the Impact of Networked Electronic
Resources (MINES) is an online transaction- based survey that
collects data on the purpose of use of
electronic resources and the demographics
of users.
ARL Statistics™
ARL Statistics™ is a series of annual publications that
describe the collections, expenditures, staffing,
and service activities for Association of Research Libraries (ARL) member
libraries.
ClimateQUAL™
ClimateQUAL™: Organizational Climate
and Diversity Assessment is an online
survey that measures staff perceptions about:
(a) the library's commitment to the
principles of diversity, (b) organizational
policies and procedures, and (c) staff attitudes.
http://www.minesforlibraries.org/
What is MINES?What is MINES?
Action Research Historically rooted in indirect cost studies Set of recommendations for research design Set of recommendations for web survey
presentation Set of recommendations for information
architecture in libraries Plan for continual assessment of networked
electronic resources An opportunity to benchmark across libraries
http://www.minesforlibraries.org/
What is MINES? What is MINES? (cont’d)(cont’d)
Measuring the Impact of Networked Electronic Services (MINES)
MINES is a research methodology that measures the usage of networked electronic resources of a library or consortium by a specific category of the patron population.
MINES is a Web-based survey form consisting of 5 questions that is administered at the time of transaction.
MINES measures: User status and discipline/affiliation (who) Physical location (where) Primary purpose and reason of use (why)
MINES a part of Association of Research Libraries’ New Measures & Assessment Initiatives (since 2003).
MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQUAL+®).
MaShana Davis
Discussion randomization here?
http://www.minesforlibraries.org/
Early Data Collection ActivitiesEarly Data Collection Activities
Academic Medical Libraries U of Connecticut Health
Center U of North Carolina U of Texas Medical Branch U of Texas Southwestern U of Utah U of Virginia Washington U
Main Universities U of Colorado U of Connecticut U of North Carolina Oregon State U U of Utah U of Virginia Washington U
Data was collected at seven main campus libraries and seven academic health sciences libraries in the U.S. between 2003 and 2005.
http://www.minesforlibraries.org/
Early Data Collection ActivitiesEarly Data Collection Activities
More than 45,000 networked electronic services uses were surveyed.
At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities.
http://www.minesforlibraries.org/
Recent Data Collection Recent Data Collection Activities via ARLActivities via ARL
Ontario Council of University Libraries (OCUL)
University of Iowa Libraries University of Macedonia
http://www.minesforlibraries.org/
Questions AddressedQuestions Addressed
How extensively do sponsored researchers use the new digital information environment?
Are researchers more likely to use networked electronic resources from inside or outside the library?
Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?
Are particular network configurations more conducive to studies of digital libraries patron use?
http://www.minesforlibraries.org/
MINES for LibrariesMINES for Libraries®® Survey Form Survey FormFive Questions and a Comments BoxFive Questions and a Comments Box
Methodological Considerations:Methodological Considerations:Experience with the MINES SurveyExperience with the MINES Survey
Terry PlumAssistant DeanSimmons GSLIS
Library Assessment Conference 2008University of Washington
Seattle, WA
http://www.minesforlibraries.org/
Issues with Web surveysIssues with Web surveys
Research design Coverage error
Unequal access to the Internet Internet users are different than non-users
Response rate Response representativeness
Random sampling and inference Non-respondents
Data security
http://www.minesforlibraries.org/
MINES StrategyMINES Strategy
A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.
Random moment/web-based surveys are employed at each site.
Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time. IRB waiver or approval
Libraries with database-to-web gateways or proxy re-writers offer a comprehensive networking solution for surveying all networked services users during survey periods.
http://www.minesforlibraries.org/
MINES StrategyMINES Strategy
Placement Point of use Not remembered, predicted or critical incident
Usage rather than user What about multiple usages Time out ? Cookie or other mechanism with auto-population or
more recently counting invisibly with a time out. Distinguish patron association with libraries.
For example, medical library v. main library. But what if the resources are purchased across
campus for all. Then how to get patron affiliation?
http://www.minesforlibraries.org/
Web Survey Design GuidelinesWeb Survey Design Guidelines
Web survey design guidelines that MINES followed: Presentation
Simple text for different browsers – no graphics Different browsers render web pages differently
Few questions per screen or simply few questions Easy to navigate Short and plain No scrolling Clear and encouraging error or warning messages Every question answered in a similar way - consistent
Radio buttons, drop downs ADA compliant Introduction page or paragraph Easy to read
Must see definitions of sponsored research. Can present questions in response to answers – for example if
sponsored research was chosen, could present another survey
http://www.minesforlibraries.org/
Quality ChecksQuality Checks
Target population is the population frame – surveyed the patrons who were supposed to be surveyed - except in libraries with outstanding open digital collections.
Check usage against IP. In this case, big numbers may not be good. May be seeing the survey too often.
Alter order of questions and answers, particularly sponsored and instruction.
Spot check IP against self-identified location Spot check undergraduates choosing sponsored research –
measurement error Check self-identified grant information against actual grants Content validity – discussed with librarians and pre-tested. Turn-aways – number who elected not to fill out the survey Library information architecture -- Gateway v. HTML pages –
there is a substantial difference in results.
http://www.minesforlibraries.org/
Documenting the Purpose and Use of Documenting the Purpose and Use of Electronic Resources:Electronic Resources:
Experience with the MINES SurveyExperience with the MINES SurveyBrinley Franklin
Vice Provost for University LibrariesUniversity of Connecticut
Library Assessment Conference 2008University of Washington
Seattle, WA
http://www.minesforlibraries.org/
““It is useless to tell the acquisitions librarian that half the It is useless to tell the acquisitions librarian that half the monographs ordered will never be used, unless we can specify monographs ordered will never be used, unless we can specify
which 50% to avoid buying.” (Galvin and Kent, 1977)which 50% to avoid buying.” (Galvin and Kent, 1977)
http://www.minesforlibraries.org/
Reliance on Vendor StatisticsReliance on Vendor Statistics Vendor statistics, while more reliable than in the past, are still maturing.
http://www.minesforlibraries.org/
The most popular current method of measuring usage of electronic resources by libraries is not through web-based usage surveys, but through vendor supplied data of library patron usage or transaction usage.
Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs.
Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for Libraries®)” IFLA Journal 32 (1) March, 2006
Measuring Digital Content UseMeasuring Digital Content Use
http://www.minesforlibraries.org/
A web-based transactional survey that collects data on users’ demographics and their purpose of use. It is administered in real time over the course of at least a year using a random moments sampling plan.
MINES for Libraries® has been administered at 40 North American universities in the last four years. More than 100,000 North American networked services users have been surveyed using a standard protocol.
Measuring Digital Content UseMeasuring Digital Content Use
http://www.minesforlibraries.org/
Library User SurveyLibrary User Survey
http://www.minesforlibraries.org/
Library User SurveyPatron Status
http://www.minesforlibraries.org/
Library User SurveyAffiliation
http://www.minesforlibraries.org/
Library User SurveyLocation
http://www.minesforlibraries.org/
Library User SurveyPurpose
http://www.minesforlibraries.org/
Sample Survey Data File GeneratedSample Survey Data File GeneratedOther UConnInstruction/Education/Departmental (Non-Funded)
Off CampusNon-UConn131.128.89.54EducationInstruction/Education/Departmental (Non-Funded) Researchhttp://www.euromonitor.com/womdas/12:57:4412/3/2004
Off CampusNon-UConn134.241.135.70Non-UConnInstruction/Education/Departmental (Non-Funded) Researchhttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First13:28:5212/3/2004
Off CampusNon-UConn134.74.42.82Business AdministrationOther Activitieshttp://homerweb.lib.uconn.edu/cgi-bin/Pwebrecon.cgi?DB=local&PAGE=First12:56:4612/3/2004In the LibraryUConn Faculty137.99.1.122
Liberal Arts & SciencesOther Activitieshttp://www.siam.org/journals/simax/simax.htm12:52:1712/3/2004On Campus - StorrsUConn Graduate Student137.99.1.122