E(p)owering Your Institution

E(p)owering Your InstitutionA Mixed-Model Approach to

Assessment

© 2004 Douglas Joubert

Douglas James Joubert

Greenblatt Library

Objectives

• Creating a culture of assessment

• H1 and H3

– Data Considerations

– Sample Characteristics

– Statistical Analysis

• Collaboration & Innovation

Illusions of Certainty1

Retaining and growing their customer base, and focusing more energy on meeting their customers’ expectations is the only way for academic libraries to survive in this volatile environment

--Rowena Cullen, 2001

Creating a Culture of Assessment

Culture of Assessment

Benchmarking

ARL Service Academy

AAHSL Task Force Questions

LibQUAL+

“Aspirational” “Comparator”

Speaking a common languageQuantitative

Input

Influential Players

Input

Mixed Model Approaches

Linear Model of Research

Theory

H1

Sampling

Analysis

Validation

Adapted from Flick, 2002

Circular Model of Research

Adapted from Glaser & Strauss, in Flick, 2002

Comparing

ComparingComparing

Sampling Sampling

CaseTextual analysis



PA T

Quantitative Research

ToMaze

Quick! get to the

cheese

Qualitative Research

I really don’t like cheese

I need a vacation

Do these frames

make me look smart?

ToMaze

Why am I here?

Qualitative Research

“Triangulation”

“Grounded Theory”“Bricoleur”

TwoVariables

?

Bothcontinuous

?

Prediction?

r/s between 2variables?

Both variables normal?

Pearsoncorrelation

Rankcorrelation

Linear regression

Y Y

N

Y

Y

N

Y

Statistical Inference Flowchart

Adapted from Rosner, 2000

Overall Data Considerations

24th AAHSL 25th AAHSL

Local SPSS Tables Local SPSS Tables Local SPSS Tables

2002 LibQUAL+




N > 13,000 2002 LibQUAL+




Descriptive

2002 LibQUAL+


• Around 120 cases• Formatted for

reading, not for statistical analysis

• Formatting must be monitored

• Are missing scores “systematic” or “random”?

AAHSL Data


• Over 13,000 cases• In SPSS, built for

statistical analysis• Formatting and

Missing scores built into the design of the variables

LibQUAL+ Data


• Use independent reviewer before data migration• Determine pattern of missing data before migration• How are you going to deal with missing scores• Easier to spot errors with 120 cases than with 13,000

AAHSL Data

LibQUAL Data

The First AAHSL Group

1. How do satisfaction ratings differ for various reporting structures?

2. How does the size of library staff affect satisfaction ratings?

3. How does the number of constituents affect satisfaction ratings?

4. How does the ratio of staff to constituents affect satisfaction ratings?

Source: AAHSL Task Force on Quality Assessment

Hypothesis 1

Hypothesis I (H1) was operationalized as, “What is the affect of library reporting structure on the mean

measure of overall quality of service?”

H1 Data Considerations

• Not originally included in the study design• Included to rectify a priori considerations for

the remaining three hypotheses

“person-levelsubscales”

“institution-levelsubscales”

means by dimension for each person

means by dimension for each institution


• Medical School• Other Health Science

School• University Library• Health Science Center

Administration• University

Administration• Other

• Question 3 (Section 5.6) of the 2002 LibQUAL

Library Reporting Structure

Overall Quality of Service+

Defined by the 2001-2002 AAHSL Library Statistics

Survey

H1 Sample Characteristics

• 35 AAHSL institutions contributed data for the 24th Annual Survey and participated in 2002 LibQUAL+.

Health Science (14) University Administration (2)

Medical School (12) Other (2)

University Library (3) Missing (2)

H1 Unit of Analysis

• This analysis needed to demonstrate if OVERSAT1 differed by reporting structure.

• H01 required an analytical method for dealing with the institutional differences

Variable reserved from LibQUAL data

Library reporting structure

Differences not explained by the grouping

Differences explained by the grouping +

H1 Unit of Analysis

• A Linear Mixed Model (LMM) allowed me to control for fixed and random effects

OVERSAT [DV] = LIBSTRUC [IV1] + INSTID(LIBSTRUC) [IV2]

Fixed-effectsRandom-effects

Descriptive Statistics for Overall Quality of Service

How would you rate overall quality of the service provided by the library?

Library Reports To Count Mean SD CV (%)

Medical School 4868 7.23 1.451 20.1%

University Administration 879 7.18 1.583 22.0%

Health Science Center 5972 7.50 1.323 17.6%

Other 649 7.22 1.457 20.2%

University Library 759 7.19 1.393 19.4%

Total 13127 7.35 1.408 19.2%

Type III Tests of Fixed Effects for Overall Quality of Servicea

Source Num df Den df F Sig.

Intercept 1 26.927 6640.536 .000

LIBSTRUC 4 27.038 .924 .464

a. Dependent Variable: How rate overall quality of the service?

Hypothesis 3

Hypothesis 3 (H03) was operationalized as, “How does the number of constituents served affect

satisfaction ratings?”


Secondary Clientele Served

Table Q4

Academic and Primary Hospital Staff (STAFFTOT) = NO

Primary Clientele Served

Table Q3A-Q3C

Total constituents served (CONSTOT)

1. Faculty (FACTOT),

2. Interns, residents, and fellows (IRF)

3. Students (STUTOT)


• From Question 3 (Section 5.6) of the 2002 LibQUAL computed mean overall quality of service by institution

“person-levelsubscales”

“institution-levelsubscales”

Mean OQS for each person

Mean OQS for each institution

H3 Unit of Analysis

• Because of the findings from H1, H3 could be analyzed with correlation and simple linear regression models

CONSTOT M_OQS by Inst

IV DV

Descriptive Statistics for H3

Dealing with Outliers• Distinct score relative to the bulk of

the distribution• Not all outliers are influential• Cook’s D is an index based on the

F-statistic• Cook’s D > 1 deemed an outlier• Cook’s D = 3.10679

X-DirectionIV

Dealing with Outliers• Studentized (SRESID) residual provides

information about outliers in the “Y” direction

• Framed within the context of the SND, so scores > ± 3

• Sign is irrelevant because we are only interested in how far the point is in the tail of the distribution

• SDRESID =1.75842

Y

Direction

DV

Dealing with Outliers

• Removing outliers in an objective method is ethical

• Filter variable ($_filter) created to exclude score 4 from the analysis

• Correlation and linear regression was performed with (non-transformed) and without (transformed) the data from case 4

DV

IV

32N

.027Sig. (2-tailed)

-.391*Pearson Correlation

Total Constituents

Served

LibQUAL+ Mean Overall Quality of Service by Institution

* Correlation is significant at the 0.05 level (2-tailed).

Transformed Correlations

Total Constituents as a Predictor of Measures of Overall Quality of

Service

.153.3911,305.423-0.3910.355-8.10

R2RdfFStd. ErrB

• Predictors: (Constant), Total Constituents Served• Dependent Variables: Mean Overall Quality by Institution

Total Constituents as a Predictor of Measures of Overall Quality of

Service

Total Constituents accounts for 15.3% of the variance of M_OQS

X Y .153.391

R2R

Further Investigation

• Controversy on Significance Testing– Evaluating Results Replicability– Confidence Intervals for Effect Size– Emphasize Effect-size Interpretation– β versus Structured Coefficient exploration

for effect size

Thompson, 2002, 1996, and 1992

BuildingConnections

BiostatisticsBiostatistics

AAHSLAAHSL

Peer NetworksPeer Networks

NewDirections

Qualitative

2005 LibQUAL

Local Projects

Collaboration & Innovation

E(p)owering Your Institution

Education

statistical

level subscales

research adapted

descriptive

total constituents

data considerations

outliers

analysis