1 Working Paper 13-24 Departamento de Economía de la Empresa Business Economic Series 03 Universidad Carlos III de Madrid July 2013 Calle Madrid, 126 28903 Getafe (Spain) Fax (34-91) 6249607 A framework for analyzing performance in higher education Lola C. Duque 1 Abstract Drawing on Tinto’s dropout intentions model (1975), Bean’s socialization model (1985), Astin’s involvement theory (1999), and the service marketing literature, this research presents a conceptual framework for analyzing students’ satisfaction, perceived learning outcomes, and dropout intentions. This framework allows for a better understanding of how students assess the university experience and how these perceptions affect future intentions. This article presents four studies testing fragments of the framework using data sets come from three countries and various undergraduate programs (business, economics, geography, and nursing). The models are tested using structural equation modeling with data collected using a questionnaire adapted to the specific contexts. The models have the ability to explain the studies’ dependent variables and offer practical utility for decision making. Applicability of the conceptual framework is evaluated in various contexts and with different student populations. One important finding is that student co-creation can be as important as perceived service quality in explaining students’ cognitive learning outcomes, which in turn explain a high percentage of satisfaction and affective learning outcomes. The studies also shed light on the roles of variables such as emotional exhaustion and dropout intentions. Keywords: subjective measures, satisfaction, perceived quality, performance, higher education The author thanks Davinia Palomares-Montero for her feedback on an earlier draft of the paper, and acknowledges support received from the Spanish Ministries of Education and Science, and Economy and Competitiveness (Projects SEJ2007-65897, EA2007-0184 and ECO2011-27942) and the collaboration of the universities and departments involved in the study. 1 Department of Business Administration – Carlos III University Calle Madrid, 126 – 20903 Getafe (Madrid) – Spain Tel: +34.91.624.8971 – Fax: +34.91.624.9607 E-mail: [email protected]
40
Embed
A framework for analyzing performance in higher education · A framework for analyzing performance in higher education ... assurance about the instructor’s competence and knowledge,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Working Paper 13-24 Departamento de Economía de la Empresa Business Economic Series 03 Universidad Carlos III de Madrid July 2013 Calle Madrid, 126 28903 Getafe (Spain) Fax (34-91) 6249607
A framework for analyzing performance in higher education
Lola C. Duque1
Abstract
Drawing on Tinto’s dropout intentions model (1975), Bean’s socialization model (1985), Astin’s involvement theory (1999), and the service marketing literature, this research presents a conceptual framework for analyzing students’ satisfaction, perceived learning outcomes, and dropout intentions. This framework allows for a better understanding of how students assess the university experience and how these perceptions affect future intentions. This article presents four studies testing fragments of the framework using data sets come from three countries and various undergraduate programs (business, economics, geography, and nursing). The models are tested using structural equation modeling with data collected using a questionnaire adapted to the specific contexts. The models have the ability to explain the studies’ dependent variables and offer practical utility for decision making. Applicability of the conceptual framework is evaluated in various contexts and with different student populations. One important finding is that student co-creation can be as important as perceived service quality in explaining students’ cognitive learning outcomes, which in turn explain a high percentage of satisfaction and affective learning outcomes. The studies also shed light on the roles of variables such as emotional exhaustion and dropout intentions.
The author thanks Davinia Palomares-Montero for her feedback on an earlier draft of the paper, and acknowledges support received from the Spanish Ministries of Education and Science, and Economy and Competitiveness (Projects SEJ2007-65897, EA2007-0184 and ECO2011-27942) and the collaboration of the universities and departments involved in the study.
1 Department of Business Administration – Carlos III University Calle Madrid, 126 – 20903 Getafe (Madrid) – Spain Tel: +34.91.624.8971 – Fax: +34.91.624.9607 E-mail: [email protected]
2
Introduction
Reporting on performance indicators of higher education has become a normal
practice of institutions nowadays; such reporting responds to demands for academic
accountability to communities and governments, requirements for regional or professional
accreditation, competition for resources and students, as well as implementing internal
practices for institutional performance evaluation and improvement (Nichols, 1995;
Peterson & Einarson, 2001; Quinn et al., 2009; Terenzini, 1989). Establishing standard
criteria of performance indicators is difficult given the multiple objectives of higher
education institutions and the variety of stakeholders involved (García-Aracil & Palomares-
Montero, 2010); however, it is necessary to develop models that can assist policymakers in
evaluating institutions’ performance, allowing for comparison between institutions and for
comparison of performance over time.
Numerous assessment tools might be employed; they usually complement one
another. While the traditional ones involve comparison of inputs-outputs in terms of
teaching, research, and third-mission activities (García-Aracil & Palomares-Montero,
2010), there are also approaches that evaluate stakeholders’ perceptions and satisfaction.
These subjective measures (i) have been proven to be good predictors of students’
performance and behavioral intentions (Lizzio et al., 2002), and (ii) allow for making
comparisons, which highlights their usefulness in the educational context.
In line with subjective approaches (based on perceptions), there are simple models
trying to understand how different perceptions of quality areas affect student satisfaction,
while other models use more complex relationships involving factors such as student
learning outcomes and student persistence intentions. Table 1 presents examples of studies
3
relating dimensions of perceived quality in higher education with student satisfaction, and
some other variables as determinants (e.g. perceived value, institution image, and
commitment) and consequences of student satisfaction (e.g. loyalty, trust in management
and support, reputation and perceptions of learning).
The aim of this research is to present a framework that reports on higher education
indicators (students’ learning outcomes, satisfaction, and dropout intentions) based on the
students’ perceptions of various factors (educational, environmental, psychological, and
their own involvement) to better understand the students’ complete experience at
university. This framework builds on Tinto’s dropout intentions model (1975), Bean’s
socialization model (1985), Astin’s involvement theory (1999), and the service marketing
literature. These models, the theory, and the literature have given insight into different areas
of knowledge, and we propose that a framework that incorporates insights from all of them
can better explain the role of different factors on students’ perceptions, intentions, and
feelings of their overall educational experience.
We first introduce the general framework and put forward specific hypotheses to be
tested. Then, four studies are presented and empirically tested with different data sets. We
conclude by summarizing the results of the studies and the implications of this approach.
Table 1 about here
Conceptual framework
Learning outcomes and dropout intentions have been central concepts studied in the
higher education literature. However, few studies approach them simultaneously. Building
on Tinto’s conceptual schema for dropout from college (1975), Bean (1985) proposes a
4
socialization model in which academic/educational, environmental, and social/
psychological factors predict students’ dropout intentions. Astin (1999) proposes the
involvement theory (effort and dedication) as a mechanism to explain the dropout
syndrome. Astin argues that dropout results from students’ low integration both
academically and socially. More recent studies coming from service marketing literature
suggest that quality perceptions of higher education have an influence on students’
satisfaction and behavioral intentions (Douglas et al., 2006; Eagle & Brennan, 2007;
Helgesen & Nesset, 2007; Petruzzelis et al., 2006), and a new perspective in marketing
highlights the student’s active participation as co-creator of service value (Dann, 2008;
Gummesson, 2008; Vargo & Lusch, 2004), which is in line with higher education theories.
Thus, integrating these streams of literature and both cognitive and affective learning
outcomes (Terenzini, 1989) into a single framework may prove a more general and
comprehensive approach, and one which better describes the students’ viewpoint on their
university experience. Figure 1 presents the integrative framework.
Figure 1 about here
Hypotheses development
Determinants of student satisfaction
Overall satisfaction is the consumer's general dis/satisfaction with the organization
based on all encounters and experiences with that particular organization (Bitner &
Hubbert, 1994). This definition represents a cumulative approach, which is preferred over
the specific-transaction one because it assesses the complete student experience; thus,
overall student satisfaction is based on the students’ general experience of the university.
5
Perceived service quality can be measured at the overall level, by dimensions or by
service attributes. Overall service quality is defined as the consumer's overall impression of
the relative inferiority/superiority of the organization and its services (Bitner & Hubbert,
1994). In higher education, many classifications and factors have been used, and typologies
vary depending on the conception of education quality, the expected achievements as result
of education quality and the methods of analysis (De Jager & Gbadamosi, 2010). Table 1
shows a variety of dimensions used to capture perceptions of quality in higher education.
Stodnick and Rogers (2008) found that the most important dimensions of quality that
impact satisfaction with the course are reliability on the instructor’s way of lecturing,
assurance about the instructor’s competence and knowledge, and empathy of the instructor.
Mai (2005) found that lecturers’ expertise, lecturers’ interest in their subject, quality and
accessibility of IT facilities, and prospects of the degree furthering students’ career are
correlated with the overall perception of education quality. Sojkin, Bartkowiak, and Skuza
(2012) found that the most important factor determining satisfaction from studying a
business major is “social conditions”, which includes aspects such as university coffee bars,
good sport facilities, subsidized accommodation and parking spaces. Yeo and Li (2012)
propose that the overall learning experience in higher education is enhanced by support
services provided; thus, better facilities, systems and processes that support learning will
increase student satisfaction. Douglas, McClelland and Davies (2008) classify various
service quality aspects as satisfiers (its presence leads to satisfaction, and absence does not
lead to dissatisfaction), dissatisfiers (lack of it leads to dissatisfaction, but presence does not
cause satisfaction), criticals (they are both satisfiers and dissatisfiers), and neutrals (aspects
whose presence does not lead to satisfaction and absence does not cause dissatisfation).
6
In services marketing a general classification of perceived service quality consists of
a functional and a technical dimension (Grönroos, 1984), which would correspond to
educational quality and administrative quality in the higher education context. Educational
quality concerns teaching and program quality perceptions (professors well prepared
academically who make the courses to be interesting, program and course contents clear
and with a coherent structure, and appropriate social environment), which relates to the
core objective of studying. Administrative quality concerns the quality perceptions of
necessary resources for learning (classrooms and course schedules appropriate for learning,
library services, laboratories, sport facilities, cafeteria, etc), including the functioning of
administrative offices. The use of two overall dimensions (tangible and intangible) for
measuring student perceptions of service quality in higher education has also been
supported by Nadiri, Kandampully and Hussain (2009), who found that these dimensions
are good predictors of student satisfaction. Because perceived service quality has been
found to affect consumer satisfaction in both the services marketing and the higher
education literatures, we expect that educational quality and administrative quality
influence student satisfaction.
H1a: Perceptions of educational quality will influence student satisfaction positively.
H1b: Perceptions of administrative quality will influence student satisfaction positively.
Performance assessment has been regarded as a component of quality (Koslowski, 2006).
In the higher education context, performance assessment evaluates student learning and
gains as a way to improve the quality of higher education (Palomba & Banta, 1999). The
European Foundation for Quality Management (EFQM, 1995) points out that institutions
need to know whether they are being successful in achieving learning outcomes in terms of
7
students’ value added to knowledge, skills and personal development. There are various
classifications of learning outcomes. A general definition is provided by Frye (1999):
cognitive learning outcomes concern the student’s acquisition of specific knowledge and
skills, whereas the affective learning outcomes concern how the higher education
experience has influenced the student’s values, goals, attitudes, self-concepts, worldview,
and behavior. DeShields, Kara and Kaynak (2005) found that student partial college
experience determines satisfaction for business student; this partial college experience is
composed by cognitive development (personal learning such as problem solving ability),
career progress (the extent to which students believe the program help them to get ahead in
their career plans), and business skills development. Sojkin, Bartkowiak, and Skuza (2012)
found that the second most important factor determining satisfaction from studying is
“professional advancement”, which includes aspects such as development of professional
skills, and opportunity of intellectual and personal development. Thus, students acquire
knowledge (cognitive outcomes) during their learning process, which is the main objective
of the time spent at university, so their perception of knowledge and skills learned is
expected to influence their satisfaction with the university experience. Therefore, we
expect:
H1c: Perceptions of cognitive learning outcomes will influence student satisfaction
positively.
Determinants of cognitive learning outcomes
Terenzini (1989) notes that doing an assessment requires reconsidering the essential
purposes and expected academic and non-academic outcomes of higher education. The
8
cognitive learning outcomes can be measured in terms of specific academic achievements
set by the career program or the institution. For instance, Besterfield-Sacre et al (2000),
define the specific learning outcomes for engineering: (a) ability to apply knowledge of
mathematics, science and engineering, (b) ability to design and conduct experiments, as
well as to analyze and interpret data, (c) ability to design a system, component, or process
to meet desired needs, (d) ability to function on multi-disciplinary teams, (e) ability to
identify, formulate and solve engineering problems, (f) an understanding of professional
and ethical responsibility, (g) ability to communicate effectively, and (h) acquiring a broad
education necessary to understand the impact of engineering solutions in a global and
societal context. A department of Geography has set the following as cognitive outcomes
for its majors: (a) interpret maps and other geographical interpretations, (b), analyze the
spatial organization of people, places and environments on the earth’s surface, (c)
comprehend relations between global and local processes, (d) analyze the characteristics,
distribution and mobility patterns of human population on the earth’s surface, (e) apprehend
the complex relations between nature and culture/society, (f) demonstrate knowledge of
geospatial analysis methods and techniques (qualitative and quantitative), (g) present
opposing viewpoints and alternative hypotheses on spatial issues (Duque & Weeks, 2010).
Cabrera, Colbeck and Terenzini (2001) factor-analyze a list of gains reported by
engineering students and found three main learning outcomes: group skills, problem
solving skills and occupational awareness. Thus, cognitive outcomes can be measured at a
more specific or general level. Lizzio, Wilson and Simons (2002) study them as generic
skills developed: problem-solving, analysis, team work, confidence tackling unfamiliar
problems, written communications and planning own work; and they found that the
learning environment (good teaching and appropriate workload) are associated with these
9
self-reported generic skills. Because development of skills and acquisition of knowledge
are dependent on a variety of quality aspects of the university, they are expected to be
influenced by the students’ perception of educational quality (professor competence,
courses, program structure, social environment) and administrative quality (classrooms,
administration, laboratories, library, sport facilities, etc.).
H2a: Perceptions of educational quality will influence perceived cognitive learning
outcomes positively.
H2b: Perceptions of administrative quality will influence perceived cognitive learning
outcomes positively.
Acquiring knowledge (cognitive outcomes) depends on not only the perceptions of
educational and administrative quality. Eagle & Brennan (2007) note that students should
take an active role in their academic experience. This view is coherent with a recent theory
in marketing (The Service Dominant Logic – Vargo & Lusch, 2004), which posits that the
consumer is an actor who co-creates the service by interacting with other actors (in this
case, faculty, classmates, administrative personnel, etc.). Accordingly, one would have a
balanced-centricity view of value creation (Gummesson, 2008) as opposed to a customer-
centricity view whereby students would take a passive role in their educational experience.
Student involvement is a concept recognized in the college engagement literature
(Kuh et al., 2005; Braxton, 2000); and student engagement has been found to be positively
related to student learning outcomes (Pike, Smart & Ethington, 2012). Astin (1999) posits
that students who put more effort and energy into their academic experience obtain better
learning and better personal development. Such involvement would include energy devoted
to studies, time spent on campus, active participation in student organizations, and
10
interaction with faculty members and other students. Thus, in line with other authors (Dann,
2008; Kotzé & Plessis, 2003) we expect that student involvement (co-creation) influences
students’ cognitive learning outcomes.
H2c: Student co-creation will influence perceived cognitive learning outcomes positively.
From the psychological factors, we study emotional exhaustion that is one of the two
components of the burnout syndrome, the other being cynicism (Schaufeli & Taris, 2005).
Emotional exhaustion reflects feelings of fatigue, frustration, burnout, and discontent with
studies (Neumann et al., 1990; Schaufeli et al., 2002). This is, a psychological state where
students have negative thoughts and anxiety regarding their capabilities, which can further
lower perceptions and generate more anxiety, thus reinforcing the probability of inadequate
performance (Bresó, Schaufeli, & Salanova, 2011). Bandura (1982) proposes the social
cognitive theory that relates the student’s well-being (low burnout and high engagement)
with self-efficacy, which then affects academic tasks’ performance and the efficient use of
the acquired knowledge and skills (Bresó, Schaufeli, & Salanova, 2011). Thus, we expect
that emotional exhaustion influences negatively the acquisition of knowledge and skills
(cognitive outcomes):
H2d: Student feelings of burnout (emotional exhaustion) will influence perceived cognitive
learning outcomes negatively.
Determinants of affective learning outcomes
Education involves more than learning facts and skills (cognitive outcomes).
Education also importantly involves affective learning – understanding how the world
11
works and developing a worldview that guides behavior and shapes how people acquire and
use knowledge (Duque & Weeks, 2010). The expected academic outcomes represent the
more concrete cognitive goals, whereas the nonacademic outcomes represent more general
results (affective outcomes) of the students’ whole educational experience (values, goals,
attitudes, self-concepts, worldview, and behavior). Therefore, we expect that if students feel
well prepared academically, this will make them to be more confident about their
achievements, self-concepts and future performance:
H3: Perceptions of cognitive learning outcomes will influence perceptions of affective
learning outcomes positively.
Determinants of student dropout intentions
Dropout intention is the inclination, conscious and discussed, to leave the university
or to end one’s studies (Bean, 1985). Suhre, Jansen, and Harskamp (2007) note that few
dropout studies consider student satisfaction as a key variable, and claim that this is a very
likely factor influencing students’ persistence at university. These authors found that
degree-program satisfaction has a strong negative effect on students’ dropout intention.
Their study also showed that satisfaction plays a role in students’ motivation, which affects
study habits, tutorial attendance and performance. De Jager and Gbadamosi (2010) also
found a significant and negative relationship between overall satisfaction with the
university and the intention to leave it. Metzner (1989) found that satisfaction is negatively
related to intent to leave, which has a direct impact on real dropout from college. We thus
expect that student satisfaction together with the more general evaluation of the university
experience learning (affective outcomes) directly influence dropout intention: the more
12
satisfied and the higher the perception of affective learning outcomes, the lower the
intention to leave the university. Therefore, we expect:
H4a: Student satisfaction will influence students’ dropout intention negatively.
H4b: Perceptions of affective learning outcomes will influence students’ dropout intention
negatively.
Methodology
The conceptual model includes variables coming from different streams of research;
variables which we propose will affect the students’ perception of their experience at
university. We examine the model’s applicability to various contexts, with different student
populations and at different levels (departmental and program level), to assess if the model
is appropriate for use, if it has the ability to explain the dependent variables in the model
across institutions, and if it can offer practical utility for decision making.
We develop four studies that test fragments of the framework. Study 1 presents a
basic model that includes overall service quality, overall learning outcomes, student co-
creation, and student satisfaction, and is tested using a sample of 235 Spanish students of
economics. Study 2 considers the same variables, but overall quality is separated at the
dimension level (educational and administrative). This model is tested using 191
Colombian students of business administration. Study 3 considers the same variables as
those considered in Study 2, but instead of overall learning outcomes, they are separated in
cognitive and affective outcomes. This more complete model is tested using 79 American
students of geography, and cognitive outcomes are measured in a very geography-specific
way. Finally, Study 4 considers the same variables as those considered in Study 3, and adds
13
two variables: a psychological factor of emotional exhaustion (burnout), and dropout
intention as the final dependent variable in the model. In this study, to validate the more
complete questionnaire, cognitive outcomes are measured in a general way to fit two
programs: Study 4_bus is tested using 284 Spanish students of business administration, and
Study 4_nur is tested using 192 Spanish students of nursing. Figures 2 and 3 present the
paths summarizing these studies. The sample sizes of these studies are not representative of
the students’ population of each university or country; thus, estimation results are not
comparable. The studies will show the applicability of the framework to different programs
and to different university levels (departmental and program level). Table 2 shows the
descriptive of the studies’ data sets.
Table 2 about here
The methodological approach consists of a base questionnaire adapted/extended to
the specific contexts and undergraduate programs. Traditional measures from the literatures
are included in the questionnaire or are adapted for this specific context: service quality
(Dabholkar et al., 2000; Hennig-Thurau et al., 2001; Martensen et al., 2000), co-creation
(Neumann et al., 1990; Kotzé & Plessis, 2003; and designed items to cover diverse facets
from Astin, 1999), exhaustion-burnout (Neumann et al., 1990; Schaufeli et al., 2002),
learning outcomes (Lizzio et al., 2002; Lundberg, 2003; Bean, 1985; Zhao, et al., 2005),
student satisfaction (Selnes, 1993; Martensen et al., 2000), and dropout intentions (Bean,