Page 1
1
Distance Education
Technology in Asia
Edited by
Jon Baggaley & Tian Belawati
on behalf of the PANdora network
and its research teams in
Bhutan, Cambodia, China, Hong Kong (China),
India, Indonesia, Laos, Mongolia, Pakistan,
the Philippines, Sri Lanka, Thailand, and Viet Nam
The project
PANdora: Distance and Open Resource Access
is supported by
the PAN Asia Networking (PAN) initiative of the
International Development Research Centre (IDRC)
under a grant (2005-08) to the
Virtual University of Pakistan (VUP)
February 2009
Page 2
2
Contents Page
vii Authors
xii Dedication: Vidya Jyothi Professor V. K. Samaranayake
(1939 – 2007)
(Sri Lanka): A.R. Weerasinghe
1 Foreword
(Canada): Jon Baggaley
(Indonesia): Tian Belawati
(Pakistan): Naveed Malik
(Philippines): Felix Librero
(Singapore): Maria Ng Lee Hoon
15 1) The Emergence of Distance Education in South Asia
(Bhutan): Sangay Jamtsho & Sonam Rinchen
(India): Sanjaya Mishra & Zeba Khan
(Pakistan): Nazir A. Sangi & Sheeraz Ahmed
(Sri Lanka): V.K. Samaranayake (sub-project leader),
P. Wimalaratne, K.P. Hewagamage &
Dilhari Attygalle
51 2) Accessibility, Acceptance and Effects of Distance
Education in South Asia
(Bhutan): Sangay Jamtsho & Sonam Rinchen
(Pakistan): Nazir A. Sangi & Sheeraz Ahmed
(Sri Lanka): Gihan N. Wikramanayake (sub-project leader),
P. Wimalaratne, K.P. Hewagamage &
Dilhari Attygalle
73 3) Development of ICT-based Distance Education
in Bhutan
(Bhutan): Sangay Jamtsho (sub-project leader)
(Canada): Mark Bullen (sub-project advisor)
Page 3
3
105 4) E-learning in Chinese Schools and Universities
(China): Chen Li (sub-project leader),
Wang Nan & Chen Hui Na
133 5) Attitudes to Distance Education in China
(China): Chen Li (sub-project leader) & Wang Nan
153 6) Distance Education Policy and Practice in Mongolia
(Mongolia): Amarsaikhan Dashtseren (sub-project leader)
& Oyun Sanjaa
(Hong Kong): Weiyuan Zhang (sub-project advisor)
173 7) Distance Education Policy and Awareness in
Cambodia, Lao PDR, and Viet Nam
(Cambodia): Doung Vuth (sub-project leader) &
Chhuon Chanthan
(Lao PDR): Somphone Phanousith & Phonpasit Phissamay
(Viet Nam): Tran Thi Tai & Vu The Binh
195 8) Open-Source Software for Learning Management
(Mongolia): Batpurev Batchuluun (sub-project leader)
(Sri Lanka): K.P. Hewagamage, K.H.R.A. Peiris &
W.A.U.C. Weerakoon
211 9) Developing an Asian Learning Object Repository
(Cambodia): Chhuon Chanthan
(Indonesia): A.P. Hardhono & Tian Belawati
(sub-project leader)
(Thailand): Sunee Silphiphat & Tanit Pusiri
231 10) Mobile Technology in Nonformal Distance Education
(Philippines): Angelo Juan O. Ramos (sub-project leader) &
Jerome P. Triñona
257 11) Training Asian Instructional Designers
(Philippines): Felix Librero (sub-project leader)
273 12) E-assessment Methods for Student Evaluation in Asia
(Pakistan): Nazir A. Sangi (sub-project leader)
Page 4
4
301 References
315 PANdora publications
317 Index
(…)
Page 5
5
Chapter 12
E-assessment Methods for
Student Evaluation in Asia
(Pakistan): Nazir A. Sangi (sub-project leader)
The chapter discusses issues and problems relating to online
assessment in teaching and learning. The benefits and problems of
e-assessment are discussed, and issues specific to Asian
institutions, including the availability of appropriate technology.
The potential of e-assessment in open and distance learning is
examined via a survey of academic faculty and administrators. An
e-assessment procedures checklist is offered to focus discussion on
the issues and problems of e-assessment in Asian countries.
The need for assessment
Reliable assessment methods are essential for grading and certifying
students and as a basis for selecting appropriate pedagogical methods
and materials. E-learning is now widely used as a supplementary or
exclusive educational approach, and requires complementary
methods for the assessment of teaching and learning. As information
and communication technology (ICT) continues to develop, online
assessment (e-assessment) offers particular promise for the evaluation
of students at a distance. It has also become a topic of discussion and
difference among educationists, for it presents specific problems in
relation to academic, social, administrative, technological, and moral
issues. In the flexible context required of open and distance learning,
the challenge of reliable student assessment poses significant
problems. Fairness to the student, objective testing of knowledge, the
capacity of students to respond in electronic mode, and the possibility
Page 6
6
of online impersonation and cheating, are significant challenges for e-
assessment research and practice. These issues are being studied in a
PANdora research project named E-assessment methods and models for
student evaluation in Asia, with specific reference to e-assessment
models and methods being used in Pakistan. This report reviews the
current approaches to e-assessment, and their problems and
potential.
What is assessment? Why it is needed? What types of assessment are
there? What are the pros and cons of e-assessment? What are the tests
and methods through which assessment can be made? What are the
goals of assessment? How should assessment itself be assessed?
Many researchers have considered the effects of assessment in the
curriculum. Barnes et al. (2000) have shown direct causal links
between changes in assessment method and classroom practice. Klein
et al. (2000) indicate that changes in measurement criteria can
dramatically, even falsely skew the outcomes of assessment
procedures, while Black & Wiliam (2002) have shown that well-
designed formative assessment can be associated with major gains in
student attainment on a wide range of conventional measures. A
major literature review commissioned by the EPPI Centre (2002)
showed that regular summative assessment can have a negative effect
on the performance of low-attaining students, but does little harm to
high-attaining students.
In other words, without a reliable mechanism, assessment may not
achieve the effects planned and expected by faculty members and
institutions. It may actually work against these objectives. It is
therefore important to develop high-quality measurement procedures
which ensure the sanctity of the evaluation process. The assessment
should reflect core educational goals and achievable rewards for the
students (and teachers) which will be of long-term benefit to them
and to the global society in which they are expected to work
(Ridgway et al., 2004). As with any pedagogical approach, it is
important to align teaching and learning activities and assessment
tasks, particularly where the intention is to encourage deep, rather
Page 7
7
than superficial approaches to learning (Biggs, 2003; MacDonald,
2005). Efficient feedback and interaction between designers,
deliverers and assessors of learning outcomes produces a mechanism
for continuous improvement.
1. Types of assessment .
• Formative
Formative assessment is performed during the learning process
for the purpose of having a positive early effect upon it. Such an
assessment involves two steps: 1) the student’s learning is
identified, and 2) feedback is provided to the student. The data
can be obtained by either the teacher or student (self-assessment).
• Summative
Summative assessments measure the learning achieved at the end
of the process. Detailed data are usually required so that the
student’s learning can be thoroughly checked. This form of
assessment does not normally involve feedback to the student.
• Diagnostic
Diagnostic assessment is often used as an initial step in
programme development, as a means to check whether the
student has certain entry-level prerequisite skills. The numbers of
students assessed in entry-level exams can be huge, and an
automated assessment methodology may be necessary for their
timely evaluation.
• Adaptive
Adaptive assessments change themselves as the assessment
progresses. For example, a student’s response to early questions
may alter the direction or difficulty of subsequent questions.
Page 8
8
Adaptive testing can be used for formative, diagnostic, and (less
commonly) summative purposes, and automated procedures are
vital for its smooth operation.
2. Question formats
E-assessment methods can provide different question/answer formats
to support many types of test. In some cases, the evaluation can be
fully automated, whereas in others requiring, for example, detailed
descriptive answers an instructor may be needed for scoring and
grading. E-assessment questions can be formatted in three ways.
• Closed-ended choice formats: e.g.,
- simple binary choices (yes/no, true/false);
- multiple-choice (choose one correct response);
- multiple-response (choose all correct responses);
- numeric entry (enter the correct number);
- slider (move pointer to the correct value); and
- hotspot (click on an image to identify a correct response).
• Open-ended formats
These usually require text-based answers varying from a single
word or value to sentences and paragraphs; e.g.,
- fill in the blank (insert the missing words or values);
- short answer (free-form text field); and
- essay answer (long-response text field, likely to require manual
grading).
• Ordering formats
The question demands a response involving rank ordering: e.g.,
Page 9
9
- drag and drop: place objects into the correct locations;
- order objects: rank objects according to the given criteria;
- match item: connect the objects in pairs; and
- connect the points: create an ordered connection of a set of
points.
3. Test formats
• Attitude surveys
Opinion-related assessment is obtained through attitudinal
surveys. Attitudes in this context relate to different aspects of
learning such as the course and its elements, the learning process,
and the discipline. The simple Likert Scale or modifications of it
are commonly used. Attitudinal surveys take many forms though
usually involve a statement requiring a scaled agree/disagree
response. Obtaining accurate attitudinal ratings can be difficult,
however, as opinions can be formed by obscure biases and
subjective elements such as family pressure, and economic or
political conditions (Seymour et al., 1997).
• Student Assessment of Learning (SALG)
.
The SALG (Seymour et al, 1997) is an example of a web-based
instrument used to grade levels of achievement or
understanding. It consists of statements about the degree of ‘gain’
(on a point scale) which students perceive they have made in
aspects of the class. Instructors can add, delete, or edit questions.
The instrument can be administered electronically with summary
results made instantly available in statistical and graphic form.
Such instruments are powerful tools and can easily be customised
for efficient formative evaluation during a course or for faculty
assessment.
Page 10
10
• Conceptual diagnostic tests
Students are often required to memorise theory, and can easily
overlook sub-concepts. A conceptual diagnostic test presents
items in a multiple-choice or short-answer format designed with
common misconceptions in mind. Such tests can be applied to
assess how well students understand and apply key concepts.
Immediate feedback on the level of class understanding is
available. Instructors have reported substantial improvements in
class attendance and attitude toward the course through the use
of such tests (Zeilik, 2007). Self-diagnostic tests can be created,
and can aid in refining the thought process and overcoming
biases. Diagnostic tests can easily be automated in an e-
assessment mode, and can have a direct impact on the student’s
development.
• Performance assessment
Performance assessments measure a student’s ability to use
specific knowledge and skills. They usually require the student to
manipulate available knowledge to solve specific problems or to
perform a conceptual analysis of various situations. Multiple-
choice and fill-in-the-missing items questions can be used. An
efficient performance assessment can reveal various problem-
solving approaches, and provide insights into the student's level
of conceptual and procedural knowledge (Slater, 1997).
Performance tests can be useful in disciplines involving scientific
problems solving. However, a student’s analytical growth can be
difficult to measure accurately, and a more complete picture of
student achievement can be achieved when performance
assessments are used in conjunction with traditional forms of
assessment.
Page 11
11
• Rubrics
Rubrics (scoring tools) are a way of describing evaluation criteria
(grading standards) based on the student’s expected performance
and outcomes. Typically, rubrics are used in grading written
assignments or oral presentations, though they may be used to
score any type of performance. Each rubric consists of a set of
scoring criteria and point values associated with them. In most
rubrics, the criteria are grouped into categories so that the
instructor and student can identify the categories with specific
levels of performance. In classroom use, the rubric aims to
provide an objective external standard against which student
performance can be compared. Rubrics sometimes generate
conflicting scores on a given evaluation criterion, but generally
provide an accessible means of communicating and developing
assessment learning goals.
• Portfolios
Portfolios are a collection of evidence prepared by the student
and evaluated by the faculty member, to demonstrate mastery,
comprehension, application, and synthesis of concepts. In
creating a portfolio, students organise, synthesise and describe
their achievements, and communicate what they have learned
(Slater, 2007). Portfolio assessment strategies provide a useful
structure for long-duration, in-depth assignments. The use of
portfolio techniques transfers much of the responsibility for
demonstrating concept mastery from the teacher to the student.
Online methods of e-portfolio development are rapidly emerging.
Survey
To investigate current awareness of these problems, and possible
solutions to them, a baseline survey was conducted in higher
education institutions of Pakistan, with the following main objectives:
Page 12
12
a) to collect information on operational policies and practices in e-
assessment;
b) to identify the current scope of e-assessment activities; and
c) to collect information on current practices and preferences
relating to e-assessment.
1. Methodology.
A survey of e-assessment experiences in public and private
educational institutions of Pakistan was conducted between October
2006 and March 2007. The first examined issues relating to the
administration of e-assessment, and was targeted at institutional
controllers, examination officers, faculty, and managers officially
engaged in examination planning, implementation, and results
reporting activities. The second questionnaire targeted faculty
members who evaluate students in their specific courses. The third
questionnaire targeted students, to find out if they had been involved
in an ICT-based assessment. Certain questions were repeated in the
three questionnaires, in order to obtain input on common issues and
practices. The questionnaire containing multiple-choice, closed-end
questions were distributed to each group of participants. In a few
cases, electronic copy was e-mailed to the participants. The
questionnaires were sent to approximately half of the 113 degree-
awarding institutions in Pakistan, and mainly to those with 2,000 or
more student enrolments.
Responses to the questionnaires were received from 85 staff members
and 60 students in 31 institutions, including Allama Iqbal Open
University (AIOU), the Virtual University of Pakistan (VUP), and 29
traditional universities. A few of the latter had initiated pilot
programs in DE and online education. The responses were analysed
in four categories:
1) assessment practices (as defined by the institutions’ policies);
2) the scope of assessment activities;
Page 13
13
3) the current status of e-assessment activities; and
4) preferences/ reservations related to e-assessment.
The first two of these sections mainly represented the views of the
institutional administrators, while the faculty viewpoint was
represented in section 3. The responses of all three of the sample
groups (institutions, faculty, and students) were represented in
section 4.
2) Results.
• Assessment practices
Pakistan’s educational institutions have adopted assessment
policies and practices covering a wide range of activities,
generally relating to summative assessment performed by their
examinations controllers. They have also defined standard
practices for the implementation of formative assessment, and
post-assessment analyses. Issues as to who should prepare the
question papers, and when, where and how the assessments
should take place are described in the institutions’ examination
regulations.
� Examination system
From the responses of institutional examinations controllers
about general exam procedures, the study identified that 84%
universities follow a semester-based exams system, only 6%
follow an annual exams system, and 10 % use a mixed approach
(both examination systems). The semester system was introduced
in Pakistani universities in the early 1970s. Previously, all public-
and private-sector educational universities in the country
followed an annual examinations system. Gradually the semester
system dominated, and many institutions converted their
programmes to a semester basis. In a few cases, this development
Page 14
14
increased the workload beyond the capacity of the institutions’
examination departments, and the publication of results was
delayed. For this reason, some universities reverted to the annual
system, and to this day the semester system is used in some
programmes and an annual system in others.
� Number of final papers assessed annually
Forty-eight percent of the responding institutions conduct their
terminal/ final (summative) examinations twice a year, while 23%
of them conduct terminal examinations three times a year.
Universities in the latter category add a summer semester, and
therefore have three semesters annually instead of two. One
university practises an unusual quarterly system, conducing
examinations four times a year.
� Marking schemes
Which marking scheme is used to prepare the student’s final
result? The survey responses show that most of the responding
institutions (68%) use a Grade Point Average (GPA) system to
generate the student’s results, whereas 29% do not follow a GPA
system. If we remove from the analysis the 10% of institutions
which follow an annual system, we find that 19% of the
institutions use students’ percentage scores to determine their
grades (A, B, C, etc) or divisions (first, second or third), as in most
annual systems. Interestingly, many students and institutions
favour both types of results presentation, and the final student
transcripts are prepared accordingly.
� Evaluation methods
As a common practice, Pakistani universities use either external
or internal assessment methods of student evaluation. Most
institutions that use a semester system have internal procedures
for both formative and summative examinations. In annual
Page 15
15
examination systems, however, external evaluations are usually
conducted. A few institutions deviate from this rule in
implementing the generation of question papers and answer
scripts with the help of an external examiner. Summative
assessment is conducted using a centrally controlled examination
system at almost all of the responding institutions.
� Paper Setting
a) Internal Examinations: The question was asked: who prepares
the question papers in internal/ formative assessments? The
survey results show that 68% of the institutions use internal
faculty members as paper setters, while 10% of institutions use
external examiners for this purpose. An appreciable number of
institutions (23%) use a combination of external and internal
examiners for paper setting and moderation.
b) Final Examinations: The procedure used for summative
assessment is almost the same as the above. The survey responses
show that 61% of the responding institutions use internal faculty
members as paper setters, while 13% use external examiners.
Twenty-six percent use a combination of the two approaches.
In the semester system, the internal faculty members usually
prepare the formative and summative question papers. They are
regarded as more reliable for maintaining secure standards of
efficiency and secrecy within the institutions. This method is
occasionally criticised, however, for relying on individuals for all
academic aspects of the process, and for the monotonous results
that can arise when exam questions are used repetitively over
extended periods. These are common reasons that some
institutions favour a combination of external and internal faculty
members as paper setters.
Page 16
16
� Final Paper Implementation.
The final (summative) examination process is usually centralised.
The survey results indicate that 71% of the institutions use
internal faculty members to assist in this centralised process,
whereas in 13% of the institutions, where university operations
are spread over many locations or there are many students, a
decentralised examination system is used. Sixteen percent of the
institutions responding to the survey did not answer this
question.
• Assessment activities.
Analysis of the survey responses identifies the activities on which
basis students are electronically assessed. In general, the semester
system uses assignments, quizzes, tests, and laboratory projects
as part of formative assessment. In summative assessment, a final
paper is administered for theoretical courses. Otherwise, term
papers and reports, laboratory tests, projects, and oral
presentations are assessed. When asked about these assessment
activities, the survey responses indicated that individual faculty
members use different assessment methods in different courses.
All of the reported activities have a high frequency of usage:
assignments (97%), class tests (90%), quizzes (84%), class
presentations (81%), class attendance (74%), and projects (71%).
� Types of question in summative assessment
Traditionally, essay-type questions have generally been the main
method used in final/ summative assessments. This trend appears
to be changing, however. In the current survey, 55% of the
responding institutions stated that they use an essay-type format
for only approximately 1:4 of the questions in their final papers,
and an equal mixture of fill-in-the-blanks, multiple-choice,
true/false and short-answer formats in the remaining questions.
This result indicates the value of automated assessment systems
Page 17
17
to support paper generation and evaluation of these different
assessment formats. A further 17% of the institutions stated that
3:4 of their questions are essay-type; and 14% reported that about
1:2 of their questions are essay-type.
• Are essay type questions necessary?
Descriptive questions can help in assessing the knowledge and
intellectual level of the student. It can be difficult and time-
consuming, however, to evaluate essay-type descriptive
questions; and the marking scale may vary between the
individual graders. These are serious practical problems,
especially in DE institutions with mass enrolments. The survey
asked about the need for descriptive/ essay-type questions. A
majority (65%) of the faculty respondents regards descriptive
questions as necessary in assessment, and only 19% of them do
not. Sixteen percent of the faculty did not respond to this
question.
• Present status of e-assessment
Useful information was obtained on how far institutions have
progressed in the development of e-assessment methods, and
about their experiences and difficulties. The majority of
responding institutions (58%) do not have a computerised
examination system owing mainly to lack of sufficient funds,
resources, and skilled manpower; while 28% felt strongly that
computerisation is not developed because they do not need it.
A question was asked about the examination-related activities
that are conducted manually, electronically, or in mixed mode. In
most institutions, activities are performed manually (paper-
setting, examination implementation, script evaluation, grading,
and results sheet preparation). The status quo of examination and
e-assessment activities in Pakistani universities is shown in
Tables 1 and 2.
Page 18
18
Table 1. Current status of examination activities (%).
Activities Manual Electronic Mixed No Resp
Paper Setting 48 26 26 0
Examinations
implementation 61 20 16 3
Script evaluation 77 10 10 3
Grading 58 23 16 3
Results sheet
preparation 13 61 23 3
Results display 20 45 32 3
Student records
maintenance 16 39 32 13
Table 2. Institutional e-assessment activities (%).
E-assessment activities Yes No No Resp
E-tests 58 32 10
T-tests by other organisations 42 48 10
Formative assessment 32 58 10
Summative assessment 26 64 10
E-assessment support for faculty 29 61 10
Contribution to e-testing projects 13 77 10
• Technology failures
E-assessment methods appear to be emerging in various
institutions at an experimental level, restricted to isolated
courses. Most institutions use e-assessment for pre-admissions
tests. Only a few use electronic means of assessment, however,
for they lack the technology infrastructure for them. When asked
about the common causes of failure in e-assessment technology,
only 35 faculty members (41%) replied, identifying the following
Page 19
19
problems: network faults (30%), software (27%), power failure
(22%), and hardware (8%).
• Faculty preferences for e-assessment
The survey indicated preferences for e-assessment methods
among the faculty members of the responding institutions, and a
very positive about the potential of e-assessment. Several issues
and concerns need to be addressed before it can be effectively
implemented, however. A major issue is the cost-effectiveness of
e-assessment systems. The survey showed that 58% of
respondents consider it an expensive option, and that 10%
consider it very expensive. On the other hand, 23% respondents
regard it as cheaper than traditional assessment methods. The
latter respondents are likely to be the early adopters of e-
assessment, as long as their other concerns can be met. The main
investment items for institutions in relation to e-assessment are:
hardware, software, and manpower. The importance of these
factors was positively rated (‘strongly agree’ and ‘agree’) by 84%
of the sample. The issue of maintenance was rated as important
by 74%. Thirteen percent of the sample did not respond to this
question.
• Types of evaluation support needed
As identified earlier, institutions use different combinations of
question formats in assessment, including descriptive/ essay-type
and objective types (multiple-choice, fill-in-blanks, true/false,
etc.). Multiple-choice assessment is the most commonly
automated, while the most difficult to automate is the essay-type,
descriptive answer which requires individual evaluation by a
faculty member. Considering the large volume of essay scripts in
DE institutions, this is a critical area for consideration in e-
assessment planning. The survey asked about the need for
specific types of e-assessment support, and the responses were
highly in favour of support for multiple-choice question formats,
Page 20
20
with good demand for support of case study formats (Table 3).
The importance of the formats was rated on 5-point Likert scales
from ‘strongly agree’ (SAgr) to ‘strongly disagree’ (SDis), with a
midpoint of ‘uncertain’ (Unc).
Table 3. Question formats needing e-assessment support (%).
Importance of
exam formats SAgr Agr Unc Dis SDis
No
Resp
Institutional responses (%)
Essay (subjective) 48 14 3 21 3 11
Objective (MCQ) 79 10 0 0 0 11
Case Studies 34 21 14 10 0 21
Faculty responses (%)
Essay (subjective) 42 22 11 11 4 10
Objective (MCQ) 51 43 6 0 0 0
Case Studies 34 50 9 4 3 0
• Software support for e-assessment administration
Institutional administrators indicate that their exams
administration systems have three major challenges: software
and operational support for the implementation process, secrecy
of assessment documents, and authentication of students at
distance. Common implementation issues include the
development of examination centres, and the need for assessment
documents and staff at the centres, and student lists. Security and
secrecy of assessment documents (question paper and answer
scripts, etc.), and the authentication of students are common
institutional issues. When the faculty members were asked about
these issues, however, the responses were more divided, possibly
owing to their lesser experience in examinations administration
and implementation. Their responses nonetheless indicate the
general importance of software support for these activities (Table
4). The importance of the types of support was rated on 5-point
Page 21
21
Likert scales from ‘strongly agree’ (SAgr) to ‘strongly disagree’
(SDis), with a midpoint of ‘uncertain’ (Unc).
Table 4. Software support needed by e-assessment (%).
Importance of
software support SAgr Agr Unc Dis SDis
No
Resp
Institutional responses (%)
Implementation 69 10 4 0 0 17
Security 72 4 4 0 3 17
Authentication 65 7 0 0 0 28
Faculty response s (%)
Implementation 28 47 12 9 1 3
Security 43 28 16 3 5 5
Authentication 43 28 9 6 3 11
These results indicate the major need for basic electronic methods
in the assessment implementation process. Support for such
issues as student authentication and secrecy is considered
necessary for the reliability, uniformity and transparency of
assessment, especially at a distance. In online DE institutions, this
requirement needs to be carefully handled, as students are in
large number and in remote areas. The implementation of e-
assessment faces major challenges with respect to the high
probabilities of technological failure and ineffective student
authentication.
• Student authentication in e-assessment
Authentication of students is a sensitive issue in e-assessment
implementation. The survey showed almost unanimous
agreement on the importance of this issue, and on the need for all
the authentication methods proposed. Sixty-five percent of
faculty members accept (agree or strongly agree) the value of
traditional, supervised authentication. Other methods such as
password access, finger-printing and camera-based
Page 22
22
authentication are also highly desired (Table 5). The importance
of the authentication methods was rated on 5-point Likert scales
from ‘strongly agree’ (SAgr) to ‘strongly disagree’ (SDis), with a
midpoint of ‘uncertain’ (Unc).
Table 5. Student authentication methods in e-assessment.
Importance of
authentication method SAgr Agr Unc Dis SDis
No
Resp
Supervised /manual 23 42 3 23 3 6
Password access 65 29 0 3 0 3
Finger printing 71 7 6 10 3 3
Camera/ image 45 19 10 13 7 6
Each of these methods now needs to be researched with regard to
its technological implementation, ease of use, reliability, and the
effects of network congestion.
Discussion
The literature review and survey of e-assessment practices illustrate a
series of issues.
1. Economic issues.
E-assessment can play a valuable role in educational measurement,
and especially in DE. Assessment data are often costly to collect when
travel is required, and if an analysis can be automated by coding the
questions and responses, such costs can be avoided. On the other
hand, e-assessment involves the costs of electronic equipment (e.g.
computers), reliable software, bandwidth, specialised manpower,
training, and the time involved in developing test questions.
Additional costs of maintenance, secrecy, security, and trouble-free
operations are also involved (Scottish Qualifications Authority, 2004).
In the short to medium term, e-assessment tools can be expensive to
Page 23
23
implement, and traditional assessment methods may be less
expensive. Once created, however, e-assessment applications become
less expensive, being easy to operate and score, and reuseable from
year to year in different combinations (Ridgway et al., 2004). Most of
the administrative colleagues surveyed during the current project
suggested that e-assessment systems can be more cost-effective than
traditional assessment methods, but others felt that the hardware,
software, manpower and maintenance costs represent difficult
obstacles to e-assessment in developing countries.
2. Technological problems.
E-assessment systems must be stable in order to generate valid and
reliable assessments results (Qualifications & Curriculum Authority,
2007). The system requires reliable hardware, software, network and
power systems at all testing centres. Data management and
operational security issues can jeopardise the system’s reliability.
When conducting tests at multiple locations, synchronisation between
centres is also important. Depending on the specific e-assessment
system in use, dedicated testing centres may be needed with an
ongoing requirement for technical support throughout the
assessment period. Responses by administrators to the sub-project
survey have indicated the importance of alternative network links
and increased security measures while online exams are being
conducted.
3. Administrative/ operational issues.
The administration of e-assessment requires careful and continual
record-keeping within a failure-free/fault-tolerant system. To ensure
the system’s reliability, a well-trained staff is needed at all testing
centres (Scottish Qualifications Authority, 2004). In addition,
automated polices and security procedures are essential, with
electronic and manual student authentication procedures. Specialised
Page 24
24
skills are required to produce high quality e-assessments, and many
teachers fear that e-assessment will de-skill their profession. Some are
threatened by the introduction of e-learning and e-assessment, and
suspect the motives involved. Such fears need to be handled by the
institution. In the current project survey, students were asked which
assessment components (assignments, tests, etc) they would welcome
in an e-assessment context. The majority favoured the electronic
submission and grading of assignments, quizzes, class tests, and final
exams. This indicates an early acceptance of e-assessment systems
among this group of stakeholders.
4. Social/ ethical issues.
The electronic manipulation of information in e-learning systems
raises major social and ethical issues (Marais et al., 2006). An e-
assessment system should above all be fair to students. The student’s
ability to respond electronically can affect his or her capacity to
succeed in the assessment, and the collection of e-assessment data
may prejudice the interests of non-computer literate students. In
addition, losses of service in automated systems through power,
equipment, software or network failure may cause loss of time and
mental composure for the students, as well as actual loss of data.
Thirdly, there is the danger that expert computer users could gain
access to, manipulate, copy, and misuse the answer scripts. These
problems require careful policy decisions and preventive measures.
5. E-assessment in Pakistan.
The results of the project’s survey confirm that uses of e-assessment
in Pakistan are currently limited, located in individual sections of the
institution and used experimentally without full acceptance by the
institutional management. The Virtual University of Pakistan is using
a relatively comprehensive e-assessment system. The National
Testing Services in Pakistan, Allama Iqbal Open University (AIOU),
Page 25
25
and a few other institutions are using e-assessment as an occasional,
alternative option. The wider use of e-assessment systems clearly
depends on the identification of solutions to the technical and
procedural hurdles that threaten their stability. In order to develop
better and robust e-assessments, a checklist of e-assessment
procedures is required to anticipate these problems, and to advise
institutional policy, assessment, security and authentication
procedures. Such a checklist might not initially cover all constraints
and questions, but it could focus attention on the common issues and
suggest the best approaches for specific institutions and types of
faculty member. In the next section, an e-assessment procedures
checklist is proposed, and the work being undertaken in the current
project to implement the procedures via a software prototype is
discussed.
An e-assessment procedures checklist
An e-assessment system has three active stakeholders: the student,
faculty, and institution. The students and faculty members are
regarded as end-users in the system while the institution is
represented as its administrator/ manager. Faculty may support the
development of assessment objects and generate assessments on the
institution’s behalf. The above survey and discussion has indicated
that all aspects of the system involve challenges. An e-assessment
procedures checklist (EPC), based on the current project’s priorities, is
provided to anticipate many of these issues and challenges (Sangi &
Ahmed, 2008). The checklist is divided into three main stages: Pre-
Assessment, Assessment and Post Assessment (Figure 1).
Page 26
26
Figure 1. The e-assessment procedures checklist.
Pre-Assessment
Policies, rules and
procedures
Institutional interfaces
Assessment objects
Assessment software
Security measures
Assessment
Assessment generation
Assessment
administration
Assessment capturing
Storage and
transmission
Student authentication
Post-Assessment
Marking and grading
Results certification
Post-assessment audit
Assessment analysis
Assessment feedback
Assessment Support System
Registration Authentication Plagiarism Check
Access Controls Record Keeping Support and Maintenance
1. Pre-assessment.
The major tasks in the Pre-assessment stage are the creation of
policies, rules and procedures. Three main policy decisions relate to
e-assessment setup, institutional setup, and rules for conducting the
assessments. In the software currently being designed to implement
the EPC, an Assessment Policy Configuration module enables the
administrator to set up necessary policies and to make decisions
regarding number of tests, quizzes, and assignments in a given
period. With the help of the Personalisation/ Configuration module,
the administrator can personalise the system’s look-and-feel and
make other adjustments according to institutional needs. The
Conduction Policy Configuration module is an important component
of the system, handling assessment generation, centralised and
Page 27
27
distributed testing, prevention of cheating, and control of backup
mechanisms. A security maintenance module handles decisions
relating to encryption, network access control, firewall settings, and
user security checks. Measures to prevent cheating include
randomisation of assessment questions and generation of multiple
exam papers.
2. Assessment.
The Assessment stage handles the generation of the actual tests.
Assessment generation procedures control the selection and assembly
of objects and test items, and their archival storage. Objects/ items
include assignments, quizzes/ tests, mid-term papers, and final term
papers. Assessment objects can be defined at varying levels of
difficulty, and assembled in random formats. Continual storage,
record-keeping mechanisms, and backup support facilities need to be
provided. Encryption rules, editing privileges, and transmission
between servers in distributed assessment environments need to be
carefully controlled. Assessment Storage deals with the real-time
compilation, capturing and archiving of assessment items with
respect to individual students and topics. This function aids the
examinations department in the administration, transmission and
storage of assessments. Secure transmission is an essential and ethical
responsibility of the system. Student authentication procedures
ensure that the right student is present for the test at the specified
place and time. Automated and/or manual authentication should be
available according to the institution’s preferences. The general
procedure of authenticating the students by login names and
password is a minimal requirement.
3. Post-assessment.
The first major component of post-assessment is grading. Automated
or teacher-assisted scoring mechanisms may be required, followed by
Page 28
28
results compilation and certification processes. Institutional templates
for printing certificates and result cards can be defined. A particularly
important component is the post-auditing of the processes and
controls used by the system during an assessment. Post-assessment
audits involve the collection of system performance data. They
provide information about the assessment objects used, student
scoring, and issues relating to the adjustment of grading levels,
certification, and test validity and integrity. They also provide data
on student performance and on whether course objectives have been
achieved. Post-assessment feedback to students, teachers and
administrators assists in the system’s subsequent improvement.
4. Assessment Support Systems.
An important consideration for software based on an EPC is its
interface with existing university systems and processes. These
include the traditional registration procedures, access controls,
record-keeping databases, methods of communication, and software
uses. Specialised routines include student authentication and
plagiarism checking.
Conclusions
This project has conducted a detail review and survey of e-
assessment practices in Pakistan. It has identified a general need for
e-assessment methods, particularly in institutions with high student
enrolments, and in distance education. The survey has highlighted
various assessment practices currently followed by Pakistani
educational institutions. In most institutions, a semester system is
followed, with variations, and many assessment components are
employed: assignments, quizzes, class tests and/or presentations are
the most commonly used, together with other assessment activities
(e.g. lab and project work, and oral examination). Essay-type
questions are common and regarded as an integral part of assessment
Page 29
29
schemes. Electronic systems need to cater to these requirements, and
to be flexible in defining and considering each e-assessment
component.
At present, electronic support for assessment in Pakistan is usually
limited to the examinations section of institutions. Other uses of e-
assessment are located in individual institutional units, and used
experimentally without full acceptance by the institutional
management. This is changing, however, as many institutions are
beginning to recognise the value of e-assessment methods. The
preferences for e-assessment systems vary between electronic
applications for operational purposes: implementation, secrecy and
authentication of students, and maintenance of students’ results and
records. Faculty members would like the scope of e-assessment
methods to be expanded to include new assessment forms other than
simple quizzes and tests.
The survey has highlighted, however, major problems in the
adoption of e-assessment methods. Their current usage is very
limited in Asian education, even in distance education and at
institutions which feature online learning. The survey conducted
during this project has revealed poor conditions for e-assessment in
many Pakistani institutions: lack of Internet access, software and
power failures, overloaded servers and networks, etc. These are major
problems for effective e-assessment, which are likely to increase as
the volume of students and frequency of tests place increasing
pressure on institutional servers and networks: e.g. security of data,
electronic plagiarism, authentication of students at multiple and
distant locations, and operational reliability. The technical and
procedural problems highlighted by other PANdora projects suggest
that these conditions are common in Asian institutions generally.
Cost-effective, secure, efficient, and reliable ICT infrastructure with
nationwide coverage will be crucial for all electronic access, learning
and assessment processes. Their development will need clear
plagiarism/ authentication control policies, training for faculty
members, students, and administrators, and vigilance by faculty
Page 30
30
members and technical staff. As a segment of Asian educators is
totally opposed to e-assessment, and can negatively influence and
manipulate the political will to overcome these barriers, support for
pilot studies must be obtained and the viability of e-assessment and
the growing need for it publicised.
E-assessment security, secrecy and operational issues can be
addressed immediately with the development of open source e-
assessment software. The current project has developed an e-
assessment procedures checklist and software prototype of this type.
The checklist covers a range of assessment practices, levels and
policies, and aims to provide a flexible basis for the reliable e-
assessment practices demanded by Asian institutions. It focuses
attention on the common issues and best approaches for specific
institutions and types of faculty member. To implement the checklist,
the project has generated an open-source, configurable e-assessment
software at AIOU, enabling institutions to adopt the specific
components of the system that they require. The software contains
varied test formats supporting different sorts of formative and
summative assessment, and adding time-saving automation to the
assessment process in the hope that acceptance of e-assessment
among faculty members and students will be increased. Both the e-
assessment checklist and software will be capable of expansion and
refinement as more institutions begin to use them; and it is hoped
that they will mature into a generalised e-assessment model for use
across the Asian region. (Access to this freely shareable software may
be sought from the author.)
The need for e-learning and assessment in Asia is clear. The drive for
a higher quality of education at a relatively low cost, and the
availability of better computing and technical facilities, especially in
remote regions, are major driving forces. As Asian DE institutions
provide education in multiple local languages, a generalised e-
assessment model will be needed to address these and other cultural
factors. The institutions which prepare themselves for this challenge
at an early stage will benefit more from e-learning in future, as their
Page 31
31
student enrolments increase and issues of quality are imposed upon
them by society.
Acknowledgements. The author of this chapter thanks the members
of the e-Assessment team at Allama Iqbal Open University, Dr. T.
Rahman, Sheeraz Ahmed, Dr. Zafar Iqbal, and Ms. Shazia. Many
thanks go to Dr. Naveed A. Malik and Mr. Javed Younis of the
Virtual University of Pakistan, Dr. A.P. Hardhono of Universitas
Terbuka, Jakarta, and to the late Professor V.K. Samaranayake and his
team at the University of Colombo for their support for the research.
Note: An earlier version of this chapter has been published by Sangi
(2008).