Drivers and Barriers for the adoption of Computer Assisted Assessment Geoff Chapman Thomson Prometric July 5th, 2005
Drivers and Barriers for the adoption of Computer Assisted Assessment
Geoff Chapman
Thomson Prometric
July 5th, 2005
Drivers and Barriers to CAA – Thomson Prometric
Rationale
QCA Blueprint for e-Assessment
Vision and Timeline
Systematic and independent Awarding Body survey
Understanding of needs and concerns
Drivers and Barriers to CAA – Thomson Prometric
Research Context
Charts show research findings on a question-by-question basis
Findings broken by sub-group with significant differences– Currently delivering versus not currently delivering e-Assessment– Preferred testing location– QCA vision - Achievable versus Not Achievable– Reasons for not developing/piloting e-Assessment– Willingness to share test delivery locations and test design
with third parties
Few significant differences found amongst these sub-groups
Majority are those who are and aren’t currently delivering e-Assessment
Drivers and Barriers to CAA – Thomson Prometric
Statistical Confidence
87 Awarding Body respondents
Findings give 95% confidence level with an approximate 5% confidence interval
26% of respondents currently deliver e-AssessmentOf these
% of programme 0-30% 31-60% 61-90% Over 90%
% of respondents 61% 22% 9% 4%
Average 29% of current programme being delivered
Drivers and Barriers to CAA – Thomson Prometric
e-Assessment Delivery
Of the 74% of Awarding Bodies who do not currently deliver e-Assessment
– 46% are in the process of developing or piloting
Reasons given by the 54% not in the process of developing/piloting e-Assessment are provided on the right
– Perceived not suitable for nearly a third
9%
9%
9%
11%
17%
20%
20%
Small/new AB
Not suitable - deliver NVQ's
Not suitable - hold practicalassessments
Investigating with tech/swproviders
Resource/cost issues
Still at very early stages
Potential for the future
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Robustness
84% believe e-Assessment to be equally as or more robust than paper-based
– There are few significant differences in responses given by those currently delivering and those not
– Those who believe that over 60% of their programme is suitable for e-Assessment are more likely to see it as more robust than a paper-based system
• 6%
39%
56%
6%
13%
13%
69%
6%
8%
19%
67%
6%
17%
22%
57%
6%
10%
20%
64%
Less robust
Don't know
More robust
Equally as robust
Overall
Currently running
Not currently running
Up to 60% of programmebelieved suitable
Over 60% of programmebelieved suitable
Q. Do you think that e-Assessment is a more robust, less robust or equally as robust method of assessment as paper-based assessment, on the whole? (Prompted)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Leading Awarding Bodies
Majority unable to strongly associate leadership with any UK Awarding Bodies
City & Guilds associated most
Minimal mention of SQA, OCR & AQA
Other ‘Awarding Bodies’ mentioned less frequently were
– Open University– American Certified Public Accountants – Teacher Training Agency
24%
24%
25%
13%
13%
53%
8%
71%
14%
3%
12%
9%
6%
8%
5%
SQA
AQA
OCR
Edexcel
City & Guilds
Other
No mention/don'tknow
1st mention
2nd mention
3rd mention
Q. Which Awarding Bodies do you associate with leadership in the field of e-Assessment? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Market insight: e-Assessment Examples
Overall, the majority of respondents who have seen existing programmes listed, thought that the City & Guilds ‘Gola’ system was the best demonstration
– The Theory Test delivered by the Driving Standards Agency received highest recognition for having the most encouraging outcome
21%
25%
29%
37%
58%
60%
10%
12%
12%
15%
14%
21%
ConstructionIndustry Health &
Safety Test
Institute of theMotor Industry
EDI system forLondon Chamber
of Commerce
City & Guilds’‘Gola’ system
Edexcel’s KeySkills Assessment
Driving StandardsAgency Theory
Test
Seen
Read/heard about
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Q. Which of the following e-Assessment prototypes, trials and existing programmes have you either seen in action, read or heard about? (Prompted)
Drivers and Barriers to CAA – Thomson Prometric
E-Assessment Suitability
Adult basic skills testing perceived to be suited to e-Assessment by the largest proportion of respondents
Other differences shown not significant
3%
10%
24%
40%
16%
6%
12%
33%
28%
13%
6%
16%
30%
24%
15%
3%
15%
39%
23%
10%
Not at allsuited
Not very wellsuited*
Neutral*
Quite wellsuited*
Extremely wellsuited
Vocational
Professional
Schools
Adult basic skills *Definitions derived from 5 point numeric scale
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Q. On a scale of 1-5 where 5 is extremely well suited and 1 is not at all suited, how well suited do you feel e-Assessment is to each of the following types of education? (Prompted)
Drivers and Barriers to CAA – Thomson Prometric
E-Assessment: Suitable Circumstances
0%
10%
20%
30%
40%
50%
Overall Currently delivering Currently not delivering
Q. In your opinion, in what types of situations or circumstances does e-Assessment work particularly well? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
E-Assessment: Unsuitable Circumstances
0%
10%
20%
30%
40%
Data Currently delivering Currently not delivering
Q. In your opinion, what types of situations or circumstances is e-Assessment less well-suited to? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Advantages
0%
10%
20%
30%
40%
50%
60%
70%
80%
Overall Currently delivering Currently not delivering
Q. What are the three main advantages of using e-Assessment, in your opinion, overall? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Disadvantages
0%
10%
20%
30%
40%
50%
Overall Currently delivering Currently not delivering
Q. What are the three main disadvantages of using e-Assessment, in your opinion, overall? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
E-Assessment: Risks associated
0%
10%
20%
30%
40% Overall Currently delivering Currently not delivering
Q. What risks do you associate with the introduction of e-Assessment, if any? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Costs
Some comments for reasoning
Cost effective as long as scaleable – i.e. larger AB’s who deliver more tests could justify high start-up costs, whereas smaller AB’s who run fewer exams may not be able to
Too early for some respondents to determine long-term savings or otherwise (may be easier to realise once up and running for a while)
16%
8%
11%
66%
0%
17%
13%
61%
14%
10%
12%
64%
Don't know
Less
Approx. equalto
More
Overall
Currently delivering
Not currently delivering
Q. Three years after implementation, will e-Assessment be more, less or equally cost-effective as paper and pencil tests?
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Driving Uptake
0%
10%
20%
30%
40% Overall Currently delivering Currently not delivering
Q. Within the vocational/professional assessment arena, what do you believe to be the top three factors driving the uptake of e-Assessment? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Further Driving Uptake
0%
10%
20%Data Currently delivering Currently not delivering
Q. In your opinion, what additional factors could further drive take-up rates of e-Assessment? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Culture for e-Assessment
0% 5% 10% 15% 20% 25% 30%
1
2
3
4
5
6
7
8
9
10
Cu
ltu
re f
or
e-A
ssessm
en
t
%
Not running e-Assessment
Already running e-Assessment
Q. On a scale of 1-10 where 10 is completely and 1 is not at all, how ready is the culture within your Awarding Body for e-Assessment?
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
E-Assessment: Importance Factors
44.5
55.5
66.5
77.5
88.5
99.5
Overall Importance Already running e-AssessmentNot currently running e-Assessment
Q. On a scale of 1-10 where 10 is extremely important and 1 is not important at all, how important are each of the following factors to you in relation to e-Assessment? (Prompted)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Location Preference
Further spontaneous responses include:It depends
– Flexibility/suitability for assessment or candidate (17%)
– Candidate preference (5%)
Place of learning– Convenience/candidate familiarity (10%)– Practicalities/convenience for AB (10%)
Dedicated test centre– Quality controlled conditions (8%)– The only option/tradition/policy (7%)
Place of work– Policy/tradition (5%)– Office facilities available (3%)
13%
19%
31%
38%
17%
39%
17%
22%
14%
24%
28%
33%
Place of work
Dedicated testcentre
Place of learning
It depends
Overall
Currently running e-Assessment
Not currently runninge-Assessment
Q. Where does your organisation prefer to test? (Prompted)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Technology Providers – Assessment Location
0% 10% 20% 30%
1
2
3
4
5
6
7
8
9
10
Sh
are
d D
eli
very
Lo
cati
on
s
%
Not running e-Assessment
Currently running e-Assessment
Q. On a scale of 1-10, how willing would you be to work collaboratively with third party technology solutions providers on shared e-Assessment locations? (Prompted)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Technology Providers – Test Design and Development
0% 10% 20% 30%
1
2
3
4
5
6
7
8
9
10
Willin
gn
es
s t
o C
olla
bo
rate
%
Not running e-Assessment
Currently running e-Assessment
Q. On a scale of 1-10 where 10 is completely and 1 is not at all, how willing would you be to work collaboratively with third party technology solutions providers on test design and question development? (Prompted)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Technology Provider Competencies
• Cost effectiveness was highest 1st mention
• Overall adaptability, IT skill/support and affordability are top three attributes required in a technology provider
• Understanding of the market, reliability and flexibility feature as lower mentions
10%
12%
15%
16%
12%
17%
9%
6%
7%
9%
8%
14%
6%
10%
1%
2%
3%
6%
7%
13%
20%
Reliability
Understanding ofmarket/industry
Reputation/trackrecord/credibility
Experience/expertise
IT skills/support/customerservice
Adaptability/ability tolisten/tailor solutions
Costeffectiveness/affordability
1st mention
2nd mention
3rd mention
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Q. What are the top three competencies, attributes or qualities would you seek in a third party technology provider? (Spontaneous)
Drivers and Barriers to CAA – Thomson Prometric
QCA & FAB
Yes 22% (Currently delivering: 39%, Not currently delivering: 16%)
No 76% (Currently delivering: 57%, Not currently delivering: 83%)
Yes 36% (Currently delivering: 35%, Not currently delivering: 36%)
No 39% (Currently delivering: 26%, Not currently delivering: 44%)
Don’t know 25% (Currently delivering: 39%, Not currently delivering: 20%)
Yes 77% (Currently delivering: 78%, Not currently delivering: 77%)
No 8% (Currently delivering: 0%, Not currently delivering: 11%)
Don’t know 15% (Currently delivering: 22%, Not currently delivering: 13%)
Q. Do you believe that the QCA’s vision of full e-Assessment adoption by 2009 is achievable in the vocational/professional market?
Q. Do you perceive QCA to have the competencies to assess the technical capabilities and securities of e-Assessment used by Awarding Bodies?
Q. Do you believe that the FAB should adopt a significant and active role in driving e-Assessment knowledge?
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Future of e-Assessment
2%
7%
15%
31%
39%
0%
100%
60 - 99%
31 - 59%
1 - 30%
6%
9%
11%
25%
42%
0%
100%
61 - 99%
31 - 59%
1 - 30%
Q. Approximately what percentage of your assessment programme do you realistically expect to be e-Assessment by 2009? (Spontaneous)
Q. What percentage of your programme do you believe could potentially be assessed by sophisticated multiple choice questions? (Spontaneous)
Base: 87 (23 Already running e-Assessment; 64 Not currently running e-Assessment)
Drivers and Barriers to CAA – Thomson Prometric
Further Feedback
Verbatim:
“There is a need for a clear definition of e-assessment. People tend to think it is very simplistic which is clearly not the case”
“It would be useful to receive information on what is going on with the developments and providers of e-assessment”
“E-Assessment is not a solution for everything but it is a very good solution for some things, for us this is only a small portion of we need”
“We are very willing to have discussions as to what an outside body could recommend for e-assessment for our subject”
“I have a worry that Government cuts in funding for qualifications will cause a conflict between cost and benefit”
“The QCA do not have a high enough profile to add to this campaign. A worry is that this is the way education is going to be taken over by businesses”
3%
5%
5%
5%
6%
8%
Reservation about ITcapabilities/range
Willing to look into/developfurther
Still working todevelop/investigate e-
Assessment
Organisation is positivetowards e-Assessment
e-Assessment is only partof the solution
Not enough informationavailable/publicised/shared