Department of Information & Communications Online Information Literacy Audits: a longitudinal study Bob Glass & Jill Griffiths.

Department of Information & Communications

Online Information Literacy Audits:a longitudinal study

Bob Glass & Jill Griffiths


The activities undertaken address the following questions:

1. Is there value in administering online information literacy audits?

2. What do the results of these audits tell us?

3. Does intervention make a difference in the results?

4. Is there a need in for a UK designed, generic, online Information Literacy Audit?

A three year study of two undergraduate cohorts

The testing is based around an online web based Information Literacy Test from Steven Wise & Lynn Cameron(2004).

The test is hosted by The Institute for Computer Based Assessment, James Madison University (2006)


Who are the students?

Information & Communications

65 students on 5 degree routes all studying the same six first year units

Economics

19 students from1 degree route all studying the same six first year units

A very mixed group socio economically, ethnically and chronologically

Some students were working librarians

Background


What is the test?

• An Online, web based, multiple choice, using radio buttons for answer selection

• Asks 65 questions (60 live plus 5 trial questions)

• Uses the “Adaptex” Assessment System (Wise, S. L. &

Yang, S. 2003) to manage the responses. • Measures four of the five ACRL standards

• Testing takes around an hour to complete, no time limit set

• Students receive a score on completion of the test

• Tutors receive detailed performance data in a range of formats

• Devised by library and psychology practitioners

What is the test?What is the test?

ACRL Objectives – designed around ACRL standards for information literacy:

•ACRL1 – Determines the nature and extent of the information needed

•ACRL2 – Accesses needed information effectively and efficiently

•ACRL3 – Evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system

•ACRL4 – Assessed differently (essay)•ACRL5 – Understands many of the economic, legal and social issues surrounding the use of information and access and uses



What is the test? Format:


Is there value in administering online information literacy audits?

• can provide a cohort baseline for tutors/ practitioners

• can provide interaction, information & feedback for the students

• can highlight areas for development with students

• can highlight areas for development in the curriculum

• can highlight areas for development for library practitioners


What do the results tell us?

1. the majority of students scored well above the scorerequired for “Proficient”

2. the mean score for ICO students was in the “Advanced” classification the mean score for Economics was almost identical

3. Only one in every 18 of the students failed to reach the Proficient grade (4 ICO students and 1 Economics)

4. just over a quarter of students were classed as Proficient two thirds were classed as Advanced

Preliminary resultsPreliminary results

Results for all students

0%

5%

10%

15%

20%

25%

30%

80-85 70-79 60-69 50-59 40-49 30-39 20-29

Score

Pe

rce

nta

ge

of

stu

de

nts



Mean scores for students

ICO59.6

Economics60.8



Results for students by cohort

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

80-85 70-79 60-69 50-59 40-49 30-39 20-29

Score

Pe

rce

nta

ge

of

stu

de

nts

ICO

Economics



1: can the student define and articulate the nature and extent of the information needed.

2: can the student access needed information effectively and efficiently.

3: can the student evaluate information and its sources critically and incorporate selected information into his or her knowledge base and value system.

5: can the student understand many of the ethical, legal and socio-economic issues surrounding information and information technology.

What is the test? Standards assessed :

Standard 4 (written work) cannot be assessed by this type of test


Performance across each ACRL objective - all students

0%

10%

20%

30%

40%

50%

60%

70%

80%

ACRL1 ACRL2 ACRL3 ACRL5

Objectives

Re

su

lts

Incorrect

Correct



Results for each objective by cohort

0%10%20%30%40%50%60%70%80%

ACRL1 IC

O

ACRL1 E

CON

ACRL2 IC

O

ACRL2 E

CON

ACRL3 IC

O

ACRL3 E

CON

ACRL5 IC

O

ACRL5ECON

Objective

Pe

rse

nta

ge

of

stu

de

nts

Incorrect

Correct



ACRL1 – Determines the nature and extent of the information needed

0%

10%

20%

30%

40%

50%

60%

70%

80%

ACRL1 ICO ACRL1 ECON

Cohort

Pe

rce

nta

ge

of

stu

de

nts

Incorrect

Correct



ACRL2 - Accesses needed information effectively and efficiently

42%44%

46%48%

50%52%

54%56%


Cohort

Pe

rce

nta

ge

of

stu

de

nts

Incorrect

Correct



ACRL3 - Evaluates information and its sources critically and incorporates selected information into

his or her knowledge base and value system

0%10%20%30%40%50%60%70%80%


Cohort

Pe

rce

nta

ge

of

stu

de

nts

Incorrect

Correct



ACRL5 - Understands many of the economic, legal and social issues surrounding the use of information and

access and uses

0%

10%

20%

30%

40%

50%

60%

70%

ACRL5 ICO ACRL5ECON

Cohort

Pe

rce

nta

ge

of

stu

de

nts

Incorrect

Correct




1. the majority of students were able to answer questions concerningdefining and articulating the nature and extent of the information needed correctly

2. around half were not able to answer questions concerning accessing needed information effectively and efficiently correctly

3. two thirds were able to answer questions concerning evaluating information and its sources critically and being able to incorporate selected information into his or her knowledge base and value system

4. a similar number were able to answer questions concerning understanding many of the ethical, legal and socio-economic issues surrounding information and information technology correctly.



ACRL2:Students struggled with answering questions about:

Selecting the most appropriate methods and strategies for accessing the needed information, for example:

Did not understand about different types of database and what a library catalogue wasCould not determine the provenance of the information or sources

Constructing and implementing effectively designed search strategiesDid not know how to expand searches or apply them to different databases

Refining the search strategy if necessaryUnable to use boolean operators, truncation, phrase searching

Extracting records and managing the information and its sourcesCould not recognise different item formats from bibliographic searchresults presented to them from databases


The answers to our remaining questions

Q. Does intervention make a difference in the results?

Q. Is there a need in for a UK designed, generic, online Information Literacy Audit?

Stage two testing completed 14th March 2008

Stage two analysis completed by the end of May 2008

Stage three testing commences November 2008

In progress


References

The Association of College and Research Libraries (2000). Information Literacy Competency Standards for Higher Education. ACRL Wise, S. L. & Yang, S. (2003). The Adaptex Assessment System (Version 1.5) . U.S. Department of Education Fund for the Improvement of Postsecondary Education. Wise,S,.L. Cameron, L. ,Yang, S. & Davis, S. (2005) Information Literacy Test: Test Development and Administration Manual. Harrisonburg: Institute for Computer-Based Assessment; Center for Assessment & Research Studies, James Maddison University.

The Institute for Computer Based Assessment, James Madison University(2006) Information Literacy Test (ILT) [Online: http://www.jmu.edu/icba/prodserv/instruments_ilt.htm accessed 21st February 2008)

Department of Information & Communications Online Information Literacy Audits: a longitudinal study Bob Glass & Jill Griffiths.

Documents

use of information

information feedback

selected information

ico students

chronologicallysome

quarter of students

majority of students

test tutors