Julie Gedeon, Coordinator of Assessment Barbara Schloman, Project Director Libraries and Media Services Profiling Students’ Understanding: Using TRAILS.

Julie Gedeon, Coordinator of AssessmentBarbara Schloman, Project Director

Libraries and Media Services

Profiling Students’ Understanding:Using TRAILS to Assess 9th Grade

Information Literacy SkillsOELMA Conference 2007, Dayton, Ohio

1. TRAILS Development Overview

2. TRAILS-9 Results: School Year 2006-07

3. Your Feedback

Presentation Objectives

ILILE goals:

–To foster successful collaboration among K-12 teachers and library media specialists

–To advance library and information literacy in the K-12 school curriculum.

Institute for Library and Information Literacy

Education

Federally funded through the Institute of Museum and Library Services (IMLS) and the U.S. Department of Education.

High School – College Collaborations

•Transitioning to College (T2C) (http://www.transtioning2college.org)

•TRAILS: Tool for Real-Time Assessment of Information Literacy Skills(http://www.trails9.org)

University Libraries’ ILILE Projects

Goal: To develop a tool to assist library media specialists in measuring the information literacy competencies of students.

Specific objectives:–Standards-based–Available on the Web at no cost–Easy to administer–Insure privacy–Report outcomes by student and by class

TRAILS

Coming in 2008!TRAILS-6

• Recurring request from librarians.• Potential to track the development of information

literacy competencies in students across their K-12 school years and into college, providing the opportunity for more targeted instruction at all levels.

• Available funding; expandable online system.

TRAILS-6 Development Timeline

2007:• June-July: item development• August: item refinement; develop pilot assessments• October: pilot testing items• November-December: analyze data; revise items;

develop final assessments

January 2008: go live

TRAILS Development Process

Reviewed standards for information literacy:

– Ohio Academic Content Standards: Identified all Ohio standards, benchmarks, indicators that relate to information literacy at the 6th/9th grade level.

– AASL Information Power: Reviewed standards and indicators.

Information Literacy Categories

Standards were categorized into five measurable information literacy categories:

Develop topic

Identify potential sources

Develop, use, and revise search strategies

Evaluate sources and information

Recognize how to use information responsibly, ethically, and legally

Developing Assessment Items

Establishing Priority Competencies Within each category the information literacy skills common across

the standards were identified and prioritized.

Developing ObjectivesStudent learning objectives were developed to address the priority competency areas.

Writing ItemsItems were then written to address the objectives for each category.

Testing and Revising Items

Field Testing of Items, Followed by Revisions

– Are the items understandable as written?

– Are they measuring what was intended?

TRAILS-9: Field testing of items by volunteer library media specialists working with a small number of 9th grade students.

TRAILS-6: Sample assessments being administered to 6th grade students in over 130 schools nationwide during October 2007.

Available Assessments

TRAILS-9:• Two 30-item general assessments covering all five of

the information literacy categories.• Two sets of 10-item assessments for each of the five

categories.

TRAILS-6:• ?? Offer general assessments of differing lengths (20-

or 25- items)??

TRAILS-9 Use

• TRAILS-9 went live in January 2006: 400 account holders by April; 1,000 by June.

• In 2006/2007 school year: Over 2,500 account holders; administered to more than 32,500 students.

• Currently: 3,200 + account holders

• Geographic distribution: All 50 states plus the District of Columbia and the Virgin Islands; more than 40 countries

• Feedback: Influences improvements to items and user interface.

TRAILS-9 ResultsAll Students

Data• From General Assessment 1 (30 items)• 2006-2007 school year • 18,323 students

Findings• Higher grades performed better than lower grades• Overall assessment slightly easy for target audience

– 4% of 9th grade top students “unmeasured”– Works great for 6th grade

6th Grade Person-Item Map

6th grade student scores are within the bounds of the TRAILS-9 item measures.

GA1gr6PImap.txt

9th Grade Person-Item Map

• Some 9th grade students were not measured because the range of difficulty of TRAILS-9 items was inadequate.

• More difficult information literacy items are needed.

GA1gr9PImap.txt

TRAILS-9 ResultsOhio Students

• General Assessment 1, Ohio• 6,929 students

– 6th 149– 7th 202– 8th 857– 9th 4,038– 10th 328– 11th 866– 12th 212– Other 277

Performance by Category

Category % correct5. Recognize how to use information

responsibly, ethically, and legally74%

2. Identify potential sources 70.4%

3. Develop, use, and revise search strategies 60.7%

1. Develop topic 55.6%

4. Evaluate sources and information 49.5%

1. Develop topic—difficult

4. When you research a topic, it is important to know its relationship to other concepts. Which word or phrase represents the broadest (most general) subject under which all of the other topics would fit?      

studying (22%)  note taking (19%)  strategies for academic success (38%)  rewriting your notes (4%)  group work (6%)  get started early (6%)  share contact information (5%)

1. Develop topic—easy

3. Read the original topic and the revised topic. Decide if the revised topic is broader (less specific) or narrower (more specific) than the original topic.

Initial Topic: Describe the elements needed in an auditorium to reduce echoes.

Revised Topic: Describe the elements needed in an auditorium for good sound quality.

Broader (63%)  

Narrower (37%)

2. Identify potential sources—difficult

6. Which one of these resources would most likely have a current article on steroid use in professional baseball?      

a school library catalog (9%)  

a science database such as Access Science (16%)  

a newspaper database such as Newspaper Source (56%)  

a biography database such as Gale Biography Resource Center (7%)  

a biography database such as American National Biography (12%)

2. Identify potential sources—easy

9. You just learned your favorite author has a new book out. Where could you most likely check out a copy?      

hospital library (1%)  

public library (94%)  

government library (4%)  

museum library (1%)

3. Develop, use, revise search strategies—

difficult

18. If you wanted to search for a topic that has several synonyms (for example, young people, adolescents, teenagers, teens), which Boolean operator would you use?    

adj (20%)  

and (30%)  

near (14%)   

not (4%)  

or (32%)

3. Develop, use, revise search strategies—

easy16. If you need to know what chapters are in a book,

which part of the book provides the best information?      

cover of the book (1%)  

glossary (6%)  

introduction (5%)  

table of contents (88%)

4. Evaluate sources and information—

difficult19. The following is a quote from an article you found for your research

paper on stem cell research.

"Stem cell research offers the possibility of understanding many different diseases. Until this research is well-funded, the number of Americans with health insurance is not expected to change."

Which of the following is an accurate assessment of this information?This is credible information because it is written as a quote, indicating this is an

expert opinion. (21%)  

This is credible information because it presents current scientific understanding. (37%)  

This is not credible information because the relationship between the statements is not logical. (32%)  

This is not credible information because the author is biased against stem cell research. (9%)

4. Evaluate sources and information—easy

22. You hear on a radio talk show that Mad Cow Disease may have been found in the United States. How might you best determine the truth of this statement?      

Call for a transcript of the program from the radio station. (4%)  

Check the fbifiles.com Web site for information the government itself might not release to the public. (8%)  

Discuss the news with co-workers who might have heard the program. (5%)  

Look up the topic at the American Council on Beef Web site for current news. (22%)  

Search for Mad Cow Disease on the U.S. Department of Agriculture Web site. (61%)

5. Recognize how to use information

responsibly, ethically, legally--difficult

28. Which of the following concepts makes it legally wrong for government agencies to deny you access to official information under most circumstances?       Fair use (8%)  

Freedom of information (45%)  

Intellectual freedom (15%)  

Intellectual property (7%)  

Right to privacy (25%)

5. Recognize how to use information

responsibly, ethically, legally--easy

29. Plagiarism means:    

  quoting someone else’s work and giving that person

credit (10%)  

writing a short story with a group of classmates (5%)  

the use of another's original words or ideas as though they were your own (83%)  

developing a handbook on copyright laws (3%)

TRAILS-9 Results: Seeking Your Input

Review handout with 3 easy and 3 difficult items.

Why do you think students would find these items easy or difficult?

At the end of the discussion, please leave your sheets with us!

The “Lifers” on the TRAILS Project:Barbara Schloman, TRAILS Project Director

Julie Gedeon, TRAILS Assessment Coordinator

Wendy Torrence, TRAILS Item Development Coordinator

David Bird, Software Engineer Extraordinaire

We thank you for your interest and ongoing support!

http://www.trails-9.org

Wiki for TRAILS http://ilfortrails9.wikispaces.com/

We welcome your use of TRAILSand your comments and suggestions.