Research Practices Survey Presentation

2008- 2009 Research Practices

Survey Analysis- an Assessment of

the impact of the QEP

Department of Institutional Research

Jessica Larsen, M.Ed.

Diane Saphire, Ph.D.

QEP Goals• As outlined in the QEP, there are five goals for

information literacy:

• Understand

• Access

• Use Ethically

• Evaluate

• Create

The QEP and Assessment

• QEP identifies tests, surveys, embedded questions, and rubrics as tools to use in assessing the progress on each of the stated goals.

• FYILLA- First Year Information Literacy in the Liberal Arts Assessment, now The Research Practices Survey is identified as one of these tools in the QEP.

• For more information on the history and development of the Research Practices Survey please see the assessment portion of the QEP

• Questions on The Research Practices survey address the first four goals

Research Practices Survey

• 34 Questions • Examines the information literacy

experiences, attitudes, approaches, and skills of first year students▫ Part test of skills and knowledge▫ Part survey of their beliefs and attitudes about

academic research

• QEP Data- 2008/2009▫ Administered in Fall of 2008▫ Re-Administered in Spring of 2009 108 students participated in both administrations.

• Pre-QEP Baseline Data- 2006/2007 academic year▫ Administered in Fall of 2006▫ Re-Administered in Spring of 2007 63 repeated subjects

Benchmark scores were computed by scoring relevant survey questions, summing the scores for each individual, and creating a percentage score, by dividing earned score by total possible score.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall'08 Spring '09

Gr

ad

e E

qu

iva

len

cy

Understand

Use Ethically

Access

Evaluate

•Understand the varieties of information sources availableStudents will demonstrate an understanding of the nature of informationsources, such as books, journals, newspapers, Websites, and media, andan understanding of how they vary in audience orientation and authority.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall Spring

Gr

ad

e E

qu

iva

len

cy

2006-2007 2008-2009

Percentages represent the proportion of the sample that responded that they had used the following sources

Dramatic decline in the use of encyclopedias, almanacs, and dictionaries.

The lack of an increase in 08-09 academic and research journals may be due to the increase in online resources

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Library BooksEncyclopedias, alamanacs, dictionariesAcademic or research journalsNewspapers or Magazines for General Public

Of

sa

mp

le

Fall '06 Spring '07 Fall '08 Spring '09

PRINT sources used for research during

past academic year?

ONLINE sources used for research in

the past academic year?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

General Search EnginesJournals, Magazines, Newspapers, or EncyclopediasLibrary Catalog Indexes or databases Google Scholar

Of

sa

mp

le


Identify source:

Jorgenson, Lars W. “Reinterpreting Navajo Rites.”

Navajo Culture 6 (1946): 469-78.

Question Prompt:

Indicate whether the item is an entire book, a journal

article, or a portion of a book

• In 08/09 here is a 4.8% increase in the number of students who answered correctly by the spring semester

• By the Spring in 2009, 67.6% of the respondents could not identify a journal article citation.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Co

rr

ec

tly

Id

en

tifi

ed

Correctly Identify

Sources from a CitationStudents were presented with three citations:

• An academic journal article,

• a book

• A portion of a book

•Question asked them to correctly identify the source

•The graph represents the percentage of the respondents that answered correctly.

•Differences were not significant

Citations:

Allen, Glover Morril. Bats. Cambridege: Harvard University Press, 1939.

Tanaka, Kazuko. “The New Feminist Movement in Japan, 1970-1990.” In Japanese Women, edited by KumikoFujimura-Fanselow. New York: Feminist Press, 1995.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Book Portion of a Book

An

sw

er

ed

Co

rr

ec

tly


Defining CitationIn a scholarly article or research paper, a citation is:

1. A direct quotation from someone else’s written work

2. Source information from any ideas or text from someone else’s written work (credited response)

3. The physical location of a source (book, journal, etc.)

4. All of the above

5. Don’t know

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

An

sw

er

ed

Co

rr

ec

tly


1. Primary sources are more scholarly than secondary sources.

2. Primary sources are old; secondary sources are new.

3. (Credited Response) Primary sources examine subjects first-hand; secondary sources examine the findings of other scholars.

4. Primary sources are more appropriate for academic projects than are secondary sources.

5. All of the above are equally accurate

6. Don’t know0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6

Of

sa

mp

le


Distinguish between a Primary and a Secondary Source

When two researchers disagree, one

must be wrong

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

strongly agree agree disagree strongly disagree

Of

Re

sp

on

da

nts


Good research yields clear results; poor research

yields ambiguous results

•Question asks: “To what extent to you agree or disagree with this statement.”

•Disagree / strongly disagree are the credited responses.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Dis

ag

re

e/S

tro

ng

ly D

isa

gr

ee


•Access information efficiently and effectivelyStudents will understand and apply techniques for accessing information which mayinclude general searching principles, accessing appropriate Web-basedresources, becoming familiar with specialized collections, and using Interlibrary Loan.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall Spring

Gr

ad

e E

qu

iva

len

cy

2006-2007 2008-2009

Application of Skills Which of the following is the correct way to truncate a search word?1. Typing in only the first

syllable of the word as a keyword

2. Combining search words with “and,” “or,” or “not”

3. Using * or ! Symbol in place of the last few letters of the word (credited)

4. None of the above5. Don’t know

Which of the following searches would retrieve the MOST results in an online search?1. Movies OR films

(credited)2. Movies AND films3. Movies NOT films4. Movies INSTEAD OF

films5. Don’t know

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Truncate Most Results

an

sw

er

ed

co

rr

ec

tly


You retrieve the following information

from a library catalog search:

Gothic modernisms / edited by Andrew Smith and Jeff Wallace.

New York, N.Y. : Palgrave, 2001.Stacks PR888.M63 G67 2001 AVAILABLExii, 232 p. ; 23 cm.Includes bibliographical references and index.English fiction -- 20th century -- History and criticism.Modernism (Literature) -- Great Britain.Modernism (Literature) -- United States.Gothic revival (Literature)Smith, Andrew, 1964-Wallace, Jeff, 1958-

Application of Skills Which of the following would be the MOST efficient way to find a comprehensive listing of other books in the catalog on this topic?

1. Identifying other books written by the same authors

2. Examining the bibliography in the book (Partial Credit)

3. Scanning the shelves where the book is located to see what books are nearby (Partial Credit)

4. Searching again using the subject headings that most closely match your research topic (Full Credited Response)

5. Don’t know.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Full Credit Partial Cred

An

sw

er

ed

Co

rr

ec

tly


Yield a list of relevant scholarly articlesWhich of the following is likely to yield the most comprehensive list of relevant scholarly articles for a research project?

1. Searching an electronic Index or database in a specific academic field (History, Biology, Music, etc.) (Credited Response)

2. Using a general Internet search like Google or Yahoo

3. Paging through print volumes of an academic journal in a specific academic field (Partial Credit)

4. Searching the library catalog

5. All of the above are equally effective

6. Don’t know

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Full Credit Partial Credit

Of

sa

mp

le


•Understand the concept of intellectual property and theeconomic, legal, and social contexts of information, and useinformation ethicallyStudents will understand the concepts of plagiarism and copyright and willappropriately use citation/documentation systems in their work. Students willdemonstrate an understanding of the University Academic Honor Code.

0%10%20%30%40%50%60%70%80%90%

100%

Fall Spring

Gr

ad

e E

qu

iva

len

cy

2006-2007 2008-2009

A citation is not required when?

• You are paraphrasing, rather than quoting a source

• More than one source says the same thing

• You are describing your own findings or analysis (correct response)

• All of the above

• Don’t know

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

An

sw

er

ed

co

rr

ec

tly


Evaluating information and its sources

0%

20%

40%

60%

80%

100%

Fall Spring

2006-2007 2008-2009

A peer-reviewed or refereed journal is BEST described as:

0% 50% 100%

Publishes reviews of other articles

Publishes articles approved by other scholars (credited …

Includes only articles written collaboratively by peers

Includes references for each article it publishes

Don't know


Researchers must distinguish between academic journals

and popular magazines. Which of the following

statements is LEAST ACCURATE?0% 20% 40% 60% 80% 100%

Academic journal articles provide objective facts (credited response)

Academic journal articles usually include a list of scholarly references

The intended audience for academic journals is mainly scholars

Editors & authors of acad. journal articles are usually employed in higher ed.

Don't know


Which of the following is the best way to determine

whether to use a particular source?

0% 20% 40% 60% 80% 100%

Whether it is a Print or internet source

How recently it was published

How easily you can get the source

Whether the source is scholarly

Whether the author is highly respected

All of the above are equally valid


Identify the least appropriate source

for a research paper0% 20% 40% 60% 80% 100%

University Press book

Encyclopedia Volumes

Book retrieved from electronic web archive

Journal article

Personal Webpage (credited response)


Library Use

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Once a week or more Once or twice a month

A few times a year Never

Of

sa

mp

le


• Prompt: How often in the past academic year did you use resources from a school library for school assignments, research projects or other academic purposes?

• Fall represents the students’ use of their high school library

• Spring represents the students’ use of their college library

Number of assignments requiring at

least 3 sources

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Five or more Three or four One or two None

Of

sa

mp

le


How many assignments, papers

, or research projects have you completed in the

past academic year that required you to include at least three

sources in a bibliography, References, or works cited

list?

Assignments requiring a specific

format (APA, MLA, Chicago, or some

other)• In the past academic

year, when you were given research project assignments, how often were you required to use a specific format (such as APA, MLA, Chicago, or some other style) for the sources in your bibliography

• Possible Responses: Almost always, Often, Sometimes, Rarely, Never/not applicable

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall '06 Spring '07

Fall '08 Spring '09

Which of the following best describes

the way you pace your work on a

research assignment0% 20% 40% 60% 80% 100%

I do most of the work soon after an assignment is given

I divide the work pretty equally across the available time

I do a little work soon after the assignment is given, but do most of

it toward the end

I do all of the work just before or on the due date


Seeking help or advice0% 20% 40% 60% 80% 100%

Teachers or Professors

Librarians

Parents/Other Adult Family

Friends, Classmates, or Siblings

Writing labs, centers, or help groups

Help screens, online tutorials or other


In the past academic

year, when you were working on

research assignments, how often did you

seek help or advice from each of the following?

% of students who answered

“almost always” or “often”

Consulting a Librarian

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Five or more times Three or four times Once or twice Never

Of

sa

mp

le


• Prompt: Over the course of the past academic year, how often did you talk with a librarian about a research project you were doing?

Conclusions:

• Students are making significant gains over the course of their freshman year.

• However, students are still demonstrating a lack of basic research skills and competencies.

• The Research Practices Survey may not be capturing student progress. Faculty in-course assessments may be the best method to collecting valid data that truly reflects student outcomes.

2008/2009 TU Repeat Participant

Gender Breakdown

Total

Freshman

Class

All Respondents Repeated

Fall Spring

Females 53.1% 64.6% 61.0% 67.6%

Males 46.9% 35.4% 39.0% 32.4%

n 655 161 292 108

2006-2007 TU Participant

Gender Breakdown

Total

Freshman

Class

All Respondents Repeated

Fall Spring

Females 53.9% 62.9% 69.8% 70.5%

Males 46.1% 37.1% 30.2% 29.5%

n 660 167 129 63

Research Practices Survey Presentation

Education

use of print sources

year information literacy

use of online sources

research process

past academic year

knowledgepart survey

s079online sources

year progress