Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning Renee A. Meyers Coordinator, UWS SoTL Leadership Site Faculty College, May.

Gathering SoTL Evidence: Gathering SoTL Evidence: Methods for Systematic Methods for Systematic

Inquiry into Student LearningInquiry into Student Learning

Renee A. MeyersRenee A. Meyers

Coordinator, UWS SoTL Leadership Coordinator, UWS SoTL Leadership SiteSite

Faculty College, May 2009Faculty College, May 2009

Gathering EvidenceGathering EvidenceAssessment of Prior Knowledge:Assessment of Prior Knowledge:

Have you ever completed an IRB protocol?Have you ever completed an IRB protocol? Have you ever had instruction in survey design?Have you ever had instruction in survey design? Have you ever written a survey? Have you ever written a survey? Have you ever administered a survey?Have you ever administered a survey? Have you ever had instruction in how to conduct Have you ever had instruction in how to conduct

interviews or focus groups?interviews or focus groups? Have you ever conducted an interview or focus group Have you ever conducted an interview or focus group

for research purposes?for research purposes? Have you ever had instruction in how to conduct a Have you ever had instruction in how to conduct a

Protocol Analysis (Think Aloud)? Protocol Analysis (Think Aloud)? Have you ever conducted a Protocol Analysis (Think-Have you ever conducted a Protocol Analysis (Think-

Aloud)?Aloud)? Have you ever collected student assignments for Have you ever collected student assignments for

research evidence of learning?research evidence of learning?

Gathering EvidenceGathering Evidence

Assume you want to conduct a SoTLAssume you want to conduct a SoTL

investigation having to do with studentsinvestigation having to do with students

learning research skills. learning research skills.

Frame a SoTL research question where Frame a SoTL research question where thethe

type of data we just collected might betype of data we just collected might be

useful.useful.

Your pre-assessmentYour pre-assessment

Write down 2-3 questions that you Write down 2-3 questions that you might want to ask your students as a might want to ask your students as a type of pre-assessment for your SoTL type of pre-assessment for your SoTL investigation.investigation.

Overview of SessionOverview of Session

Prior to Gathering Evidence: IRBPrior to Gathering Evidence: IRB Orientation to SoTL ResearchOrientation to SoTL Research Methodological OrientationsMethodological Orientations Types of QuestionsTypes of Questions Forms of EvidenceForms of Evidence Tools for, or Methods of, Gathering Tools for, or Methods of, Gathering

EvidenceEvidence Developing an initial evidence-Developing an initial evidence-

gathering plan for your projectgathering plan for your project

IRB ApprovalIRB Approval

What is IRB, and why do I have to do What is IRB, and why do I have to do it?it?

– IRB QuizIRB Quiz

– Whitepaper on UWS SoTL website:Whitepaper on UWS SoTL website: http://sotl.uwm.eduhttp://sotl.uwm.edu

Orientation to SoTL Orientation to SoTL ResearchResearch

Orientation to SoTL Orientation to SoTL ResearchResearch

Interpretive OrientationInterpretive Orientation Quasi-Objective OrientationQuasi-Objective Orientation

– What are research goals of each?What are research goals of each?– How is human behavior viewed? How is human behavior viewed? – What types of methods typically characterize What types of methods typically characterize

each orientation?each orientation?– What are strengths/difficulties of each What are strengths/difficulties of each

orientation? orientation? – Where do you stand? Is there a disciplinary Where do you stand? Is there a disciplinary

stance?stance? Fill out handout on research orientationsFill out handout on research orientations

Quasi-ObjectiveQuasi-Objective InterpretiveInterpretive

Research GoalsResearch Goals Explain and predictExplain and predict Understand and Understand and interpret meaninginterpret meaning

Motivation for Motivation for Human BehaviorHuman Behavior

Natural forces Natural forces shape behaviorshape behavior

Humans have Humans have conscious choiceconscious choice

Types of MethodsTypes of Methods Often QuantitativeOften Quantitative

-Surveys, content -Surveys, content analysis, analysis, experimental designexperimental design

Often QualitativeOften Qualitative

-Interviews, Focus -Interviews, Focus Groups, Groups, Observation. Observation. Discourse AnalysisDiscourse Analysis

Strengths/Strengths/DifficultiesDifficulties

Useful/practicalUseful/practical

Detached; overly Detached; overly objectiveobjective

New insightNew insight

Can’t generalizeCan’t generalize

Difficult to Difficult to validatevalidate

Orientation to SoTL Methods Orientation to SoTL Methods for Gathering Evidencefor Gathering Evidence

Thinking it through . . . Thinking it through . . .

Suppose you wondered how students use Suppose you wondered how students use instructor feedback to write the second draft instructor feedback to write the second draft of a paper. How might you design methods of a paper. How might you design methods of evidence gathering to answer this of evidence gathering to answer this question? question?

Qualitative methods?Qualitative methods?

Quantitative methods?Quantitative methods?

Pros/Cons of each? Pros/Cons of each?

Types of SoTL QuestionsTypes of SoTL Questions

Asking Good SoTL QuestionsAsking Good SoTL Questions

SoTL questions are focused on SoTL questions are focused on student learningstudent learning– Consequently we look to students for Consequently we look to students for

evidence to answer our questionsevidence to answer our questions Handout of some of your questions Handout of some of your questions

(and others)(and others)– Type/Form of research question guides Type/Form of research question guides

the type of evidence you collectthe type of evidence you collect

Types of QuestionsTypes of Questions

Descriptive questions (What is)Descriptive questions (What is)– Lessons From the BestLessons From the Best article on website article on website

Confirmatory questions (What works)Confirmatory questions (What works)– Practice Versus Review ExamsPractice Versus Review Exams article on article on

websitewebsite

Can you identify the SoTL questions Can you identify the SoTL questions on the handout?on the handout?

Types of Questions and Types of Questions and Time Orientation in Time Orientation in

ResearchResearch

Time Orientation and Research Time Orientation and Research QuestionsQuestions

All research takes place within time All research takes place within time and spaceand space

Researcher needs to decide whether Researcher needs to decide whether question being asked is best question being asked is best investigated as a one-time investigated as a one-time phenomenon, or over time phenomenon, or over time (longitudinally)(longitudinally)– Cross sectional research-snapshot in timeCross sectional research-snapshot in time– Longitudinal research-goes across timeLongitudinal research-goes across time

ExampleExample

Suppose you wanted to study whether Suppose you wanted to study whether group members use “explaining” group members use “explaining” communication in their group discussions.communication in their group discussions.– If you were to do a cross-sectional study, how If you were to do a cross-sectional study, how

would you proceed?would you proceed?– If you were to do a longitudinal study, how If you were to do a longitudinal study, how

would you proceed?would you proceed? What are the advantages/disadvantages of What are the advantages/disadvantages of

undertaking each of these time undertaking each of these time orientations in SoTL research?orientations in SoTL research?

Types of EvidenceTypes of Evidence

Types of Evidence Types of Evidence

Secondary dataSecondary data– Secondary data are data collected for some Secondary data are data collected for some

purpose or need other than the one being purpose or need other than the one being currently pursued by a SoTL researcher.currently pursued by a SoTL researcher.

– Secondary data can be found within your own Secondary data can be found within your own university and also within external organizations.university and also within external organizations.

Primary dataPrimary data– Primary data are collected to specifically answer Primary data are collected to specifically answer

the questions posed in the current SoTL research the questions posed in the current SoTL research project.project.

– Primary data are collected by you in your own Primary data are collected by you in your own classroom, or others’ classroomsclassroom, or others’ classrooms

Sources of SoTL Secondary Sources of SoTL Secondary DataData

– National Center for Education StatisticsNational Center for Education Statistics– UWS Office of Policy and Analysis ResearUWS Office of Policy and Analysis Resear

chch– The Chronicle of Higher EducationThe Chronicle of Higher Education– U.S. Department of EducationU.S. Department of Education– NSSE and FSSE dataNSSE and FSSE data– Campus Institutional ResearchCampus Institutional Research

Using Secondary DataUsing Secondary Data

How might you use secondary data How might you use secondary data in a SoTL investigation? in a SoTL investigation?

Credibility of Secondary Credibility of Secondary DataData

Whenever you use secondary data, you Whenever you use secondary data, you must consider how credible it ismust consider how credible it is

Questions to ask:Questions to ask:– Does the report reveal who sponsored the Does the report reveal who sponsored the

research?research?– Does the report reveal who conducted the Does the report reveal who conducted the

research? Is this a respected source?research? Is this a respected source?– Are the research methods disclosed and Are the research methods disclosed and

explained? (Size of sample? Characteristics of explained? (Size of sample? Characteristics of respondents? Margin of error?)respondents? Margin of error?)

– When was the study conducted?When was the study conducted?

Initial ReflectionInitial Reflection

What is your orientation (and/or your What is your orientation (and/or your disciplinary orientation) to doing disciplinary orientation) to doing research?research?

What is your comfort level with What is your comfort level with qualitative and quantitative methods? qualitative and quantitative methods? Do you have a preference for one over Do you have a preference for one over the other?the other?

What is your research question?What is your research question? What forms of evidence/data do you What forms of evidence/data do you

think you might collect?think you might collect?