Using National Using National Studies of Student Studies of Student Engagement to Engagement to Support Support Institutional Change Institutional Change Nathan Marti, CCSSE Nathan Marti, CCSSE Todd Chamberlain, NSSE Todd Chamberlain, NSSE FAIR Conference FAIR Conference June 23, 2004 June 23, 2004
46
Embed
Using National Studies of Student Engagement to Support Institutional Change Nathan Marti, CCSSE Todd Chamberlain, NSSE FAIR Conference June 23, 2004.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using National Using National Studies of Student Studies of Student
Engagement to Engagement to Support Institutional Support Institutional
““The research is unequivocal: The research is unequivocal: students who are actively students who are actively involved in both academic and involved in both academic and out-of-class activities gain more out-of-class activities gain more from the college experience than from the college experience than those who are not so involved.”those who are not so involved.”
Ernest T. Pascarella & Patrick T. Ernest T. Pascarella & Patrick T. Terenzini, Terenzini, How College Affects How College Affects StudentsStudents
Lessons from the Lessons from the ResearchResearch
What matters most to desired What matters most to desired outcomes is what students outcomes is what students dodo, , not who they arenot who they are
A key factor for student A key factor for student learning is the learning is the quality of effortquality of effort students devote to students devote to educationally purposeful educationally purposeful activitiesactivities
National Survey of National Survey of Student Student EngagementEngagement(pronounced “nessie”)(pronounced “nessie”)
Community College Community College Survey of Student Survey of Student EngagementEngagement(pronounced “sessie”)(pronounced “sessie”)
College student surveys that College student surveys that assess the extent to which assess the extent to which students engage in students engage in educational practices educational practices associated with high levels of associated with high levels of learning and developmentlearning and development
GenderGender Race/EthnicityRace/Ethnicity Class LevelClass Level Enrollment StatusEnrollment Status
Comparisons by Carnegie/size, Comparisons by Carnegie/size, national and consortium when national and consortium when relevantrelevant
Benchmarks: High Level Benchmarks: High Level Views of the DataViews of the Data
NSSE/NSSE/CCSSECCSSE developed five indicators, or benchmarks, to represent the multi-dimensional developed five indicators, or benchmarks, to represent the multi-dimensional nature of student engagement at the institutional, sector and national levelsnature of student engagement at the institutional, sector and national levels
Level of Academic Challenge
Active & Collaborative
Learning
EnrichingEducational Experiences/Student Effort
SupportiveCampus
Environment/Support for
Learners
Student-Faculty Interaction
Benchmark Benchmark ReportReport
Institution Consortium Carnegie Classification All NSSE Institutions
Preparing for class (studying, reading, writing, rehearsing, and other activities related to your academic program)
Number of assigned textbooks, books, or book-length packs of course readings
Number of written papers or reports of 20 pages or more
Number of written papers or reports of between 5 and 19 pages
Number of written papers or reports of fewer than 5 pages
Coursework emphasizes: Analyzing the basic elements of an idea, experience or theory
Coursework emphasizes: Synthesizing and organizing ideas, information, or experiences into new, more complex interpretations and relationships
Coursework emphasizes: Making judgments about the value of information, arguments, or methods
Coursework emphasizes: Applying theories or concepts to practical problems or in new situations
Worked harder than you thought you could to meet an instructor's standards or expectations
Campus environment emphasizes spending significant amounts of time studying and on academic work
The NSSE survey, The College Student Report, measures student engagement in many important activities that research studies show are positively related to learning and personal development. Forty-one questions from the survey are assigned to five clusters of similar activities and conditions to make up the national benchmarks of effective educational practice. The benchmarks are created on 100-point scales to make it easier to compare performance within and across sectors and institutional types.
These benchmarks are: (1) level of academic challenge, (2) active and collaborative learning, (3) student interactions with faculty members, (4) enriching educational experiences, and (5) supportive campus environment. The NSSE benchmark analysis is based on more than 105,000 randomly selected students at 467 four-year colleges and universities that participated in the spring of 2000 or 2001. The students represent a broad cross-section of first-year and senior students from every region of the country. The institutions are similar in most respects to the universe of four-year schools. More detailed information about the benchmarks can be found in the national report that was sent with this mailing and on the NSSE website at www.iub.edu/~nsse.
This report summarizes your institution’s performance in these five areas of effective educational practice. Your institution’s benchmark scores are presented and compared to schools in your consortium, your Carnegie Classification, and the NSSE national norms. Page 4 provides some additional information, including a standard score that represents the magnitude of the difference between your institution's score and the respective comparison group, and page 5 presents a table of National and Carnegie classification deciles against which you can gauge the relative performance of your institution on each of the benchmarks.
Asked questions in class orcontributed to class discussions
Total
Never
Sometimes
Often
Very often
Made a class presentation
Total
Never
Sometimes
Often
Very often
Prepared two or more drafts of apaper or assignment before turning itin
Total
Never
Sometimes
Often
Very often
Worked on a paper or project thatrequired integrating ideas orinformation from various sources
Total
Never
Sometimes
Often
Very often
Came to class without completingreadings or assignments
Total
Never
Sometimes
Often
Very often
Worked with other students onprojects during class
Total
Never
Sometimes
Often
Very often
Worked with classmates outside ofclass to prepare class assignments
Total
Never
Sometimes
Often
Very often
Tutored or taught other students(paid or voluntary)
Total
Never
Sometimes
Often
Very often
Participated in a community-basedproject as part of a regular course
Total
Never
Sometimes
Often
Very often
Used an electronic medium (list-serv,chat group, Internet, etc.) to discussor complete an assignment
Total
Never
Sometimes
Often
Very often
Used e-mail to communicate with aninstructor
Total
Never
Sometimes
Often
Very often
Discussed grades or assignments withan instructor
Total
Never
Sometimes
Often
Very often
Talked about career plans with afaculty member or advisor
Total
Never
Sometimes
Often
Very often
Discussed ideas from your readingsor classes with faculty membersoutside of class
Total
Never
Sometimes
Often
Very often
Received prompt feedback fromfaculty on your academicperformance (written or oral)
Total
Never
Sometimes
Often
Very often
Worked harder than you thought youcould to meet an instructor'sstandards or expectations
Total
Never
Sometimes
Often
Very often
Worked with faculty members onactivities other than coursework(committees, orientation, student lifeactivities, etc.)
Total
Never
Sometimes
Often
Very often
Discussed ideas from your readingsor classes with others outside of class(students, family members,coworkers, etc.)
Total
Count Col %
Nesseville State Univ
Count Col%
Carnegie
Count Col%
National
First-year Students
Count Col %
Nesseville State Univ
Count Col%
Carnegie
Count Col%
National
Seniors
Means & Frequency Means & Frequency ReportsReports
Look at the Items that Make Look at the Items that Make Up Each BenchmarkUp Each Benchmark
Which Items Have Which Items Have Significantly Higher/Lower Significantly Higher/Lower Responses than Comparison Responses than Comparison Groups?Groups?
Practical Significance: Practical Significance: Identify Standardized Effect Identify Standardized Effect Sizes greater than .2Sizes greater than .2
Review Benchmark and Means Review Benchmark and Means Summary ReportSummary Report
Small groups by benchmarkSmall groups by benchmarkExamine a benchmark Examine a benchmark Which items differ?Which items differ?
Digging Deeper: Using the DataDigging Deeper: Using the Data
Identify important subpopulationsIdentify important subpopulations Determine the outcome that mattersDetermine the outcome that matters Determine the factors that influence the Determine the factors that influence the
outcomeoutcome
Principles for Data-Driven Principles for Data-Driven Learning-Centered Learning-Centered
ChangeChange
1. Get the ideas right1. Get the ideas right
Focus on a real problem Focus on a real problem (e.g., persistence, raising (e.g., persistence, raising expectations, success in expectations, success in major field courses)major field courses)
Concentrate on effective Concentrate on effective educational practiceseducational practices
Characteristics of Characteristics of Educationally Effective Educationally Effective
High expectations High expectations Respect for diverse talents Respect for diverse talents Emphasis on early years of Emphasis on early years of
studystudy
Characteristics of Characteristics of Educationally Effective Educationally Effective
CollegesCollegesCurriculumCurriculum Coherence in learningCoherence in learningSynthesizing experiencesSynthesizing experiencesIntegrating education Integrating education
and experienceand experienceOngoing practice of Ongoing practice of
learned skillslearned skills
Characteristics of Characteristics of Educationally Effective Educationally Effective
CollegesCollegesInstructionInstruction Active learning Active learning Assessment and feedbackAssessment and feedback CollaborationCollaboration Adequate time on task Adequate time on task Out-of-class contact with Out-of-class contact with
facultyfaculty
2. Get grass roots buy-in2. Get grass roots buy-in
Leaders endorse, but Leaders endorse, but don’t dictatedon’t dictate
Structures not (nearly) Structures not (nearly) as important as as important as relationshipsrelationships
Validate pockets of Validate pockets of qualityquality
The 10% ruleThe 10% rule
2. Get grass roots buy-in2. Get grass roots buy-in
ExamplesExamples Ask deans about their Ask deans about their
concernsconcerns Focus groupsFocus groups Get students “engaged” in Get students “engaged” in
the improvement effort the improvement effort (Illinois State, Oregon State)(Illinois State, Oregon State)
Faculty version of NSSE Faculty version of NSSE surveysurvey
3. Keep the stakes and 3. Keep the stakes and volume lowvolume low
Suspend disbeliefSuspend disbelief Denial managementDenial management Go public later than Go public later than
soonersooner
4. It’s the culture 4. It’s the culture (stupid)(stupid)
Culture is (almost) always Culture is (almost) always (at least) part of (at least) part of the the problemproblem
Focus on “reculturing” Focus on “reculturing” and “revisioning” and “revisioning”
Use familiar (or at least Use familiar (or at least understandable) language understandable) language
5. Think and act 5. Think and act systemicallysystemically
Link innovations and Link innovations and change efforts from change efforts from different parts of the different parts of the campus (e.g., Greater campus (e.g., Greater Expectations, Gen Ed Expectations, Gen Ed reform, SOTL, NSSE, reform, SOTL, NSSE, service learning, diversity)service learning, diversity)
Potential Audiences? Potential Audiences? Consider Audiences Before Consider Audiences Before
Administration—It May Administration—It May Influence DecisionsInfluence Decisions
Internal Use & External Internal Use & External ReportingReporting
Be StrategicBe Strategic
Internal Sharing of NSSE 2003 Results
%
President 80
Administrative Staff 68
Department Chairs 59
Academic Advisors 51
Faculty 71
Governing Board 34
Students 32
Other (web site, fact book, etc.) 20
Internal Audiences
Source: NSSE 2003 Report Card
External AudiencesExternal AudiencesExternal Sharing of NSSE 2003 Data
%
Accreditation Agencies 31
State Agencies 11
Media 13
Web Site 25
Prospective Students 13
Parents 13
Alumni 13
Other 7
No External Disclosure 35
Source: NSSE 2003 Report Card
Ways to Communicate Ways to Communicate ResultsResults
Customize Report to Reach Campus Customize Report to Reach Campus AudiencesAudiences
Organize NSSE data by colleges, depts, Organize NSSE data by colleges, depts, programsprograms
Focus on items of interest to stakeholder and Focus on items of interest to stakeholder and communicate understandable percentagescommunicate understandable percentages
Use data to start discussionsUse data to start discussions Use data to correct misunderstandingsUse data to correct misunderstandings Assemble and distribute detailed information Assemble and distribute detailed information
on method and credibility of surveyon method and credibility of survey
Activity: Connecting Activity: Connecting Findings with Relevant Findings with Relevant
AudiencesAudiencesWhat was the most What was the most
interesting finding?interesting finding?Who needs to know about Who needs to know about
this?this?How can you let them know?How can you let them know?
National ContextNational Context
Putting Your Results in Context:Putting Your Results in Context: How good is good enough?How good is good enough? What would it mean to have 80% of What would it mean to have 80% of
your students indicate that they are your students indicate that they are satisfied with the college?satisfied with the college?
Sampling MethodsSampling Methods
Sampling Consistency Across Sampling Consistency Across Colleges:Colleges:
CCSSECCSSE takes a stratified random takes a stratified random sample of coursessample of courses
NSSE takes random samples of first- NSSE takes random samples of first- and fourth-year studentsand fourth-year students
Results are representative of the Results are representative of the populations from which they are drawnpopulations from which they are drawn
Over-sampling for targeted populationsOver-sampling for targeted populations
Comparing Your ResultsComparing Your Results
Sampling Consistency Across Sampling Consistency Across Colleges:Colleges:
Because results are comparable across Because results are comparable across institutions, institutions can situate their institutions, institutions can situate their resultsresults
Anything on the survey can potentially be Anything on the survey can potentially be benchmarkedbenchmarked
BenchmarkingBenchmarkingTwo Approaches:Two Approaches: Normative - compares your students’ Normative - compares your students’
responses to those of students at responses to those of students at other colleges and universities. other colleges and universities.
Criterion - compares your school’s Criterion - compares your school’s performance against a performance against a predetermined value or level predetermined value or level appropriate for your students, given appropriate for your students, given your institutional mission, size, your institutional mission, size, curricular offerings, funding, and so curricular offerings, funding, and so forth.forth.
Most valued activities Most valued activities Ask faculty & staff what is most Ask faculty & staff what is most valued in institution, department, valued in institution, department, then present relevant datathen present relevant data
Eliminate “Nevers”Eliminate “Nevers”Work on reducing or eliminating Work on reducing or eliminating reports by students of reports by students of nevernever doing specific engagement doing specific engagement activitiesactivities
Student-faculty contactStudent-faculty contact Active learningActive learning Prompt feedbackPrompt feedback Time on taskTime on task High expectationsHigh expectations Respect for diverse learning Respect for diverse learning
stylesstyles Cooperation among studentsCooperation among students
Lessons from the Lessons from the ResearchResearch
What matters most is what What matters most is what students do, not who they arestudents do, not who they are
A key factor is the quality of A key factor is the quality of effort students expendeffort students expend
Educationally effective Educationally effective institutions channel student institutions channel student energy toward the right energy toward the right activitiesactivities
Principles For Promoting Principles For Promoting Student EngagementStudent Engagement
Intentionality - Not Intentionality - Not leaving serendipity to leaving serendipity to chancechance
Both instruments have been shown to Both instruments have been shown to be reliable and validbe reliable and valid
Good construct reliability: benchmarks Good construct reliability: benchmarks items measure the same constructitems measure the same construct
Structural equation models have been Structural equation models have been used to demonstrate that there is used to demonstrate that there is strong consistency across strong consistency across subpopulations (gender & part- v. full-subpopulations (gender & part- v. full-time) and across yearstime) and across years
NSSE data used to form clusters of NSSE data used to form clusters of studentsstudents
http://mypage.iu.edu/~chuzhao/air03dpi.pdfhttp://mypage.iu.edu/~chuzhao/air03dpi.pdf Psychographic data consistently was Psychographic data consistently was
more effective at predicting outcomes more effective at predicting outcomes than demographic datathan demographic data
The Florida OpportunityThe Florida Opportunity
How does engagement relate to How does engagement relate to outcome?outcome?
Activity: Expectations and Activity: Expectations and Desires in Student Desires in Student
EngagementEngagement How many students responded often or How many students responded often or
very often?very often? What would you like to see?What would you like to see? How did they respond?How did they respond? How can you increase engagement?How can you increase engagement?
Questions and SuggestionsQuestions and Suggestions
Questions for NSSE and Questions for NSSE and CCSSECCSSE Suggestions for NSSE and Suggestions for NSSE and CCSSECCSSE