Using IDEA for Assessment, Program Review, and Accreditation Texas A & M University November 8, 2012 Shelley A. Chapman, PhD.

Using IDEA for Assessment, Program Review, and Accreditation Texas A & M University November 8, 2012 Shelley A. Chapman, PhD Plan for this Session Program Evaluation & Assessment of Student Learning Group Summary Reports Aggregate Data File Benchmarking Reports Accreditation Guides What makes IDEA unique? 1.Focus on Student Learning 2.Focus on Instructors Purpose 3.Adjustments for Extraneous Influences 4.Validity and Reliability 5.Comparison Data 6.Flexibility Underlying Philosophy of IDEA Teaching effectiveness is determined primarily by students progress on the types of learning the instructor targets. Faculty Information Form Diagnostic Report Overview Page 1 Big Picture How did I do? Page 3 Diagnostic What can I do differently? Page 2 Learning Details What did students learn? Page 4 Statistical Detail Any additional insights? Your Average (5-point Scale) RawAdj. A.Progress on Relevant Objectives 1 Four objectives were selected as relevant (Important or Essentialsee page 2) If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average. The Big Picture Progress On Relevant Objectives Summary Evaluation: Five-Point Scale Report Page 1 Your Average Score (5-point scale) RawAdj. A.Progress on Relevant Objectives Four objectives were selected as relevant (Important or Essentialsee page 2) Overall Ratings B. Excellent Teacher C. Excellent Course D. Average of B & C Summary Evaluation (Average of A & D) % 25% Using Evidence to Improve Student Learning Individual Reports to Group Reports The Group Summary Report How did we do? How might we improve? Defining Group Summary Reports (GSRs) Institutional Departmental Service/Introductory Courses Major Field Courses General Education Program GSRs Help Address Questions Longitudinal Contextual Curricular Pedagogical Student Learning- focused Adding Questions Up to 20 Questions can be added Institutional Departmental Course-based All of the above Local Code Use this section of the FIF to code types of data. Defining Group Summary Reports Local Code 8 possible fields Example: Column one Delivery Format 1=Self-paced 2=Lecture 3=Studio 4=Lab 5=Seminar 6=Online Example from Benedictine University Example Using Local code Assign Local Code 1=Day, Tenured 2=Evening, Tenured 3=Day, Tenure Track 4=Evening, Tenure Track 5=Day, Adjunct 6=Evening, Adjunct Request Reports All Day Classes Local Code=1, 3, & 5 All Evening Classes Local Code=2, 4, & 6 Courses Taught by Adjuncts Local Code=5 & 6 Description of Courses Included in this Report Number of Classes Included Diagnostic From 42 Short Form 27 Total 69 Number of Excluded Classes 0 Response Rate Classes below 65% Response Rate 2 Average Response Rate 85% Class Size Average Class Size 20 Page 1 of GSR Assessment of Learning What are our faculty emphasizing? How do students rate their learning? How do our courses compare with others? How do our students compare with others (self- rated characteristics)? What efforts can we make for improvement? (How can we close the loop?) Texas A & M University Student Learning OutcomesPossible IDEA Learning Objectives Master the depth of knowledge required for a degree 1, 2, 3, 4, 7 Demonstrate critical thinking11 Communicate effectively8 Work collaboratively5 Practice personal and social responsibility 10, Extra Question Demonstrate social, cultural, and global competence 7, Extra Question Prepare to engage in lifelong learning 9, 12 Are we targeting TAMU SLOs in Core Curriculum? TAMU Core Curriculum Courses ENGL 203XXXX MATH 241XXX BIOL 101XXXX RELS 211XXXX ARTS 103XXX ANTH 201XXX IDEA Learning Objectives Are we targeting TAMU SLOs in Core Curriculum? TAMU Core Curriculum Courses ENGL 203XXXX MATH 241XXX BIOL 101XXXX RELS 211XXXX ARTS 103XXX ANTH 201XXX IDEA Learning Objectives What are We Emphasizing? Percent of Classes Selecting Obj. as Important or Essential This GroupInstitutionIDEA System Objective 116%70%78% Objective 213%59%75% Objective 341%58%75% Objective 432%35%55% Objective 523%19%32% Objective 632%14%25% Objective 722%27% Objective 878%43%47% Objective 919%23%41% Objective 107%11%23% Objective 1128%42%49% Objective 1220%23%41% Average # of Obj. Selected Page 2 What are We Emphasizing? Page 9 Section B Number Rating Percent indicating amount required None or Little SomeMuch Writing662%17%82% Oral Communication666%42%52% Computer Application6650%44%6% Group Work6627%59%14% Mathematics/Quantitative Work 6597%3%0% Critical Thinking6630% 40% Creative/Artistic/Design6661%33%6% Do Students report of learning meet our expectations? Pages 5 and 6 Raw Average Adj. Average # of Classes This Report Institution 4.2 3,963 IDEA System ,991 Objective 1: Gaining factual knowledge (terminology, classifications, methods, trends ) How do students rate their learning? Page 3 Part 1: Distribution of Converted Scores Compared to the IDEA Database Overall Progress Ratings (Courses) Page 3 Percent of Classes at or Above the IDEA database Average Overall Progress Ratings (Courses) Part 3: Percent of Classes at or Above This Institutions Average Page 4 Which teaching methods might we use to improve learning? Page 7 Teaching Methods and Styles Stimulating Student Interest# ClassesAv.s.d. 15. Inspired students to set and achieve goals which really challenged them Relationship of Learning Objectives to Teaching Methods How do students view course work demands? Page 8B Student Ratings of Course Characteristics Diagnostic Form Item # & ItemAverage% Classes Below 3.0 % Classes 4.0 or Above 33. Amount of Reading This Report3.421%24% Institution3.331%19% IDEA System3.233%15% 34. Amount of work in other (non-reading) assignments This Report3.324%10% Institution3.423%20% IDEA System3.421%18% 35. Difficulty of subject matter This Report3.219%0% Institution3.513%19% IDEA System3.420%18% Aggregate Data File Allows you to Use Excel Spreadsheet Use with SAS or SPSS Ask other types of questions Display data in different ways Instructors Reports on Course Emphases: Selected Pairings-Writing and Oral Communication Instructors Reports on Course Emphases: Selected Pairings-Critical Thinking & Writing Benchmarking Institutional and Discipline Reports Benchmarking Reports Comparison to 6-10 Peers Same Carnegie Classification IDEA database Benchmarking Reports The student, rather than the class, is the unit of analysis Percentage of positive ratings is given rather than averages Report Summary Comparison Groups Your University Peer* Carnegie National * Peer group is based on 6-10 institutions identified by your institution Students Perceptions: Gen Ed Background for Specialization Instructional Objectives Selected by Instructors Instructors Intentions/ focus Students Self-Reported Progress on Learning IDEA Objective 3 Learning to apply course material (to improve thinking, problem solving, and decisions) IDEA Objective 8 Developing skill in expressing oneself orally or in writing Teaching Methods and Styles Reported by Students Fostering Student Collaboration Encouraging Student Involvement Using Aggregate Data for Assessment TAMU Student Learning Outcomes Sub-Group Summary Reports Institutional Group Summary Report, Include Extra Questions Benchmarking: One Year or 3-5 Year Trend Report Benchmarking: Discipline Report Core Curriculum Courses in the Major Graduate Level Course Learning Outcomes Course Learning Outcomes Course Learning Outcomes Course Learning Outcomes Accreditation Guides SACS NCATE Guide CACREP Guide Questions?