Enhancing & Enhancing & Evaluating Student Evaluating Student Success: Integrating Success: Integrating LMS and Assessment LMS and Assessment Delia Heck Delia Heck Director, Assessment and Institutional Research Director, Assessment and Institutional Research Assistant Professor of Environmental Science Assistant Professor of Environmental Science Christine Stinson Christine Stinson Chief Information Officer Chief Information Officer Ferrum College, Ferrum, Virginia Ferrum College, Ferrum, Virginia Presentation at AIR Forum, June 6, 2007 Presentation at AIR Forum, June 6, 2007 www.ferrum.edu/dheck www.ferrum.edu/dheck
39
Embed
Enhancing & Evaluating Student Success: Integrating LMS and Assessment
Enhancing & Evaluating Student Success: Integrating LMS and Assessment. Delia Heck Director, Assessment and Institutional Research Assistant Professor of Environmental Science Christine Stinson Chief Information Officer Ferrum College, Ferrum, Virginia - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Success: Integrating Success: Integrating LMS and AssessmentLMS and Assessment
Delia HeckDelia HeckDirector, Assessment and Institutional ResearchDirector, Assessment and Institutional ResearchAssistant Professor of Environmental ScienceAssistant Professor of Environmental Science
Christine StinsonChristine StinsonChief Information OfficerChief Information Officer
Ferrum College, Ferrum, VirginiaFerrum College, Ferrum, VirginiaPresentation at AIR Forum, June 6, 2007Presentation at AIR Forum, June 6, 2007
www.ferrum.edu/dheckwww.ferrum.edu/dheck
MotivationMotivation
• Many educational institutions use Learning Many educational institutions use Learning Management Systems (LMS) to deliver Management Systems (LMS) to deliver and enhance student learning. and enhance student learning.
• These same institutions are under These same institutions are under increasing pressure to establish effective increasing pressure to establish effective processes for evaluating (and improving) processes for evaluating (and improving) student learning outcomes.student learning outcomes.
Motivation continuedMotivation continued
• How do you use your LMS to get How do you use your LMS to get assessment data?assessment data?
• Can LMS aggregate and export Can LMS aggregate and export information useful in evaluating student information useful in evaluating student success?success?
OverviewOverview• Purpose of today’s presentation:Purpose of today’s presentation:
– Review how we are integrating Assessment Process and LMS.Review how we are integrating Assessment Process and LMS.– Share what we are doing, get your feedback and suggestions.Share what we are doing, get your feedback and suggestions.
• Background on Ferrum CollegeBackground on Ferrum College• Developing a Culture of Assessment at Ferrum College.Developing a Culture of Assessment at Ferrum College.• Process of Choosing Assessment SoftwareProcess of Choosing Assessment Software• Ferrum’s experience with LMS: 3 systems in last 5 years.Ferrum’s experience with LMS: 3 systems in last 5 years.
• What What areare we doing now: Assessment, ANGEL 7.1, and ePortfolio. we doing now: Assessment, ANGEL 7.1, and ePortfolio.• What What willwill we be doing: Assessment, ANGEL 7.2, and ePortfolio we be doing: Assessment, ANGEL 7.2, and ePortfolio
• Your suggestions, questions, comments, and discussion.Your suggestions, questions, comments, and discussion.
Ferrum College
• Liberal arts college affiliated with the Methodist Church
• Mainly residential undergraduate population.• 1000-1100 students, 60% male, 40% female• SW Virginia (eastern Appalachian region).• Member of 36-school Appalachian College
Association.• Accredited by Southern Association of Colleges and
Schools (SACS).
Overview and review of Overview and review of Assessment.Assessment.
AssessmentAssessment
• What is assessment?What is assessment?– Systematic process to gather, analyze & Systematic process to gather, analyze &
interpret information for continuous interpret information for continuous improvement of student learningimprovement of student learning
• What are learning outcomes?What are learning outcomes?– Know, Do, & Be LikeKnow, Do, & Be Like
• What do we do with assessment What do we do with assessment results?results?– Improve student learningImprove student learning– Improve pedagogyImprove pedagogy– Collect learning artifactsCollect learning artifacts
Assessment ProcessAssessment Process
• Assessment should:Assessment should:– Set standards for programs & coursesSet standards for programs & courses– Evaluate progress & provide feedback Evaluate progress & provide feedback
to students and program coordinatorsto students and program coordinators– Motivate & improve student learningMotivate & improve student learning– Help the institution enhance its Help the institution enhance its
effectivenesseffectiveness
Assessment SystemsAssessment Systems• Assessment Systems provide valuable Assessment Systems provide valuable
feedback feedback – How students are being shaped and transformed How students are being shaped and transformed – Faculty learn how they can improve their own Faculty learn how they can improve their own
pedagogypedagogy• Coordination of LMS and Assessment Systems Coordination of LMS and Assessment Systems
is potentially valuable, but…is potentially valuable, but…– faculty can be reluctant to embrace new programs faculty can be reluctant to embrace new programs
that require large investments of time. that require large investments of time. • Need to design assessment systems that meet Need to design assessment systems that meet
the simultaneous needs ofthe simultaneous needs of– Individual facultyIndividual faculty– Programs or departmentsPrograms or departments– Divisions or unitsDivisions or units– Overall institution Overall institution
Evaluation of Assessment Evaluation of Assessment software alternatives.software alternatives.
• Key issues:Key issues:– How to capture learning artifactsHow to capture learning artifacts– How to tie learning artifacts to student How to tie learning artifacts to student
learning outcomeslearning outcomes– How to aggregate data about student How to aggregate data about student
learning outcomes across courses, learning outcomes across courses, programs & institutionprograms & institution
– Any redundancy of work between LMS Any redundancy of work between LMS and assessment system will render and assessment system will render process “dead-in-the-water”process “dead-in-the-water”
• Two on-site, open demonstrations Two on-site, open demonstrations for campus communityfor campus community– TaskStreamTaskStream– True OutcomesTrue Outcomes
• Participants at demonstrations Participants at demonstrations completed evaluation sheets for completed evaluation sheets for each product.each product.
Sample Evaluation FormSample Evaluation FormAssessment Software Evaluation Form
Software Package : Evaluator :
Criteria Ranking
1 = poor, 5 = excellent Develops a matrix that tracks stated outcome objectives from the course to the program to the institution
1 2 3 4 5 NA
Ease of inputting learning goals & outcome objectives at the course level 1 2 3 4 5 NA Ease of inputting learning goals & outcome objectives at the program level 1 2 3 4 5 NA Ease of inputting learning goals & outcome objectives at the campus level 1 2 3 4 5 NA Provides a library of learning outcome samples 1 2 3 4 5 NA Assess any program, whether curricular or co-curricular 1 2 3 4 5 NA Provides online instructions for writing learning goals and outcome objectives 1 2 3 4 5 NA Provides online instructions for entering data and/or completing forms 1 2 3 4 5 NA E-portfolio module 1 2 3 4 5 NA Track and analyze individual students 1 2 3 4 5 NA Specific measurement results by student 1 2 3 4 5 NA Track summative or formative evaluation 1 2 3 4 5 NA Track data across multiple programs or outcomes 1 2 3 4 5 NA End-user surveys (can administrator, faculty or others test or survey others, e.g., students, alumni, businesses, etc.)
1 2 3 4 5 NA
"Closing the loop" reports 1 2 3 4 5 NA Time per person/per semester required to administer 1 2 3 4 5 NA Interface with our current student records and registration? 1 2 3 4 5 NA Server platform and requirements, if any? 1 2 3 4 5 NA Does it require on-site technical support person? 1 2 3 4 5 NA If so, what percentage of that person's workload should be allocated? 10% 20% 30%
40% 50% 60% 70% 80% 90%
100% NA Cost - First year including setup with data input? 1 2 3 4 5 NA Cost - Following years? 1 2 3 4 5 NA Other costs? 1 2 3 4 5 NA Overall Impression of the software 1 2 3 4 5 NA Comments:
• Learning Outcomes • Sample Learning Outcomes• National Standards• Co-curricular• On-line instructions• ePortfolio• Analyze & Track individual outcomes• Summarize across programs• Track data across outcomes• Interface with ERP• Technical Support• Time requirements• Costs – initial, direct, indirect
Ferrum College’s ProcessFerrum College’s Process
• All assessment software products All assessment software products were either too limited or too were either too limited or too complex.complex.
• If the assessment process isn’t If the assessment process isn’t simple, faculty and staff members simple, faculty and staff members won’t do it well or happily.won’t do it well or happily.
Overview and review of Overview and review of Learning Management Learning Management
Systems.Systems.
Ferrum College and LMSFerrum College and LMS
• We had been searching for LMS that combined flexibility, ease of use, and allowed us to achieve institutional goals.
• 2003-2004, 2004-2005 – used a Moodle-based system.
• 2005-2006 – WebCT Vista (with Appalachian College Association)– Not pleased with cost/performance combination.– Needed LMS that integrated with our Assessment Process.
Evaluation of LMS Evaluation of LMS AlternativesAlternatives
<--- <--- Open sourceOpen source ---- ---- ----- ----- Private butPrivate but ----- -----
Database completely Database completely accessible to usersaccessible to users
Database partially Database partially accessible for feeaccessible for fee
No current support for No current support for Learning Outcomes.Learning Outcomes.
Newest/pending Newest/pending releases support releases support multiplemultiple LOs per LOs per
Assignment.Assignment.
BlackboardBlackboard: next : next release supports release supports oneone LO per assignment.LO per assignment.
WebCTWebCT: no LOs.: no LOs.
Ferrum College and LMSFerrum College and LMS
• We have been searching for LMS that combined flexibility, ease of use, and allowed us to achieve institutional goals.
• 2003-2004, 2004-2005 – used a “free” Moodle-based system.
• 2005-2006 – WebCT Vista (with Appalachian College Association)– Not pleased with cost/performance combination.– Needed LMS that integrated with our Assessment Process.
• 2006-2007, 2007-2008 – Happly home with ANGEL 7.1 (and, soon, 7.2).
Outcome and progress Outcome and progress at Ferrum College.at Ferrum College.
Ferrum College and Ferrum College and ANGELANGEL
Faculty Perspective…Faculty Perspective…
• Extensive training available for faculty.Extensive training available for faculty.• Strong (voluntary) adoption in first Strong (voluntary) adoption in first
year.year.• 60% of sections used ANGEL.60% of sections used ANGEL.• High faculty (and student) satisfaction.High faculty (and student) satisfaction.
How are we integrating How are we integrating Assessment and ANGEL?Assessment and ANGEL?
• Developing faculty culture of Developing faculty culture of assessment.assessment.
• Encouraging and supporting use of Encouraging and supporting use of ANGEL 7.1 to deliver course content.ANGEL 7.1 to deliver course content.
• Encouraging and supporting use of Encouraging and supporting use of ePortfolio to store artifacts (evidence ePortfolio to store artifacts (evidence of Learning Outcomes).of Learning Outcomes).
Faculty Culture of Faculty Culture of AssessmentAssessment
• Faculty owned and managed the process of Faculty owned and managed the process of developing assessment goals and methods.developing assessment goals and methods.
• Used pilot projects (course-level and program-Used pilot projects (course-level and program-level) to gradually introduce process.level) to gradually introduce process.
• Identified key players, both on early adopters-Identified key players, both on early adopters-side as well as the objectors and worked to get side as well as the objectors and worked to get their buy-in and find out their concerns.their buy-in and find out their concerns.
Example of using ANGEL Example of using ANGEL to undertake Course to undertake Course
AssessmentAssessment
• Professor used ANGEL Gradebook to Professor used ANGEL Gradebook to record grades for subsets of major record grades for subsets of major exams.exams.
• Each subset corresponded to course Each subset corresponded to course Learning Outcome.Learning Outcome.
ACC 202 A - Learning ACC 202 A - Learning OutcomesOutcomes
1. Be able to record economic events as accounting journal entries.
2. Be able to aggregate journal entries into a new Income Statement, Balance Sheet, and Statement of Cash Flows.
3. Be able to compute and interpret simple financial accounting ratios.
ACC 202 A - ResultsACC 202 A - Results
1. 19 out of 20 students who took the Final Exam scored 70% or higher on Learning Outcome 1.
2. Only 3 of those 20 students scored 70% or higher on Learning Outcome 2.
3. 14 of 20 students scored 70% or higher on Learning Outcome 3 (5 of the 6 who scored poorly ran out of time on the exam).
ePortfolio and ePortfolio and AssessmentAssessment
• Used by independently-accredited Used by independently-accredited program now (Teacher Education).program now (Teacher Education).
• Used by Director of Assessment now Used by Director of Assessment now to collect evidence for College-level to collect evidence for College-level Learning Outcomes.Learning Outcomes.
••Explore an area of interest Explore an area of interest through reading and research through reading and research
••Evaluate the validity, Evaluate the validity, perspectives, and contexts of perspectives, and contexts of information information
••Use problem-solving skills Use problem-solving skills creatively to synthesize an creatively to synthesize an effective response effective response
••Articulate a personal Articulate a personal aesthetic with regard to aesthetic with regard to human expression and the human expression and the natural worldnatural world
••Demonstrate a depth of Demonstrate a depth of knowledge in their chosen knowledge in their chosen fields fields
••Demonstrate the skills to Demonstrate the skills to enhance the transition to enhance the transition to employment and/or graduate employment and/or graduate and professional educationand professional education
••Work cooperatively and Work cooperatively and effectively with others. effectively with others.
••Use technology appropriate Use technology appropriate to their chosen fields to their chosen fields
••Integrate personal integrity Integrate personal integrity with professional ethics with professional ethics appropriate to their fields appropriate to their fields
••Apply the concepts and skills Apply the concepts and skills gained in the liberal arts core gained in the liberal arts core to the major to the major
Professional Capability
ePortfolio and ePortfolio and AssessmentAssessment
• Used by independently-accredited programs Used by independently-accredited programs now.now.
• Used by Director of Assessment now to collect Used by Director of Assessment now to collect evidence for College-level Learning Outcomes.evidence for College-level Learning Outcomes.
• Next year: use by Program Coordinators to Next year: use by Program Coordinators to collect evidence of Program Learning collect evidence of Program Learning Outcomes.Outcomes.
• Next year: use by students for personal Next year: use by students for personal portfolios with Academic Advisor & Student portfolios with Academic Advisor & Student Affairs.Affairs.
Knowledge of Knowledge of Fundamental Fundamental
Ecological Ecological PrinciplesPrinciples
Quantitative Quantitative FoundationFoundation
Basic Basic Environmental Environmental
ParametersParameters
ESC 110 Introduction to ESESC 110 Introduction to ES
GEOG 221 Geographic GEOG 221 Geographic Information SystemsInformation Systems
Environmental Science Program
What are our ANGEL 7.2 What are our ANGEL 7.2 plans?plans?
• Import College-level and Program-level Import College-level and Program-level Learning Outcomes.Learning Outcomes.
• Train faculty to use these for their Train faculty to use these for their assessment activities. assessment activities.
• Develop scheduled export of results to Develop scheduled export of results to Program Coordinators, Associate Deans, Program Coordinators, Associate Deans, Dean, and Director of Assessment.Dean, and Director of Assessment.– Some reports built into system, others built Some reports built into system, others built
by queryby query
What potential problems What potential problems exist?exist?
• Differences in vocabulary between our internal Differences in vocabulary between our internal culture and ANGEL 7.2 (e.g., “Standards”).culture and ANGEL 7.2 (e.g., “Standards”).
• Affects faculty perception of difficulty of Affects faculty perception of difficulty of adoption.adoption.
• Not all faculty use LMS.Not all faculty use LMS.
• Not all assessments are conducive to on-line Not all assessments are conducive to on-line environment, but grades & feedback can be environment, but grades & feedback can be entered.entered.
SummarySummary
• ANGEL 7.1 & 7.2 and ePortfolio are ANGEL 7.1 & 7.2 and ePortfolio are already supporting assessment of already supporting assessment of college learning outcomes.college learning outcomes.
• Implemented solutions are institution-Implemented solutions are institution-specific.specific.
• Process is key for our faculty and staff.Process is key for our faculty and staff.• Software has not yet caught up with our Software has not yet caught up with our
needs, but we are getting there slowly.needs, but we are getting there slowly.
NotesNotes• Blackboard: Blackboard: http://www.fool.com/news/commentary/2006/commentary06081803.htmhttp://www.fool.com/news/commentary/2006/commentary06081803.htm• eCollege: eCollege: http://www.fool.com/news/mft/2006/mft06081123.htmhttp://www.fool.com/news/mft/2006/mft06081123.htm• 2004 review of LMS: 2004 review of LMS: http://www.obhe.ac.uk/cgi-bin/keyresource.pl?resid=24http://www.obhe.ac.uk/cgi-bin/keyresource.pl?resid=24• UK Market share: UK Market share: http://mfeldstein.com/index.php/weblog/comments/359/http://mfeldstein.com/index.php/weblog/comments/359/
– combined market share of Blackboard and WebCT declined from 56.8% in 2001 to combined market share of Blackboard and WebCT declined from 56.8% in 2001 to 43.7% in 2005.43.7% in 2005.
– share for Open Source went from 0% to 11%, with 8% of the gains happening in the last share for Open Source went from 0% to 11%, with 8% of the gains happening in the last 2 years.2 years.
– number of homegrown VLE’s jumped from 7% in 2001 to 30% in 2005.number of homegrown VLE’s jumped from 7% in 2001 to 30% in 2005.• Desire2Learn: Desire2Learn: http://www.desire2learn.com/http://www.desire2learn.com/
– Canada's 18th fastest growing technology company (2006 Deloitte Technology Fast 50 Canada's 18th fastest growing technology company (2006 Deloitte Technology Fast 50 program). program).
on-line-learning-market-may-be-monopolized.htmlon-line-learning-market-may-be-monopolized.html• Moodle: As of July 15, 2006 there were some 13,526 Moodle sites involving Moodle: As of July 15, 2006 there were some 13,526 Moodle sites involving
nearly 45,000 courses, five million users, over seven million enrollments, and nearly 45,000 courses, five million users, over seven million enrollments, and over 150 countries. over 150 countries. http://www.isu.edu/itrc/resources/moodle-info.shtmlhttp://www.isu.edu/itrc/resources/moodle-info.shtml