“Strategies for On-Line Assessment” Presented by: JulieAnne Adamich Ph. D., CPA St. Petersburg College E-mail address: [email protected] Florida Association.

“Strategies for

On-Line

Assessment”Presented by:

JulieAnne Adamich Ph. D., CPA

St. Petersburg College

E-mail address: [email protected]

Florida Association of Accounting

Educators

Fall 2004

Define the purpose of the Assessment Process

• It is a Formative evaluation process

• based on the Continuous collection and analysis of data

• with the primary purpose of emphasizing Improvement of:- student learning- student development- the program within the institution

Assessment accomplishes two important goals

“Improvements in learning will depend on how well assessment, curriculum, and instruction are aligned and reinforce a common set of learning goals” Pellegrino, et al (2001,p. 25)

It also is a way of ensuring that students have been exposed to and have “acquired” the major learning outcomes …. essential for successful job placement.

Assessment is an integral part of the Planning and Evaluation

Process

Program/Course Assessment

Program ReviewsStrategic Planning

(Budgeting)

Formative Goals: Continuous Improvement

Summative Goals: Accountability, Performance Evaluation

Both Formative and Summative Goals:

Focus: Allocating Resources

Another reason- “Accreditation” Requirements for

SACS Reaffirmation• “The institution engages in ongoing, integrated, and

institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement and (b) demonstrates that the institution is effectively accomplishing its mission.” (Core Requirement 5)

• “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.” (Comprehensive Standard 16)

Goals for Developing On-Line Assessments

• Select questions that assess learning objectives taught in a course/program.- Textbook End of Chapter assignments and test banks are typically categorized by learning objective.

• Select questions that match the taxonomy level of the student’s learning development.

• Utilize technology incorporated into Learning Management Systems (such as WebCT) which include quizzing modules to minimize instructor intervention during the assessment process to permit time for instructor focus on the analysis of detailed reports available to the instructor and the provision of student feedback.

• Incorporate quiz features/settings that discourage or prevent cheating.

On-Line Assessment Resources

Quizzing Assessments using WebCT (which is similar to other Learning Management Systems such as Blackboard, eCollege, ANGEL etc.)

Pre-formatted matching and multiple-choice

quizzes are provided to assess each chapter in

Thomson Learning’s Course E-Packs.

Quizzes can also be created directly in WebCT (or imported from a Test Bank, Respondus etc.)

For matching questions, select

the correct answer by clicking on the drop down box.

Answers are saved by clicking on “Save Answer”

Students know the status of

each question

at all times.

For multiple choice questions, students

click on the button to the left of the correct

answer.

On-Line Quizzing Security Features

• Develop question sets and randomize answers to generate a “unique” quiz for each student.

• Deliver questions one at a time to discourage students from printing the entire quiz.

• Limit the quiz duration to discourage students from using/sharing resources.

• Passwords, quiz availability, selective release settings limit the availability of the quiz.

There are a variety of question delivery options to promote a

secured environment.

Question sets can also be developed in

WebCT. For example, the system randomly

selects either MC#1 OR MC#2. Each exam is therefore unique to

discourage cheating.

There are also settings for each question, where the answer order

can be randomized.

Students verify that all

questions have been

answered …

… and upon completion of the quiz click “Finish”

Webct provides the opportunity for students to

receive immediate feedback.

There are a variety of options for

student feedback.

Scores can be released at various stages

Question feedback (such as

the correct answer, the total

score, and grader’s

comments) are available options.

Instructors can also view statistical information

regarding the performance of the entire class on each quiz.

Test Item AnalysisReporting Features available in WebCT

* Item Difficulty Index

* Item Discrimination Index

• Group Performance Indices:

- Standard Deviation

- Mean

Item Difficulty IndexUpper 25%: the percentage of students in the upper 25% of

quiz scorers who answered this question correctly.

Whole Group: the percentage of students

who answered the question correctly.

Lower 25%: the percentage of students in

the lower 25% of quiz scores who answered this

question correctly.

Item Discrimination Index

Discrimination: an indication as to how well the question relates

between high versus low scorers. Higher values indicate better

discrimination. A negative value indicates that the lower-scoring portion of the class performed better on this question than the

high-scorers.

Good Discrimination

Low Discrimination

Group Performance

N: the total number of all

student responses for the

question.

Mean: the average score of all students

for the question.

SD: the standard deviation of the results

from the average score.

Analyze assessment results and prepare an

assessment report

Analyze each major learning outcome, based on the statistical results. Include the following:

• Criteria for Success, i.e., expected statistical results

• Summary of Assessment Findings, i.e., actual statistical results

• Use of Results, i.e., analyze results to determine if planned program improvements are deemed necessary

• Action Plan and Timetable for Implementation• Budgetary and Planning Implications

AICPA Educational Competency Assessment (ECA) Web Site

The Educational Competency Assessment (ECA) Web site is an online resource for educators. The software is designed to guide accounting educators through the process of assessing a program or course.The purpose of the site is to help educators develop or reform curricula and courses to support the development of a set of competencies, consistent with the findings of the CPA Vision. Register free at:

http://eca.aicpaservices.org

Organizers(A series of templates to develop a database for your

institution with the capability to sort by selected course and/or program)

• Evaluate Competency Performance (ECC) Organizer:

- Documents your comparison of the actual level to your desired level of coverage of competencies to determine the degree to which the AICPA Core Competencies are covered in your courses and programs.

- Organized by Competency Categories, evaluate your perceived degree of:* Actual Competency Coverage* Desired Competency Coverage* Desired Learning Outcome* Overall Conclusion

• Assess Student Performance (ASP) Organizer:

- Documents the process of gathering, analyzing and using assessment information to determine how well students are developing the AICPA Core Competencies.

- Areas of input include:* Describing the Assessment Activity* Documenting Findings* Drawing Conclusions* Recommending Improvements

Quizzing Assessments Major Pros

• Explicit demonstration of knowledge

• Efficiently examines a wide range of content knowledge

• Can use well-established practices to evaluate reliability and validity

• Inexpensive to administer and evaluate for a large number of students

• If embedded in courses, allow for more representative or complete student samples

• If test is used multiple times, can obtain longitudinal/trend or pre/post data

Source:

AICPA Educational Competency Assessment (ECA) Web Site

Register free at:http://eca.aicpaservices.org

Major Cons – Mitigated using On-line Assessments

• Labor-intensive to develop valid and reliable questions – Test Bank – Test Bank questions can be imported into WebCT.questions can be imported into WebCT.

• May be unable to generalize performance to other tasks/ Less “authentic” because they do not simulate conditions of work – – Personal Trainer “simulates” detailed exercises and problems. Personal Trainer “simulates” detailed exercises and problems.

• Student participation rates and motivation levels may be low for non-graded examinations. Limited feedback to students if they are not allowed access to results for individual questions – Quiz settings – Quiz settings provide options of hints, solution, and IMMEDIATE feedback upon provide options of hints, solution, and IMMEDIATE feedback upon completion of the quiz. completion of the quiz.

• Standardized tests may encourage instructors to ‘teach to the test’ and direct student focus away from competencies – Question sets – Question sets may be developed to randomize questions based on criteria (such may be developed to randomize questions based on criteria (such as learning outcome, competency level, etc.).as learning outcome, competency level, etc.).

Resources for Faculty

* BEEP – Best Educational E-Practiceshttp://www.spcollege.edu/eagle/research/beep/beep38.htmArchive includes information associated with the following

resource categories:- Assessment and Evaluation- Innovative Technologies- Instructional Resources

• Merlot. Well-known and growing collection of peer-reviewed online learning materials and more. Officially titled Multimedia Educational Resource for Learning and Online Teaching. http://www.merlot.org