Top Banner
1 ChemEd 2003 Auburn University, July 27- 31, 2003
50
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

1

ChemEd 2003Auburn University, July 27-31, 2003

Page 2: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

2

A Methodology for the Simultaneous Assessment of

Cognitive and Content Domains

Ray A. Gross, Jr.

Prince George’s Community CollegeLargo, Maryland

Page 3: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

3

My Final Objective for Today

• To provide a methodology for identifying student weaknesses--both in content areas and cognitive skills

• So that you may use the methodology to find weaknesses and then devise strategies to overcome the weaknesses

Page 4: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

4

My Intermediate Objective

• To show how to develop a set of desired learning outcomes (DLOs) that reflect what is actually tested and can also be assessed

Page 5: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

5

Outline of Presentation

1. Define terms

2. Develop content categories

3. Marry content and cognitive categories in a matrix

4. Write DLOs from the matrix

5. Code exam questions for assessment

6. Prepare an assessment matrix

Page 6: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

6

What is a DLO?

A statement written by an instructor to a student telling the student by an action verb (related to a cognitive skill) what the instructor expects the student to be able to demonstrate during a semester for a specific topic (related to a content area).

Page 7: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

7

What is a DLO?

• A statement the instructor can use to code exam questions so that the questions can be assessed by content area and cognitive skill in a systematic and consistent fashion.

Page 8: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

8

Example of a DLO

• Predict products for name reactions such as the Grignard and Wittig reactions.

• A DLO completes the phrase—“Upon successful completion of this course, a student should (or will) be able to:”

Page 9: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

9

A Set of DLOs

• The number of DLOs determined by the instructor to cover the content domain of the course.

Page 10: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

10

Two Elements in a DLO

• Cognitive Skill—A mental process used by a person in problem solving (general – applies to all courses)

• Content Area—A portion of a course defined by the instructor (specific – applies to a given course)

• Time—A semester

Page 11: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

11

What are the Cognitive Skills?

• Some person’s idea of the mental processes humans use to solve problems (i.e., learn).

• Bloom’s Taxonomy: Six ways of thinking or six cognitive skills humans use to solve problems, rank ordered by complexity. These six skills span all disciplines and are adopted for use herein.

Page 12: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

12

Bloom’s Cognitive Domain

A. Knowledge--rote

B. Comprehension—grasp meaning

C. Application—use material in new situations

D. Analysis—break material into components

E. Synthesis—put parts together

F. Evaluation—use judgment

Page 13: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

13

Content Domain

• A list of content areas that can be subdivided into course topics

• The content domain is course specific and must be determined by the instructor.

Page 14: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

14

Content Domain

• Course topics are derived from content areas in the same way action verbs are derived from cognitive areas.

• An example of a content domain is shown on the next slide for organic chemistry.

Page 15: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

15

A Content Domain (Orgo)

1. Nomenclature2. Equations3. Mechanisms4. Problems 5. Synthesis6. Explanations7. Correlations8. Classifications9. Definitions

Page 16: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

16

Your Content Domain

• Determined initially by what you have tested in the past

Page 17: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

17

Limits of Assessment

• The cognitive domain (six alphabetic items) and content domain (nine numeric items) define the level to which the course is subdivided for assessment purposes.

• For the example, a 6 x 9 matrix defines the parameters for assessment and the corresponding DLOs.

Page 18: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

18

What is a Cognitive-Content Matrix?

• A way of showing how the content and cognitive domains are interrelated

• A way of showing how the two independent variables that make up a set of DLOs are related

• A tool for qualitatively or quantitatively assessing an evaluation or combined evaluations

Page 19: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

19

Cog

Cont

Kno

A

Com

B

App

C

Ana

D

Syn

E

Evl

F

1-- x x x

2-- x x

3-- x

4-- x

5-- x x x

6-- x

Hypothetical Cognitive-Content Matrix

Page 20: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

20

Before Developing Content Domain--we will use Organic Nomenclature to:

• Explore how a given content area may be assessed by all six cognitive skills

• Learn how to dissect a question into its cognitive and content components

• Learn how to code a question by its content and cognitive components

Page 21: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

21

Six Cognitive Codes for each Content Code

• Content area 1 = Nomenclature

• DLOs may be coded from 1-A to 1-F

Page 22: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

22

A. Knowledge—rote

Q1. Write the name of CH4?

Code = 1. Nomenclature

A. Knowledge

• The student is intimately familiar

with the item.

Page 23: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

23

B. Comprehension—grasp meaning

Q2. Give the IUPAC name of CH3CH(CH3)CH3.

Code = 1. Nomenclature

B. Comprehension

• The formula is not new to the student; the instructor discussed it prior to the exam.

Page 24: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

24

C. Application—use material in new situations

Q3. Give the IUPAC name for CH3C(CH3)2CH(CH3)CH3.

Code = 1. Nomenclature

C. Application

• Student has never seen this structure before; nomenclature rules are applied.

Page 25: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

25

D. Analysis—break material into components

Q4. What is the parent hydrocarbon of 3-isopropyl-4-methylheptane.

Code = 1. Nomenclature

D. Analysis

• The name is broken into its component parts to find parent name.

Page 26: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

26

E. Synthesis—put parts together

• Q5. What is the IUPAC name of the structure on your right?

Code = 1. Nomenclature

E. Synthesis

• Component parts must be assembled into a name; requires analysis then synthesis.

Page 27: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

27

F. Evaluation—use judgment

O

N

OH

O

OHCH3O O

N

OH

O

CH3Odehydration rearrangement

?

8 9

Figure 1

• A research group studied the mechanism for the dehydration and rearrangement of the aglucone 2,4-dihydroxy-7-methoxy-2H-1,4-benzoxazin-3(4H)-one (8) into 3-formyl-6-methoxybenzoxazolin-2(3H)-one (9) by the reagent N-ethoxycarbonyltrichloroacetaldimine.

Page 28: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

28

F. Evaluation—use judgment(continued)

• Q6. The carbon atom in 8, indicated by the question mark in Figure 1, should be given what number by nomenclature rules?

• Code = 1. Nomenclature

F. Evaluation

• Judgment is required in comparing two ring systems, which students haven’t seen before.

Page 29: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

29

Summary

• All six cognitive skills (codes A-F) might be tested against nomenclature (code 1).

• In theory, 54 (6 x 9) question types might be formulated for organic chemistry.

• For any course in which Bloom’s taxonomy is adopted, a similar matrix is theoretically possible.

• In practice, only a fraction of the possible codes is needed.

Page 30: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

30

How is the Content Domain Developed?

• By analysis not synthesis

• Analyze previous exams, categorize each Q by content

• Consolidate the content in the questions into a content domain

• Also, categorize each Q by cognitive category as you go (Bloom’s A-F)

Page 31: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

31

Content Development Process

1. Make a content domain of about 8 categories.

2. Code previous questions into one of the categories by number.

3. Add categories if necessary.

4. Consolidate categories if necessary.

Page 32: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

32

The Process is a Retro-Analysis of each Question

5. Concurrently, code each Q by letter—a Bloom cognitive skill (e.g., C).

6. Use the codes to make a qualitative content—cognitive matrix (see slide 18).

Page 33: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

33

Build a Qualitative Cognitive-Content Matrix

• Lay out a content vs cognitive matrix by number and letter codes.

• Each dual-coded Q (e.g., 7D, 4C, and 6F) equates to a box in the matrix.

• Place an X in each box that is represented by a dual-coded Q.

• The result is a qualitative matrix• It shows what you actually tested by content

and cognitive categories.

Page 34: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

34

Qualitative Matrix

• Shows which content and cognitive categories are to be assessed or were assessed

Page 35: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

35

Cog

Cont

Kno

A

Com

B

App

C

Ana

D

Syn

E

Evl

F

1-- x x x

2-- x x

3-- x

4-- x

5-- x x x

6-- x

Page 36: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

36

Subordinate Cognitive Domain to Content Domain

• In order to minimize the number of DLOs, the cognitive domain is subordinated to the content domain, so that only nine DLOs are needed for the 54 possibilities.

• The cognitive skill for a specific question will be evident by the letter code you assign when the question is formulated.

Page 37: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

37

Use the Qualitative Matrix to Develop DLOs

Cognitive

Content

Know.

A

Comp.

B

Appl.

C

Nomen.

1.

X

Identify

Name or Draw

X

(Re)write

Name or Draw

X

Construct or Formulate

Name or Draw

Page 38: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

38

Consolidation of Cognitive Skills

• One or more verbs that encompass all of the cognitive skills tested are married with the content category to arrive at a single DLO for nomenclature.

Page 39: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

39

Construct DLO

• Name chemical species from structures and draw structures from names.

• This is the DLO for nomenclature.

• Develop DLOs for each content category in the same way.

Page 40: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

40

Cognitive-Content Linkage

Application

Applies principles to new situtations

CognitiveGeneral Instructional Objectives Verb for

Specific LO

Predict

Reaction products

Content

Equations

Page 41: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

41

Process Marries Domains

• For the next exam, dual code each Q.

• Assign points for each Q and develop a quantitative matrix.

• Determine max points for each box.

• Calculate student percentages for each box.

• Both content and cognitive skills are assessable.

Page 42: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

42

Dual Coding Process

• Allows instructor to predetermine the mix of content and cognitive skills to be assessed for each evaluation

Page 43: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

43

Coding Example 1

• Which compound, ethanol, ethane or diethyl ether has the highest boiling point?

• Content = 7. Correlation (property—structure) • Cognitive = D. Analysis or B. Comprehension

Page 44: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

44

Note

• In the previous example, an instructor who covered the specific question in class would code the question as a comprehension, whereas an instructor who covered the principles involved but not the specific question would code the question as an application.

• (Added per ChemEd 2003 participation!)

Page 45: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

45

Coding Example 2

• Calculate the degrees of unsaturation in C5H10O.

• Content = 4. Problem• Cognitive = C. Application

Page 46: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

46

Coding Example 3

• Explain why the nitration of bromobenzene is slower than that of benzene yet produces an ortho,para product.

• Cognitive = 6. Evaluation • Content = F. Explanation

Page 47: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

47

A Partial Quantitative Matrix

CognitiveContent

Totals ↓ A B C D E F

1 80% 90 70 80

2 70% 70

3 48% 48

4 70% 66 67 75

Totals %

90 70 55 69 80 75

Page 48: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

48

Quantitative Matrix

• See the Word document (in the folder with these PowerPoint slides) for an actual quantitative matrix.

• The quantitative matrix identifies student strengths and weaknesses.

• A strategy for improving weaknesses can be developed from the identified weaknesses.

Page 49: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

49

Summary• Content areas may be assessed against each

applicable cognitive skill by dual coding test questions.

• Content-cognitive weaknesses can be identified and actions formulated to improve student learning in those areas.

• The skill levels (i.e., degree of higher-order thinking) for a given exam can be determined in advance by systematically coding exam questions as they are prepared.

Page 50: 1 ChemEd 2003 Auburn University, July 27-31, 2003.

50

Comments

• The DLOs provide a methodology for doing a macro-assessment that is useful for instructors.

• For the DLOs to be meaningful for students, examples of coded test questions for each teaching block are probably necessary.