Identifying and Identifying and Assessing Learning Assessing Learning Outcomes in Psychology Outcomes in Psychology Jerry Rudmann, PhD Jerry Rudmann, PhD Irvine Valley College Psychology Department Irvine Valley College Psychology Department Strengthening Student Success Conference Strengthening Student Success Conference San Jose Fairmont Hotel San Jose Fairmont Hotel October, 2007 October, 2007
41
Embed
Identifying and Assessing Learning Outcomes in Psychology Jerry Rudmann, PhD Irvine Valley College Psychology Department Strengthening Student Success.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Identifying and Assessing Identifying and Assessing Learning Outcomes in Learning Outcomes in
PsychologyPsychology Jerry Rudmann, PhDJerry Rudmann, PhD
Irvine Valley College Psychology DepartmentIrvine Valley College Psychology Department
Psychology’s 10 Categories of Psychology’s 10 Categories of Student LearningStudent Learning
From theFrom theTask Force on Learning Goals & OutcomesTask Force on Learning Goals & Outcomes
Board of Educational Affairs, American Psychological AssociationBoard of Educational Affairs, American Psychological Association20022002
Each of the 10 major SLOs include a subset of specific SLOs Each of the 10 major SLOs include a subset of specific SLOs for the psychology majorfor the psychology major
1 - Theory and Content of PsychologyMajor concepts, theoretical perspectives, empirical findings, and historical trends.
2 - Research Methods in PsychologyUnderstand and apply basic research methods in psychology, including research design, data analysis, and interpretation.
3 - Critical Thinking Skills in Psychology Respect and use critical thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes.
4 - Application of PsychologyUnderstand and apply psychological principles to personal, social, and organizational issues.
5 - Values in PsychologyWeigh evidence, tolerate ambiguity, act ethically, and reflect other values that are the underpinnings of psychology as a discipline.
First 5 Categories
6 - Information and Technological Literacy6 - Information and Technological LiteracyDemonstrate information competence and the ability to use computers and Demonstrate information competence and the ability to use computers and other technology for many purposes.other technology for many purposes.
7 - Communication Skills7 - Communication SkillsCommunicate effectively in a variety of formats.Communicate effectively in a variety of formats.
8 - Sociocultural and International Awareness8 - Sociocultural and International AwarenessRecognize, understand, and respect the complexity of socio-cultural and Recognize, understand, and respect the complexity of socio-cultural and international diversity.international diversity.
9 - Personal Development9 - Personal Development Insight into their own and others’ behavior and mental processes and apply Insight into their own and others’ behavior and mental processes and apply effective strategies for self-management and self-improvement.effective strategies for self-management and self-improvement.
10 - Career Planning and Development10 - Career Planning and DevelopmentEmerge from the major with realistic ideas about how to implement their Emerge from the major with realistic ideas about how to implement their psychological knowledge, skills, and values in occupational pursuits in a psychological knowledge, skills, and values in occupational pursuits in a variety of settings. variety of settings.
Category 2Category 2: Knowledge, Skills and Values Consistent with a Liberal : Knowledge, Skills and Values Consistent with a Liberal Arts Education that are Further Developed in PsychologyArts Education that are Further Developed in Psychology
5
A Second Taskforce Met to A Second Taskforce Met to Further Refine the APA SLOs Further Refine the APA SLOs The Tripartite Curricular ModelThe Tripartite Curricular Model
Three increasing levels of proficiency that Three increasing levels of proficiency that students encounter throughout the students encounter throughout the undergraduate psychology curriculum:undergraduate psychology curriculum:– BasicBasic (skills acquired in the introductory course)(skills acquired in the introductory course)
– DevelopingDeveloping (skills emerging as students (skills emerging as students progress through lower and upper division progress through lower and upper division psychology curriculum)psychology curriculum)
– AdvancedAdvanced (skill levels consonant with (skill levels consonant with completing capstone educational experiences and completing capstone educational experiences and nearing the end of a psychology major)nearing the end of a psychology major)
Aesthetic AwarenessCommunicationCritical Thinking, Problem Solving,
and Information Access and AnalysisGlobal AwarenessPersonal, Professional and Civic
Responsibilities
7
8
Our StepsOur Steps
1. Mapped APA SLOs to IVC SLOs2. Mapped psychology courses to
– APA SLOs– IVC SLOs
3. Identified priority SLOs for our department
4. Created an assessment plan for each SLO
9
American Psych Association (APA) SLOs American Psych Association (APA) SLOs
Mapped to IVC Institutional SLOsMapped to IVC Institutional SLOs
10
Psych Courses Mapped to IVC Psych Courses Mapped to IVC Institutional SLOs Institutional SLOs
11
American Psychological Association (APA) American Psychological Association (APA) SLOs Mapped to IVC Psychology CoursesSLOs Mapped to IVC Psychology Courses
12
Psyc Program SLO Assessment SummaryPsyc Program SLO Assessment Summary
13
A total of five combined institutional and APA SLOs are assessed each semester.
One or more SLOs are being assessed in six of 13 (46%) of the program’s courses.
Four of the five SLOs simultaneously qualify as course, program, and institutional SLOs.
14
Examples of SLO Assessment Examples of SLO Assessment ResultsResults
15
Introductory PsychologyIntroductory Psychology
SLO: Introductory knowledge of psychology – Students will have an introductory familiarity with basic psychological concepts regarding: the brain and nervous system, psychology’s alternative perspectives, names of psychology’s pioneers, research methodology, states of consciousness, developmental psychology, operant and classical conditioning, memory, psychological testing, motivation, personality, and social psychology.
85 students in three Introductory classes completed 30 multiple-choice pre and post-test questions
16
Spring ResultsSpring ResultsMean Total Scores on Psychology Knowledge
Critical Thinking– SLO: Given data sets produced from alternative
research designs, students will correctly identify, apply, and justify the most appropriate statistical procedure.
Communication– SLO: Students will demonstrate the ability to
communicate with statistics by correctly expressing the results of statistical analyses in narrative form and through APA format tables and graphs.
Authentic Assessment in a Statistics Authentic Assessment in a Statistics Course - Course - Five problem final examFive problem final exam
Sample ProblemSample Problem: A psychologist
administered two different forms of a short-memory test to a sample of 8 participants. The first memory test was administered before exposing participants to a stressful situation. The second memory test was administered after the stress exposure.
Question: Do the results show if
stress influenced short-term memory?
Memory Test Scores
Pre-Stress Post Stress
11 10
15 13
14 15
19 15
12 12
16 14
13 12
15 15
This aThis answer sheet accompanies each problemnswer sheet accompanies each problem
1. What statistical test would you use for this problem? ____2. Why would you use that test? (Reflect on the DV’s level of
measurement, the parametric assumptions, and the question the researcher needs to answer)____
3. Ho ___Ha ___
4. Degrees of freedom (if applicable) __5. Critical value of the statistic (.05) ___6. Computed value of the statistic (show work unless you are using
SPSS, StatDisk, or statistical calculator ___7. Decision regarding Ho (accept or reject) __8. How would you report your statistics in the results section of
a journal article? ___9. How would you interpret the results (in everyday language)? __
ID Test Justification Hypothesis DF & CV ObsValue Ho_Decision Journal_Report Eng_Interp TotalPossible Points
Test of Students’ Knowledge of A.P.A. Formatting—Fall 2005
Scores on the APA Formatting Test
30.027.525.022.520.017.515.012.510.07.5
Distribution of Students’ Performances10
8
6
4
2
0
Std. Dev = 5.56
Mean = 24.6
N = 29.00
Frequency
26
Critical Thinking in Introductory Critical Thinking in Introductory Psychology CoursePsychology Course
SLO on Pseudoscience skepticism: Students will correctly identify non-scientific explanations of human behavior and explain why those explanations are not based upon science and do not provide reliable or valid explanations of behavior or predictions of future behavior.
The Pseudoscience Belief TestThe Pseudoscience Belief Test(adapted from…Walker, Hoekstra, & Vogl, (2002). Science education is no guarantee of skepticism, Skeptic, vol 9, no 3.
Please rate how much you believe the following statements. Use the 7-point scale provided.
1 – Do not believe in this at all.2 – I doubt very much that this is real.3 – I doubt that this is real.4 – I am unsure if this is real or not.5 – I believe that this may be real.6 – I believe that this is real.7 – I strongly believe this is real.
__ 1. A person’s personality can be easily predicted by their handwriting. __ 2. A person can use their mind to see the future or read other people’s thoughts.__ 3. A person’s astrological sign can predict a person’s personality and their future.__ 4. An ape-like mammal, sometimes called Bigfoot, roams the forests of America.__ 5. The body can be healed by placing magnets on to the skin near injured areas.__ 6. Healing can be promoted by placing a wax candle in your ear and lighting it.__ 7. A dinosaur, sometimes called the Lock Ness Monster, lives in a Scottish lake.__ 8. Sending chain letters can bring you good luck; ignoring them can bring you bad luck.__ 9. The government is hiding evidence of alien visitation at places such as Area 51.__ 10. Voodoo curses are real and have been known to kill people.__ 11. A broken mirror can bring you bad luck for many years.__ 12. Houses can be haunted by the spirits of people who have died in tragic ways.__ 13. Water can be accurately detected by people using “Y” shaped tree branches.__ 14. Animals, such as cats and dogs, are sensitive to the presence of ghosts.
28
Critical Thinking Experiment Using a Critical Thinking Experiment Using a SLO as the Dependent VariableSLO as the Dependent Variable
Pseudoscience Belief Pre-test
Randomly Assigned 90
Students
Calibrated Peer Review Lesson on Graphology
Pseudoscience Belief Post-test
Calibrated Peer Review Lesson
on Classical Conditioning
Statistical SummaryStatistical Summary
Graphology Belief ScoresGraphology Belief Scores
TreatmentTreatment
GroupGroup
Pre-test Average
Post-test
Average
Paired t-tests
GraphologyGraphology 4.41 2.33 t(26) = 6.40
p < .01
Conditioning Conditioning 4.12 3.69 t(25) = 1.31
p = ns
t(51) = 0.67
p = ns
t(46.7) = 2.93
p < .01
Mean Pre and Post-Test Scores on Mean Pre and Post-Test Scores on Graphology Belief QuestionGraphology Belief Question
00.5
11.5
22.5
33.5
44.5
5
Pre-test Post-test
Graphology
Conditioning
Statistical SummaryStatistical Summary
Astrology Belief ScoresAstrology Belief Scores
GroupGroup
Pre-test Average
Post-test
Average
Paired t-tests
GraphologyGraphology 3.30 2.52 t(26) = 2.42
p < .05
Conditioning Conditioning 3.27 2.73 t(25) = 1.49
p = ns
t(51) = 0.45
p = ns
Mean Pre and Post-Test Scores on Mean Pre and Post-Test Scores on Astrology BeliefAstrology Belief QuestionQuestion
0
0.5
1
1.5
2
2.5
3
3.5
Pre-test Post-test
Graphology
Conditioning
33
Calibrated Peer Review in Calibrated Peer Review in Psychology 101Psychology 101
http://cpr.molsci.ucla.edu/
34
Example EssayExample EssayThe Detection of a Pseudoscience: Graphology
Elaine Quigley’s posting on the website www.businessballs.com is littered with “red flags” that expose graphology as the pseudoscience/pseudopsychology that it is. While an attempt to promote graphology, Quigley’s posting fails to measure up to several of Cotton and Scalise’s guidelines for “baloney detection.” This paper will examine four areas in which graphology fails to live up to its claim of being “science.”In an attempt to display graphology’s validity, Quigley cites the notion that it is “a very old and respected science.” The fact that it has existed for approximately 3,000 years is used to justify Quigley’s notion that graphology is a science. However, one educated in the definition of science knows that the age of a theory is not a factor used to determine its validity. In fact, there are many beliefs that have been around for thousands of years that cannot be tested and therefore cannot be deemed as scientifically reliable. Graphology is just one of many ideas that cannot be justified despite their age.Quigley also fails to tell how the “science” of graphology has been tested and proven. Instead, she simply states that graphology is a “reliable indicator of personality and behavior” and expects her readers to accept this statement as fact. She also mentions that “the science is still being researched and expanded.” This is the extent to which she approaches the issues related to the research of graphology. Without explaining the testing that was done to prove the methods reliability, how is one to know that graphology is indeed reliable? Indeed, the answer is simple. It is impossible to be sure of the reliability of a measure of personality if the measure itself cannot be tested.In addition to not presenting methods for testing the claims of graphology, Quigley also fails to present evidence in support of its validity. Instead, she simply states that “it is not easy to explain how and why graphology works, nevertheless it continues to be used, respected and appreciated by many.” Could it be that the only “evidence” for the reliability of graphology is the satisfaction that its users experience? Unfortunately, being “used” and “accepted” characteristics required of a science.Finally, the vast majority of information provided by Quigley is anecdotal and leads up to a sales pitch for her services. She provides vague stories about how graphology has been used to produce more successful hiring processes and personal relationships. The information is presented more as an advertisement than a scientific work. Quigley goes into more detail on her experience as a graphologist than she does on the aspects of graphology that would qualify it as a science.In conclusion, it is quite clear that based on the evidence presented in this paper, graphology qualifies as a pseudoscience rather than a science. The claims of graphologist Elaine Quigley fail to show that graphology is indeed a science. Instead, she relies on the age of graphology and anecdotal evidence in support of graphology while ignoring issues related to methods for testing graphology’s claims and the results that have resulted in tests of its validity. Looking critically at “discoveries” is no doubt a useful tool that extends beyond the subject of graphology. The methods for recognizing pseudosciences compiled by Cotton and Scalise are certainly tools that would empower all people and prevent them from being fooled by pseudoscientific claims.
35
Questions and Answers for CPR Peer Questions and Answers for CPR Peer ReviewersReviewers
1. Did the essay begin with a topic sentence?2. Was the essay free of spelling and grammatical errors?3. Did the essay present at least four (4) different reasons for supporting or
denying the validity of graphology (or handwriting analysis) as a method of assessing personality and/or predicting behavior?
4. Did the essay have balance? Although this may seem subjective, do you feel that it provided a balance among each of the points made? For example, was each point was explained in the same amount of detail.
5. Did the author's arguments seem convincing to you?6. Did the author conclude with any reflection about whether this assignment
was or was not helpful to his or her learning? In other words, did the author indicate that this assignment might help him or her judge the validity explanations of behavior encountered in the popular media (newspaper, radio, TV, magazines, etc.)?
7. How would you rate this text? (Scale of 1 – 10)