Student Teacher Evaluation Dennis E. Clayson Professor of Business University of Northern Iowa.

Student Teacher Evaluation

Dennis E. ClaysonProfessor of Business

University of Northern Iowa

What is a Construct?

Good teaching is not a thing,it is a “construct.”

This the Ayers Rock in Australia. It can be measured.

It has a location. It has a weight, length, width,and height. It has a temperature, density, and a

chemical composition.

But what about love?

Or motivation?

Or personality?La personalidadPersonlichkeit

Or happiness?

Or satisfaction? satisfacción,Befriedgung

These are called “constructs.”

They are necessary becausetwo people in the exact environment

will not act the same.

WHY?

“Motivation” explains why one person’s behavioris stronger, and more focused than another person’s.

“Personality” explains why one person’s behavioris more animated, or more “sad” than another person’s.

They are called “hypothetical constructs”because they can only be hypothesized.

Hypothesized and measured by watching andmeasuring behavior.

This typically takes multiple measures.

And this can be very be very difficult to do….

But it matters and it can be very important In the real world….

A professor claimed that everyone is a genius at something….

All are types of intelligence:

1. Bodily-kinesthetic2. Musical3. Spatial4. Linguistic5. Logical-mathematical6. Interpersonal7. Intrapersonal 

The problem for education was to discoverwhat type of genius you are and then

develop the knowledge or skill that magnifies that genius……

 But what if it were not true… all of these abilities have a correlation with Spearman’s factor g…

Consider bodily-kinesthetic:  Almost everyone can be taught to play tennis, but few of us could ever be a varsity gymnast. 

There are things that we cannot do, or at least never do as well as others. Perhaps there will always be people left behind…

It could be that:

Half of all people are in the bottom half…One quarter is in the bottom quarter, and ten percent of everyone is in the bottom 10 percent.

And no amount of spending will every change that.

So how to measure somethingcan have powerful impacts

on individuals and on society!

In practical terms, a construct is defined byhow it is measured.

This is called an:

“Operational Definition.”

Operationalization is the process of defining a fuzzy concept so as to make the concept measurable in form of variables consisting of specific observations. In a wider sense it refers to the process of specifying the

extension of a concept.

Percy W. Bridgman 1882-1961

“Intelligence” could be defined as how fast (RT)a person solves a puzzle….

Or “intelligence” could be defined as not getting cheated at the car dealership.

So how does a “good”teacher’s behavior differfrom a “bad” teacher?

Good teaching is not a thing,it is a “construct.”

But more important:What behavior separates a

“GOOD” teacher from a “OK” teacher?

Perhaps, the behavior of teachers is irrelevant. “Good” teachers produce “good” students, “bad” teachers produce

“bad” students.

Irrespective of all this….

Student evaluation of teaching (SET) is usedby almost all modern colleges and universities.…..

But why?

There seems to have beenlittle thought given about WHY we use the method we do.

SET is assumed to be necessary because of theunion of 4 ideas and trends:

1.Philosophically, many educations have adopted the stance thatwhat happens in the classroom is the responsibility of the instructor.

This common sense notion has been taken to the illogicalconclusion that the instructor must be held accountable for

EVERYTHING that happens in the classroom.

2.   Political pressure has been increasing  for educational accountability.

Political actions create bureaucracies where the process and controlling the process is more important than actual outcomes.

When I informed an administrator that our instrumentsshould be abandoned because they were invalid

and may lead to law suits….

He replied, “I am require to evaluate my faculty. Give mesomething better.”

3.The adoption of a student-customer model.

Ironically, the model adopted is consider a bit archaic by marketers.

4.  Over time, individuals have developed vested interest in the instruments.

At least 49 percent of all professorshave above average SETs.

And 20 percent of all professors are in the top 20 percentof all professors.

But that also means that 49 percent of all professorshave SETs below average.

And 20 percent of all professors are in the bottom 20 percentof all professors.

And THAT’S a problem!

If fact, instructors with lowscores are most likely tosay nothing at all…..

So…. We have moved ourselvesinto a position that many do not wish to be in.

“I am still extremely disappointed I took this class. I took it as an elective just because I thought it would be interesting...wrong teacher. The tests were so hard; I'm a straight A student and ended up with my first B. Don't take him…”

Student evaluation of teaching taken at randomfrom the internet as an example.

Teacher A (picked at random… named Smith)Student 1

Teacher AStudent 2

“I like her as a person ,however, I wasn't very fond of her teaching style. The last month all we did was presentations and were left on our own. If you don't learn well this way, don't taker her. Otherwise she is pretty good.”

Teacher AStudent 3

“HE is a guy. His name is Roland Acosta. He's actually pretty good, but his classes take FOREVER. Bring a laptop or small GameBoy and you will be fine.”

Is Teacher A:

a good teacher, or a bad teacher….

Questions:

Is Teacher A:

For that matter:

a man or a woman?

As we would expect from any systemthat could establish merit pay, tenure, andreinforce or humiliate a profession known

for their arrogance….

There have been thousands of articles published on the topic.

This research has raised a number of interesting questions.

1. Who is a good teacher? 

This research has raised a number of interesting questions.

1. Who is a good teacher? 2. How do we measure something? And how do we do thatwith enough accuracy to make decisions that can change another person’s position, salary, job, reputation, and self esteem?  

This research has raised a number of interesting questions.

1. Who is a good teacher? 2. How do we measure something? 3. Do these measurements make someone a better teacher? 4. Do these measurements create an improvement in education? 

Has the quality of American education improvedsince the widespread adoption of SET?

This research has raised a number of interesting questions.

1. Who is a good teacher? 2. How do we measure something? 3. Do these measurements make someone a better teacher? 4. Do these measurements create an improvement in education? 5. In society?

What Has Been Learned?

What Has Been Learned?1. There has been much disagreement…

Myths!!

What Has Been Learned?1.There has been much disagreement…2.Research methodologies have been stretched…

What Has Been Learned?1.There has been much disagreement…2.Research methodologies have been stretched… 3.Defensive camps have been established…

What Has Been Learned?1.There has been much disagreement…2.Research methodologies have been stretched… 3.Defensive camps have been established…4.BUT… in the last 15 to 20 years, a consensus seems to be immerging…..

And that consensus has become increasingly negative,especially when researcher from non-educational

disciplines came into the picture.

A Brief Review of the Literature

Information will come from wherever it can be found.

Do men get better evaluations than women?

1. NO2. But… students do seem to prefer instructors of their

own gender.3. Female instructors may need to display more gender stereotypical behavior than men.4. If a male and female instructor get the same global eval the reasons are likely to be different.

Do physically attractive instructors getbetter evaluations?

Do physically attractive instructors getbetter evaluations?

1. Of course!2. Maybe….

Research with business students showed thatbeing attractive didn’t result in better evals.

But

Being unattractive was a definite negative.

Do professors who do research getbetter evaluations?

Do professors who do research getbetter evaluations?

1. NO!

Research productivity appears to beunrelated to the evaluations.

Do personality traits influence the evaluation process?

Do personality traits influence the evaluation process?

1. Early researchers from education largely denied thatpersonality influenced evaluations.

2. Detractors maintained that SET was not only influenced by personality, but overwhelmingly so.3. Newer research backs the detractors.

The association is so strong, that some have suggestedthat the SETs could be replaced with a personality

inventory with little loss of validity…

Some have suggested that SET essentially creates a

likeability scale.

Does academic rigor lowerthe evaluations?

1. Yes2. No3. Maybe4. Findings are highly dependent upon methodology5. It may also depend upon student norms

Eval

Rigor

Rigor seems to have a level that is “just right.”

And students don’t respect easy classes.

But… they stay away in large numbers from hard classes….

Do the grades students receive changethe evaluations?

Both instructors and students believe that grades change the evaluations!

This issue has been hotly debated. Terms like:

MYTHSInadequate SamplesBad Methodologies

Have been:

tossed around…

As it stands today,the major debate is not

whether there is a relationshipBut

What Causes It.

Six theories have been proposed…

1. Leniency

2. Statistical artifact

3. Teaching effectiveness

4. Motivation

5. Attribution

6. Reciprocity

All of these have been found, butreciprocity seems to be theprimary reason for the relationshipbetween grades and SET.

Are SETs related to Student Learning?

One attempt to establish validity has been the general assumption that students will

learn more from a "good" instructor.

As Peter Cohen affirms, “Even though there is a lack of unanimity on a definition of good teaching, most researchers in this area agree that student learning is the most important criterion of teaching effectiveness.”

Are SETs related to Student Learning?

1. SET is related to the students’ perception of learning.2. Research findings show that the relationship weakens

+ As methodologies and statistical controls have increased + Over time3. There is NO association if learning is objectively measured.4. There is a negative association with deep learning

and for material that must be used later.5. There may be one interesting exception.

Do students tell the truth when responding to SET?

So students tell the truth when responding to SET?

1. Very limited research

Which is VERY odd…

2. Students will purposely falsify answers.

Students estimate that 30% of allsubmitted forms contain informationthat the student knows is false.

Students do not think that this cheating!

Are the evaluations reliable?

Instruments cannot be valid unless they are reliable.

Instruments can be reliable and not valid, but…

Are the evaluations reliable?

1. SET is remarkably consistent A. Between class averages of evaluationsB. Over time

2. SET is remarkably inconsistent when looking at within-class raters

Are student evaluations of teaching valid?

This is a difficult question to answer.

Are student evaluations of teaching valid?

It depends upon a number of factors.

1.How are the evaluations to be used? Instructional improvement Performance evaluations Feedback to third parties

Are student evaluations of teaching valid?

It depends upon a number of factors.

1.How are the evaluations to be used?2.Philosophical implications

Are student evaluations of teaching valid?

It depends upon a number of factors.

1.How are the evaluations to be used?2.Philosophical implications

For example, are students:customers?products?partners?

Are student evaluations of teaching valid?

It depends upon a number of factors.

1.How are the evaluations to be used?2.Philosophical implications3.Definition of the construct4.What is meant by “valid”

There a numerous types of validity,all of which have different purposes and applications.

What is meant by “valid”

Do the evaluations haveFACE validity?

1. Yes

Face validity exists when an instrument appears to be measuring what the respondent thinks it should be measuring.

Since SET instruments are created and sanctioned by the institution, ask questions generally associated with instruction, and are administrated in formalized manners, it would be assumed that SET has face validity.

Do the evaluations haveCONTENT validity?

1. Maybe

Content validity is said to exist if the questions on an instrument can be logically said to cover the domain of the construct that the instrumentis intended to measure.

Two problems:1.No clear definition of the construct.2.Students report having problems using SET to say what they really want to say.

Do the evaluations haveCONCURRENT and PREDICTIVE validity?

1. Yes

Especially if the comparative instruments are also SETs.

Do the evaluations haveCONSTRUCT validity?

Do the evaluations haveCONSTRUCT validity?

It depends upon how thisvalidity is measured.

Do the evaluations haveCONVERGENT validity?

1. Yes

SET appears to have convergent validity in that the results are related to other measures that could logically be assumed to be associated with the domain of “effective teaching.”

Problem:1.Convergent validity alone does not establish construct validity.2.A measure could be associated will all aspects of a construct simply because it is associated with almost everything!

Do the evaluations haveDIVERGENT validity?

1. No

Construct validity must also show that the instrument is not associated with aspects of unrelated constructs (divergent validity).

Problem:1.SET is highly related to factors that few would association with

good teaching.2. SET appears to be unrelated to objective learning.

Do the evaluations haveDISCRIMINANT validity?

1. No

Is the instrument able to identify differences between the construct of interest and other constructs that are distinct but similar (discriminant validity)?

Problems:1.No research in the last 15 years that has investigated this issue

has found evidence of discriminant validity. 2. This is EXACTLY the area that SET is applied with personnel usage.

Do the evaluations haveNOMOLOGICAL validity?

1. Maybe

Nomological validity is the degree to which a construct behaves as it should within a system (network) of related constructs.

Problems:1.No construct definition. 2.It would be dependent upon a standard of teaching excellence being present in students’ cognitive structures and perceptions.

Do the evaluations haveUTILITARIAN validity?

1. Maybe

An instrument could, hypothetically, be useful as a tool to achieve an end irrespective of any validity to related theoretical constructs.

This is the essence of the debate

Do the evaluations haveUTILITARIAN validity?

1. SET does have utilitarian value for administrators.

2. There is little evidence that teaching has improved since widespread use of SET.

3. There is no conclusive evidence that:Students know moreStudents are more prepared for society or employmentSudents are happier

Do the evaluations haveUTILITARIAN validity?

1. SET does have utilitarian value for administrators.

2. There is little evidence that teaching has improved since widespread use of SET.

3. There is no conclusive evidence that student conditions have improved.

4. There is no conclusive evidence that the quality of education has improved.

Problems and Ironies

The “duh” factor

One of the frustrations of studying an area such as SET is the necessity of researching the obvious.

Valen Johnson

Wanting X

Wanting X Rewarding Y

Institutions say they produce skilled and productive studentswho hopefully can use reason and logic….

Institutions reward teachers who keep students happy!

Diversity and Civil Rights

Diversity and Civil Rights

If SET instruments are to be taken at face value:

1.What is the fate of an instructor who presents material or ideas to students that they currently don’t like?

Diversity and Civil Rights

If SET instruments are to be taken at face value:

2. Or what is the fate of an instructor who comes from a religious, racial, or ethnic groups that the students find unacceptable?

Diversity and Civil Rights

Actual Example:

Same professor, same class: Professor was a religious man, but did not mention his religion in class.

Students in both schools pointed to exactly the same things: 

X UNV Y UNVOverall Quality 4.2 4.7Helpfulness 4.6 5.0Clarity 3.9 4.4Easiness 2.1 2.6 

Diversity and Civil Rights

X UNV The guy should retire and become a therapist or preacher. I thought I was doing decent in the class, but ended up doing horrible. Preaches about nothing pertaining to the book, which is fine but makes for an unbelievably difficult test. Damn you XXX. The class was boring, and lastest forever. XXX really gets in touch with his students, and really cares about them. The tests have little to do with the book. Nice guy, but hard class! I'm a 3.3 student, and I've never studied so hard for a class and done so utterly bad. He is an incredibly nice guy, but teaches you hardly anything out of the book, but by god you WILL know what a sociopath is by the end of the class..he only spends 30 years on the symptoms. Nice guy but HARD.

Also a 3.3 student, XXX is a nice guy but seriously what a pointless class if it wasnt for pop quizes i would have never went. Don't know what to study for, presentation guidelines vague and he grades far to hard, never saw any perfects on anything. Feels like you are in theropy, he should retire.

Diversity and Civil Rights

YUNV Dr XXX is by far the best teacher in the School of Business. Rather than than sliding by, Dr XXX will inspire you to want more of yourself. I would suggest him to anyone who needs to take organizational behavior! AMAZING Professor! The tests were very hard and he requires a group presentation and a 10-page paper. However, he cares about you, he cares about your future, and he cares about your understanding of the material. Great Professor. Dr. XXX has brought his worldwide experience to the classroom. He's not an easy A teacher but is well worth your time if you want an education. The impossible XXX is the man. His class has been one of my favorites. You get to debate ethical dilemmas almost every class period. He gives you all semester to turn in your ten page paper over and over again til you get an A. Most people (like me) waited until the last day. DON'T! Tests are super specific and difficult. I recommend him.

Diversity and Civil Rights

If SET instruments are to be taken at face value:

When, and on what basis, can a decision-maker say that the students are wrong?

Or to use SET to deny tenure or merit pay to one instructor, while claiming the students are correct when they don’t like an instructor on religious grounds, but are incorrect if they

don’t like a minority female as an instructor?

Strange Assumptions

1. Administratively, students are considered to behonest, while faculty will cheat.

Strange Assumptions

2. Even though faculty are experts and students are students, they are considered experts in

determining what they have learning in the experts’ class.

Strange Assumptions

3. Information that can change a instructors’ pay, chances of promotion, and reputations is

considered public information.

Strange Assumptions

4. Contrary to all HR normative behavior, in a faculty/administrative dispute involving SET,

the faculty member generally has to demonstrate that the instruments are invalid.

Administrators are seldom required to demonstrate that the instruments are valid.

Strange Assumptions

5. Tautologies…

Minimum requirements for

validity.

Minimum Requirements for Administrative Validity

1. The questions or statements on the SET encapsulate the construct of good or efficient teaching as understood and defined by the institution.

2. The students’ interpretation of the questions can be shown to be similar to those utilizing the instrument.

3. The scales produced by the SET instrument are appropriate for measuring good and/or efficient teaching.

4. The instrument must be shown to have not only convergent, but also discriminant, and divergent validity.

Minimum Requirements for Administrative Validity

5. Any mathematical or statistical use of the data from the SET is appropriate to the scale created by the instrument.

6. Since the data comes from a certain subset of students, it must be shown that this sample extracted from the set of all potential students of this Instructor is representative of that set.

7. If the scores are utilized comparatively with other instructors, it must be shown that this comparison is methodologically and statistically equivalent to all instructors teaching the same students under identical conditions, or the differences cannot be attributed to the instructors.

Can the evaluations be fixed?

Can they be tweaked?

NO