A couple of weeks ago on Twitter, someone posted a link to a blog post discussing ways to attract more majors in the philosophy department. Attracting majors is really important, especially in departments without graduate students where the number of majors is a determining factor in the amount of funds you get, the potential for new hires, etc..
Today, I was reading up on some of the factors that influence teacher evaluations, and was rethinking some of the discussions I’ve had with colleagues about whether taking advantage the psychological factors that influence student’s evaluations of teachers has any advantage beyond making the teacher look good.
When I first came to grad school, a professor in the department gave us an informal chat about the kinds of factors that can influence student evaluations. For example:
Williams and Ceci conducted a related experiment. Professor Ceci, a veteran teacher of the Developmental Psychology course at Cornell, gave the course consecutively in both fall and spring semesters one year. In between the two semesters, he visited a media consultant for lessons on improving presentation style. Specifically, Professor Ceci was trained to modulate his tone of voice more and to use more hand gestures while speaking. He then proceeded, in the spring semester, to give almost the identical course (verified by checking recordings of his lectures from the fall), with the sole significant difference being the addition of hand gestures and variations in tone of voice (grading policy, textbook, office hours, tests, and even the basic demographic profile of the class remained the same). The result: student ratings for the spring semester were far higher, usually by more than one standard deviation, on all aspects of the course and the instructor. Even the textbook was rated higher by almost a full point on a scale from 1 to 5. Students in the spring semester believed they had learned far more (this rating increased from 2.93 to 4.05), even though, according to Ceci, they had not in fact learned any more, as measured by their test scores. Again, the conclusion seems to be that student ratings are heavily influenced by cosmetic factors that have no effect on student learning (Huemer, M. here).
As it turns out (see here), how a teacher looks, dresses, and how much a teacher moves their hands and changes their tone, can all have an impact on student’s evaluations of a teacher (and on the course, and the texts!). Of course, we all wanted to get better teaching evals (after all our teaching profile will make a difference in our job prospects), so we wanted to take advantage of these psychological factors.
But, apart from these selfish reasons we also debated whether taking advantage of these psychological factors had any other benefits. My initial thought upon hearing about the impact of cosmetic factors on teaching evals was annoyance. I thought it was annoying that evaluations weren’t solely a measure of my effectiveness as a teacher, and that any good evals I would get might be merely a product of the fact that I bought some new shirts and had a tendency to wave my arms about like a mad man when talking philosophy. But, a colleague pointed out that doing those things might actually contribute to my effectiveness as a teacher. There is a correlation between, for example, moving one’s arms and higher scores, but this could be because people are paying more attention and therefore learning more. After reflecting on this for awhile, I came to agree that evaluations shouldn’t just be a function of the content and how clearly it is presented, but also of the teacher’s ability to get people to pay attention to that content. If that meant taking advantage of certain psychological tendencies, so be it. Insofar as I chose to take advantage of those tendencies to get my students to learn more, I was a more effective teacher.
As it turns out (see above), taking advantage of those psychological factors might not actually result in better content uptake. While students might perceive themselves to have learned more when a teacher does a little jig, if their grades are the same when the content is presented in a less exciting manner (and the grades are really just a function of the students grasp of the material), it seems that dancing the jig makes little real difference in whether the students learn. So much for taking advantage of psychology to increase student learning.
Still, there might be another reason to dance a little jig. I take it that if students perceive a class to be enjoyable, and take themselves to have learned a lot, they are more likely to take other classes with that teacher, in that teacher’s department, and, ultimately, more likely to major in the department of which that teacher is a part. I don’t have anything but anecdotal evidence for those claims, but I’ll settle for that for now.
I take it that having a thriving philosophy department (or other departments for that matter) is a good thing both for the department and for the student body. If training teachers to take advantage of these psychological factors can increase the number of majors and thereby increase the quality of the philosophy department, this seems like a prima facie reason to train teachers to take advantage of these factors in the same way that a teacher’s being good is a reason to have them teach intro courses.
I’m not sure if it is ethical to take advantage of these psychological factors in order to promote department health (something tells me maybe not…), but for those wondering about how to increase the number of majors, it may be worth some investigation. It would be good to know whether there is any correlation between people majors and the evaluations of teachers in a given department. If anyone has this data, I’d be glad to have it and post some links.
10 Comments
I’ve had a colleague in psych. admit to ‘priming’ the students before handing out the evaluate forms. Informally, if you force the students to look at the syllabus in the few minutes before handing out the form (through some contrived excuse), you’ll see a signficant increase in your ratings for ‘organization’, ‘clearly laying out expectations’, etc.
I’m uncomfortable with such actions, but since I’m at a small school where I am evaluated in comparrison not to other Philosophers, but to people who appear to be regularly manipulating their data, it’s hard to take a principled stand. After all, doing so would be tantamount to my job at risk in order to protect the integrity of data that is pretty much meaningless anyway!
Peter,
I know a few people who have primed their students before evals. I’m also not comfortable with certain kinds of priming right before evals.
I think it is tough to decide what to do in circumstances that you mention. I don’t feel much pressure to prime my evals in that way, and so don’t. As it turns out, the philosophy department at UW tends to get higher evals than most other departments (probably because the subject is more interesting, and because, I think, we train teachers well), so if other departments are priming, it isn’t so successful that we need to do it to keep up.
But, there is one kind of priming that I am comfortable with. When I hand out my evals, I tell them that the most important thing for me is the written comments and I do my evals at the beginning of class, telling my students that the reason for this is so that they don’t rush through them in a hurry to get out the door. I find that this yields more written comments on my evals and thereby gives me a more substantive way to evaluate my own teaching. I don’t really feel any qualms about doing this (maybe I should?).
I wonder whether the unchanged grades were weighty evidence for students learning being unaffected by the teachers adding gestures, tone modulation, etc, to their delivery. Claiming that the unchanged grades were weighty evidence requires some strong assumptions about the tests being excellent indicators of learning. But maybe the improved learning wasn’t captured by the tests. This seems especially plausible if there is a correlation between increased student approval of teaching and increased student memory for taught material. Maybe the psychologically manipulated students didn’t score better on some tests, but would score better in a few more months because the psychological manipulatiosn improved their memory for the material (by changing the way they interacted with the taught material, for example–a possibility made plausible by accounts of performative accounts of memory).
Of related interest is a provocative piece that Clark Glymour wrote, called “Why the University Should Abolish Faculty Course Evaluations” (see the link to it here: http://www.hss.cmu.edu/philosophy/faculty-glymour.php). It would be interesting to see whether his considerations make it “any more ethical” to prime students before handing out evals.
Hey Matt,
Good points. You could be right about the tests and I think if you are right, then there might be a really good reason for dancing the jig.
Thanks for the link to the Glymour. It was flying about Twitter, but I hadn’t taken a look yet.
John and Matt – Thanks for the links!
John – interesting discussion. I think it is important to not only inform the students, but to capture the imagination, and insofar as even superficial aspects of a professor’s performance help us do that, we should be paying attention to these things.
Matt,
I agree with that, but if the capturing is short lived, I’m not sure how much attention should be paid. In any case, it is worthwhile to know what impacts evaluations, even if we aren’t yet sure how to use that knowledge.
I’m intrigued by this “ethics of priming” issue. John–What types of priming do you see as ethically problematic? I’m curious if someone could say explicitly what about priming students they find discomforting.
Hey Matt,
I think it is ethically problematic to take advantage of the psychology of your students for the sole purpose of making yourself look good without their consent. That probably has to be cleaned up a bit. Still something in the vicinity seems right.
If that isn’t right, the following at least would seem problematic. Imagine that I hand out evals and say “these evals mean a lot for my career. Bad scores could ruin me.” This might get me better eval scores, but this seems to me ethically problematic. I think knowingly doing something like this in a sneakier way is just as ethically problematic.
That sounds about right. I’m curious how widespread a phenomenon unethical priming actually is. I prime my students, in a way, by telling them what’s at stake with the evaluations, not for the purpose of getting higher evals, but simply because, if I don’t, many of them won’t take it seriously at all. I also, like you, encourage comments over just filling in bubbles. I know other people who prime with somewhat different strategies. For instance, one person I know tells her students that, when she grades them, grades less than an A require justification; she then tells them that, likewise, eval scores less than 5 require justification. Do you think these types of priming are problematic? Or is it just having a particular motive?
Hey Matt,
I don’t think it is motive alone. For example, I think we can prime students to take the evals seriously without telling them how much is at stake with respect to careers. Instead, we can tell them that the evals are important to us because we want to be good teachers and that we read them carefully for suggestions. This might be priming them, but it seems to put less pressure on them than other ways of encouraging them to take them seriously.
I think that the person you mention is certainly engaged in something unethical. One grading standard you might have is that less than A must be justified, but you might start with an F and say higher grades have to be earned (and give points for positive qualities). It seems to me that by telling students they have to justify scores lower than 5, the person a) is trying to, or is knowingly, taking advantage of the laziness of students and/or b) is likely to come off as an ass and just invite getting slammed. I’d be surprised if that kind of priming worked. In any case, maybe I’m wrong, but I wouldn’t feel right saying that to my students, just because I don’t think it would engender honest responses.
I’m not sure how widespread intentional priming is. It would be interesting to know, but I doubt people would be very forthcoming if they were primers.