After 17 years, College considers overhaul of faculty, course evaluations

Tenure and salary decisions at Vassar are most heavily determined by the Course Evaluation Questionnaires (CEQs) that students complete at the end of each course. These CEQs, however, are becoming increasingly problematic in the eyes of students, and even professors and administrators.

At the April 5 Vassar Student Association (VSA) Council meeting—which three professors and Dean of the Faculty Jonathan Chenette attended—Class of 2011 President Joseph Redwood-Martinez, among others, worried that the CEQs were outdated and ineffective in providing professors with substantial feedback.

"There is concern on three different levels about CEQs," said VSA Vice President for Academics Camille Friason '09, who explained that the CEQs are often ambiguous for both students and faculty. "There is concern from the students, concern from faculty and also concern from administration."

"From my experience," said Redwood-Martinez. "It seems that students and faculty are equally upset with the CEQs. They are poorly written. I do not understand how the current format provides any constructive feedback."

Most of the unease, it seems, comes from the first of the two parts of the questionnaire—the "Scantron" portion, which poses questions about the professor and his or her course and asks students to respond with a ranking from one to five—five standing for "high" and one for "low."

"A lot of the questions are still ambiguous and do not give students direction," said Friason. "There are so many ways to think about what each question and each word in the question means."

Many professors and students are dissastisfied with the questionnaire. Assistant Professor of Political Science Zachariah Mampilly is a junior professor finishing his second year of the seven-year tenure-track process. At this stage, Mampilly and other junior professors rely heavily on the CEQs as their primary method of feedback and instruction.

"The data-driven part," said Mampilly, "is not necessarily the most useful in terms of being a faculty member, because when you get it, you just see a series of scores—of numbers—that are used to evaluate you in relation to other faculty members. The part I find more useful is the second part—the space in which students can critique your course rather than answering specific questions that are represented in the Scantron portion."

The Scantron half of the CEQs, however—and not the written statements from students—is the portion with which the Faculty Appointments and Salary Committee (FASC) and the department chairs make their tenure recommendations to the President. Although the Governance of the College gives the President full authority over granting tenure, the recommendations of FASC and the academic department guide the outcome.
Mampilly explained that though the Scantron portion of the CEQs is taken so heavily into account, the collected data is not specific or guiding enough to give substantial information about a professor or his or her teaching style.

"On either side of things," said Mampilly, "I don't think you necessarily have that much clarity as to why one person gets fours and fives and why another person gets twos and threes. You can't point to it and say, ‘Well, it's because you did this, or you didn't do this.' That's not how the CEQs work."

"It's these vague, generic sentences," Mampilly continued, "and that is the basis of your score. They do not actually give you direct feedback as to why students really like you or, more importantly, why they dislike you."

Chenette explained that one possible way of making the CEQ Scantron data more constructive and less vaguely defined would be to give professors a report with individual scantron data from each student, paired with the same student's free-response comments.
"At the very least," said Chenette, "I think we could link the free text comments that students make to the numerical evaluations they give—not necessarily for review purposes, but just for the faculty member's own information. That would be a small step that would help the faculty member get more information." Chenette, however, went on to explain that there are "some logistical issues to doing that." Professors receive not individual Scantron reports, but rather an average score of each of the 18 questions on the evaluation.

When the CEQs were first issued in 1972, there were three parts of the evaluation, consisting of only nine questions. Since then, the questionnaire has been revised once—23 years later in 1995—into the draft with which Vassar students currently evaluate professors.

The revisions, explained Friason, mostly added or expanded on questions in the same style as those featured in the original evaluation—adding little new or different to the updated draft. While the 1972 draft of the questionnaire asked students to rate the instructor on, for example, "Openness to student questions or disagreement" and "The overall effectiveness of the instructor," the current CEQ similarly asks students to assess professors based on "Openness to students' points of view" and "Overall effectiveness of the instructor."

The major addition to the current CEQ was Part IV—a section that attempts to address more specific aspects of education as they vary across departments. It asks, for example, about the effectiveness of laboratories, as well as the effectiveness of the "performance, studio or production aspect" of a course.

Fraison and the rest of the VSA Academic Committee, however, are no longer satisfied with the revised CEQ. This year they created a new draft of the evaluation that replaced the short questions or statements—like "rate the course on organization"—with full sentences that provided more detail and less ambiguity.

"Instead of having a question or a one-word explanation, we wanted to have a full sentence like, ‘The professor came to class prepared and organized,' or ‘The professor explained difficult material,' and then you rank them on a scale of one to five. That way, it is not as ambiguous," Friason explained.

Redwood-Martinez also drafted a preliminary copy of a new CEQ, which he created based off of input from the Academic Committee, the Sophomore Class Council and Vassar faculty members from various departments.

The primary difference between Redwood-Martinez's draft and the current CEQ lies in the way in which students are asked to evaluate a professor. While the current CEQ asks students to rate an aspect of the class from "low" to "high"—or from numbers one through five—Redwood-Martinez's draft uses a scale that equated numbers one through four with ratings from "never" to "all the time," since his draft asks students to determine how often a professor fulfills certain ideals of teaching. For example, the draft includes statements such as, "The readings were constructive and relevant," and "The course met its objective as stated by the instructor or syllabus."

In addition to changing the way in which students approach the Scantron section, Redwood Martinez's draft also transforms the blank page into a section in which students respond to three questions with more lengthy hand-written comments. The page also includes a space were students can leave "additional comments."

Chenette, as well as Friason and Redwood-Martinez, wants to critically examine the way in which the College evaluates its professors. "From the moment I arrived here [this summer as Dean of the Faculty]," said Chenette, "I have wanted to look at how we evaluate and how we rely on the CEQs. I wanted to be confident that the current CEQs are evaluating what we want to evaluate."

Chenette went on to explain that since the CEQ was revised, Vassar professors have taken on a more diverse role and relationship with students that is not accounted for in the current questionnaire. "There is a lot that has changed since 1995 in terms of pedagogy and in terms of what we value—we do a lot more one-on-one student-faculty research, senior theses and field work, and advising is increasingly important at Vassar. The CEQs do not measure any of this," Chenette said.  "It is really important to me that we find a way of letting the faculty know that we value these things and that we come up with ways of assessing them."

Mampilly similarly explained that his week-to-week work consists of much more than what he does in his political science classes. "Our jobs extend way beyond the classroom. A lot of what we do is not just how we interact with students in class, but also how we interact with them in all these different forums," Mampilly said. "So how do you account for those types of interactions, for all the learning that goes on outside of the classroom? We do not have any way of getting at these aspects which are, I think, downplayed quite a bit if we just focus on the numerical scores of the CEQs," he continued.

In order to touch on the out-of-the-classroom aspects that Chenette and Mampilly addressed, Redwood-Martinez suggests that the College explore alternative methods of evaluation that would stand alongside the CEQs. "In addition to a significantly revised CEQ," said Redwood-Martinez, "I think students and faculty at Vassar would benefit from other methods of evaluation, like peer-to-peer visits and major reports."

"I would love to have a conversation," said Chenette, "about whether or not there are other methods [Vassar could use], such as the faculty-peer visits to classes. My understanding is that the faculty has taken up some of these issues in the past and that they haven't translated into change for various reasons."

Over the past few years, various proposals for alternative methods of evaluation—primarily peer visitations from professors—and for revised drafts of the current CEQs have been brought to the faculty floor for a vote. Friason, however, explained that getting a majority vote amongst every faculty member often proves to be difficult. "The faculty cannot agree. It is so hard to get anything passed," said Friason.

Many of Vassar's peer institutions employ the two techniques mentioned by Redwood-Martinez in addition to other methods of evaluating professors. Like Vassar, these colleges use a student questionnaire similar to the CEQs—yet they use the questionnaire in addition to these alternative methods such as peer visits from faculty and testimonials from student majors and alumnae/i.

Barnard College uses what they call "Student Ratings Questionnaires"—an evaluation similar to Vassar's CEQs—as well as reviews by department chairs and eight to 12 letters from former students and reviews performed by peer professors. Similarly, Williams College uses a "Student Course Evaluation Survey," interviews of students, "exit interviews" of senior majors and peer visits from professors.

Students at Hamilton College complete a "Teaching Evaluation Form" online at the end of each course. At the VSA Council meeting last Sunday, Andrew Bennett '10 proposed that Vassar put its course evaluations online as well. Both the Council members and administrators present worried that if CEQs were made online-only, fewer students would ultimately complete the evaluation.

"[Registrar] Dan Giannini and Computing and Information Services (CIS) were talking about trying to get the CEQs online because we use 40,000 sheets of paper each year," said Friason. "The problem with that is that we must have a certain percentage response rate. Right now it is 50 percent. We are already having problems getting that many students to fill it out. So if it goes online, we might see even bigger issues with that."
According to Joseph Urgo, however, who is Hamilton's Acting President and Dean of the Faculty, Hamilton has not seen problems with the online evaluations since they were launched during the 2008-09 academic year. Since the switch to online, Urgo has seen a 92 percent response rate to the Teaching Evaluation Forms, a rate that, he explains, "is not much different [than what] we had with paper—given class absences."

"We have seen that with the online system students say more in response to the questions," said Urgo. He also noted that "they tend to be a little more critical than they were on paper—I mean this in a positive sense." Urgo went on to explain that with evaluations online, "Possibilities for research expand." Because its evalutaions are now online, Hamilton can sort its evaluations by almost any characteristic—for example, by the gender or class year of  students, or even the grades given to students for each course. "Even though the responses are confidential," said Urgo, "we now have the capacity look at responses from A students versus B students, and so forth. Such responses might tell us more about teaching effectiveness, and ultimately help us improve our work in the classroom."

To sort a professor's evaluations by grades received in his or her class would perhaps enable the College to mitigate grade inflation, an issue that is increasingly becoming a concern at Vassar and institutions nationwide. As stated in "Vassar, peers have faced loosing battle against grade inflation" in the 4.9.09 issue of The Miscellany News, there is a proven correlation between CEQ rankings and the level of grades.

With an online system like Hamilton's, Vassar would perhaps be able to do what Chenette describes by sorting its CEQ results online.

Putting CEQs online, however—and even making less substantive changes to the questionnaires—is not something that the College is prepared to do overnight. Two weeks ago, a subcommittee comprised of three people—Friason, Giannini and Professor of History Robert Brigham—was created just to revise the first paragraph on the Scantron portion of the CEQ. The paragraph is descriptive and asks no questions of the students taking the survey—the subcommittee has been charged with revising this four-sentence paragraph as their single undertaking. This year, there have been no drafts or official proposals presented to the faculty with regard to evaluating Vassar professors. In order for a proposal to reach the faculty floor, that proposal would have to be endorsed by three committees—the Committee on Curricular Policy (CCP), the FASC and the Faculty Policy and Conference Committee. Only then would it go to the faculty. "On the faculty floor," said Friason, "all faculty present would read [the proposal], discuss it, recommend changes, and then we would revise it for a second read for maybe the next month. So that is already something like a four-month process."

The issue of faculty evaluations has also been on hold because the College and its senior officers have been busy mitigating the effects of the fiscal crisis. "The CEQs are something that I spoke to Chenette about in August when I got here," said Friason. "We both agreed that we needed to seriously look at them and change them—and then the whole economic crisis came, and everything that we were talking about went to the back burner. Next year, the faculty cuts will still be a big topic, but CCP can go back to working on the CEQs, which was one of their major agenda items."

Chenette, too, recognized that although CEQs may not be a priority compared to the ongoing financial crisis, they are nevertheless a major concern within the Vassar community and a highly important issue. "The students have been persistent and persuasive that we need to look at this and do things about it. I've seen them trying to play a constructive role in this, and I know that it's frustrating when things don't happen," said Chenette.

"As of right now—with the financial situation—it is a very hard time to tackle this important discussions about how we evaluate faculty," Chenette said. "It seems absolutely crucial to me, though, that we have this conversation and we ask if we are happy with the CEQs as they are now."