Scriven Talks about Evaluating Teaching at McMaster

That was the verdict of Michael Scriven, world-renowned authority on evaluation, when he spoke on December 5th to about 90 MUFA members, and a handful of faculty from neighbouring universities, about the evaluation of teaching, and about McMaster's system for evaluating teaching.

The one good thing, he said, is that McMaster's system is among the best he's seen in North America for avoiding questions about style.

"It is completely inappropriate for administrative decisions to be based on style variables at all -- style being the way you do it as opposed to getting it done," Scriven said.

On the other hand, Scriven faulted McMaster's system in a number of respects:

• Many of the student ratings forms take too long for students to fill out, and there is overkill in requiring them in every course in every semester.

• A lot needs to be done on making clear what it is legitimate to put in teaching dossiers, and what instructors are encouraged to put in them.

• There is sloppiness about where to send which parts of the three-part form used in many Faculties, adapted from one devised by Scriven. According to Scriven, the only part the administration should see is the answer to the last global question: "Overall, how do you rate the effectiveness of the instructor as a teacher?"

• There should be a requirement in the system that instructors demonstrate a commitment to studying how they can improve their instruction. In this study, they should use the same standards of scholarly evidence they use in their research.

• For graduate courses, McMaster's policy favours interviewing over other methods of getting student ratings of teaching effectiveness. "Interviewing is not a superior form, it is an inferior form."

Scriven interspersed his comments on McMaster's evaluation system (based on a reading of 37 officially adopted policies, procedures and rating forms) with provocative remarks about current fashions in the evaluation of university teaching. Some examples:

• You cannot use a statistically supported secondary indicator, like appearing enthusiastic about the subject-matter, as a criterion of merit in teaching. You must use the primary indicator. Using enthusiasm about the subject-matter as a criterion of merit in an individual teacher is exactly the same fallacy as saying, "You're First Nations and therefore dumb", just because IQ tests on First Nations individuals generally speaking produce lower scores.

• The whole notion that somehow across-the-board student ratings are valuable is absurd. Student ratings range from useless, completely useless, irrelevant and legally unacceptable to very, very good indeed.

• Peer visits are absolutely essential for the documentation of incompetence. They are no use at all for anything more interesting than that.

• It is completely absurd to use student ratings as the only information about teaching quality. The content has got to be independently validated. The quality of the testing must be fair, fairly enforced and comprehensive. And the quality of the grading must be appropriate.

• There is no getting away from the fact that the student is a consumer. But then the mistake is to think that student satisfaction is all we need to bother about. In the business world, it is nice that somebody is bothering about customer satisfaction, and so they should. But we better look at whether the thing is Underwriter Laboratory safe for electrical shock, and a few other things.

• Why on earth do we worship innovation? What we need to be doing is something worthwhile. The general pattern in hotshot multimedia fields is: "God, you know, the classrooms look just like they did in the thirteenth century in Paris." Well, yes. Right. Now what? I mean, I want to hear them saying that they are no good, and I don't hear it. The worship of innovation is just a sign of trashy standards in the business.

• But the worship of checking to see how well you are doing and somewhere having a programme of experimentation, perhaps in quite modest respects -- changing the way you test, changing the feedback on the test, changing the frequency of the test, and then finding out whether it produced anything -- is much more serious.

As a follow-up to his visit, Scriven has volunteered to send copies of his annotations of McMaster's teaching evaluation policies and procedures to the responsible individuals who requested them. The Provost, Harvey Weingarten, has formed a small group to look at University-wide policies, with a view to proposing some changes which would eventually go to Senate. He has invited the Faculty Association to nominate a representative to this group.

Anyone who wishes can obtain from the Faculty Association office (Hamilton Hall 103A, mufa@mcmaster.ca, ext. 24682) a transcript of Michael Scriven's talk, of the reports of the four discussion groups which followed it, and of Scriven's response to those reports. A videotape of the same material is available from the Instructional Development Centre (General Sciences 217, ext. 24540).

I would like to thank all those who worked with me to make Scriven's visit such a success (as indicated by very positive ratings and comments on the evaluation form for the event). The organizing committee included Barbara Brown (nursing), Dale Roy (instructional development), Yufei Yuan (information systems), Stephen Link (psychology), Geoff Norman (clinical epidemiology and biostatistics), and Phyllis DeRosa Koetting (Faculty Association executive assistant). The four discussion groups were chaired by Pat Chow-Fraser (biology), Marilyn Parsons (nursing), Stefania Miller (political science) and Susan Denburg (psychiatry). Their rapporteurs were Dauna Crooks (nursing), Alan Harrison (economics), Christel Woodward (clinical epidemiology and biostatistics) and Sue Inglis (kinesiology). Resource persons for the groups were Bill Garland (engineering physics), Dale Roy (instructional development), Stephen Link (psychology) and Barbara Brown (nursing).

I would also like to express the thanks of the Faculty Association to the office of the Provost for moral and some financial support; and to the University Committee on Teaching and Learning for an instructional development grant of $1,000 to help with the expenses.

David Hitchcock
Chair, Academic Affairs Committee