STUDENT EVALUATION AND PERFORMANCE
Photo by: CEASAR
Some observers argue that student evaluations are the most important evaluations since students are their teachers' "customers." Other observers object, insisting that student evaluations are merely popularity contests. As you will see from the links below, more than 80 years of research confirms that well-designed, correctly administered, and properly interpreted student ratings are valid and reliable measures of effective teaching (Alaemoni, 1999), especially if they are obtained from multiple courses over multiple semesters. Unfortunately, most "homemade" rating forms are not valid and reliable. Therefore, the following resources will show you how to design, administer, and process your own questionnaires or how to find vendors that will do the job for you.
A word of advice: If you are currently teaching, don't leave the task of soliciting student evaluations up to the administration and don't wait until the end of the term. Solicit student feedback throughout the term so that you can use that input to improve your course for your current students. For instance, use Blackboard surveys to administer anonymous mid-semester course evaluations. Or set up a Teaching & Learning Forum on Blackboard's Discussion Board, where you can respond to students' anonymous comments on your teaching and their learning.
If you wish to design your own student rating form, first, sign out CETLA's CD "Student Ratings: Their Design, Construction, and Use," Magna's 90-minute recording of Raoul Arreola's online seminar. Drawing upon nearly 85 years of research on student ratings, Arreola explains how to design valid and reliable questionnaires. Once you understand the psychometric procedures for designing student rating forms, you can use the following resources most effectively:
Designing a Rating Form: Syracuse University's Center for Support of Teaching and Learning has developed an extensive website for faculty and administrators who wish to design their own rating forms. The site includes step-by-step instructions for creating a rating form and an Item Bank of nearly 500 questions. Like the Syracuse site, the Teaching, Learning, and Technology (TLT) Group's Flashlight Online site also helps faculty or administrators construct course evaluations. Subscribers can take advantage of "a huge item bank of validated questions" and "a large collection of model surveys created by Flashlight staff and other subscribers." Subscribers can not only use, adapt, or create surveys on their own but collaborate on survey development online.
Testing Your Form: After designing a form, don't forget to test it for reliability and validity before conducting an official evaluation. Click these links to find out how:
Administering Online Surveys: Once you have tested your questionnaire, you can choose one of many web-based companies to deliver your survey online. Online surveys offer students "anywhere, anytime" access and offer you immediate results: The survey software computes statistics and records students' comments as students respond. If you need only descriptive statistics and downloads in Excel format, CETLA recommends Blackboard and SurveyMonkey.
Choose Blackboard if you are a faculty member wishing to conduct midterm evaluations. You and your students can access your password-protected Blackboard coursesite immediately and at no extra charge. Moreover, although your students' responses will remain anonymous, you can tell who has responded so that you can award homework credit for prompt responses and then email reminders to the other students. To learn how to set up evaluations in Blackboard, sign up for CETLA's BB12 Administering Course Evaluations via Blackboard or BB06 Creating Tests and Surveys.
On the other hand, choose SurveyMonkey if you are a committee member or administrator evaluating your unit's teaching at the end of the term. SurveyMonkey is inexpensive, reliable, user-friendly, and controlled solely by the survey administrator. Also, unlike Blackboard, SurveyMonkey can give anyone you choose access to the survey (via an email, web link, or kiosk), so you can survey alumni as well as current students. To learn how to use SurveyMonkey, watch the video tutorials or sign up for CETLA's CA10 Collecting Assessment Data via SurveyMonkey.
However, if you need inferential statistics, using SPSS may prove to be well worth the money. You can pick up a free SPSS CD in the User Support Services Office on the 2nd floor of the iLab. For assistance with SPSS, sign up for CETLA's CA08 Using SPSS for Assessment or watch CETLA's online Guest WorkshopVideo.
Regardless of whether you choose Blackboard, SurveyMonkey, or SPSS, make sure someone is available to provide technical support if students run into technical difficulties. Also, remember that achieving an adequate response rate for online evaluations normally requires incentives if your students cannot take online surveys in your classroom. To encourage students to complete online surveys on time, some faculty offer homework points or extra credit. On the other hand, some institutions limit students' access to survey results or course grades until the students have completed the surveys.
Processing Paper Surveys: After considering the limitations of online surveys, you may decide to administer a paper survey instead to ensure that students complete the survey during the class period. If so, vendors such as Scantron and Questionmark can assist you with the printing and scanning of forms. Or you can buy your own forms and scanner from vendors such as Apperson Education Products and train an employee to scan your forms.
Instead of designing your own evaluations, you can select a vendor who offers valid and reliable forms as well as data collection, data analysis, and reporting services. Arreola (2000) recommends three all-purpose course evaluations (click the titles to access the sites):
A non-profit organization, the IDEA Center was established at Kansas State University in 1975 to assist colleges and universities with the assessment and improvement of teaching and learning. According to the Center, "The IDEA Student Ratings of Instruction system takes a positive approach to soliciting student input. Rather than emphasizing the instructor's teaching techniques or personality, the IDEA system focuses on student learning. Research has shown that there is no one correct way to teach, so The IDEA Center tailors each report to fit the instructor's teaching objectives. Teaching effectiveness is determined by student progress on goals chosen by the instructor, not by an outdated, rigid model of what effective teaching is supposed to be. Research based on our very large national database allows us to provide diagnostic assistance for those with disappointing results. Fairness is improved by taking into account the influence of factors outside the instructor’s control. The IDEA Student Ratings of Instruction system is unique in its emphasis on using the results constructively. Reader-friendly faculty reports are produced that include not only ratings of teaching effectiveness, but also diagnostic assistance to help the instructor improve. These reports gain even more usefulness when combined across classes or years by using the Group Summary Report. No other evaluation service offers such comprehensive, longitudinal reporting services."
Sample Form: http://www.theideacenter.org/
Sample Report: http://www.theideacenter.org/node/5
Sample Group Summary Report: http://www.theideacenter.org/node/5
Developed by the Educational Testing Service (ETS), the Student Evaluation of Teaching Effectiveness (SIR II) not only reports students' responses but compares them to mean ratings from an appropriate group of peer institutions. According to ETS, "SIR II™ is the second generation of the Student Instructional Report -- a teacher evaluation instrument used for more than 30 years that objectively captures student assessment of teacher effectiveness. Available in both Web-based and paper-and-pencil delivery methods, the SIR II is a course evaluation that is highly acclaimed for its fairness and validity. The research-based SIR II clearly identifies strengths and weaknesses in teaching and delivers the information that faculty and administrators need and use."
Sample Form: unavailable
Sample Report: unavailable
Used at more than 500 institutions,the Aleamoni Course/ Instructor Evaluation Questionnaire (CIEQ) is, according to the website, "a student rating form and statistical analysis package designed for use as part of a program for assessing both course and faculty teaching performance. The form is designed to provide primarily two kinds of information; (1) diagnostic information for professional enrichment and faculty development; and (2) statistical aggregate data, based on appropriate norms, which may be used as one part of the information used in [personnel] decision making."
Sample form: http://www.cieq.com/forms.htm
Sample Report: http://www.cieq.com/index.htm
In addition to these well-established surveys, CETLA recommends two other surveys, the CLASSE and the SALG:
Adapted from the National Survey of Student Engagement (NSSE), the Classroom Survey of Student Engagement (CLASSE) assesses the level of student engagement at the course level. According to the website, "CLASSE is a pair of survey instruments that enable one to compare what engagement practices faculty particularly value and perceive important in a designated class with how frequently students report these practices occurring in that class."
Funded by the National Science Foundation, The Student Assessment of Learning Gains (SALG) survey, according to the website, "allows instructors to gather learning-focused feedback from students. The SALG survey asks students to rate how each component of a course (e.g., textbook, collaborative work, labs) helped them to learn, and to rate their gains toward achieving the course goals. The SALG survey can be customized to fit any college-level course, and can be administered multiple times per course. A baseline instrument allows faculty to compare gains relative to incoming student characteristics."
Even the best data will prove useless or even harmful if you and your evaluators do not interpret or use the data correctly. As William Cashin (1995) reminds us in "Student Ratings of Teaching," "Student ratings are data that must be interpreted. We should not confuse a source of data with the evaluators who use student rating data--in combination with other kinds of data--to make their judgments about an instructor's teaching effectiveness." Moreover, once interpreted, the data must be used by the instructor as well to improve his or her teaching. For assistance, see the following sources.
"Learning from Student Feedback": Written for the instructor, this bulletin from Washington State's Center for Instructional Development and Research suggests how you might interpret and respond to student evaluations.
"Using Student Evaluations to Improve Teaching." The Stanford University Newsletter on Teaching offers advice on how to make sense of student feedback and use it to improve specific teaching skills.
"Dealing with Hurtful Student Comments." Nancy Givens shares insights from faculty who've "been there"!
"Student Ratings of Teaching: Recommendations for Use." William Cashin at Kansas State University's IDEA Center lists 34 short recommendations that are of special value for peer and administrative evaluators.
"Student Evaluation of Teaching Summary." Here is a sample form from Wayne State University for summarizing student evaluations (see Appendix 1). The form requires evaluators to consider student evaluations of multiple courses over more than one semester and to compare the results with departmental norms.
While rating forms are the best means of collecting student feedback for summative evaluation, here are a few other ways to solicit student feedback for formative evaluation:
"Arrange for a Midterm Class Interview (SGID)": According to Washington State University, a "midterm class interview" (or Small Group Instructional Diagnosis) is "an open-ended, mid-course oral interview process" in which an outside consultant asks the students two questions: "'What is helping you learn in this class?'" and "'What changes would assist you in your learning?'" If you would like to schedule such an interview, email firstname.lastname@example.org so that CETLA can assign a Faculty Mentor as a consultant.
Student Focus Groups: After the semester has ended, a peer or consultant can interview small groups of students about their learning. The University of Wisconsin's Teaching Academy provides a list of procedures and a timetable for "interviewing students about their learning experience" or "interviewing students about what they have learned."
Student Observer Program : Since Carleton College launched its Student Observer Program, numerous faculty have benefited from the observations of trained student consultants who visit their classes to take notes and talk with the students. If you would like to work with a student observer, CETLA recommends that you recruit a student who is pursuing a teaching certificate or education degree. Then the experience will contribute to the student's career development as well as yours.
Teaching & Learning Forum: Set up a Teaching & Learning Forum on Blackboard's Discussion Board where students can post anonymous comments on your teaching and their learning. Then students can comment at any time during the term, and, best of all, you can respond to their comments.
Although student evaluations can tell us much about the quality of teaching, we must remember that they are simply evaluations. What about tests of student performance? Is there any relationship between effective teaching and students' performance on departmental, senior comprehensive, and college-wide proficiency exams or portfolio assessments? What about students' pass rates on board, licensure, or certification exams or their scores on admission exams for graduate or professional schools? Should colleges and universities hold faculty accountable for their students' performance?
Spearheaded by the Spellings Commission and the "No Child Left Behind" legislation, the "Accountability Movement" has compelled colleges and universities as well as public schools to reassess the relationship between student performance and teacher evaluation. Like an increasing number of legislators, public school administrators, and higher education accreditors, James Stronge and Pamela Tucker (2000) argue that student learning outcomes should play a role in the evaluation of teaching because of "the clear and undeniable link that exists between teacher effectiveness and student learning." However, in their book Teacher Evaluation and Student Achievement, Stronge and Tucker insist that evaluators should consider student learning data only under certain circumstances because so many factors influence student learning that are beyond a teacher's control (e.g., lack of resources, poverty, or overcrowded classes). Arreola (2000) is even more cautious, contending that evaluators should use student learning data only "to improve student learning" and "identify teachers for merit recognition" (p. 66). Therefore, if you decide to include student performance as one source of data for the evaluation of teaching, please note the conditions for use explained below.
"How Does Student Achievement Relate to Teacher Accountability?" : In this article, the Virginia Education Association introduces an excerpt from Stronge and Tucker's book with these words of advice: "...an education professor from the University of Virginia and another from William & Mary list a set of recommendations for the use of student achievement in educator evaluation. Those recommendations include using student learning as only one component of a teacher evaluation system based on multiple data sources; using measures of student growth instead of fixed achievement goals; comparing the same group of students over time instead of groups of different students; and selecting student assessments that are closely aligned to existing curriculum."
"Can and Should Teachers Be Evaluated Using Student Learning Data?": In this essay, Kenneth Peterson (2000), the author of Teacher Evaluation: A Comprehensive Guide to New Directions and Practices, proposes six conditions for using student learning data for teaching evaluation.
"Analysis of Pretest and Posttest Differences" : Stronge, Tucker, Peterson, and others agree that evaluators should focus on measures of student progress rather than fixed standards for achievement when evaluating teaching. Such measures require that each student complete similar tests over time and that the statistician analyze the pretest and posttest scores by computing gain scores or a repeated measures ANOVA. In this lesson, a University of Colorado psychology professor, Lee A. Becker, explains how to use SPSS to analyze the scores.
Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924-
1998. Journal of Personnel Evaluation in Education, 13(2), 153-166.
Brace, I. (2004). Questionnaire design: How to plan, structure, and write survey material for effective market research. Sterling, VA: Kogan Page.
Cashin, W. E. (1990). Student ratings of teaching: Recommendations for use. IDEA Paper: No. 22. Manhattan, KS: Kansas State University.
Cashin, W. E. (1995). Student ratings of teaching: Recommendations for use. IDEA Paper: No. 32. Manhattan, KS: Kansas State University.
Centra, J. A., (2000). Is there gender bias in student evaluations of teaching? Journal of
Higher Education, 70(1), 17-33.
Centra, J. A., (2003). Will teachers receive higher student evaluations by giving
higher grades and less coursework? Research in Higher Education, 44(5), 495-518.
Cohen, P.A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multi-section validity studies.
Review of Educational Research. 51(3) 281-301. Excerpt.
Doyle, K.O. (1975). Student evaluation of instruction. Lexington, MA: Lexington Books.
Feldman, K.A. (1998). Reflections on the study of effective college teaching and student ratings: One continuing quest and two unresolved issues. In Smart, J.C. (Ed.) Higher education: handbook of theory and education Volume XIII (35-65). Memphis: Agathon Press.
Knapper C. & Cranton, P. (2001). New directions for teaching and learning: 88.Fresh approaches to the evaluation of teaching. San Francisco: Jossey-Bass.
Lewis, K. (Ed.). (2000). New directions for teaching and learning: 87.Techniques and strategies for interpreting student evaluations. San Francisco: Jossey-Bass.
Marsh, H.W. (1987). Students' evaluation of university teachers: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11(3), 255-388.
Onwuegbuzie, A. J., Collins. K. M., Filer, J. D., Moore, C. W, Wiedmaier, C. D., &
Witcher, A. E. (2007). Students’ perceptions of characteristics of effective college teachers: A validity study of a teaching evaluation form using a mixed-methods analysis. American Educational Research Journal, 44(1), 113-160.
Peterson, K. (2000).
evaluation: A comprehensive guide to new directions and practices.
Memphis, TN: Corwin Press.
Sorenson, D.L., & Johnson, T.D. (Eds.). (2003). New directions for teaching and
learning: No. 96. Online student ratings of instruction. San Francisco, CA: Jossey-Bass.
Stronge, J.H. & Tucker, P.D. (2000).
Teacher evaluation and
student achievement. Annapolis, MD: NEA Professional Library.
Stronge, J.H. & Tucker, P.D. (2005). Linking teacher evaluation and student learning.
Alexandria, VA: Association for Supervision & Curriculum Development.
Theall, M. & Franklin, J.L. (Eds.). (1990). New directions for teaching and learning: 43. Student ratings of instruction: Issues for improving practice. San Francisco: Jossey-Bass.
Theall, M., Abrami, P.C. & Mets, L. (Eds.). (2001). New directions for institutional research: 109. The student ratings debate: Are they valid? How can we best use them? San Francisco: Jossey-Bass.