Study finds flaws in teacher performance observations

Study finds flaws in teacher performance observations

(Penn.) As the components of teacher evaluations remain under debate among policy makers, a new study suggests the results of classroom observation may hinge more on the students’ capabilities than the teacher’s.

Analysis from the American Institutes for Research and the University of Pennsylvania’s Graduate School of Education found that students’ behavior and prior academic achievement weighs heavily on teacher performance and can skew the results of an evaluation.

“When information about teacher performance does not reflect a teacher’s practice, but rather the students to whom the teacher is assigned, such systems are at risk of misidentifying and mislabeling teacher performance,” reported Rachel Garrett of the American Institutes for Research and Matthew Steinberg from the University of Pennsylvania’s Graduate School of Education.

Classroom observations are a component of every state’s evaluation system in some capacity, according to the Virginia-based Center for Public Education. Many states seeking federal grants or No Child Left Behind waiver exceptions restructured teacher evaluation systems to include student test scores – in effect linking the scores to decisions regarding dismissal, promotion, compensation and professional development.

Prior to the push to include test scores in the evaluation process, studies suggested that teacher evaluation scores across the country were inflated. In Michigan, Florida and Tennessee, approximately 98 percent of new teachers were being rated as effective or better.

Those in support of high-stakes evaluations argued that the system had allowed ineffective teachers to remain in the profession and was a disservice to students. Opponents, including the teachers’ unions, countered calling such evaluations unfair.

A number of states, including Michigan, have since taken steps to lessen the impact test scores have on teacher evaluations, often citing factors outside an educator’s control such as problems at home that can affect a student’s academic performance.

Yet in this most recent study, published last week in the trade journal Educational Evaluation and Policy Analysis, even the process of classroom observation may be reliant on factors beyond a teacher’s control. Researchers concluded that when teachers were assigned a more engaged class with higher incoming achievement levels, they were more likely to see an increase in their measured performance.

Garrett and Steinberg used data from an earlier two-year Measures of Effective Teaching study that sampled 834 teachers in grades 4 to 9 across six districts in New York, Florida, Texas, Colorado and Tennessee. Researchers in the original study concluded that observation scores would be more reliable when paired with student feed-back and achievement gains.

Going one step further, Garrett and Steinberg concluded teacher performance observations should rely more on specific instructional skills used in the classroom rather than measures such as student participation or class climate, which can be skewed by student behavior.

In order to account for the influence student behavior and ability can have on the different factors included in teacher observation scores, researchers recommend the scores be adjusted based on student demographic characteristics.

“If higher achieving students are more easily instructed, then a teacher’s observation scores will, in part, reflect these easier-to-teach students,” researchers wrote.

Garrett and Steinberg found that math teachers were six times more likely to be highly rated based on classroom observations when assigned students who were the highest achievers the previous year. English language arts teachers were twice as likely to be so rated.

On the flip side, only 18 percent of math teachers assigned the lowest-performing students were rated as effective or better based on observation, as were 37 percent of English teachers.

...read more