Value-Added Models' Fatal Flaw

Much has been written about the corporate education reformers pet teacher evaluation tool the value-added model (VAM). A VAM is a statistical measure that attempts to account for a teacher’s effectiveness through student performance on standardized tests. These measures have been nearly universally discredited by study after study, most recently by the American Statistical Association, the leading professional organization for statisticians in the country. For an outstanding review of all that is bad about VAMs, I would recommend Rethinking Value-Added Models in Education: Critical Perspectives on Tests and Assessment-Based Accountability, by Audrey Amrein-Beardsley.

While VAMs do not come close to meeting the sniff test for reliability and validity, they have been forced down states’ throats as an accountability measure through the Obama administration’s Race to the Top program. Each of the 46 states that signed on is supposed to come up with a teacher evaluation system using VAMs in order to get a waiver from the impossible to achieve goals of the No Child Left Behind Act.

Whether VAMs are statistically suspect or not, they are doomed to fail. They have a fatal flaw that will insure that failure. Here is why.

What is the purpose of a program of teacher evaluation? For more than 95% of teachers the chief purpose of an evaluation is formative. In other words the evaluation is designed to help that teacher get better at teaching. Any smart school district leadership recognizes that by the time even a first year teacher is evaluated, the district has invested heavily in that teacher. The investment came in the form of recruiting, interviewing and training costs. The new teacher has value and, as in any good organization, a school district seeks to protect that investment through ensuring, to the extent possible, that the teacher is successful.

In order for an evaluation program to be successful in helping a teacher improve, three critical conditions must be met.

1.    Trust – Feedback is a two way street; it requires a giver and a receiver. Feedback does not lead to improvement if the teacher does not trust the source of the feedback. A teacher must be open to whatever evaluative feedback is provided. An administrator providing feedback earns that trust through being knowledgeable, fair and open to dialogue with the teacher.

2.    Actionable Feedback – A formative evaluation must provide a teacher with feedback that is doable. Recommendations must take into account the unique characteristics of the context in which that teacher is working. This means that the feedback is situational, responsive to the realities of the classroom setting, including such things as class size, grade level, special needs of the students and other demands on the teacher’s time.

3.    Adequate Resources – The evaluation must also take into consideration resources of time, money and materials. If a refinement in teaching is recommended by the administrator, then that administrator must be able to point to the resources that are available to implement the recommendation. This may include sending the teacher to a professional development opportunity, providing time for the teacher to view other classrooms implementing the suggestion or providing support materials for the teacher to use in the classroom.

The fatal flaw of value added models is that they do not have and can never have the trust of teachers and the do not provide actionable feedback. Let’s leave aside for a moment their statistical volubility. VAMs are simply too far removed from any classroom reality to be trusted by teachers. The classroom teacher, working with the child every day, knows much more about the individual child as a learner than any one shot standardized test can ever reveal.

VAMs also fail to provide actionable feedback. What do you do with a number that describes the growth scores of students who are no longer in your classroom? How do you use that number to improve instruction? If your scores are low does that mean you need to do more test prep? Does it mean something about your instruction? Does it mean something about last year’s group of students? How does that apply to this year’s group of students?

Finally, VAMs waste resources because they are simply not a useful tool for a school district seeking to provide an evaluation program that supports the development of skilled and contributing teachers. Even if an evaluation program were simply designed to ferret out under performing teachers, the VAM measures are too unreliable to be used for dismissal purposes.

By forcing schools to focus on this unproductive approach, the federal Department of Education and the education reformers are hampering the opportunity for a richer more meaningful approach to evaluation.