The Reliability of Classroom Observations and Student Surveys in Non-Research Settings: Evidence from Argentina. EdWorkingPaper No. 25-1322

Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/

N

44

Inter-American Development Bank (IDB)

Foreign Countries; Clinical Supervision (of Teachers); Student Evaluation of Teacher Performance; Teacher Effectiveness; Evaluation Methods; Student Surveys; Classroom Observation Techniques; Observation; Response Rates (Questionnaires); Research Problems; Feedback (Response); Preservice Teachers

There is a growing consensus on the need to measure teaching effectiveness using multiple instruments. Yet, guidance on how to achieve reliable ratings derives largely from formal research in high-income countries. We study the reliability of classroom observations and student surveys conducted by practitioners in a middle-income country. Both instruments can achieve relatively high reliability (0.6-0.8 on a 0-1 scale) when averaged across raters and occasions, but the reliability of observations varies widely (from 0.4 to 0.8) based mostly on how raters are assigned to teachers. We use Generalizability Theory to estimate how reliability improves by increasing the number of times teachers are observed or the number of respondents to surveys. We recommend that practitioners design their teacher feedback systems based on analyses of their own data, instead of assuming that instruments and rubrics will generate scores with the same reliability as research settings.

As Provided

2026

ED678303