Empirical Issues in the Design of Group-Randomized Studies to Measure the Effects of Interventions for Children. MDRC Working Papers on Research Methodology

MDRC. 16 East 34th Street 19th Floor, New York, NY 10016-4326. Tel: 212-532-3200; Fax: 212-684-0832; e-mail: publications@mdrc.org; Web site: http://www.mdrc.org

N

85

William T. Grant Foundation
Abt Associates, Inc.
Annie E. Casey Foundation
Rockefeller Foundation, New York, NY.
Grable Foundation, Pittsburgh, PA.

Pilot Projects; Research Methodology; Intervention; Sampling; Preschool Children; Preschool Evaluation; Evaluation Research; Correlation; Item Analysis; Weighted Scores; Evaluation Methods; Test Construction; Psychometrics; Information Sources

This paper provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children. To do so, the paper: (1) provides new empirical information about the values of parameters that influence the precision of impact estimates (intra-class correlations and R-squares); (2) examines the implications of planning group-randomized studies for three-level hierarchical situations, using empirical information obtained by estimating two-level hierarchical models (which under many conditions appears to not be problematic); and (3) assesses the implications of the uncertainty that exists when the design of group-randomized studies is based on estimates of intra-class correlations. Data for the paper come from two studies: the Chicago Literacy Initiative: Making Better Early Readers study (CLIMBERs) and the School Break-fast Pilot Project (SBPP). The analysis sample from CLIMBERs comprised 430 4-year old children from 47 preschool classrooms in 23 Chicago public schools. The analysis sample from the SBPP study comprised 1,151 third-graders from 233 classrooms in 111 schools in six school districts. Four appendixes are included: (1) Description of Outcome Measures; (2) Definition of the Multiplier M; (3) Complete Set of Results for Three-Level vs. Two-Level Model Comparisons: Nonstandardized Unconditional Variance Components; and (4) Proofs of the Relationship between Three-Level Models and Two-Level Models in Terms of Precision. (Contains 20 footnotes, 1 figure and 14 tables.) [This document is also sponsored by Judith Gueron Fund for methodological Innovation in Social Policy Research at MDRC, and the Jerry Lee foundation.]

As Provided

35

2008

ED502531