Causal Inferences From Observational Studies in Education Policy: Toward Pragmatic Social Science

Abstract

In the complex world of education policy, causal inferences will be debated. As the authors review nonexperimental designs in education policy, they focus on how to clarify and focus the terms of debate. They begin by presenting the potential outcomes/counterfactual framework and then describe approximations to the counterfactual generated from the general linear model, regression discontinuity designs, the propensity to be treated, and instrumental variables. Noting the advantages and limitations of each method, they turn to literature showing that conventional linear models, which can be extended to fixed and random effects models and difference-in-differences designs, are particularly useful when covariates include pretests. Recognizing that no technique fully resolves concerns about inferences, they quantify the conditions necessary to invalidate an inference. They review additional considerations, including accounting for context, heterogeneous treatment effects, and mediation. Throughout, they focus on the role of the researcher in making analytic choices, accentuated in the Quantcrit framework.

Keywords: causal inference; observational studies; pragmatic; confounding variables