Rely (only) on the rigorous evidence” is bad advice

Pritchett, Lant (2023) Rely (only) on the rigorous evidence” is bad advice. Review of Development Economics. ISSN 1363-6669

Text (Pritchett_rely-only-on-the-rigorous-evidence--published) - Published Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (2MB)

• Author
• Scopus publication

Abstract

A popular interpretation of “evidence-based” decision-making is “rely (only) on the rigorous evidence” (RORE) via “systematic” reviews that: use objective protocols to generating the potentially relevant papers from the literature; then filter those to retain only the small subset that provide impact estimates regarded as “rigorous”; and summarize only those estimates. I use two sets of cross-country impact estimates—on wage gains for migrants and private school learning gains—to illustrate this seemingly attractive approach is both empirically and conceptually unsound. First, the cross-country variation in the rigorous estimates of impact is very large, which implies the average(s) from a systematic review is of little predictive use. In both empirical examples the “systematic review of the rigorous estimates” approach leads to worse predictions of impact across countries than the naïve use of country-specific ordinary least squares estimates. Second, I contrast a systematic review—RORE approach with an “understanding” approach—which seeks to encompass all of the available evidence into coherent understandings in forming judgments. In both examples the notion that the impact effects are constant across countries—“external validity”—is easily rejected. Insisting on privileged reliance on “rigorous” estimates in making context-specific decisions is logically incoherent and deeply anti-scientific.

Item Type:	Article
Official URL:	https://onlinelibrary.wiley.com/journal/14679361
Additional Information:	© 2023 The Author
Divisions:	School of Public Policy ?? SCPP ??
Subjects:	H Social Sciences > HB Economic Theory
JEL classification:	O - Economic Development, Technological Change, and Growth > O1 - Economic Development > O10 - General C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C90 - General C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods: General > C10 - General
Date Deposited:	24 Jul 2023 15:54
Last Modified:	25 Apr 2024 16:45
URI:	http://eprints.lse.ac.uk/id/eprint/119818

Actions (login required)

View Item