RCTs Against the Machine: Can Machine Learning Prediction Methods Recover Experimental Treatment Effects?
RFF researchers examine how well machine learning counterfactual prediction tools can estimate causal treatment effects.
We investigate how well machine learning counterfactual prediction tools can estimate causal treatment effects. We use three prediction algorithms—XGBoost, random forests, and LASSO—to estimate treatment effects using observational data. We compare those results to causal effects from a randomized experiment for electricity customers who faced critical-peak pricing and information treatments. Our results show that each algorithm replicates the true treatment effects, even when using data from treated households only. Additionally, when using both treatment households and nonexperimental comparison households, simpler difference-in-differences methods replicate the experimental benchmark, suggesting little benefit from ML approaches over standard program evaluation methods.
Click "Download" above to read the full paper.
Brian C. Prest
Fellow; Director, Social Cost of Carbon Initiative
Brian Prest is an economist and fellow at Resources for the Future specializing in climate change, oil and gas, and energy economics.
Casey J. Wichman
Casey Wichman is a university fellow at RFF. He performs research at the intersection of environmental and public economics, with an emphasis on examining the ways in which individuals make decisions in response to environmental policies.
Resources Radio — Oct 3, 2023
Cutting the Queue and Emissions in the US Power Sector, with Will Gorman
Will Gorman discusses the interconnection queue (the waiting list for developers that hope to connect power plants to the US electric grid), the reasons for recent growth in wait times, and reforms that could speed up the queue.
Common Resources — Sep 28, 2023
Designing Climate Policy for the Electricity Sector
Recent federal policy incentivizes low-carbon electricity generation. This conversation shares insights from experts about additional policy approaches that can help decarbonize the electricity sector while addressing environmental justice.
Resources Radio — Jul 4, 2023
The Impact of Coal’s Decline in West Virginia, with Jamie Van Nostrand
Jamie Van Nostrand discusses West Virginia’s support of the coal industry, how reliance on coal has affected residents, and the state’s energy future.