RCTs against The Machine: Can Machine Learning Prediction Methods Recover Experimental Treatment Effects?
RFF researchers investigate how successfully ML prediction algorithms can be used to estimate causal treatment effects in electricity demand applications with nonexperimental data.
We investigate how successfully ML prediction algorithms can be used to estimate causal treatment effects in electricity demand applications with nonexperimental data. We use three prediction algorithms—XGBoost, random forests, and LASSO—to generate counterfactuals using observational data. Using those counterfactuals, we estimate nonexperimental treatment effects and compare them to experimental treatment effects from a randomized experiment for electricity customers who faced critical-peak pricing and information treatments. Our results show that nonexperimental treatment effects based on each algorithm replicate the true treatment effects even when only using data from treated households. Additionally, when using both treatment households and nonexperimental comparison households, standard two-way fixed effects regressions replicate the experimental benchmark, suggesting little benefit from ML approaches over standard program evaluation methods in that setting.
Brian C. Prest
Fellow; Director, Social Cost of Carbon Initiative
Brian Prest is an economist and fellow at Resources for the Future specializing in climate change, oil and gas, and energy economics.
Casey J. Wichman
Casey Wichman is a university fellow at RFF. He performs research at the intersection of environmental and public economics, with an emphasis on examining the ways in which individuals make decisions in response to environmental policies.
On the Issues — Jan 19, 2024
On the Issues: New Magazine, City-Splitting Highways, and More
A biweekly newsletter connecting global current events, pressing climate and energy policy news, and economics research from RFF scholars. This week: US emissions trends, city-splitting highways, and more.
Resources Radio — Jan 2, 2024
2023 Year in Review: Energy and Environmental Policy, with Karen Palmer and Joseph Majkut
Karen Palmer and Joseph Majkut discuss the important and overlooked stories in energy and environmental policy from 2023, along with stories and issues to watch for in 2024.
Working Paper — Dec 18, 2023
Widening the Scope: The Direct and Spillover Effects of Nudging Water Efficiency in the Presence of Other Behavioral Interventions
This working paper describes the results of a social information campaign designed to nudge water conservation.