A Retrospective Assessment of COVID-19 Model Performance in the USA
This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modeling.
Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that—(i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.
Gabriela Florentina Nane
Delft University of Technology
Senior Fellow Emeritus
Roger Cooke is a senior fellow emeritus at RFF. He joined RFF in September 2005 as the first appointee to the Chauncey Starr Chair in Risk Analysis.
Report — Jun 5, 2023
Valuing a Reduction in The Risk of Asthma: A Large-Scale Multi-Country Stated Preference Approach
RFF research contributed to this OECD working paper estimating willingness to pay to avoid asthma or reduce its severity.
On the Issues — May 19, 2023
On the Issues: US Debt Limit, Climate Hits Home, and More
A biweekly newsletter connecting global current events, pressing climate and energy policy news, and economics research from RFF scholars. This week: US debt limit, climate change hits home, and more.
Working Paper — May 10, 2023
The Impact of the Clean Air Act on Particulate Matter in the 1970s
This working paper reviews the definition of nonattainment status under the Clean Air Act to determine the designation's relationship with total suspended particulate readings.