Congress’s Role in Promoting Environmental Policy Evaluation

In principle, there is a long-standing and broad bipartisan consensus in favor of robust ex-post evaluation of federal environmental policy. Proponents of ex-post evaluation note its many possible benefits—these include identifying (and eliminating) ineffective programs; improving the quality of ex-ante analysis; and facilitating regulatory experimentation. In reality, a number of political, institutional, and cultural barriers have blocked the emergence of any consistent practice of ex-post review. These barriers have persisted for decades, through administrations of both political parties and during periods of both united and divided government. They arise from consistent features of the administrative state, rather than the temporary politics of any particular moment. Overcoming them will likely require sustained reforms efforts.

Based in part on a series of interviews with senior government officials with decades of experience in environmental policymaking, this report identifies a set of ten interrelated barriers that have inhibited the development of a robust practice of environmental policy evaluation at federal agencies. These barriers are partially the result of political dynamics, but they also arise from agency institutional constraints, incentives, and cultures. Our research identifies the following leading barriers to ex-post evaluation:

1. Many agencies lack dedicated funding to carry out ex-post evaluation.
2. Agencies have highly constrained agenda space, making it difficult to prioritize ex-post evaluation.
3. Agencies face adverse political incentives to engage in self-scrutiny that may open them to political attacks.
4. Although agency personnel often have essential domain knowledge, they may lack the expertise to carry out rigorous ex-post analyses.
5. Agencies may lack (or believe they lack) legislative authority to carry out certain types of ex-post review.
6. It is often not clear how agencies should integrate the results of ex-post evaluation into decisionmaking, given that major regulatory decisions are rare.
7. The self-evaluations of agencies, even if carried out in good faith, may not be perceived as trustworthy by relevant stakeholders.
8. Ex-post evaluation often raises difficult questions of causal inference that agencies may be ill-positioned to address.
9. It is often difficult for agencies to collect the data needed to engage in ex-post evaluation.
10. Many agencies have not developed a culture of evaluation.

Given the identified barriers, we propose a suite of potential reforms for Congress to consider. These proposals are based on prior experience with ex-post review, analogous efforts within and outside the government, or suggestions that interviewees raised. We offer options at different levels of ambition, recognizing that enthusiasm to dedicate substantial congressional effort to the project of improving ex-post evaluation may vary. These levels of ambition can also be understood as scaling with time, so that more ambitious proposals, while perhaps unrealistic in the short term, may be more plausible over longer time horizons.

At the lowest level of ambition, Congress could use the appropriations process to direct funds to environmental policy evaluation. So as not to interfere with agency operations, and to avoid the impression that policy evaluation necessarily comes at the expense of programmatic goals, such funds should be made as additions to agency budgets, rather than as reallocations of existing resources. Appropriations directed to ex-post review would, most obviously, address the barrier of a lack of dedicated funds. Such appropriations would also require agencies to prioritize, to at least some degree, ex-post evaluation, regardless of their limited agenda space. Additional funding would also facilitate the acquisition of expertise as needed.

At a similar level of ambition, Congress could contemplate small-bore reforms that could ease the process of ex-post evaluation. For example, the Paperwork Reduction Act places procedural and substantive limits on agencies’ ability to collect data—a core prerequisite of ex-post evaluation. Given limited agency resources and competing priorities, these burdens act as additional roadblocks that can sap momentum from evaluation efforts. Targeted reforms to the Paperwork Reduction Act could reduce these burdens while still ensuring that the core purposes of the Act are protected. Efforts to identify other similar facilitating reforms may be warranted.

At a higher level of ambition, Congress could create legal requirements for agencies to engage in policy evaluation, either of specific programs or more broadly. The most extensive ex-post analysis of environmental policy making that has been carried out to date was prompted by section 812 of the Clean Air Act Amendments of 1990. In response to this congressional directive, the Environmental Protection Agency developed considerable expertise and dedicated significant resources to estimating the costs and benefits of air quality policies. Similar mandates could be effective, especially if coupled with additional funding and facilitating reforms. Legislative mandates would require agencies to devote at least some of their finite agenda space to ex-post evaluation, and, based on prior experience, would likely result in agencies accessing the expertise needed to carry out these evaluations in a rigorous fashion. Mandates would also clarify that agencies have authority to carry out ex-post evaluations.

A further measure at a similar level of ambition would be the creation of a specialized group of policy evaluation experts, within the Executive Office of the President, tasked with coordinating and facilitating ex-post evaluation efforts. Ideally, such a Policy Evaluation Group (PEG) would operate across a wide range of agencies (not only those with environmental portfolios), would be structured to be as politically neutral as possible, and would serve a supporting, rather than overseeing, role. The primary function of the PEG would be to provide human resources and expertise for agencies that are interested in engaging in policy evaluations. In this way, it would address issues of funding and expertise. A reform along these lines could also help increase the perceived trustworthiness of ex-post evaluations, given that outside experts would be part of the team that designed and implemented the analyses.

A more ambitious step that Congress could take would be the creation of a National Institute for Policy Evaluation and Review (NIPER), modeled on other scientific funding institutes. This new agency would fund collaborations between academic institutions and government agencies to engage in large-scale policy evaluations designed to identify and address key areas of uncertainty across a range of policymaking domains. Environmental policies would be one of several portfolios that could be overseen by a funding agency of this kind, which could also support research in areas such as education and criminal justice. A NIPER-like entity would have considerable potential to address most of the barriers to ex-post evaluation.

Finally, Congress could contemplate deeper statutory reforms designed to encourage agencies to integrate the results from ex-post evaluations into regulatory decisionmaking. Ex-post evaluations are only useful if they provide information that serves as an input into agency decisionmaking. However, the laborious rulemaking process, which includes litigation and the attendant risk of judicial invalidation of agency decisions, means that agencies are not well-poised to update rules based on new information. Recent moves by the US Supreme Court to reduce agency discretion has further constrained the ability of agencies to act on the information from ex-post evaluations. Congress may wish to contemplate legislation to ensure that the process of judicial review does not interfere unduly with flexible and responsive agency decisionmaking.

Reforms of some significant ambition are likely needed to address barriers that, for decades, have inhibited ex-post evaluation. But such reforms require considerable study and coalition building before they would become politically feasible. For this reason, we recommend that Congress begin by creating a Commission on Policy Evaluation, modeled on the successful Commission on Evidence-Based Policy Making. This new Commission would be charged with studying policy evaluation across the federal government, examining state and comparative analogies, and consulting with experts to generate a set of concrete recommendations for additional congressional action. The Commission would help generate the knowledge base necessary to support well-informed congressional action and could also act as a forum for bipartisan coalition building.