Yesterday, we released a survey of 215 shale gas development experts that found a surprising amount of agreement between experts from academia, industry, government, and environmental NGOs on what potential environmental risks due to hydraulic fracturing for natural gas are of most pressing need for attention. Why, though, is a survey of experts a useful exercise in this policy context, and, furthermore, how do you go about sampling experts?
The shale gas boom has seemingly dropped out of the sky into the laps of policy makers and the public who until very recently had little idea what hydraulic fracturing was, and certainly did not know how to feel about its potential costs and benefits. So, what should we do? Some states have decided there is not enough yet known and instituted moratoriums on certain practices or indeed on all shale gas development, but far more have decided the known (or unkown) risks can be managed and are likely to be overwhelmed by the rewards. Who should these regulators turn to when they are in the hot seat?
In his 2005 book, The Wisdom of Crowds, James Surowiecki noted that another group in the hot seat, contestants on the hit game show Who Wants to Be a Millionaire were far more successful when they polled the audience (91% correct answer) than when they phoned a friend who they thought would be an expert (65% correct). But what if you could do both? Polling a large group of experts could give you the benefit of crowds and the benefit of expertise.
Without a ready-made expert studio audience, or even a way to randomly sample the population of experts, RFF had to create our own population by searching news stories, academic literature, government hearings, conference participant lists, and recommendations of an expert about who else to contact, as well as identifying every organization, firm, and governmental body that has a prominent role in the public discussion. This gave us a pool of over 1000 experts. After eliminating those who disclaimed expert status, and extra people from organizations that elected to give a single response, just over 700 remained. Of these, 30% (215 experts) took part in our lengthy survey. Remarkably, the response rate was nearly identical across the four groups of experts (industry, environmental NGO, academic, and government experts), so we are confident that non-response bias from an institutional perspective is not a problem.
The report details how our responses came from experts representing or affiliated with nearly every big firm, headline environmental NGO, state government, federal department, and university that has played a prominent role in shale gas development. We choose to group the experts by organization (NGOs, industry, government, and academia) because that was clearly the characteristic that mattered most—organization was a strong predictor of how many high priorities experts had, their views on boomtowns, whether government or industry should take the lead on addressing risks, and just how severe the risks of certain accidents are. As long as group was controlled for, years of experience in oil and gas or in shale development itself as well as specific expertise or education had little to no explanatory value.
This suggests that regulators and the public, when hearing from a specific expert, should ask “who do you work for?” just as often as they ask “what do you know?” when trying to build their own panel of diverse advice.
This makes it even more surprising, however, that we found so much consensus between the four groups of experts on 12-15 specific risk pathways that are in need of more government or industry attention. For a regulator unable to assemble her own TV studio full of shale gas experts, these risks are the ones she should keep most in mind. For more on that, see our summary report.
