Some time ago I participated in a discussion about responding to a request for information. The requesters wanted the formula for the model developed by myself and others for a state agency. At the time, I mused that if we had posted all the code and data for the project on github, the request would be redundant. I was told that it was necessary to only give requesters exactly what they asked for. So we did. But.
Why not make the code and data open?
I can think of one reason: maintain agency authority over the decision making process. If no one else has the information, then no one else can criticize or take over the decision. But this is patently false. Ballot initiatives tie the hands of Agencies across the country. These initiatives were not stopped by agencies keeping their data and analyses secret. I don’t know if having the data in the open would have stopped a ballot initiative either, but it seems a reasonable hypothesis to try out.
Keeping the data and analyses secret provides another advantage: it hides the trade offs that decision makers are making. There are unavoidable trade offs in natural resources management. There is no way to make all stakeholders perfectly happy. If there were, then those decisions aren’t going to be challenged. If an agency simply tells stakeholders what the decision is, then perhaps some will be happy, some merely satisfied, and others outraged. But no one will be able to tell the extent to which their concerns were traded away to advantage someone else.
Reasons to make the data and code open
I like to think about these issues in the context of a three-legged stool model of the science-policy interface: credibility, salience, and legitimacy.
Increase credibility
Scientists are used to building credibility by exposing their work to peer review as part of the publishing process. But a lot of agency work doesn’t get peer reviewed, because it isn’t really pushing science forward. Putting the data and code out in the open invites constructive criticism – stakeholders can see what choices were made in the analysis. In fact, they can take the data and make different choices and see if that affects the results. Making the work public will also lead to better work – no room to be sloppy and no carpet to hide sketchy assumptions under.
Expose salience
The analysis should already be salient to the decision, because the work is done to support the decision. But it might not be obvious to stakeholders how the work is connected to the decision. Agencies might consider that a feature rather than a bug, but I think it is something that should be fixed.
Increase legitimacy
I think of legitimacy of the analysis as the “3rd rail” of the science policy interface – touch it and your contribution is dead to the stakeholders. One way to increase legitimacy is to involve stakeholders in the process of defining the science. Opening a repository with code and data for an analysis explicitly provides a place for that conversation to happen. It might not be the best place, but something seems better than nothing.
Conclusion
Increasing credibility, salience, and legitimacy increases trust. The more trust stakeholders have in an agency, the more likely they are to comply with a decision. Placing the code and data in the open should increase trust, because look ma, nothing up either sleeve! But that only works if it is also clear how the analyses contributed to the decision, and ultimately that means exposing the trade offs involved.
I don’t have rose-colored glasses on; radical openness won’t make conflict with stakeholders go away. But I think it could change the nature of that conflict for the better.
Postscript
I learned some more details about the request for information on a phone call. One stakeholder group, representing fur trappers, wants a particular action to be taken. As far as I can tell, the action would cost the agency nothing, and the stakeholders in question are the only ones who would experience a (slight if not zero) negative effect. Nonetheless, the agency doesn’t think the action is warranted. The stakeholders feel that a population of trappable animals is declining precipitously, and they want the population closed to trapping. The agency thinks there are very few animals there (poor habitat), and there have always been few animals there. Closing the population to trapping admits that the stakeholders’ view is correct.
There is an opportunity here to have a different kind of conversation about the management. There are clearly two hypotheses: the population is declining because of trapping, or the population is low because of poor habitat. These hypotheses make very different predictions about what will happen after closing the management area. If trapping is the problem, then after closure the population will start to increase. If habitat is the problem, then after closure nothing will change. Alternatively, closure will lead to increasing populations. We could do adaptive management! If we really wanted stakeholder good will, set up a program for trappers to earn money monitoring track stations and camera traps in both the closed and nearby open areas.
