Forthcoming paper by Snoeij and Stauffer (2019)
The topical focus of policy-making depends on agenda-setting processes; meaning the process of problems and solutions entering the policy-making process where they are defined, debated, and then prioritized through the combination of various forces such as power, political interests, funding, shared discourse, external events, and media coverage.
In contrast to agenda-setting, 'priority-setting' has been employed as a method to prioritize problems and solutions more systematically in the sector of health policy. This approach comes close to the prioritization models that are advocated within the Effective Altruism community. They include the Scale-Solvability-Neglect (SSN) and the expected value (EV) frameworks. Generally, they serve as broad heuristics to guide one's decision-making in the allocation of limited resources.
This paper tries to push the envelope by analyzing how these two frameworks would apply to policy-making in general, across contexts, sectors and institutions. For this purpose, we select four case-studies that respectively cover local, national, regional and international policy-making and rely on different types of resources (legal, financial, human) and policy instruments. We run both prioritization models through these case-studies and identify whether they would apply and be useful, fall short, or provide misleading guidance. We use ex-ante and ex-post policy reports and impact evaluation for our analysis.
Our preliminary findings suggest that these models are not well-suited to inform macro policies (such as national policies) because of systemic effects that inflate confidence intervals and the unprecedented nature of some policies which prevent us from establishing useful counterfactuals. In some instances, these prioritization models are useful to guide data collection; incentivize stakeholders to make educated guesses; or at least state assumptions in an explicit manner. In multiple cases, the 'neglect' criterion is misleading because it is based on the assumption of diminishing marginal returns which do not necessarily apply when a policy entails the investment of billions of dollars or the reform of a law. In these cases, we could sometimes assume increasing returns or at least acknowledge non-linear dynamics of impact.
We conclude that prioritization models that mainly come from microeconomics fall short when applied to policy-making. While researchers and practitioners might suggest to turn to economic theory to deal with large questions with limited data, we recommend the Effective Altruism community and policymakers to complement their approaches with further modelling techniques such as computer simulations.