Return on investment: Evidence-based options to improve statewide outcomes (Document No. 12-04-1201)

This technical appendix describes the latest version of the Washington State Institute for Public Policy (Institute) benefit-cost model. The Institute built its first model in 1997 to estimate the economic value of programs that reduce crime. Later, as the Institute received additional and varied assignments from the Washington legislature, the benefit-cost model was revised
and expanded to cover additional public policy outcomes. The model described here reflects our current approach to computing benefits and costs for a wide array of outcomes and contains several enhancements over earlier versions. Our ongoing goal is to provide Washington policy makers with better “bottom-line” estimates each successive legislative session. The 2009 Washington State Legislature directed to update and extend its review of the benefits and costs of prevention and intervention programs. 1

The Legislature directed the Institute to “calculate the return on investment to taxpayers
from evidence-based prevention and intervention programs and policies.” Specifically, the Legislature asked the Institute to identify public policies that have been shown to improve these broad outcomes of public interest:

• Crime,
• K–12 education,
• Child maltreatment,
• Substance abuse,
• Mental health,
• Public health,
• Public assistance,
• Employment, and
• Housing.

A principal objective of the Institute’s model is to produce a “What Works?” list of public policy options available to the Washington State legislature—and to rank the list by estimates of return on investment. The ranked list can then help policy makers choose a portfolio of public policies that are evidence based and that have a high likelihood of producing more benefits than costs. For example, if the public policy objective is to reduce crime, then a portfolio of evidence-based policies can be selected from the list—from prevention policies, juvenile justice policies, and adult corrections policies— that together can improve the chance that crime is reduced and taxpayer money is used efficiently.

There are three basic steps to the analysis.

1. What Works? First, we conduct a systematic review of the research literature to identify policies and programs that have demonstrated an ability to improve the outcomes. In Appendices B and C, we describe the methods we use to screen and code research studies, the meta-analytic approach we use to estimate the effectiveness of policy options to achieve outcomes, and the procedures we use to compute monetizable units of change. The
objective of the first step is to draw statistical conclusions about what works—and what does not—to achieve improvements in the outcomes, along with an estimate of the statistical error involved.
2. What Makes Economic Sense? The second basic step involves applying economic calculations to put a monetary value on the improved outcomes (from the first step). Once monetized, the estimated benefits are then compared to the costs of programs to arrive at a set of economic bottom lines for the investments. Appendix D describes the processes we use to monetize the outcomes.
3. How Risky are the Estimates? Part of the process of estimating a return on investment involves assessing the riskiness of the estimates. Any rigorous modeling process, such as the one described here, involves many individual estimates and assumptions. Almost every step involves at least some level of uncertainty. Appendix F describes the “Monte Carlo” approach we use to model this uncertainty. The objective of the risk analysis is to access the odds that an individual return on investment estimate may offer the legislature the wrong advice. For
example, if we conclude that, on average, an investment in program XYZ has a ratio of three dollars of benefits for each dollar of cost, what are the odds, given the uncertainty in this estimate, that the program will not even generate one dollar of benefits for each dollar of cost?
Thus, our analytical goal for each evidence-based investment option we analyze is to deliver to the legislature two benefitcost bottom-line measures: an expected return on investment and, given the uncertainty, the odds that the investment will at least break even.

Citation: Lee S, Aos S, Drake E, Pennucci A, Miller M, & Anderson L. 2012. Return on investment: Evidence-based options to improve statewide outcomes (Document No. 12-04-1201). Olympia: Washington State Institute for Public Policy.