¿Cómo incide el método de las evaluaciones en su uso?
How Does the Method of Evaluations Affect their Use?
Resumen (es)
Las crecientes necesidades de los ciudadanos en diferentes aspectos de la vida diaria, junto a las enormes restricciones presupuestales para cubrirlas adecuadamente, obligan a los gerentes públicos a llevar a cabo políticas mucho más eficientes y efectivas. Una de las principales funciones de las evaluaciones es servir de insumo en la toma de decisiones para mejorar las políticas públicas y en procesos relacionados con la rendición de cuentas del gobierno. No obstante, en Latinoamérica, y específicamente en Colombia, existen escasos análisis sobre el uso de la información de las evaluaciones de programas y políticas públicas. Para avanzar en el cierre de esta brecha se realizó un estudio descriptivo y un análisis multivariado en Colombia que analiza los usos potenciales de la información evaluativa y la incidencia del método utilizado en los diferentes tipos de uso. Para efectuar análisis se utilizó información de 251 evaluaciones provenientes de los sectores educación, salud, inclusión social, agricultura, transporte e infraestructura, y vivienda a nivel nacional para el periodo comprendido entre 2008 y 2019. Los principales resultados de este análisis son: i) la mayoría de las evaluaciones muestran un uso potencial de tipo instrumental, es decir, sus hallazgos y recomendaciones pretenden ser empleadas para tomar decisiones directas e inmediatas sobre el programa evaluado; ii) 23,5% de las evaluaciones presenta un uso potencial de tipo conceptual, es decir, fueron diseñadas para brindar información para ejercicios de rendición de cuentas o mejorar el conocimiento sobre la forma como funciona u opera el programa analizado; iii) el uso instrumental tiene una relación estadísticamente significativa y positiva con los métodos cualitativos, la contratación pública y la evaluación concomitante; y iv) el uso conceptual tiene una relación estadísticamente significativa y negativa con los métodos cualitativos y con los sectores agricultura, transporte y vivienda.
Resumen (en)
The growing needs of citizens in different aspects of daily life, together with the enormous budgetary restrictions to adequately cover them, force public managers to carry out much more efficient and effective policies. One of the main functions of evaluations is to serve as an input in decision-making to improve public policies and in processes related to government accountability. However, in Latin America, and specifically in Colombia, there is little analysis on the use of information from evaluations of public programs and policies. To make progress in closing this gap, a descriptive study and a multivariate analysis were carried out in Colombia to analyze the potential uses of evaluative information and the incidence of the method used in the different types of use. The analysis used information from 251 evaluations from the education, health, social inclusion, agriculture, transportation and infrastructure, and housing sectors at the national level for the period 2008-2019. The main results of this analysis are: i) most of the evaluations show a potential instrumental use, i.e. their findings and recommendations are intended to be used to make direct and immediate decisions about the evaluated program; ii) 23.5% of the evaluations show a potential conceptual use, i.e. they were designed to provide information for accountability exercises or to improve knowledge about how the analyzed program works or operates; iii) instrumental use has a statistically significant and positive relationship with qualitative methods, public procurement and concomitant evaluation; and iv) conceptual use has a statistically significant and negative relationship with qualitative methods and with the agriculture, transportation and housing sectors.
Referencias
Aiello, E.; Donovan, C.; Duque, E.; Fabrizio, S.; Flecha, R.; Holm, P.; Molina, S.; Oliver, E.; y Reale, E. (2021), “Effective Strategies that Enhance the Social Impact of Social Sciences and Humanities Research”, en Evidence and Policy, Vol. 17 N° 1, pp. 131-146, https://doi.org/10.1332/174426420X15834126054137
Alkin, M. y S. Taut (2002), “Unbundling Evaluation Use”, en Studies in Educational Evaluation, Vol. 29 N° 1, pp. 1-12, http://doi.org/10.1016/S0191-491X(03)90001-0
Appleton-Dyer, S.; Clinton, J.; Carswell, P.; y McNeill, R. (2012), “Understanding Eva-luation Influence within Public Sector Partnerships: a Conceptual Model”, en The American Journal of Evaluation, Vol. 33 N° 4, pp. 532-546, https://doi.org/10.1177/1098214012447672
Arellano, D. y Hernández, J. (2014), “De la Torre de Marfil a la pertinencia. ¿La investi-gación académica en administración pública es útil para la toma de decisiones guber-namental? Un estudio exploratorio del caso mexicano”, México, Centro de Investigación y Docencia Económicas (Dossier Académico).
Askim, J. (2007), “How Do Politicians Use Performance Information? An Analysis of the Norwegian Local Government Experience”, en International Review of Administrative Sciences, Vol. 73 N° 3, pp. 453-472.
Balthasar, A. (2009), “Institutional Design and Utilization of Evaluation. A Contribution to a Theory of Evaluation Influence Based on Swiss Experience”, en Evaluation Review, Vol. 33 N° 3, pp. 226-256.
Behn, R. (2002), “The Psychological Barriers to Performance Management: or Why Isn’t Everyone Jumping on the Performance-Management Bandwagon?”, en Public Performance and Management Review, Vol. 25 N° 1, pp. 5-25.
Berman, E. y Wang, X. (2002), “Performance Measurement in U.S. Counties: Capacity for Reform”, en Public Administration Review, Vol. 60 N° 5, pp. 409-420.
Boyer, J. F. y Langbein, L. I. (1991), “Factors Influencing the Use of Health Evaluation Research in Congress”, en Evaluation Review, Vol. 15 N° 5, pp. 507-532.
Bridgeland, J. y Orszag, P. (2013), “Can Government Play Moneyball?”, en The Atlantic, Boston, July-August, https://www.theatlantic.com/magazine/archive/2013/07/can-government-playmoneyball/309389/, 06-12-2018.
Burr, E. (2009), “Evaluation Use and Influence among Project Directors of State GEAR UP Grants”, Knoxville, The University of Tennessee, Ph.D. Dissertation, http://trace.tennessee.edu/utk_graddiss/15
Cairney, P. (2019), “The UK Government’s Imaginative Use of Evidence to Make Policy”, en British Politics, Vol. 14 N°1, pp. 1-22, https://doi.org/10.1057/s41293-017-0068-2
Cárdenas, S.; González, J.; y Ramírez, E. (2022), Aprender de la evidencia. Usos de la información en la gestión de organizaciones educativas, Bogotá, Universidad de los Andes; Centro de Investigación y Docencia Económicas.
Cardozo, M. (2006), “La evaluación de políticas y programas públicos. El caso de los programas de desarrollo social en México”, México, Cámara de Diputados; Porrúa.
Cardozo, M. (2013a), “Políticas públicas: los debates de su análisis y evaluación”, en Andamios, Vol. 10 N° 21, pp. 39-59.
Cardozo, M. (2013b), “De la evaluación a la reformulación de políticas públicas”, en Política y Cultura, N° 40, pp. 123-149, http://www.redalyc.org/articulo.oa?id=26728947007
Chelimsky, E. (2006), “The Purposes of Evaluation in a Democratic Society”, en The Sage Handbook of Evaluation, I. Shaw, J. Greene y M. Mark (eds.), London, Sage Publishing.
Choi, Y. y Woo, H. (2021), “Understanding Diverse Types of Performance Information Use: Evidence from an Institutional Isomorphism Perspective”, en Public Management Review, Vol. 24 N° 12, pp. 1-20, https://doi.org/10.1080/14719037.2021.1955953
Cousins, J.; Goh, S.; Elliott, C.; y Bourgeois, I. (2014), “Framing the Capacity to Do and Use Evaluation”, en New Directions for Evaluation, Vol. 2014 N° 141, pp. 7-23, https://doi.org/10.1002/ev.20076
Cunill, N. y Ospina, S. (2003), “Una agenda de investigación sobre la evaluación de los resultados de la gestión pública”, en Evaluación de resultados para un gestión publica moderna y democrática. Experiencias latinoamericanas, N. Cunill y S. Ospina (eds.), Caracas, CLAD.
De Lancer Julnes, P. (2006), “Performance Measurement an Effective Tool for Government Accountability? The Debate Goes on”, en Evaluation, Vol. 12 N° 2, pp. 219-235.
De Lancer Julnes, P. y Holzer, M. (2001), “Promoting the Utilization of Performance Measures in Public Organizations: an Empirical Study of Factors Affecting Adoption and Implementation”, en Public Administration Review, Vol. 61 N° 6, pp. 693-708.
Fetterman, D. (1996), “Empowerment Evaluation: an Introduction to Theory and Practice”, en Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability, D. M. Fetterman, S. J. Kaftarian, y A. Wandersman (eds.), Thousand Oaks, Sage.
Fleischer, D. y Christie, C. (2009), “Evaluation Use: Results from a Survey of U.S. American Evaluation Association Members”, en American Journal of Evaluation, Vol. 30 N° 2, pp. 158-175, https://doi.org/10.1177/1098214008331009
French, R. (2018), “Lessons from the Evidence on Evidence-Based Policy”, en Canadian Public Administration, Vol. 61 N° 3, pp. 425-442, https://doi.org/10.1111/capa.12295
González, J. (2016), “Por qué se usa la información de seguimiento y evaluación de políticas públicas”, México, Centro de Investigación y Docencia Económicas. Tesis para optar por el título de Doctor en Políticas Públicas.
Halachmi, A. (2002), “Performance Measurement, Accountability, and Improved Performance”, en Public Performance and Management Review, Vol. 25 N° 4, pp. 370-374.
Heinrich, C. y Marschke, G. (2010), “Incentives and their Dynamics in Public Sector Performance Management Systems”, en Journal of Policy Analysis and Management, Vol. 29 N° 1, pp. 183-208.
Henry, G. (2000), “Why Not Use?”, en New Directions for Evaluation, Vol. 2000 N° 88 (Special Issue: The Expandings Scope of Evaluation Use), pp. 85-98.
Hojlund, S. (2014), “Evaluation Use in the Organizational Context: Changing Focus to Improve Theory”, en Evaluation, Vol. 20 N° 1, pp. 26-43.
Hyyryläinen, E. y Viinamäki, O. (2008), “The Implications of the Rationality of Decision-Makers on the Utilization of Evaluation Findings”, en International Journal of Public Administration, Vol. 31 Nos. 10-11, pp. 1223-1240, https://doi.org/10.1080/01900690801973311
Johansson, T. y Siverbo, S. (2009), “Explaining the Utilization of Relative Performance Evaluation in Local Government: a Multi-Theorical Study Using Data from Sweden”, en Financial Accountability and Management, Vol. 25 N° 2, pp. 197-224.
Kroll, A. (2012), “Why Public Managers Use Performance Information: Concepts, Theory, and Empirical Analysis”, Potsdam, The University of Potsdam. Doctoral Thesis.
Kumar, M.; Taegtmeyer, M.; Madan, J.; Ndima, S.; Chikaphupha, K.; Kea, A.; y Barasa, E. (2020), “How Do Decision-Makers Use Evidence Incommunity Health Policy and Financing Decisions? A Qualitative Study and Conceptual Framework in Four African Countries”, en Health Policy and Planning, Vol. 35 N° 7, pp. 799-809.
Lammert, J.; Heinemeier, S.; y Fiore, T. (2017), Effectively Communicating Evaluation Findings, Rockville, Westat.
Landry, R.; Lamari, M.; y Amara, N. (2003), “The Extent and Determinants of the Utilization of University Research in Government Agencies”, en Public Administration Review, Vol. 63 N° 2, pp. 192-205.
Lapuente, V. y Van de Walle, S. (2020), “The Effects of New Public Management on the Quality of Public Services”, en Governance, Vol. 33 N° 3, pp. 461-475, https://doi.org/10.1111/gove.12502
Ledermann, S. (2012), “Exploring the Necessary Conditions for Evaluation Use in Program Change”, en American Journal of Evaluation, Vol. 33 N° 2, pp. 159-178, https://doi.org/10.1177/ 1098214011411573
Leviton, L. y Hughes, E. (1981), “Research on the Utilization of Evaluations: a Review and Synthesis”, en Evaluation Review, Vol. 5 N° 4, pp. 525-548.
Lindblom, C. (1959), “The Science of Muddling through”, en Public Administration Review, Vol. 19 N° 2, pp. 79-88.
Lipsky, M. (1980), Street-Level Bureaucracy: Dilemmas of the Individual in Public Services, New York, Russell Sage Foundation.
Luna, C. (2016), “La evaluación de políticas públicas centrada en el uso de resultados y de aprendizajes”, en RevIISE: Revista de Ciencias Sociales y Humanas, Vol. 8 N° 8, pp. 7-14.
Majone, G. (2014), Evidencia, argumentación y persuasión en la formulación de políticas, México, Fondo de Cultura Económica.
March, J. (1994), A Primer on Decision Making: How Decisions Happen, New York, The Free Press; Simon and Schuster.
Mark, M. y Henry, G. (2004), “The Mechanisms and Outcomes of Evaluation Influence”, en Evaluation, Vol. 10 N° 1, pp. 35-57.
McDonnell, L. y Weatherford, M. S. (2020), Evidence, Politics, and Education Policy, Cambridge, Harvard Education Press.
Mikkelsen, M. F.; Petersen N. B. G; y Bjørnholt, B. (2021), “Broadcasting Good News and Learning from Bad News: Experimental Evidence on Public Managers’ Per¬formance Information Use”, en Public Administration, Vol. 1 N° 19, https://doi.org/10.1111/padm.12762
Moynihan, D. (2005), “Goal-Based Learning and the Future of Performance Manage¬ment”, en Public Administration Review, Vol. 65 N° 2, pp. 203-216.
Moynihan, D. (2009), “Through a Glass, Darkly: Understanding the Effects of Performance Regimes”, en Public Performance and Management Review, Vol. 32 N° 4, pp. 592-603.
Moynihan, D. y Hawes, D. (2010), “The Organizational Environment and Performance Information Use”, paper prepared for the European Group of Public Administration Meeting in Toulouse, France, September 8-11.
Moynihan, D. y Hawes, D. (2012), “Responsiveness to Reform Values: the Influence of the Environment on Performance Information Use”, en Public Administration Review, Vol. 72 N° 1, pp. 95-105.
Moynihan, D. e Ingraham, P. (2001), “When Does Performance Information Contribute to Performance Information Use? Putting the Factors in Place; Working Paper”, New York, Syracuse University. Campbell Public Affairs Institute.
Moynihan, D. y Landuyt, N. (2009), “How Do Public Organizations Learn? Bridging Cultural and Structural Perspectives”, en Public Administration Review, Vol. 69 N° 6, pp. 1097-1105.
Moynihan, D. y Pandey, S. (2010), “The Big Question for Performance Management: Why Do Managers Use Performance Information?”, en Journal of Public Administration Research and Theory, Vol. 20 N° 4, pp. 849-866.
Newcomer, K. (1997), “Using Performance Measurement to Improve Public and Nonprofit Programs”, en New Directions for Evaluation, N° 75, pp. 5-14.
Newman, J.; Cherney, A.; y Head, B. (2017), “Policy Capacity and Evidence-Based Policy in the Public Service”, en Public Management Review, Vol. 19 N° 2, pp. 157-174, https://doi.org/10.1080/14719037.2016.1148191
OCDE (2002), Glossary of Key Terms in Evaluation and Results-Based Management, Paris, OCDE.
OCDE (2020), Improving Governance with Policy Evaluation. Lessons from Country Experiences, Paris, OCDE, https://doi.org/10.1787/89b1577d-en
Ongaro, M. y Andreotti, M. (2022), “Non-Empirical Uncertainties in Evidence-Based Decision-Making”, en Perspective on Science, Vol. 30 N° 2, pp. 305-320.
Ordóñez-Matamoros, G.; Centeno, J.; Santander Ramírez, D.; Llanos Congote, O.; Sierra Daza, M; y Remolina Pulido, J. (2018), “La evaluación ejecutiva y la toma de decisiones de política pública: el caso de la E2+SGR en Colombia”, en OPERA, N° 22, pp. 53-79, https://doi.org/10.18601/16578651.n22.04
Oszlak, O. (2013), “Gobierno abierto: hacia un nuevo paradigma de gestión pública”, s.l., Red de Gobierno Electrónico de América Latina y el Caribe (Colección de Documentos de Trabajo sobre e-Gobierno; N° 5).
O’Toole Jr., L. (2004), “The Theory-Practice Issue in Policy Implementation Research”, en Public Administration, Vol. 82 N° 2, pp. 309-329, https://doi.org/10.1111/j.0033-3298.2004.00396.x
Owen, J. y Lambert, F. (1998), “Evaluation and the Information Needs of Organizational Leaders”, en American Journal of Evaluation, Vol. 19 N° 3, pp. 355-365.
Pallett, H. (2020), “The New Evidence-Based Policy: Public Participation between ‘Hard Evidence’ and Democracy in Practice”, en Evidence and Policy, Vol. 16 N° 2, pp. 209-227, https://doi.org/10.1332/174426419X15704985880872
Parkhurst, J. (2016), “The Politics of Evidence. From Evidence-Based Policy to the Good Governance of Evidence, London, Routledge, https://doi.org/10.4324/9781315675008
Patton, M. (1988), “The Evaluator’s Responsibility for Utilization”, en Evaluation Practice, Vol. 9 N° 2, pp. 5-24.
Pawson, R.; Wong, G; y Owen, L. (2011), “Known Knowns, Known Unknowns, Unknown Unknowns: the Predicament of Evidence-Based Policy”, en American Journal of Evaluation, Vol. 32 N° 4, pp. 518-546.
Peck, L. y Gorzalski, L. (2009), “An Evaluation Use Framework and Empirical Assessment”, en Journal of Multidisciplinary Evaluation, Vol. 6 N° 12, pp. 139-156.
Perry, J. y Hondeghem, A. (2008), Motivation in Public Management: the Call of Public Service, Oxford, Oxford University Press.
Poister, T. y Streib, G. (1999), “Performance Measurement in Municipal Government: Assessing the State of the Practice”, en Public Administration Review, Vol. 59 N° 4, pp. 325-335.
Preskill, H. (2004), “The Transformational Power of Evaluation: Passion, Purpose, and Practice”, en Evaluation Roots, Marvin C. Alkin (ed.), California, Sage Publications, https://doi.org/10.4135/9781412984157.n23
Pressman, J. y Wildavsky, A. (1984), Implementation. How Great Expectations in Washington Are Dashed in Oakland, Berkeley, University of California Press. 3. ed.
Rickinson, M.; De Bruin, K.; Walsh, L.; y Hall, M. (2017), “What Can Evidence-Use in Practice Learn from Evidence-Use in Policy?”, en Educational Research, Vol. 59 N° 2, pp. 173-189, https://doi.org/10.1080/00131881.2017.1304306
Rossi, P.; Lipsey, M.; y Freeman, H. (2004), Evaluation: a Systematic Approach, Thousand Oaks, Sage Publications. 7. ed.
Rutter, J. (2012), Evidence and Evaluation in Policy Making: a Problem of Supply or Demand?, London, Institute for Government, http://www.instituteforgovernment.org.uk/publications/evidence-and-evaluation-policy-making
Saliterer, I. y Korac, S. (2013), “Performance Information Use by Politicians and Public Managers for Internal Control and External Accountability Purposes”, en Critical Pers¬pectives on Accounting, Vol. 24 Nos. 7-8, pp. 502-517.
Sandison, P. (2006), “The Utilisation of Evaluations”, en ALNAP Review of Humanitarian Action: Evaluation Utilisation, J. Mitchel (ed.), London, Active Learning Network for Accountability and Performance in Humanitarian Action, pp. 89-144.
Shaxson, L. (2019), “Uncovering the Practices of Evidence-Informed Policy-Making”, en Public Money and Management, Vol. 39 N° 1, pp. 46-55, https://doi.org/10.1080/09540962.2019.1537705
Shaxson, L. y Boaz, A. (2020), “Understanding Policymakers’ Perspectives on Evidence Use as a Mechanism for Improving Research-Policy Relationships”, en Environmental Education Research, Vol. 27 N° 4, pp. 518-524, https://doi.org/10.1080/13504622.2020.1764505
Shulha, L. y Cousins, J. (1997), “Evaluation Use: Theory, Research, and Practice Since 1986”, en American Journal of Evaluation, Vol. 18 N° 3, pp. 195-208.
Simon, H. (1945), The Administrative Behavior, New York, Free Press.
Spekle, R. y Verbeeten, F. (2009), “The Use of Performance Measurement Systems in the Public Sector: Effects on Performance”, paper presented at the AAA 2009 Management Accounting Section (MAS) Meeting, https://ssrn.com/abstract=1162242
Stufflebeam, D. (1983), “The CIPP Model for Program Evaluation”, en Evaluation Models: Viewpoints on Educational and Human Services Evaluation, G. F. Madaus, M. S. Scriven y D. L. Stufflebeam (eds.), Boston, Kluwer-Nijhoff, pp. 117-141.
Taylor, J. (2011a), “Factors Influencing the Use of Performance Information for Decision Making in Australian State Agencies”, en Public Administration, Vol. 89 N° 4, pp. 1316-1334, https://doi.org/10.1111/j.1467-9299.2011.02008.x
Taylor, J. (2011b), “Strengthening the Link between Performance Measurement and Decision Making”, en Public Administration, Vol. 89 N° 3, pp. 860-878.
Tummers, L. y Bekkers, V. (2014), “Policy Implementation, Street-Level Bureaucracy, and the Importance of Discretion”, en Public Management Review, Vol. 16 N° 4, pp. 527-547.
UNEG (2016), “Normas y estándares de evaluación”, Nueva York, Naciones Unidas. Grupo de Evaluación.
Van Dooren, W.; Bouckaert, G.; y Halligan, J. (2010), Performance Management in the Public Sector, London, Routledge.
Van de Walle, S. y Van Dooren, W. (2009), “How Is Information Used to Improve Performance in the Public Sector? Exploring the Dynamics of Performance Information”, en Connecting Knowledge and Performance in Public Services: from Knowing to Doing, K. Walshe, G. Harvey y P. Jas (eds.), Oxford, Oxford University Press, https://doi.org/10.2139/ssrn.1491644
Weiss, C. H. (1979), “The Many Meanings of Research Utilization”, en Public Administration Review, Vol. 39 N° 5, pp. 426-431.
Weiss, C. H. (1982), “Policy Research in the Context of Diffuse Decision Making”, en The Journal of Higher Education, Vol. 53 N° 6, pp. 619-639.
Wholey, J. S.; Hatry, H. P.; y Newcomer, K. E. (2010), Handbook of Practical Program Evaluation, San Francisco, Jossey-Bass.
Wittgenstein, L. (2014), Recherches Philosophiques, Paris, Gallimard.
Wooldridge, J. M. (2009), “On Estimating Firm-Level Production Functions Using Proxy Variables to Control for Unobservables, en Economics Letters, Vol. 104 N° 3, pp. 112-114.
Zaltsman, A. (2010), The Multiple Uses of Performance Information throughout the Budgetary Process: Insights from Chile’s Performance-Based Budgeting System, s.l., s.e.
Cómo citar
Licencia
Derechos de autor 2022 Revista del CLAD Reforma y Democracia
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-SinDerivadas 4.0.
Los contenidos de la revista se distribuyen bajo la licencia Creative Commons Reconocimiento-NoComercial 4.0 Internacional (CC BY-NC-4.0), que permite a los usuarios copiar, usar, difundir, transmitir y exponer públicamente los contenidos, siempre y cuando se cite la autoría (https://creativecommons.org/licenses/by-nc/4.0).