Advocacy evaluation

Advocacy evaluation, also called public policy advocacy design, monitoring, and evaluation, evaluates the progress or outcomes of advocacy, such as changes in public policy.

Advocacy evaluators seek to understand the extent to which advocacy efforts have contributed to the advancement of a goal or policy. They do this in order to learn what works, what does not, and what works better in order to achieve advocacy goals and improve future efforts.

Advocacy evaluation is different from policy analysis, which generally looks at the results of the policy, or mainstream program evaluation, which assesses whether programs or direct services have been successful. Advocacy strives to influence a program or policy either directly or indirectly; therefore, the influence is being evaluated, rather than the results of that influence.

Goals of advocacy (dependent variables)

In order to evaluate something, one must know the goals of the program/activity, in this case - advocacy efforts. Policy advocacy evaluation focuses on the contribution towards achieving policy, and not on the results of that policy. Policy advocacy evaluators look at these dependent variables (many of which interrelate significantly with movement in the policy cycle):

Intermediate Goal Examples:

• Increased awareness of constituents about the need for policy (Problem Identification -> Agenda Setting)
• Change in rate of key-words use by politicians, sometimes starting from 0 (Problem Identification -> Agenda Setting)
• Increase in ratio of policy being implemented according to the adopted legislation (Adoption->Implementation)
• Developed capacity of advocacy actor or network of actors to conduct advocacy efforts

Ultimate Goals

• Policy change itself in the desired direction (of the policy cycle). This is the highest level intermediate outcome, and as an inherent best practice, is the goal of most policy advocacy efforts. Policy Advocacy works to move a policy through the policy cycle.

Typology of policy advocacy

Direct Advocacy (Directly trying to influence policy makers):

• Lobbying (also known as direct lobbying) is the act of attempting to influence decisions made by government officials, most often legislators or members of regulatory agencies. Various people or groups, from private-sector individuals or corporations, fellow legislators or government officials, or advocacy groups use lobbying.

Indirect Advocacy (Indirectly influencing policymakers by getting their constituents to advocate):

• Grassroots lobbying (also known as indirect lobbying) is a form of lobbying that focuses on raising awareness of a particular cause at the local level, with the intention of reaching the legislature and making a difference in the decision-making process. Grassroots lobbying is an approach that separates itself from direct lobbying through the act of asking the public to contact legislators and government officials concerning the issue at hand, as opposed to conveying the message to the legislators directly.
• Activism consists of intentional efforts by citizens or citizen groups, towards policymakers, to promote or prevent social, political, economic, or environmental change. Activism can take a wide range of forms including, from writing letters to newspapers or politicians, political campaigning, rallies, street marches, strikes, sit-ins, and hunger strikes.
• Astroturfing supports political, organizational, or corporate agendas, and is designed to give the appearance of a "grassroots" movement. The goal of such campaigns is to disguise the efforts of a political and/or commercial entity as an independent public reaction to some political entity—a politician, political group, product, service, or event.

Distinct challenges of advocacy evaluation

• Contribution vs. attribution: Since multiple actors campaign simultaneously for and against any given policy, it is difficult to ascertain attribution. Evaluating contributions is preferred in this case as it allows multiple actors to influence the degree of success.
• Long term nature of advocacy: Since many advocacy goals are long term, measuring impact can be a challenge. Instead, outcomes, interim progress, and intermediary goals are the preferred measures of influence.
• Shifting strategies: Since the context that advocates work within is ever-changing, advocates adapt their strategies, which creates a difficult environment in which to monitor progress.
• Complexity and theories of change: logic models and theories of change for advocacy campaigns are inherently complex; for example: protests+lobbying+media campaigns -> contribution to policy change. These kinds of theories of change have so many layers, nuances, and uncontrollable factors to them that intra and inter organizational agreement is difficult, making strategic planning, and evaluation all the more challenging.

See also

• Advocacy
• Evidence-based policy
• Program evaluation
• Policy analysis

Source documents

Advocacy evaluation:

• Coffman, J. 2009 A User's Guide to Advocacy Evaluation Planning. Harvard Family Research Project.
• Teles, T, & Schmit, M, The Elusive Craft of Evaluating Advocacy, Stanford Center for Social Innovation
• Whelan, Justin, 2008 Advocacy Evaluation: Review and Opportunities, COMM-ORG Papers – The On-Line Conference on Community Organizing
• Reisman, Jane, Anne Gienapp, and Sarah Stachowiak, 2007 A Guide to Measuring Advocacy and Policy, Organizational Research Services for the Annie E. Casey Foundation
• Chapman, J. and Wameyo, A, 2001 Monitoring and Evaluating Advocacy: A Scoping Study, Action Aid
• O'Flynn, Maureen, 2009 Tracking Progress in Advocacy: Why and How to Monitor and Evaluate Advocacy Projects and Programs, International NGO Training and Research Center (INTRAC)
• Coffman, J., 2007 Using the Advocacy and Policy Change Composite Logic Model to Guide Evaluation Decisions, Harvard Family Research Project
• Cohen, David, Rosa de la Vega, and Gabrielle Watson. "Advocacy for Social Justice: A Global Action and Reflection Guide.” Kumarian Press, 2001.
• Covey, J, 1994 Accountability and Effectiveness of NGO Policy Alliances, Institute of Development Research (IDR)
• Patton, Michael Q, 2008 Advocacy Impact Evaluation, Journal of Multi Disciplinary Evaluation
• Fox, Leslie M. and Priya Helweg, 1997 Advocacy Strategies for Civil Society: A Conceptual Framework and Practitioner's Guide, USAID
• McGuigan, Claire, 2003 closing the Circle: From Measuring Policy Change to Assessing Policies in Practice, Development Dialogue Team, and Save the Children UK
• Coe, Jim and Ruth Mayne, 2008 getting the Basics Right: Is Your Campaign Making a Difference? National Council for Voluntary Organizations
• Coates, Barry and Rosalind David, 2002 Learning for Change: The Art of Assessing the Impact of Advocacy Work, Development in Practice, Oxfam
• Puntenney, Deborah L., 2002 Measuring Social Change Investments, Women's Funding Network
• Guthrie, Kendall, Justin Louie, Tom David and Catherine Crystal Foster, 2005 The Challenge of Assessing Policy and Advocacy Activities: Strategies for a Prospective Evaluation Approach, Blueprint Research, and Design, Inc. for The California Endowment
• Guthrie, Kendall, Justin Louie, and Catherine Crystal Foster, 2006 The Challenge of Assessing Policy and Advocacy Activities: Part II Moving from Theory to Practice, Blueprint Research and Design, Inc. for The California Endowment
• Chapman, Jennifer, and Thomas Fisher, 2000 The Effectiveness of NGO Campaigning: Lessons from Practice, Development in Practice
• Coffman, Julia, 2009 Overview of Current Advocacy Evaluation Practice, Center for Evaluation Innovation

Contribution Analysis:

• Egbert, M., & Hoechstetter, S. (2006). Mission possible: Evaluating advocacy grants. Foundation News & Commentary
• Gardner, Annette L and Brindis, Claire D.(2017) Advocacy and Policy Change Evaluation: Theory and Practice, Stanford University Press
• Mayne, J. (2008). Contribution analysis: An approach to exploring cause and effect. ILAC Brief
• Mayne, John (1999). Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly. Office of the Auditor General of Canada working paper, Ottawa
• Jessica Dart, Report on outcomes and get everyone involved: The Participatory Performance Story Reporting Technique
• Fiona Kotvojs Kurrajong Hill Pty Ltd and Cardno ACIL. “Contribution Analysis – A New Approach to Evaluation in International Development.” Paper presented at the International Conference of the Australasian Evaluation Society, Inc., 2006, Darwin Australia.

External links

• International Advocacy Evaluation Community of Practice
• Aspen Institute Advocacy Planning and Evaluation Program
• The Change Agency
• Center for Evaluation Innovation: Advocacy & Public Policy
• Advocacy Evaluation Update, a Regular Newsletter
• The Advocacy Progress Planner – An online tool for advocacy planning and evaluation
• California Endowment – Advocacy Evaluation Advances

Examples of Advocacy Evaluation

Note – some of the following evaluations should be seen as forms of "Proto" Advocacy Evaluation, done prior to, or without regard for, current best practices in this field. Many simultaneously conduct policy analysis and advocacy evaluation. Most are, at the least, useful examples for anyone wishing to conduct Advocacy Evaluation.

• A Decade of Effort, A World of Difference: The Policy and Public Education Program of The California Youth Violence Prevention Initiative by Lawrence Wallack, Amy Lee and Liana Winett 2003
• Collaborations that Count, Power, Participation, and State-Based Politics An Evaluation of the Ford Foundation's Collaborations that Count Initiative. Applied Research Center, April 2004
• Anne Gienapp and Carolyn Cohen, advocacy evaluation case study- the chalkboard project – prospective evaluation of an Oregon K-12 education reform effort
• Measuring Impact in Practice: A Case Study of The Humane Society of the United States
• Byron Farmers Market: A Case Study of Local Food Advocacy Vanessa Joh
• Changing the system, what works? Insights from advocates for migrants, refugees, and asylum seekers in Queensland
• Not in our Name:An Evaluation of the AustralianAnti-War Movement, 2002–2003
• Organizational Learning and the NGO:A preliminary case study of the Refugee Action Collective Queensland-Brisbane
• Supported living campaign: Autonomy and control by people with disability and their families
• Wildlife Advocates: A case study of the wildlife preservation society of Queensland
• Assorted Ongoing Advocacy Evaluations of InnoNet
• Evaluation of Oxfam Great Britain's Climate Change Campaign
• Coffman, Julia. “Lessons in Evaluating Communications Campaigns: Five Case Studies.” Harvard Family Research Project, March 2003.
• Trenholm, Christopher. Expanding Coverage to Children: The Santa Clara County Children's Health Initiative. Mathematica Policy Research, Inc., June 2004.
• Gold, Eva, and Elaine Simon. “Strong Neighborhood, Strong Schools: The Indicators Project on Education Organizing.” Research for Action, 2002.
• “Outreach Extensions: National “Legacy” Outreach Campaign Evaluation.” Applied Research and Consulting, 2002.
• “Power, Participation, and State-Based Politics: An Evaluation of the Ford Foundation's Collaborations that Count Initiative.” Applied Research Center, April 2004
• Speer, Paul W. et al. “People Making Public Policy in California: The PICO California Project.” Vanderbilt University, May 2002.
• Wallack, Lawrence, Amy Lee, and Liana Winett. “A Decade of Effort, A World of Difference: The Policy and Public Education Program of the California Youth Violence Prevention Initiative.” School of Community Health, College of Urban and Public Affairs, Portland State University. Funded by a grant from the California Wellness Foundation, 2003.