Orientation to using Outcome Harvesting for evaluating advocacy. Klugman with Ricardo Wilson-Grau 2012

OUTCOME HARVESTING

Here's the talk I wrote with Ricardo Wilson-Grau reflecting on our experience on using OutcomeHarvesting for evaluating an advocacy network. I presented it at the American Evaluation Association 2012 Conference.

I find that the idea of harvesting outcomes helps us to be rigorous in naming who changed what when. Often we'll say 'oh that meeting was really useful because it gave people ideas'. The discipline of Outcome Harvesting is to make sure that we record who took away what ideas from what event and how did they use those ideas - that is what changed what, and how did a meeting contribute towards those changes (by giving the person new ideas and motivation to take action)....so read on....

Klugman B with Wilson-Grau R The value of outcome harvesting in evaluating advocacy organisations. AEA Conference Eval2012

Evaluating social justice advocacy: a values based approach. Center for Evaluation Innovation 2010

See my briefing document on values in social justice evaluation:

Brief: Evaluating social justice advocacy: a values based approach. Center for Evaluation Innovation 2010

AEA365.org Using SNA to quantify and visualise relationships of power and influence in advocacy networks

Reposted from AEA365.org:

Using social network analysis to quantify and visualise relationships of power and influence in advocacy networks AEA365.org

I have been using social network analysis (SNA) both for understanding shifts in the scope and depth of collaborations within networks, and to understand if and how relationships among a cohort of fellows in a leadership development programme shift over time. This blog talks to use in evaluating advocacy networks.

APC TIG Week: Using SNA to quantify and visualise relationships of power and influence in advocacy networks by Barbara Klugman

2 Comments / Advocacy and Policy Change / By Sara Vaca / June 5, 2018

Hi, I’m Barbara Klugman. I offer strategy support and conduct evaluations with social justice funders, NGOs, networks and leadership training institutions in South Africa and internationally. I practice utilization-focused evaluation, frequently using mixed methods including outcomes harvesting and Social Network Analysis (SNA).

Rad Resource: For advocacy evaluation, SNA can help identify:

• how connected different types of advocacy organizations are to each other;
• what roles they play in relation to each other such as information exchange, partnering for litigation, driving a campaign, or linking separate networks;
• if and how their positioning changes over time in terms of relative influence in the network.

The method involves surveying all the groups relevant to the evaluation question, asking if they have a particular kind of relationship with all other groups surveyed. To illustrate the usefulness of SNA, the map below illustrates an information network of the African Centre for Biodiversity, a South African NGO.  In the map, each circle is an organization, sized by the number of organizations who indicated “we go to this organization for information” – to answer one piece of the evaluation question, regarding the position and role of the evaluand in its field, nationally and regionally. Of the 55 groups advocating for food sovereignty in the region who responded, the evaluand is the main bridger between South African groups and others on the continent. It is also a primary information provider to the whole group alongside a few international NGOs and a few African regional organizations.

As another example, an SNA evaluating the Ford Foundation’s $54m Strengthening Human Rights Worldwide global initiative distinguished changes in importance and connectedness before the initiative and after four years, among those inside the initiative (blue), ‘matched’ groups with similar characteristics (orange), and five others in Ford’s portfolio (pink). It shows that the initiative’s grantees and notably those from the Global South (dark blue) have developed more advocacy relationships than have the matching groups (see larger size of nodes and more connections). However, the largest connector for advocacy remains Amnesty International – the big pink dot in the middle, demonstrating its continuing differential access to resources and influence relative to the other human rights groups.

Hot tips:

• Keep it simple: As surveys ask about each organization, responding takes time, so ask only about roles that closely answer the evaluation questions regarding the network. For example, “my organization has engaged with them in advocacy at a regional forum”; “my organization has taken cases with them”
• Work with a mentor: While SNA software like Gephi is open access, making sense of social network data requires statistical analysis capacity and SNA theory to extract meaning accurately.

Lesson Learned:

• Consider whether or not to show names of groups as your tables or maps will surface who is ‘in’ and who is on the outside of a network in ways that might have negative consequences for group dynamics or for individual groups, or expose group’s negative perceptions of each other.

Rad resources:

Wendy Church, Introduction to Social Network Analysis, 2018.

AEA365.org Six conditions that increase the likelihood & effectiveness of evaluators speaking truth to power

From Six conditions that increase the likelihood & effectiveness of evaluators speaking truth to power AEA365.org

Speaking truth to power was the topic of the American Evaluation Association's eval2018 Conference. I did this blog of a paper I presented there.

Six Conditions that Increase the Likelihood and Effectiveness of Evaluators Speaking Truth to Power by Barbara Klugman

1 Comment / Advocacy and Policy Change / By Sara Vaca / September 5, 2018

Hi, I’m Barbara Klugman. I offer strategy support and conduct evaluations with social justice funders and NGOs in South Africa and internationally. I practice utilization-focused evaluation, frequently using mixed methods including outcomes harvesting and social network analysis (SNA). My own history spans social activism, directing NGOs and both working for and being on boards of foundation.

AEA’s Conference theme is Speaking Truth to Power, something that is particularly challenging because of inequitable power relations between nonprofits and their funders, and even between boards and staff.  Evaluators can play a useful intermediary role by providing both the evidence and the facilitation to open space for honest communication.

Hot Tip: I have found the following six factors influenced the effectiveness of my communication across power divides:

1. Timing of the evaluation and a formative or developmental approach may enhance both grantee and funder interest in the outcomes.
2. Making learning rather than compliance the evaluation objective creates an environment that welcomes insights to strengthen effectiveness and removes much of the fear and risk from evaluation.
3. The evaluator needs a substantial capacity for evaluation practice that enhances trust-buildingto undercut anxiety and establish rules of engagement that allow those with least power the ability to engage, influence and use findings.
4. The production of high quality evidence while self-evident will be more effective in speaking truth to power if all parties have agreed on the questions, mix of methods and evaluation rubrics
5. A commitment to and comfortableness with the role of evaluator as social justice advocateassumes that the evaluator can navigate when it is appropriate for her to speak, and when to empower the evaluand to do so.
6. Terms of reference give the evaluator the independent right and resources to communicatefindings to audiences beyond the intended users or those to whom they disseminate findings. While recognising the concomitant ethical responsibility to do no harm, the right and resources to publish findings is critical to the ability of an evaluator to speak truth to power and for the resources that go into evaluation to contribute to broader learning in the field.

Rad Resources: As an illustrative example, see the public communications from the evaluation team of the Ford Foundation’s $54m Strengthening Human Rights Worldwide global initiative. The ToR included funds for the team to publicize findings in Spanish and English which included the summary report, a series of blogs and videos, an article for the international human rights journal SUR and a reflection in Alliance magazine.

Blogs:

The Value of Diversity in Creating Systemic Change for Human Rights

Finding Equity – Shifting Power Structures in Human Rights

Addressing Systemic Inequality in Human Rights Funding

Videos:

The Human Rights System is Under Attack – Can it Survive Current Global Challenges?

The Changing Ecology of the Human Rights Movement

Funding an Effective Human Rights Movement

AEA365.org on the Value of an Evaluation Advisory Group

See The value of an independent evaluation advisory group for developmental evaluators AEA365.org

The Value of an Independent Evaluation Advisory Group for Developmental Evaluators by Barbara Klugman

3 Comments / Independent Consulting / By Sara Vaca / October 25, 2018

Barbara Klugman

Hi, I’m Barbara Klugman. I offer strategy support and conduct evaluations with social justice funders, NGOs, networks and leadership training institutions in South Africa and internationally.  This blog is about the value of establishing an Evaluation Advisory Group when your task requires more skills that you have!

I’m currently the ‘learning and assessment partner’ of Tekano an organisation which runs the Atlantic Fellows Programme for Health Equity South Africa.  I have found this experience challenging, not least because it’s my first experience in long term developmental evaluation, because I have multiple accountabilities – to funder, Tekano board, and staff, as well as to the public good given the country’s desperate need for a strong fellowship programme on social determinants of health; and because my primary expertise is in strategizing and evaluating social justice advocacy initiatives rather than leadership development. Finally, I am a white older woman playing the role of ‘critical friend’ to an initiative whose staff and fellows are mostly black and young and whose advocacy experience is taking place in a vastly different historical moment from when I built such experience.

With these constraints in mind, when I developed my terms of reference, I motivated a budget to allow me to establish an Advisory Group. I sought members to fill three gaps in my expertise (that I was aware of) – an organizational psychologist whose expertise traverses organisational behaviour and monitoring and evaluation; a fellowship evaluation expert; and an expert in community-building and evaluation in the South African context.

I engaged the group to help me review the vast quantity of data and analysis I had produced in order to hone in on priorities for my final 18 months in this role, and to help me clarify how to manage my layers of accountability. It was fantastic!! The group brought totally independent eyes to bear on my questions. They helped me distinguish the nature and hierarchy of my various accountabilities. They confirmed my own conclusions on achievements and challenges thus far while hauling me out of the data, insights and relationships I was buried in.  This allowed me to hone in on powerful questions and ways of asking them, for the second half of my tenure in this role.

I loved every minute of the experience even while it was exhausting and challenging. I would encourage you to build in space for this kind of independent support if you’re conducting developmental evaluations.

Hot tips: You have to be a fairly confident person to open yourself up to criticism. Choose people who are ethical and don’t have oversized egos!

Lesson Learned:

• Building a formal advisory group into evaluation budget allows you to identify exactly
• what skills might complement your own, and to compensate people appropriately  for their time
• My huge thanks to Drs. Jane Reisman, Mark Abrahams and Suki Goodman for their efforts on my advisory group