How Do We Measure Impact?

Reshma Trasi

Director, Monitoring, Evaluation and Research

        From left to right: Sara Bennett, Eric Sarriot, Kenneth Sklaw and the author, Reshma Trasi.

Yesterday, I had the pleasure and privilege of moderating a stimulating discussion, hosted by the USAID-funded AIDSTAR-Two project entitled, "Organizational Capacity Development: How Do We Measure Impact?" The thoughtful and brilliant panelists Sara Bennett, Associate Professor in the Health Systems Program at Johns Hopkins Bloomberg School of Public Health; Eric Sarriot, founder and director of the Center for Design and Research in Sustainable Health and Human Development at ICF International; and Kenneth Sklaw, Team Lead for the Capacity Building Team in the Office of HIV/AIDS within USAID validated and challenged my thinking on ways to measure the results of organizational capacity development—which is core to the work that the LMG Project does around the world.

These were my take-aways—the “Seven Cs” for measuring results of organizational capacity development:

  1. Organizational capacity development is COMPLEX and dynamic; measuring its results must take this into consideration. Organizational health, like a person’s health, is influenced by the system in which it functions and is more than just the sum of the individuals that make up the organization. So, when we measure organizational capacity development results, we have to remember that these may change over time as the organization and the organization’s needs evolve or change over time. One set of measures used at the baseline may no longer be relevant a year or two down the line.
  1. Organizational development is CONTEXTUAL: organizational development results for a small community based organization maybe different from those for a larger civil society organization, district health team, university or network or federation of organizations.
  1. The results being measured must be COMPREHENSIVE: results ought to look beyond output measures and focus on the performance of the organization. One-time assessments are not useful; results need to be measured over time. Projects need to think about what happens if the change occurs after the life of the project.
  1. The process of deciding what to measure, when, and how often, and for how long must be COLLABORATIVE: evaluators are but one player in the mix. Evaluators must also reflect on whether the framework they are using to measure organizational capacity helps the organization to meet its own capacity development goals.  
  1. CAUSALITY is hard to prove. We should be looking at plausible associations rather than attribution of results to organizational capacity development efforts.
  1. The evidence base for capacity building needs to be along a CONTINUUM: Quantitative indicators must be combined with (perhaps) case studies of success and failure so we can understand not only what has changed, but how and why the change has occurred. Using mixed-methods creatively – like realist evaluation – may be the next frontier.  
  1. And most importantly, the organization whose capacity is being developed must be CENTRAL to measurement and evaluation efforts. We must remember to measure what the organization wants to see change, in its own capacity, over time.

You can watch the archived discussion here: https://sas.elluminate.com/mr.jnlp?suid=M.9D9B8325A63AC6DF0F18AE71A7EFBE&sid=2011113

Its true that measuring Impact of Organizational Capacity Programs is quiet a challenge. Being a Deputy Project Director for an Institutional Capacity Building Project in Tanzania, we have been facing a problem of tying the Organization success to our interventions. For example, during Organizational assessment for one of our partners a need to review organogram, job descriptions, HR manual and Induction manual was evident. Through consultative Technical Assistances and workshops, the organogram , JD , HR & Induction Manual were developed. The question is how can we link these management & leadership structures into the overall performance of the agency?

For the same Organisation, we implemented a resource mobilization Capacity Building resulting into mobilization of resources from private companies in the country. For this one, we can confidently link our intervention with the resources mobilized post intervention.

Leadership & management researchers & Health system strengthening specialists needs to come up with a clear way to measure & link to the overall performance of the Organisation, the results of interventions aimed at strengthening the Organizational structures & functions as part of organizational capacity building.

Dear Dr. Isangula: Thank you for your comments. One way to link management and leadership changes to the overall performance of the agency is to identify, right at the start, measures that we think will change over time. These can be through a consultative process that includes the organization staff as well as the team providing technical assistance. E.g. when resource mobilization was carried out, you had a tangible change: resources raised by the organization. What are some of these things you expected to change with the JDs or when the HR & Induction Manual was applied? Also, in my humble opinion, not everything can be measured or quantified.

If these discussions about 'change' can be had at the design stage or when the manuals or capacity building strategies are being developed, it will help identify indicators that can then be measured over time. I do not mean to oversimplify the issue - this is challenging and we cannot attribute the changes to the capacity development effort alone - but identifying indicators can allow us to do baseline assessments and then measure these indicators at different points in time to understand what has changed. It is also useful to bring these changes back to the organization and have a discussion around why and how these changes have taken place. And if the measures are still relevant and appropriate.

It sounds like your project does really interesting and important work in Tanzania. My best wishes for a successful project!

Wow that was odd. I just wrote an extremely long comment but after I clicked submit my comment didn't show up. Grrrr well I'm not writing all that over again. Anyway, just wanted to say superb blog!