Measuring impact in nonprofits
Our Nonprofit Datafolk Club is a friendly group of like-minded data folk working in or with nonprofits. Every month we get together online for a free interactive workshop discussing a data-related issue. In February, the session was all about measuring impact in nonprofits. As usual, we discussed three key questions…
Q1: What do you understand by impact measurement and how does your organisation measure impact currently?
This question led to our participants discussing the challenges and opportunities presented by using a theory of change to measure impact.
While some organisations did have a theory of change, people noted that it requires a significant investment of time and effort to develop one, and this is a potential barrier. Some felt that the wording of ‘theory of change’ could be off-putting, coming across as jargon and sometimes being misinterpreted as referring to internal change (which can make people unnecessarily worried about motives). There was also some discussion on the purpose and intended audience for a theory of change – was it to help staff rally around the mission, or to explain the work to funders?
Nevertheless, people whose organisations had a theory of change emphasised that the process could be very rewarding when done well.
People also spoke about the friction between measuring outputs (for example, what activities/interventions/services were delivered), compared with outcomes (the change or difference in attitudes, behaviours, skills or knowledge that happened as a result). Many felt that key performance indicators (KPIs) don’t always measure what is meaningful. Most agreed that you need senior buy-in and commitment to be impact-led rather than income-led.
People felt that qualitative data gave a better reflection of impact (but they didn’t always have the processes or framework in place to analyse it), yet grant-makers tend to want quantitative evidence. This is where standardised measurements could come in useful, such as the Warwick–Edinburgh Mental Wellbeing Scale (WEMWEBS) or ONS wellbeing measures.
Q2: What advice do you have around impact measurement?
We identified five top tips that appeared frequently in our participants’ discussions:
1. Start early
Think about impact from the start of a programme or project - before you begin collecting data. Ideally, use a theory of change to determine clear outputs and outcomes, and develop a plan to implement it which includes a feedback loop to share insights.
2. Regularly review and adapt
Be flexible and willing to make continuous improvements. One person suggested an annual cycle of ‘decide – pilot – review'. Strategic goals, demand and other things change, so you also need to be able to change.
3. Invest in good data collection practices
Bearing in mind that impact measurement is often low on the priority list for frontline staff and participants, ensure that data collection methods are straightforward and user-friendly. Actions to consider include:
Invest in training for frontline staff so that they can thoroughly understand and embed evaluation processes throughout their work with participants.
Identify measures that are specific and relevant, so you only require people to spend time collecting data that is absolutely necessary.
Make sure key methodologies and assumptions are clearly documented for reference.
Ensure data collection methods are appropriate for both staff and participants (whether it’s a paper or digital form filled in directly, or a discussion in person or via a phone call).
4. Keep communication clear
The benefits of collecting and reporting on data should be clearly communicated with staff from early in the process. When presenting data, it should be accessible for people to engage with so they can understand it’s importance. If you have a dashboard, show people what they're looking at and get their feedback.
5. Encourage a positive data culture
Work to embed impact measurement into the culture of the organisation, ensuring staff at all levels see the value in data. It is important to have someone at strategic or board level advocating for data and impact. Senior staff tend to see value in data, data gathering, monitoring and evaluation when they have been able to use it in lobbying and campaigning.
Q3: What concerns do you have about impact measurement?
Some people were unsure about how to build capacity around impact measurement in their organisation – for example, how best to structure roles or teams relating to evaluation and impact.
Others said it can be challenging to develop effective methodologies for collecting data from participants, particularly where interventions are short-term. They were keen to ensure that people don’t feel harassed or over-surveyed, whilst also wanting to be able to evidence change. Some people suggested using unconventional contact methods, such as WhatsApp, and sharing the impact that feedback has with participants to encourage engagement.
People also questioned how best to balance obtaining responses from as broad a range of participants as possible (for example, by having both paper and digital forms, as well as conversational feedback options), with resource limitations.
One person was concerned that the brand loyalty of long-term clients influenced their feedback, and asked the group how to encourage objective responses. People suggested a common feedback model – asking for one thing to start, stop, and continue doing – or asking for ‘a feedback sandwich’, which is one piece of constructive criticism sandwiched by two pieces of praise.
Some people were also unsure of the best analysis techniques for qualitative data, such as conversational feedback. They wanted to be able to capture robust and meaningful insight from such data in a time-efficient way. Some hoped that developments in AI would help with this over the next few years.
Support in measuring impact in nonprofits
If you’re looking for support with impact measurement, we can provide both strategic and practical support to explore, define, and articulate the impact your organisation has. Get in touch for a no-obligation chat.
Join the Nonprofit Datafolk Club
If you found this resource interesting, or if you have any curiosity in nonprofit data more generally, please come and join us at our next workshop. Each month has a different topic, and you will be able to find the details on our events page. Previous topics have included: