Impact report
Data Maturity Assessment Impact Report 2020
Written by Ben Proctor and Sian Basker, Data Orchard CIC, January 2021
Download the pdf or read the online version below.
Contents
Executive summary
The headline finding from our impact analysis is that Data Orchard’s online Data Maturity Assessment is a tool for learning and a catalyst for action.
Data maturity is the organisational journey towards improvement and increased capability in using data. Our theory of change suggests that key problems in the not-for-profit sector around data include:
a lack of knowledge about data maturity (i.e. what good and great looks like);
a sense of needing to get better but not knowing where to start;
an inability to frame the problem with data in order to engage colleagues and senior leaders in driving improvements;
frustrations with poor and outdated systems and tools;
an inability to unlock the value of data for decision making.
Our theory goes on to suggest that undertaking our Data Maturity Assessment leads to important changes in knowledge, attitudes and behaviours for these users. This ultimately enables their organisations to focus their efforts, build their data capabilities and advance their data maturity.
What users told us
Based on our survey of 73 users (approximately 20% of all validated users in the first 13 months since launching our online assessment tool in October 2019) we found the tool has been successful in numerous ways:
It stimulates dialogue, enabling people to start thinking about and talking about data in their organisations. For example, 63% of users discussed with their colleagues where change and improvement was needed, and 61% shared their results with other people in the organisation.
It focuses organisations on planning to improve. Around half of non-profits in our survey went on to develop plans to improve with data. Over a quarter successfully secured resources to build their data capabilities by investing in jobs, advice, tools and training.
Those that went on to implement their plans reported key benefits (either extensively or moderately), including: increased knowledge and expertise (86%), improved strategic planning and decision making (71%), improved products and services (57%), and increased collaboration and data sharing for strategic benefits with partners/stakeholders (50%). Other benefits included increased credibility and influence (48%), efficiency savings (37%) and increased impact (29%).
Another key user group of the tool is ‘support providers’. This group includes agencies and freelance consultants, volunteer/pro-bono services, and service and product providers who support not-for-profit sector clients. They tell us that our tool has enabled them to provide an independent diagnosis of clients’ needs, help clients work out where to start, and secure leadership support for change and improvement. Benefits to support providers included: improved understanding of factors and questions (100%), raised awareness of good practice (100%) and reassurance they were on the right track with their clients (89%).
The tool is perceived by users in the not-for-profit sector to be useful (90%) and something they would be likely to recommend to other not-for-profit organisations (90%). Support providers and other users are probably our greatest advocates with 93% saying they find it useful and 100% likely to recommend it to not-for-profits.
Next steps
It is early days for the tool, and this was a small sample. Given the online tool has only been available for just over a year, it’s reassuring that positive rewards are already being seen by users.
We hope as we continue to monitor our impact and explore how best to optimise the usefulness of our tool, we can continue to improve our impact. In the meantime this exercise has been useful in informing our next steps, which are:
Get going with the next phase of our online product development (and continue to engage partners and users in developing and testing).
Re-review how long it takes for users to complete the Data Maturity Assessment, make it shorter and quicker to complete (without losing integrity and quality).
Continue to capture user feedback in an ongoing way (and tweak our questions and theory of change here and there).
Capture stories and case studies of data maturity journeys from the not-for-profit sector.
Keep working at improving our methods for benchmarking, validating and segmenting.
Promote the Data Maturity Assessment to the not-for-profit organisations who would benefit from it, and the support providers, networks and infrastructure organisations serving them.
INTRODUCTION
Data Orchard has been researching organisational data maturity since 2015. In October 2019 we launched the world’s first free online assessment and benchmarking tool for the not-for-profit sector, our Data Maturity Assessment. In February 2020 we created a more advanced customisable version for larger organisations and those really serious about data maturity.
Of course, being data specialists ourselves, we’ve been monitoring the user data to improve our tool along the way. However, after our first year of the tool being online, with almost 400 validated users, we felt it was the right time to evaluate its impact.
Our purposes were threefold:
To listen to user feedback.
To test our ‘theory of change’.
To learn how we can improve the tool for future users.
What is data maturity?
Data maturity is defined as the journey towards improvement and increased capability in using data.
We’ve created a framework model which describes data maturity on a five stage journey. This moves from ‘unaware’ through to ‘mastering’ across seven key themes: Uses, Data, Analysis, Leadership, Culture, Tools and Skills.
The framework is freely available for non-profit use, licensed under Creative Commons Licence, and can be downloaded here dataorchard.org.uk/resources/data-maturity-framework
The online Data Maturity Assessment involves users answering a set of questions related to each of the seven themes described in Figure 2.
Figure 2 Summary of the key themes and sub themes
At the end of the assessment users receive a report with their results. The report identifies where on the five stage journey the user/s are assessed to be, how they score in each of the seven themes, where their strengths are, and suggests priorities to focus on. In addition, users can compare how they’re doing with the sector through benchmarks, and explore current and next stages for each theme. An example report is given in figure 3. You can view a live version of this report here.
Figure 3 Sample results report from our Data Maturity Assessment
What is a theory of change?
A theory of change is a way of articulating the impact you seek to create. It identifies the people and problems you intend to make a difference to, the activities you deliver and the outputs from these. Crucially, it then describes the outcomes that result from user engagement, specifically changes in behaviour, attitudes, knowledge and skills.
A key aspect of a theory of change is that it points to the long-term sustained change you want to create. Hence the theory extends from immediate outcomes at the point users leave your service (or online product in our case), into what happens afterwards.
Our theory of change for the Data Maturity Assessment tool
Key user groups
We identify three user groups in not-for-profit organisations that we seek to help:
Leader (CEO, director, trustee)
Data Champion (Data/database manager, analyst or someone responsible for data in impact evaluation, research, fundraising, digital/ICT, marketing, service management or delivery)
Data Strategist (Responsible for data across the whole organisation)
Problems
Our theory proposes these people experience a range of problems:
they don’t have much knowledge about data maturity
they feel data is not working for their organisation’s cause
they sense they’re not doing so well with data compared to others in the sector and want to check*
they think they might need to get better with data but don’t know where to start
they can’t unlock the value of data for decision making
they struggle to engage senior leaders in driving improvements
they are frustrated by poor and/or outdated systems and tools
they can’t interest their colleagues in thinking and talking about data.
*conversely, some, especially if they’ve been investing a lot of effort into building their data capabilities in recent years, might think they should be doing better than others in the sector and want to check.
Activities and outputs
There are currently three versions of the Data Maturity Assessment tool available for users:
Taster
Individual
Organisational
| Taster | Individual | Organisational | |
|---|---|---|---|
| Description | Quick 5 min taster assessment | Full 20 min assessment | Full assessment for multiple users |
| Users | One user | One user (or small group together) | One organisation/ unlimited users |
| Output | Basic report giving overall stage of maturity and scores across seven key themes. | Detailed report giving overall stage of maturity and scores for seven key themes. Assessment across seven key themes and identification of current and next stage for each. Benchmarking against the sector. | Everything in the individual plan. Customised sign up page with option to add own questions. Detailed branded individual reports for each user. Administrator dashboard and overall organisational report.Option for copy of data and bespoke detailed report. |
| Examples | Sample Basic Report. | Sample Individual Report. | Sample Organisation Report. |
| Cost | Free | Free | Paid |
Outcomes
The key changes we anticipate users will experience as a result of taking the assessment are:
Advancing knowledge
Improved understanding about important factors and questions in organisational data maturity.
Objective reflection of the stage the organisation is currently at.
Increased awareness of different perceptions about data across the organisation.
Identified clear priorities to focus on.
Shared understanding and language about data in the organisation.
Changing attitudes
Raised aspirations of what’s possible.
Increased motivation to improve data maturity.
Reassured they are on the right track.
Influencing behaviours
Invited colleagues to take the assessment.
Shared results report with colleagues.
Discussed with colleagues where improvement is needed.
Secured leadership support for change and improvement.
Sought external expert advice on data strategy.
Sought funding or resources to implement improvements.
Developed a data strategy/improvement plan.
Impacts
Following the implementation of the behaviours (actions) described above, our theory suggests organisations then go on to:
Secure funding or resources.
Invest in people (existing or new staff roles and responsibilities), tools, advice, and training.
Implement their data strategy/improvement plans.
And, ultimately, as a result of all this they advance their organisation’s data maturity.
Which means benefits and rewards like:
improved strategic planning and decision making
better services and/or products
greater impact
increased income
efficiency savings
increased credibility and influence
increased knowledge and expertise
increased collaboration and data sharing with strategic partners.
The more organisations we can encourage to make use of the Data Maturity Assessment tool, the greater the impact on the whole sector, and the beneficiaries organisations exist to serve.
Survey of users
In November 2020 we set out to evaluate the impact of our Data Maturity Assessment tool, and designed a survey to test our theory of change, gather feedback from users and listen to ideas on how we could improve.
The survey was sent out to 379 previous users who’d completed a Data Maturity Assessment between 4th October 2019 and 5th November 2020. (136 unique users of the free taster assessment, 232 unique users of the free individual assessment, 11 users of the customised organisation assessment.)
About the respondents
In total there were 73 responses to our survey, an overall response rate of approximately 20%. As expected, the response rate was lower for those who’d only completed a five minute taster assessment (11%) and higher for those who had used the customised whole organisation assessment (82%). This provides a useful indicator of how ‘engaged’ different users are on the subject of data maturity.
Type of organisation
The 73 respondents represented the following types of organisation:
56 from not-for-profit organisations (e.g. charity, social enterprise, coop, university, public sector organisation)
11 from an agency that provides support to not-for-profit organisations (e.g. consultant, data support provider, product/service company)
6 other (mostly researchers).
Findings on impact among not-for-profit organisations
Users
We asked people from not-for-profit organisations to indicate which of a selection of roles best described their role or interest in data maturity (notably linking these with our user groups in our theory of change).
The most common choice of role was ’Data Champion’ (32%) followed by ‘Leader in a not-for profit organisation’ (30%), followed by ‘Data strategist’ (16%) with 22% of people choosing other roles.
How users took the assessment
Since data means different things to different people and is used in so many ways in an organisation, getting multiple people to take the assessment is a key aspect of building engagement, learning and collective thinking about data.
We asked not-for-profit groups to indicate whether they took the assessment alone or with other people.
· 64% completed it on their own on behalf of their organisation
· 23% had multiple people in their organisation completing it individually
· 14% completed it as part of a group (i.e. several colleagues completed it together).
Why people took the assessment
We asked people what their motivations were for taking the assessment.
The most common reasons that people from not-for-profits took the Data Maturity Assessment were:
to learn more about data maturity (64%)
thought they needed to get better but didn’t know where to start (46%)
to test the suitability of the tool for others to complete (40%).
Other, reasonably common reasons were:
they were frustrated with poor tools or systems (31%)
they struggle to engage senior leads in driving improvements around data (29%)
data was not working for their cause (29%)
they sensed they were not doing so well compared to others in the sector (27%).
Other reasons people took the assessment included:
they couldn’t unlock the value of data for decision making (17%)
they thought they might be doing better than others and wanted to check (8%).
It is worth noting that the motivations for taking the assessment were different for different users.
For leaders the top reasons were:
thought we needed to get better but didn’t know where to start (77%)
to learn more about data maturity (62%)
sensed we weren’t doing so well compared to other organisations in our sector (31%)
frustrated with poor and/or outdated data systems and tools (31%)
to test suitability for colleagues to complete, to help people in the organisation think/talk about data (31%).
For data champions top reasons were:
to learn more about data maturity (64%)
frustrated with poor and/or outdated data systems and tools (50%)
to test suitability for colleagues to complete to help people in the organisation think/talk about data (50%).
For data strategists top reasons were:
learn more about data maturity (71%)
thought we needed to get better but didn’t know where to start (57%)
felt data was not ‘working’ for our organisation’s cause (57%).
Immediate benefits from taking the Data Maturity Assessment
The biggest benefits users reported from using the tool were in advancing knowledge. There were also some promising outcomes around changes in attitudes.
Advancing knowledge
Combining responses reporting moderate or extensive benefits:
85% said it improved their understanding about important factors and questions in organisational data maturity
79% said it provided an objective reflection of where their organisation was at
70% said it highlighted different perceptions about data across the organisation
68% said it identified clear priorities to focus on.
Changing attitudes
Combining responses reporting moderate or extensive benefits:
73% said it increased their motivation to improve their organisation’s data maturity
69% said it raised their aspirations about what’s possible in their organisation
56% said it reassured them their organisation was on the right track.
How users acted following assessment
Most respondents (92%) reported taking some kind of action following completion of the assessment.
63% discussed with colleagues where change and improvement was needed
61% shared their results with other people in the organisation
55% used the assessment results internally to guide plans for developing data strategy/maturity
35% secured leadership support for change and improvement
31% sought funding/resources from internal budgets (20%) or external sources (10%)
25% sought external advice/consultancy to help with data strategy/maturity.
Securing resources to implement improvements
We asked some follow up questions to those who had sought funding or resources from internal or external budgets. All 15 of these reported that they had been successful in securing this. Overall this represents about 27% of survey respondents from the not-for-profit sector.
Building data capabilities
We asked the respondents who said they were successful in securing resources how they spent these. Responses showed resources were predominantly invested in jobs, tools and skills development.
60% allocated new roles or responsibilities to existing staff
40% created new jobs
40% consultancy/external support
33% new tools
27% training.
Implementing data strategy/improvement plans
We asked everyone if they had implemented any data strategy or improvement plans since completing the assessment. Overall, 45% of the respondents from not-for profit organisations said yes.
When we analysed responses to this question based on which type of assessment users had completed, we found none of the respondents who’d completed the 5 minute assessment had gone on to implement any plans, however 54% of those who took the full assessment did so.
Rewards reaped from implementing data improvement plans
We presented the 24 respondents who had implemented improvement plans with a list of possible benefits and asked them to assess the degree to which they had achieved each of those benefits. Based on extensive or moderate responses:
The top three biggest benefits were:
increased knowledge and expertise (86%)
improved strategic planning and decision making (71%)
improved services and/or products (57%).
The next greatest benefits were:
increased collaboration and data sharing for strategic benefits with partners or stakeholders (50%)
strengthened partnerships or networks (50%)
increased credibility and influence (48%).
Other benefits included:
efficiency savings (37%)
improved impact (29%)
increased income (23%).
Findings on impact among support providers
About support providers
A key user group identified as separate to the not-for-profit sector are ‘support providers’. These include consultants, agencies, pro-bono services, and product/service companies that provide advice and support around data and data strategy to not-for-profit clients. We have a whole theory of change specifically dedicated to this user group.
9 of the 73 respondents in our impact survey responded in their capacity as support provider agencies
How they took the assessment
We asked agencies how they completed the assessments
56% completed for their own interest
44% on behalf of a client
11% together with a client
22% in another way.
Motivations for using the Data Maturity Assessment
The most popular reasons for people from agencies to take the assessment were
help clients work out a starting point (78%)
independent diagnosis of needs (78%)
engage senior leaders in improving (78%).
Followed by:
learn about data maturity (44%)
baseline for measuring my impact (33%)
demonstrating client’s data maturity in comparison with others (33%)
testing the tool’s suitability for clients (22%).
Benefits to support providers
The most significant benefits for people from agencies were:
improved understanding of factors and questions (100% said extensively or moderately)
improved motivation of clients to improve (100% said extensively or moderately)
raised good practice awareness (100% said extensively or moderately).
Followed by:
diagnosis of their clients’ needs (89% said extensively or moderately)
objective reflection of where their client was at (89% said extensively or moderately)
reassurance they were on right track with client (89% said extensively or moderately).
Actions taken by support providers’ clients
The actions taken by clients following the Data Maturity Assessment and the support provider’s advice were:
adopted new tools (71%)
created new roles or responsibilities allocated to existing staff (43%)
invested in training (43%)
secured consultancy or external support (43%)
created new jobs (29%).
User feedback
Usefulness
We asked not-for-profit users to tell us how useful they had found the Data Maturity Assessment.
Amongst respondents from not-for-profit organisations:
90% found it at least somewhat useful
63% found it very or extremely useful
10% found it not so useful.
Amongst the 15 support providers and other users 100% felt the Data Maturity Assessment was at least somewhat important with 93% saying they found it extremely or very useful.
Importance of benchmarking
A feature of our Data Maturity Assessment tool is the ability to compare your results to a benchmark for the sector as a whole, based on the results of all those that have completed the assessment.
We believe that understanding where your organisation is compared to the sector is a potential motivation to improve or keep improving.
We asked people to assess how important the benchmarking feature of the tool was.
92% of people felt it was at least somewhat important with 66% saying it was very or extremely important. The pattern was very similar for non-profits, support providers and other users.
Likelihood to recommend
We asked respondents from not-for-profit organisations how likely they would be to recommend the tool to other not-for-profit organisations.
86% would be at least somewhat likely to recommend
70% would be likely or very likely to recommend
no-one said they would be unlikely to recommend.
User Comments on benefits and why they’d recommend
We asked people who were likely or very likely to recommend the assessment tool to others to tell us how they had benefited and why they would recommend the tool.
Here is a selection of the comments we received (more comments appear in the Conclusions).
“It gives a great overview of where we are already doing well and where to focus more attention and resources. It was also thought provoking for how to continue to improve.”
“Really useful to do self-reflective practice around the way data is used in an organisation. Also the 7 themes support organisations to reflect on data in the whole rather than just how they collect/report on it.”
“The main benefit was to start the conversation on my organisation.”
“The Data Maturity Assessment tool has helped me to pinpoint where exactly my organisation needs more and better data and has prompted me to do something about it - gradually but steadily. It's been the first step in a process of finding out more.”
“It is straightforward to use and easy to interpret the results.”
“Quick, easy and insightful.”
“It has helped our organisation focus on what we need to do to improve to get a basic idea of what data maturity is and what can be worked on.”
“We have yet to fully implement all the recommendations and ideas around impact measurement, but we are getting there and I expect the value to be great. As we started from such a low data maturity base, it was important to understand where people were both in terms of where they thought they were in understanding data, and then where they discovered they were with a little more knowledge! Using the maturity tool has helped me to measure progress across the team over time.”
“Further assessments as we continue to implement changes would be hugely useful.”
“The Data Maturity Assessment tool was an extremely useful vehicle to hang internal conversations on, and to help us understand where we needed to develop our capabilities.”
“I have recommended the framework to many other not for profits, as the more we get doing this the more impactful it will become. We have put data at the heart of our charity and ensure that all initiatives and projects are data led and data informed. It is helping us to formulate an impact analysis that will greatly increase our efficacy and efficiency. Indeed, apart from our staff, data is our biggest asset.”
“Easy to use, produced systematic way of thinking about data within the organisation, gave us something to aim for - mastery and what that looks like.It was a useful tool for identifying different teams' perceptions of where they were and, as chief executive, to gain an organisation wide insight into where we needed to focus efforts.”
“It's a really clear-eyed look at all aspects of data within an organisation.”
“I found the Data Maturity Assessment very useful as a benchmark for my client in the nfp industry, and useful as a way of showing that the client was at a lower level of data maturity than previously thought to better focus the data strategy for them.”
Conclusion
We found that people’s reasons for using the tool confirm many of the problems identified in our theory of change. Notably a low level of knowledge and awareness about organisational data maturity and a sense they think they might need to get better but don’t know where to start.
It turns out not being able ‘to unlock the value of data for decision making’ is not identified by users as a big problem (only 17% thought so). Yet, for those going on to implement their plans, improved strategic planning and decision making is the biggest overall reward (95%).
It is striking that the most common reason for undertaking an assessment is to learn more about data maturity. It is reassuring that the questions and definitions provided in the assessment questions enable respondents to learn as they go. Having multiple people in the same organisation complete the assessment creates a shared language and understanding about the concepts around data maturity.
Nissa Ramsay, Consultant, Think Tech Social told us:
“It's the only tool dedicated to data maturity, which is desperately needed and supports digital maturity. It's evidence based, clear and practical.”
As an anonymous not-for-profit user said:
“It was also thought provoking for how to continue to improve. The main benefit was to start the conversation in my organisation.”
That so many went on to engage in dialogue with their colleagues and go on to develop data strategies and improvement plans is very encouraging. Moreover, the evidence of those investing in their data capabilities and implementing those plans is very satisfying. And of course the ultimate success in data maturity: increased knowledge and expertise, improved strategic planning and decision making, improved products and services, increased impact, and increased collaboration and data sharing for strategic benefits with partners/stakeholders.
Benchmarking is clearly important for users of the Data Maturity Assessment. It’s likely that the benchmarking is in itself setting the bar high, i.e. people with no interest in data maturity and at the lowest level are unlikely to engage with the tool. We suspect, though have no way of knowing, there are many organisations that are completely unaware of data maturity and completely disinterested in it. Feedback suggests the benchmarking is getting it about right when it comes to placing and comparing organisations. Note we plan to do further work on refining and tailoring the benchmarking function in the future.
As Stuart Beattie, CTO, Bookmark Reading Charity explained:
“Being able to understand where our charity is in relation to others, as well identify areas for improvement, has been invaluable.”
That said it seems as though the majority of our users had a reasonably accurate picture of how their organisation compared to the sector. It is possible that the benchmarking data is useful to communicate the need for change with colleagues.
An anonymous user from a not-for-profit said:
“It's easy to use, the results are well-presented and easy to digest and communicate, and the not-for-profit aspect of the tool makes the benchmarking really helpful and relevant.”
These findings seem to support our theory of change in terms of outcomes. We anticipated that the assessment report would be a catalyst for better conversations with colleagues and the basis for improvement planning and bidding for resources. This is very much what we found.
It does seem that for many users of our Data Maturity Assessment tool undertaking the assessment has led to real changes in the organisation. In many cases the Data Maturity Assessment has led to the implementation of action plans or data strategies.
Garry Abbott, Business Intelligence Analyst, Honeycomb Group explained:
“Being able to objectively assess our data maturity enables us to clearly identify areas for improvement and development which in turn create a data strategy action plan. This helps to break through the lingo into real, achievable actions rather than rhetoric.”
And an anonymous user from a not-for-profit organisation showed how the assessment can be used to measure progress
“We are using our data maturity scores as the key metric for our annual strategic plan related to research-evaluation-learning at our organisation. We took the assessment at the beginning of the year and have set up activities over the course of the year to drive improvements in our data maturity score; we will be retaking the assessment over the year to track whether we are improving.”
These in turn have delivered a wide range of tangible benefits.
In a smaller number of cases (27%) the Data Maturity Assessment has led to successful bids for additional resources which have delivered a range of outcomes.
While every organisation that derives value from our work is gratifying, we think we can make an impact on the sector as a whole. Previous users recommending the Data Maturity Assessment tool to others will be an important way of scaling our impact.
An anonymous user from a not-for-profit organisation recommends the tool as a way of reflecting and learning:
“The organisations I worked with really loved this tool as a place to begin thinking about what increased data maturity could look like at their organisation, and what that would make possible for how they do their work. I'd also recommend it as a tool to do to begin reflecting, and then upon learning more about data practices, returning to the assessment.”
Ultimately our theory of change relies on the intervention that we provide being a useful activity for the people taking the assessment. The fact that 90% of people from not-for-profit organisations found it useful is an important factor in validating our theory of change.
The fact that 10% of people from not-for-profit organisations found it not-so useful is troubling and we will need to undertake more work to understand how we can ensure that everyone who uses the tool finds it useful.
Next steps
This research was undertaken as part of our commitment to keep developing and improving the Data Maturity Assessment tool and framework.
As a result of this review we will:
Get going with the next phase of our online product development (and continue to engage partners and users in developing and testing).
Re-review how long it takes for users to complete the Data Maturity Assessment, make it shorter and quicker to complete (without losing integrity and quality).
Continue to capture user feedback in an ongoing way (and tweak our questions and theory of change here and there).
Capture stories and case studies of data maturity journeys from the not-for-profit sector.
Keep working at improving our methods for benchmark, validating and segmenting.
Promote the Data Maturity Assessment to the not-for-profit organisations that would benefit from it, and the support providers, networks and infrastructure organisations serving them.
If you’re a not for profit organisation or an agency that supports clients in this sector you can find out more about Data Orchard’s Data Maturity Assessment tool by visiting
Acknowledgements
We are grateful to the Digital Impact programme at the Stanford Center on Philanthropy and Civil Society, The Esmée Fairbairn Foundation and CAST (the Centre for the Acceleration of Social Technology) for their grant support towards the creation of the free version of the Data Maturity Assessment Tool.
Much appreciation also to our tech and communications partners and collaborators at Outlandish, Kindlemix and Thinkblink.