aBOUT OUR dATA mATURITY aSSESSMENT tOOL

 
 
 
 

What’s the purpose of the data maturity assessment tool?

For you, the users

The assessment tool is a diagnostic tool to help organisations understand how well they are doing with data. Each user completes a 20 minute online questionnaire and the responses are combined to generate a report on your organisation's data maturity.

The 5 minute taster will give a quick ‘rough’ indication. The full version will indicate strengths and weaknesses across the seven themes, enable comparison with a benchmark, and identify the next stages to work towards.

We hope it will stimulate thinking and dialogue about data and enable organisations to develop plans and priorities for the future. There are no right or wrong answers, it’s your assessment and it’s completely confidential, so it’s best to be as honest as possible.

For us and our not-for-profit sector

We will ask a few questions about you, your role/s and your organisation. These will not be shared with anyone but will enable us to monitor how representative our respondents are of the sectors they are from.

Anonymised aggregated data from respondents will be analysed for research purposes to help build understanding about data maturity in the not-for-profit sector. We will use this to inform funders, policy makers, and other Data4Good partners about the needs of the sector and design appropriate support services.

Who should take the assessment?

In our experience ‘data’ means different things to different people and nobody knows everything there is to know about data in an organisation. Typically the answers to the assessment questions are in many people’s heads and you/they may not all necessarily agree.

To get the best out of the tool, multiple people should be involved in the assessment. The more participants from your organisation involved, the more accurate the results will be. Many choose to involve all their staff to ensure the full range of knowledge and experience participate in the process (and share in the thinking about future improvements). Others select a diverse and representative range i.e. mixing levels of seniority, frontline, leadership and different teams/departments and project staff.

At the very least we recommend you include someone in a leadership position and the person in the most data-related role. Others may come from service delivery/planning, research and evaluation, digital technology, marketing and communications, HR, ICT, Fundraising/sales.

If you’re completing the assessment as an individual we recommend you answer questions to the best of your knowledge based on your perception of the overall organisation, rather than you as an individual or views of a particular project or department.

Three ways for organisations to take the assessment

  • Taster: The quick five minute taster offers a chance to ‘dip a toe’ and see what kind of questions are asked in the assessment. It’s something easy to share with busy leaders and colleagues before going ahead with a full assessment. (FREE)

  • Individually or as a small group: Ideal for very small organisations. EITHER get a group of staff together and complete the online self-assessment as a group. Up to six people can work well for managing discussion and reaching consensus in a timely way. OR individuals take the self-assessment on their own and then get together with colleagues to share and discuss their individual reports. (FREE)

  • Whole organisation: This approach is suitable for any organisation wanting to take a serious review of data and involve their staff in the process. A unique customised link to the organisation’s assessment is sent out via email for unlimited numbers of staff to complete online. Each staff member completes the assessment individually and gets a personal report and all the staff assessments are combined to create a single ‘ whole organisation’ data maturity report. There’s the option to brand with your own logo and add up to three extra questions tailored to your organisation’s context e.g. larger and geographically dispersed organisations might want to get a profile of data maturity by department or operating area. You can set your own time-frame for collecting responses and monitor numbers of staff participating via the online report dashboard. (from £4,999 +VAT with discounts for nonprofits with a turnover of less than £10M)

There are also options for additional support e.g. independent facilitation of the process via online workshops and more in-depth analysis, internal or tailored benchmarking and reporting of findings. Contact Data Orchard for a no obligation chat about your requirements.

How does the scoring and benchmarking work?

Essentially the measurement in our data maturity tool works by: scoring questions; weighting questions; grouping the weighted scored questions by theme and taking an average of these to giving an overall theme score; and averaging the seven theme scores to give a data maturity stage (based on a scale of 0 to 5 for Unaware through to Mastering).

All questions are normalised so that before weighting they contribute equally to the relevant theme and themes are normalised so that they contribute equally to the overall score for an individual organisation.

Scoring in more detail

Each question has scoring assigned to it in the range 0 to 5. Most are on an ‘extent’ or ‘agreement’ range though some have bespoke scoring. ‘Don’t know’ and ‘not applicable’ responses are not included in the scoring. For some questions, where multiple responses are possible (tick all that apply), the total score is normalised by 5/maximum possible scores if all options are selected so that the score is in the range 0 to 5.

Where we arrange for multiple users of an organisation to take the assessment so we can get a more detailed overview, we sum the scores and divide by the number of responses (excluding don’t know/not applicable) to give an average score for that question.

Weighting in more detail

Each question is assigned a weight: standard, high, or very high based on how important responses will be in determining important features or behaviours of data maturity. In testing we have monitored both weighted and unweighted scores for themes.

Benchmarking in more detail

Benchmarking has been used as a key feature for enabling comparison between one organisation and a number of others. For benchmarking we use the same approach for the scoring, weighting and grouping of questions by theme. However we select ‘all respondents’ rather than one organisation’s response. To date, in testing, we have been able to verify participating organisations are authentic, and validate the tool’s assessment against our own observations.

Who built the tool?

We have a long list of people and organisations to acknowledge and thank for bringing this data maturity self assessment tool into being.

The Data Orchard Team

Sian Basker, Co CEO at Data Orchard has been the driving force and lead researcher developing it since way back in 2015. Madeleine Spinks (the other Co CEO), Tony Cramp (Senior Analyst), Ben Proctor (Innovation Director); and Paul Ticher (Associate Data Protection Specialist) have been the other key Data Orchard people involved in its development.

Partners and specialist advisors

Partners: DataKind UK (notably Emma Prest) and the Project Board that supported the first pilot phase of the research in 2016 (Duncan Ross - DataKind UK/TES Global; Alexandra Rehak- DataKind UK/Ovum; Madeleine Thornton - Big Issue Invest; Gaia Marcus - Centrepoint; Ed Anderton - Access Social Investment). Tracey Gyateng, Data Science Manager, and their data science volunteers also tested the beta-phase assessment questions and provided feedback in 2019.

Online Data Maturity Assessment Tool Steering Group that supported the second phase of the research and development: Caitlin Milne and Anne Tregoning (Kindlemix Communications); Suraj Vagdama (CAST), Giselle Cory (DataKind UK), Ben Proctor (Satori Lab/ODI Cardiff).

Specialist advisors: We were enormously grateful to draw on the expertise of seven specialist advisors with experience of working with not-for-profits in building their data and digital capabilities. They kindly reviewed the beta stage assessment questions and advised on improvements to the first data maturity framework. These were: Megan Griffith Gray (NCVO); Ellie Harries (Freelancer); Tracey Gyateng (DataKind UK); Josh Cockcroft (Esmee Fairbairn Foundation); Ben Proctor (ODI Cardiff/Satori Lab); Kate White (Superhighways); and Alison Prince (Parkinson’s UK).

Special thanks to: Ellie Harries whose expertise has driven the development of some amazing data tools (Inspiring Impact; Audience Finder); Elizabeth Larkin (Epidemiologist/Data Scientist at Digital C in Cleveland US) who took our pilot Data Evolution assessment questions, adapted them, and surveyed hundreds of non-profits in the US, and kindly fed back their learning and experience to inform our next phase of development; Gaia Marcus, Alison Prince, and Jesse Mears from Parkinson’s UK who shared their experience of using the original data maturity framework and their experience of assessing and re-assessing data maturity.

User testing participants

User Testing #1 (May 2018 – July 2019): The following charities submitted themselves (in some cases ALL their staff and trustees) to the task of completing the beta phase online assessment and benchmarking (in some cases they did it twice – 12 months apart): The Cart Shed, Bracken Trust, The Key, Joseph Rowntree Foundation. In total the beta version tool was tested by 51 people from 4 organisations and we are enormously grateful to them for their feedback.

User Testing #2 (March 2019): Six not-for-profit organisations kindly took part in our user testing of different data/digital maturity tools. They were: Thomas Pocklington Trust; Founders and Coders CIC; FareShare; Parkinson’s UK; Jupiter Ltd; Into University.

User Testing #3 (July 2019): Five not-for-profit organisations took part in our third round of user testing of our online tool. They were: St John’s Ambulance; The British Science Association; Imperial Health Charity; Charity Finance Group; and ACDI/VOCA.

The web developers

Following a formal tendering process we were delighted to appoint Outlandish, a leading tech co-operative, to build the online tool. Their technical expertise both in terms of web development, user design, and in data was crucial to the project’s success. We also really liked that they were themselves part of the not-for-profit family!

Our funders

We are extremely grateful to our funding partners for their grant support towards the creation of this data maturity assessment tool. They are: