Collaborative Partner or Independent Assessor? Navigating our role in a Developmental-Outcomes Evaluation
Earlier this year, Dovetail was commissioned by the New Zealand Drug Foundation to evaluate a new educational resource designed to prevent gambling and gamblification harm for high school students. This resource is the latest in a suite of school-based resources designed and delivered by Tūturu to support rangatahi and wider school communities in exploring health issues in educational settings.
Tūturu required an evaluation that could both support real-time testing and refinement of the resource and provide independent judgements on how well it is positioned to achieve its intended outcomes. In other words, Tūturu needed a developmental and outcomes evaluation at the same time. Balancing the relational partnership focus of a developmental evaluation with the independence and rigour of an outcomes evaluation raised an important question: can an evaluation really be both developmental and outcomes-based?
In this post, we share what we learned about walking the line between two evaluation methodologies.
To bring developmental elements into our recent outcomes evaluation…
We built reflection and learning into each stage of evaluation.
We took a structured approach to learning by building multiple rounds of feedback into the evaluation process. Throughout our data collection, we spoke to academic and addiction experts, teachers trialling the resource, gambling harm sector organisations, and the Tūturu design and delivery teams. Analysing and presenting real-time insights from each group throughout the evaluation period gave Tūturu opportunities to respond to feedback and strengthen both the content of the resource and their planned approach to delivering it in schools.
We focused on utility and intended outcomes.
At every step, our focus was on intended use by intended users. We led with evaluative questions that asked for outcomes-focused data in a developmental way; instead of asking, “Did the programme work? How well, and for whom?’, we asked, “What in this programme is well-positioned to work, why, and what needs to change to achieve and sustain results in the future?’.
We remained adaptive and flexible to an emerging field of work.
Youth gambling is a new and rapidly evolving field of research in New Zealand, bringing high levels of uncertainty and complexity that strongly suit a developmental approach. Unlike traditional outcomes evaluations, this programme of work was not stable, established or complete, and was designed based on an overarching vision as opposed to pre-defined, measurable outcomes. A thorough review of the safety and relevance of the content was needed to ensure the emerging approach was engaging, strengths-based, and safe for rangatahi. We remained flexible by adapting our lines of questioning and changing directions as conversations progressed. We weren’t afraid to go back to the drawing board, at times adapting our evaluative criteria in response to new findings to ensure our assessments remained grounded in the practical realities of the current youth gambling space.
We made it clear where developmental support ended and evaluative assessment began.
Transparency was core to a successful approach, with clear expectation setting needed on our independence as evaluators and where the boundary lay between collaboration and assessment. Throughout the process, we made it clear where the resource sat against intended outcomes compared to where it could sit with developments made. We used quotes and testimonials to directly link suggested improvements back to the voices of our interviewees, which helped to instill trust and transparency in our process.
We upheld evaluative rigour.
We developed an evaluation rubric and were collecting evidence to make judgments against pre-defined criteria. We sought guidance on best practice in youth gambling harm prevention, questioned assumptions underpinning content and delivery approaches, and validated our insights by triangulating interview and survey findings with published literature. By including members of the Tūturu team in the formal interview process, we were able to document their insights and decision-making with the same objectivity applied to other stakeholder groups, which provided reassurance that their views would be captured while ensuring consistency and rigour in our data collection process.
We kept our recommendations strengths-based and actionable.
Developmental evaluation asks organisations to be comfortable with adapting, changing direction, and learning by doing. At first glance, organisations who both commission evaluations and deliver the programmes being evaluated appear best placed for this type of approach because they have the ability to change and develop. However, development also brings uncertainty, and the scrutiny and accountability experienced through outcomes evaluations places pressure on those sitting in the spotlight. The trust we built with the Tūturu team was central to ensuring our insights would be well-received and acted on, and we were intentional about keeping our findings strengths-based, practical, and supportive of the underlying values present in their work so far.
“With the input of Dovetail, we were able to effectively utilise learning gained through the process and implement those during the trials which improved the overall design and effectiveness of the programme. We found it an extremely useful and effective process which added value to the work we are undertaking in Tūturu.” – Jude Woolston, Tūturu
This type of multi-method approach gives organisations the chance to learn in real time, while still building the evidence needed to understand whether their work is well positioned to make a difference. Deliberately drawing on different approaches, and doing so with intention, care, and transparency, can allow us to remain grounded in evidence, while supporting organisations to develop, improve, and extend their impact for their communities.
To read our full report on the Tūturu website, click here
This work was commissioned by the New Zealand Drug Foundation.