Monday 27 August 2012

Kirsty Wier talks with us about using the Ako Aotearoa Impact Evaluation Framework


Mike and I met with Kirsty, Research Manager for Ako Aotearo, last Wednesday to discuss her experiences using their Impact Evaluation Framework. Kirsty had just come from spending several days at OLT in Sydney where she was working with Tilly and others to refine and develop the framework for the Australian Higher Education context.

Since our meeting with Tilly earlier in the week, this meeting provided a great chance to better understand how the Framework is currently used in practice and what are some key things to consider if we do choose to adopt it or elements of it in our evaluation process.

Some key characteristics/outcomes of the use of the Framework:
  • Provides legitimacy for project teams to actively influence others
  • Keeps the project focus on the learner
  • Provides opportunities for structured reflections (impact journal)
  • Enables funding body to see where intervention is needed or would be of value
  • Data collected can inform future funding models and requirements through developing better understandings of what constitutes a quality project
  • Expands the scope of relationships between funding agency and project teams
  • Using the framework is time consuming and it can be difficult where the project has had little impact which has not been directly related to the quality of the project

An important part of our discussion centred on answering the following questions about the impact on learning and teaching outlined in the Evaluation Framework. This was due to the Network’s distance or large degree of separation from the experience of learners and consequently, about our concern in addressing such questions.



Kirsty explained that whilst there may be quite a distance between project activities and the learning experience in many projects, such at NATA, it is important to attempt to answer this question to the best of the project's ability. For, in addressing this question it forces the project to really clarify how it fits within the learning and teaching sphere where the learner experience as being critical. This could be done through mapping the different steps by which the project's activities hope to influence the learner experience or if feasible, reflecting with teachers about changes in their teaching or reviewing student surveys for example. However, Kirsty did make it clear that it is important to keep true to the project and not lose sight of the nature and objectives of the project itself. 

In reflecting upon this question we felt that whilst quite abstracted from the learner experience, we could map how we felt the project aims to influence change in learning and teaching . Moreover, we felt we could ask association members in the research survey how they felt their involvement in professional associations may have influenced their teaching. How what we are doing seeks to impact learner outcomes could also be a key question for the executive focus groups and leader interviews. Going through these processes would also help us keep the learner in mind whilst keeping true to our project.

Ako Aotearoa currently uses the framework to assess projects as they progress 6 months, one year and two years post completion. However, they are now looking at trying to embed this framework in upcoming projects so that it helps project teams identify what impact would look like and develop strategies to achieve that at earlier stages in the project life-cycle. Thus, by adopting such an approach ourselves, we may be able to provide real experiences and feedback that aids in the design of such an approach.

No comments:

Post a Comment