Category: Evaluation & Consultancy

Virtual Reality Prosthetics Evaluation

Posted on by Liz Lister

Client: Sheffield Hallam University

Start date: January 2016

end date: November 2017

Background:

Virtual Reality Prosthetics, Body and Mind was an exhibition co-developed by a multi-disciplinary group from Sheffield Hallam University (SHU) and young people from Sheffield UTC working with local primary schools. The exhibition had its origins in a multidisciplinary project that used virtual reality (VR) as a more comfortable way for people with robotic upper limb prosthetics to learn to use their prosthetic arms.

The main aims of the exhibition were to:

  • Engage with audiences who would not normally go to a science exhibition
  • Create exhibits that will be inclusive and accessible to all
  • Increase understanding of:
    • Joint physiology and neural connections
    • Prosthetics
    • Lived experience of limb loss
    • Ethical issues associated with limb loss

Our Approach

We used a mixed methods approach to explore the effectiveness of the exhibition in achieving its aims. As well as using observation and questionnaires, we developed a range of interactive evaluative tools to ensure participation in the evaluation was as accessible as possible and to enable wider data collection in a manner that was satisfying for the participant.

Outputs

Outputs included formal reports containing both formative and summative insights, as well as creation of both traditional and tactile evaluation tools.

Great Exhibition Road Festival 2019 Evaluation

Posted on by Liz Lister

Client: Imperial College London

Start date: February 2019

end date: September 2019

Background:

The Great Exhibition Road Festival (GERF) 2019 takes Imperial College’s annual festival and partners with 21 leading institutions, museums, and organisations in the area of South Kensington known as Albertopolis. The festival invokes the spirit of the first Great Exhibition in 1851 and fuses art, science, technology and curiosity.

GERF is an extension of Imperial College’s annual festival and takes place along the length of Exhibition Road, in addition to the Imperial campus and partner hosts such as the museums and the Royal Albert Hall. The involvement of the partner institutions has broadened the festival scope to encompass more arts and culture alongside STEM focussed activity.

Our Approach

Our evaluation of the first Great Exhibition Road Festival aimed to understand the value of extending the offer beyond Imperial alone. It explored the festival’s audience, their needs, and the festival’s impacts upon them, as well as the views of volunteers and participants.

We employed a mixed methods approach to explore different elements of the festival. This included both paper and online surveys, interviews, and the creation of tactile interactive evaluation tools.

outputs

A final report including recommendations for the development of GERF, discussed in a round up meeting with the festival team, as well as development of a range of evaluative tools for use at future festivals.

Current project – Great Exhibition Road Festival 2020 Evaluation

Posted on by Liz Lister

Client: Imperial College London

Start date: February 2020

Expected end date: September 2020

Background:

Following a successful pilot year, the Great Exhibition Road Festival (GERF) is now an annual event celebrating the arts and sciences happening in the South Kensington cultural quarter. The festival aims to deliver a dynamic range of participatory visitor experiences that generate curiosity and a pioneering spirit among the audiences and partner institutions through a unique collaboration of the arts and sciences institutions taking part.

The festival also aims to celebrate diversity and engage with communities and audiences with whom the partner institutions do not often engage. The 2020 festival has a theme of ‘trailblazers’, which will unite a selection of content across the festival.

Our Approach

Our evaluation seeks to understand the value of a collaborative endeavour, combining disciplines and partner institutions. It explores the festival’s audience and any impacts upon them, as well as the views of volunteers and participants.

We are employing a mixed methods approach to explore different elements of the festival and its development from the pilot year. We are using a range of evaluative tools, including both traditional and development of new creative approaches.

Expected outputs

A final report including recommendations for the development of GERF, discussed in a round up meeting with the festival team, as well as development of a range of evaluative tools.

Current project – Inspiring Science Fund Evaluation

Posted on by Liz Lister

CLIENTS: Wellcome and UK Research & Innovation

Start Date: June 2017

Expected end date: 2024

Background

The Inspiring Science Fund is a £30M competitive grant scheme for science centres co-funded by Wellcome and UKRI. The programme objectives are to:

  • Create more opportunities for young people and the public to learn and engage with science, technology, engineering and maths (STEM) in ways which are innovative, and which reach and meet their needs
  • Sustain and extend the audience base, with an increased focus on underrepresented and underserved audiences, and increase the overall number visiting or taking part in STEM activities
  • Have the potential to improve funded organisations’ financial stability and resilience

Graphic Science are the external evaluators for this funding programme.

Our Approach

We are employing a mixed methods approach to understand the short and longer-term impacts of the funding programme across all funded projects as well as identifying learning that can be used by the sector as a whole.

This is a complex, multi-faceted evaluation with a large qualitative component as well as quantitative elements. Qualitative elements include narrative reporting and interviews with funded centres as well as analysis of strategic, policy, and communications documents. Quantitative elements include cohort-wide visitor exit and staff surveys, developed in consultation with funded organisations. These surveys are the result of extensive research looking at survey instruments from a wide range of sources including the Office for National Statistics, COVES, Public Attitudes to Science, Stonewall, the Natural History Museum, and incorporating indicators for science capital.

Advisory Board

For this project we have convened an evaluation advisory group to advise on our methods and plans for the evaluation of the impact of the Inspiring Science Fund. By contributing broad knowledge of the current evidence landscape and providing constructive challenge to evidence and findings, the group gives independent support and oversight that will contribute in due course to sector development aspirations.

The evaluation advisory group members are:

Colin Johnson (Chair)

Dele Adeyemo

Lenna Cumberbatch

Emily Dawson

Jen DeWitt

Marie Hobson

Jane Steele

Ben Ward

Expected outputs

Outputs include a Theory of Change for the programme, evaluation framework, written reports, conference sessions and presentations, surveys, and creation of a self-assessment tool for financial resilience.

Current project – Curiosity Evaluation

Posted on by Liz Lister

Clients: BBC Children in Need and Wellcome

Start date: July 2019

Expected end date: October 2022

Background

The Curiosity funding scheme is a joint programme developed by Wellcome and BBC Children in Need. It aims to enhance the opportunities for disadvantaged children and young people to engage with and enjoy informal science activities to improve their personal and social development. The programme funds projects at the intersection of the youth work and informal science learning sectors.

The main emphasis of the programme and its evaluation is identifying whether there is a ‘distinctiveness of science’ that contributes or not to improving children’s lives, and how this is influenced by informal science activities being enacted in a youth work setting.

Our Approach

Working in partnership with research company Substance we are evaluating the Curiosity funding programme, employing a mixed methods approach to understand differences that informal science activities can make to the lives of children and young people, and what aspects of project delivery enable these changes.

We are particularly interested in whether delivering informal science activities in a youth work setting can contribute to particular impacts on the lives of disadvantaged children and young people, and whether there is a distinctive contribution from informal science in achieving youth development outcomes.

We are working closely with the funded projects and their participants to ensure the presence of the youth voice, as well as those at the front line of youth work and informal science learning.

Expected outputs

Outputs include a Theory of Change model, impact framework, creation of evaluation tools, literature review, formal reporting, and case studies.

Current project – Inspiring Science for All

Posted on by Liz Lister

Client: Winchester Science Centre

Start date: September 2019

Expected end date: February 2022

Background:

Winchester Science Centre (WSC) is embarking upon a cultural and physical transformation, rethinking traditional exhibition design and how science, technology, engineering and maths (STEM) activities are delivered. The aims of the Inspiring Science for All programme are to:

  • Transform the visitor experience through a rethink about how the exhibition will support all audiences to be curious and discover STEM
  • Widen participation in STEM through understanding and overcoming barriers to participation and creating exciting opportunities to get involved
  • Become more financially sustainable through minimising financial risk, creating new income streams and generating a reliable surplus for re-investment
  • Change the culture within the organisation and sector through championing science for all, challenging stereotypes and being a benchmark organisation for inclusivity

Our Approach

We are using a collaborative, responsive approach that adapts to learning that emerges as the project develops. We are working closely will the WSC team, including support around data collection and interpretation, to ensure the team has ownership of an evaluation process they can adopt beyond our involvement.

We are supporting the team to review their strategic priorities and have co-developed a Theory of Change model for the programme. We are developing evaluative tools that are inclusive and accessible for all visitors and participants, in partnership with Sarah Bearchell, a specialist in STEM engagement with SEND audiences.

Expected outputs

Outputs will include a Theory of Change model, creation of evaluative tools, and written reports.

Evaluation workshops for Wellcome Trust centre

Posted on by adrian@webdesigncity.co.uk

Evaluation workshops for Wellcome Trust centre Public Engagement staff

Graphic Science was asked by the Wellcome Trust to run practically-focussed evaluation workshops for the public engagement leads at each of the centres that receive strategic Wellcome Trust funding.

We consulted with staff from each of the centres to find out what would be most useful to them and then developed a highly interactive workshop that took them step by step through the process of planning, designing and analysing the data from an evaluation.

As part of this, we threaded through a real example of a project we had evaluated and encouraged participants to think about how they would approach a similar task and critique our approach.

“I found it very useful and the orientation – towards practical advice and hands-on exercises was spot-on.”

“Personally, I liked the relaxed delivery style of the trainers – they were happy to let the session wander off the planned agenda into group discussions, which were equally useful and helped the group start to form constructive relationships between different centres – contacts that I’m sure will be mutually helpful in years to come.”

Engineering Everywhere Evaluation

Posted on by adrian@webdesigncity.co.uk

An EPSRC funded project in collaboration with the Centre for Science Education at Sheffield Hallam University; the University of Glasgow and 4Science.

Engineering Everywhere consists of a suite of resources, including classroom activities and videos on a DVD, to support the teaching of physical science with reference to engineering, in an effort to enhance young people’s interest in engineering and science related careers.

Preconceived perceptions and poor understanding of engineers and engineering have led to a poor uptake of engineering courses by school leavers. The DVD introduces relevant engineering ideas and concepts to form an integral part of the assessed science curriculum.

We performed a rigorous and fully-embedded formative evaluation of ‘Engineering Everywhere’ to inform development of the resource. We ran a series of iterative focus group sessions over the course of two years with our ‘standing panels’ of year 10-12 students. We probed the students for their values, opinions and regular activities, as well as testing resources in various stages of development. Our feedback was integrated into the development of the project from the outset.

We later performed a summative evaluation of the project, using a combination of timed, structured observations; informal interviews; follow up teacher interviews and questionnaires.

TEMI Evaluation

Posted on by adrian@webdesigncity.co.uk

TEMI – Teaching Enquiry with Mysteries Incorporated – is a four-year FP7 funded project involving 13 partners around Europe. The project is a CPD programme for teachers with accompanying resources designed to support enquiry based science learning through investigating mysteries.

Graphic Science is acting as the external evaluator for the project. Our role is both formative – monitoring the project as a whole, identifying risks and recommending mitigations – and summative – our final report looking at the successes and failures of the project as a whole will be submitted to the EC when the project is completed in July 2016.

You can follow the project’s progress through their website: teachingmysteries.eu/en or from @teachmysteries on Twitter

WT PE Leadership Programme

Posted on by adrian@webdesigncity.co.uk

In 2014, Graphic Science was commissioned by the Wellcome Trust to run a 10 month long leadership programme in public engagement for heads of academic departments working across the Trust’s research remit.

Each participant has a small seed fund of £2,000 to spend on activities that embed and support a culture of public engagement in their department/institute.

We recruited 14 heads to take part.

The programme began with a one-day launch workshop at Brunel’s ssGreat Britain where participants heard a range of different perspectives from other academic heads who described their experiences of doing and supporting public engagement. Participants also spent some time thinking about their departments’ own public engagement and how they could develop this and make it more strategic.

We were also pleased to welcome special guest speaker Eliza Manningham-Buller, former Director General of MI5, who shared her views on and approaches to leadership.

The programme then continued with remote support and visits to each participant to discover more about their and their colleagues opportunities and aspirations for public engagement.

Approaches have included developing a public engagement strategy, creating an internal public engagement small grants scheme and running annual community-focussed public engagement events.

The programme will end with a finale event during Cheltenham Science Festival.

>