Evaluation of the Implementation and Impact of Diplomas

1 January 2008 - 31 July 2011

Awarded to: Dr  Gill Haynes

Co-investigators: Professor William Richardson

Research partners: (In partnership with the National Foundation for Educational Research)

Funding awarded to Exeter £ 346,692

Sponsor(s): Department for Children, Schools and Families. (DCSF)

About the project

Introduction

The introduction of Diplomas for 14-19 year olds, at three levels and across 14 sectoral ‘lines’ of learning, represents a major innovation in educational opportunity for young people in England.

The Diploma consists of:

  • ‘Principal Learning’ (sector-related knowledge and underpinning skills needed to progress in relevant sectors) 
  • ‘Generic Learning’ (functional skills in English, mathematics and information and communication technology (ICT), a project, and development of personal, learning and thinking skills)
  • ‘Additional/Specialist Learning' (a selection of options can be chosen from a range of qualifications).

Diplomas also include learning in the workplace (a minimum of ten days’ work experience) and learning through realistic work environments to enable the development of practical skills and work-related application of learning.

Diplomas are available pre- and post-16 at three levels: Foundation (Level 1 – with equivalence to 5 GCSEs D-G), Higher (Level 2 – with equivalence to 7 GCSEs A*-C grades); and Advanced (Level 3 – with equivalence to 3.5 A levels). Students wishing to study a Diploma at Level 3 can also choose the Progression Diploma, equivalent to 2.5 A levels.

 Around 12000 students started Diplomas in September 2008 in the first five ‘lines’ of learning: Construction and the Built Environment; Engineering; Information Technology; Creative and Media; and Society, Health and Development. From September 2009, a further five lines of learning are being introduced in Business, Administration and Finance; Environmental and Land-based Studies; Hair and Beauty Studies; Hospitality; and Manufacturing and Product Design. Four more will become available from September 2010 in Public Services; Retail Business; Sport and Active Leisure; Travel and Tourism with, in 2011, a further three in Humanities and Social Sciences; Languages and International Communication; and Science.

In January 2008, the DCSF commissioned the National Foundation for Educational Research (NFER) and the University of Exeter to conduct the national evaluation of the implementation and impact of Diplomas over the period 2008-2013.

Research Questions

The purpose of the national evaluation is to provide policy makers and practitioners with systematic and robust evidence which will enable them to make informed judgements about the outcomes of the Diplomas for different stakeholders and to make improvements to design and delivery, if appropriate. The evaluation has two main aims:

  • to review the implementation and delivery of the Diplomas 
  • to assess the impact of the Diplomas on young people’s participation, attainment and progression.

Impact & Outcomes

The evaluation will inform the implementation of the Diplomas across the three phases and beyond. It will gather evidence to help the DCSF and its partners understand the impact of Diplomas on young people, educators, employers and higher education institutions (HEIs). The evaluation will offer numerous dissemination opportunities such as briefings for policy-makers and reports/articles for practitioners. The evaluation team will produce reports during each year of the evaluation.

Research Design & Methods

The research design provides a complementary mixed-method approach to address the complex range of issues and aims, and has three main strands: surveys of stakeholders, in-depth case-study visits and analysis of external datasets. Further details about each strand are presented below.

1. Surveys

In each phase of implementation, surveys of a range of stakeholders will be undertaken in a sample of 30 consortia.

Surveys of learners

Prior to each phase of implementation, ‘take-up’ surveys will be completed by learners in year 9 and year 11 in the selected consortia in order to explore reasons for take-up and non-take up of Diplomas and whether this changes over time (spring 2008 and 2010).

Longitudinal questionnaire surveys will also be carried out with all students who have chosen to pursue a Diploma (starting when they are in either year 10 or year 12 or equivalent) in the sampled consortia in each phase of implementation (spring 2009-2013). In addition, longitudinal surveys of a comparison group of students in the same institutions will be undertaken. These surveys will explore learners’ experiences of their Diploma journey and the impact of Diplomas on their choices and future destinations.

Surveys of teaching staff (spring 2010, 2011, 2012)

The research study will include cross-sectional questionnaire surveys of teachers and tutors employed by schools, colleges and training providers, who are engaged in delivering the Diplomas in each phase. The sample will comprise up to ten teachers for each school, and up to five tutors for each college and training provider in the sampled consortia. The surveys will explore their experiences of delivering Diplomas and the factors they think facilitate successful implementation.

Survey of employers (autumn 2010 and 2012)

A baseline telephone survey of 140 employers not actively involved in Diploma delivery will be undertaken in autumn 2010 in order to explore their levels of awareness of Diplomas and the extent to which they are planning to include Diplomas as part of their recruitment policy. A follow-up survey of the same individuals will be carried out in autumn 2012, to investigate any change in perceptions over time.

Surveys of parents (summer 2009, 2010, 2011)

Parents’ personal characteristics, their experiences, aspirations, and expectations are likely to have an impact on their child’s life choices and life chances. Therefore, in order to understand the choices young people have made in relation to Diplomas, a telephone survey will be carried out with parents of Diploma and non-Diploma students in the sample consortia at each phase.

Survey of HEIs

In order to explore how Diploma implementation is meeting its goal of securing progression to higher education (HE) at all levels of achievement, the study will survey senior managers and admissions tutors in HEIs at two time points: prior to the first cohort achieving level 3 awards (2008/09 and 2009/10) and after the first awards are made and Diploma applicants have progressed to higher education (2010/11 to 2012/13). These surveys will explore institutions’ admissions policies in relation to Diploma applicants, the size of anticipated Diploma intake, views on national progress of Diplomas and their experiences of having admitted Diploma-holders to undergraduate programmes.

2. Case Studies

In order to explore in detail the implementation of the Diplomas, a longitudinal programme of qualitative case studies will be undertaken in 15 consortia in each of the three phases of implementation (45 in total). Within each of the consortia selected, visits to four institutions will take place – to two schools and two colleges or training providers – giving a total of 60 institutions at each of the three phases. Interviews will be conducted with the strategic and operational staff of schools, colleges and training providers, learners and partners (employers and HEIs).

Four stages of research activity will be replicated with each of the 15 new consortia included at each phase. The four stages will be as follows.

  • baseline visit prior to the introduction of each phase of implementation (spring 2008, 2009 and 2010): this will involve interviews with key stakeholders and young people. The focus will be on how prepared consortia are prior to implementation, and on the exploration of the influences on young people’s choice to take up a Diploma or not.

  • Year 1 case-study visit: this will take place in the first year of each phase of implementation (January to March 2009, 2010 and 2011) to explore early implementation issues and initial experiences and perceptions of the Diplomas amongst key stakeholders and Diploma students.

  • Year 2 case-study follow up visit (January to March 2010, 2011 and 2012): will explore the lessons learned by key stakeholders, changes in perceptions about Diplomas, and the perceived impact on young people.

  • Follow-up telephone survey of young people post-16 (autumn 2010, 2011 and 12): will track a sample of young people post-16 (that is, those who started a Diploma in year 10 in September 2008) to explore the Diploma journey and the impact the Diploma has had on their choice of post-16 destinations.

3. Statistical Analysis of External Datasets

In order to explore the outcomes and impact of the Diplomas on a larger scale than would be possible through surveys or qualitative data collection, datasets such as the Diploma Aggregation Service, the National Pupil Dataset, the Individual Learner Record, Higher Education Statistics Agency data, the Ofsted database, UCAS data and the National Employer Skills Survey will be analysed.

Audience(s)

The main audience will be policy makers at DCSF, Qualifications and Curriculum Authority, the Sector Skills Development Agency, the General Teaching Council and the Local Government Association. The study will also be of interest to a wide variety of audiences such pre- and post-16 practitioners, employers, HEIs, information, advice and guidance providers, and parents.

Recent Publications

  • The first Annual Report is available at:

http://publications.dcsf.gov.uk/eOrderingDownload/DCSF-RW079.pdf

  • The report of the first survey of Higher Education Institutions is available at:

Full Report

http://www.dcsf.gov.uk/research/data/uploadfiles/DCSF-RR145.pdf

 Summary

http://www.dcsf.gov.uk/research/data/uploadfiles/DCSF-RB145.pdf

  • Gill Haynes (University of Exeter) and Sarah Lynch (NFER) will be presenting a paper at the European Educational Research Association’s Annual Conference in Vienna on 28 September 2009 entitled: Delivering curriculum change through partnership: Key lessons from the implementation of the new sector-related Diplomas for 14-19 year olds in England.