Evaluation and Assessment

It is unfortunately all too common for considerations of assessment and evaluation to be relegated to afterthoughts when any curriculum innovation is introduced. Unless those responsible for the innovation are clear as to its purpose and how they or any other interested party will be able to tell if this has been achieved, then a lack of clarity is almost certain to bedevil the project throughout its lifetime and may even doom it to failure before it has begun.

Assessment and evaluation are two different but related processes; confusion between the two can often lead to disappointing or inconclusive outcomes. Assessment is usually considered to involve some form of measurement of progress, often but not always by the application of tests. It is not the place here to provide details of different forms of assessment – summative, formative, assessment for learning, ipsative - involving normative or criterion-referenced tests. Qualitative data is often as valuable as quantitative. The point is, though, that assessment in itself can only be part of the wider process of evaluation whereby decisions are made about the worthwhile nature of the programme and whether and how to continue it in the same or a different form or to abandon it for cost-effective or other reasons.

Traditionally, approaches to assessment have entailed pre-post measurement and have tended to be set within a positivist experimental – control group paradigm often advocating the use of randomised controlled studies (RCTs). However well these are applied – and it is extremely difficult to meet all the necessary criteria – they can only provide limited information to these interested in ‘real-world’ evaluation. Essentially, evaluation of innovative projects in natural contexts requires a much more holistic approach than is offered by RCTs.

The issue at stake is the conflict between psychology’s efforts to provide scientifically respectable data and the claims of more recent socio-cultural theorists that social and historical contexts play a vitally important role in all learning and that the nature of the interactions between novices and more knowledgeable others (parents, teachers, peers) around different activities (e.g. the curriculum) mainly by means of language must all be taken into account. There are a number of ways in which this can be done, but at the CEDU we favour a form of what has come to be called ‘illuminative evaluation’. (Burden, 2008). This approach is built around the ‘SPARE wheel’ model whereby the Setting, the Plans, the Actions and Reactions of participants are all taken into account as part of the Evaluation process. The term ‘Wheel’ is meant to represent the fact that this is a cyclical process and not simply one with fixed ‘before’ and ‘after’ points.   Within this model, aspects of assessment take place at every stage. 

 

  • The Setting of the novation needs to be described in detail and possible contributing or inhibitory factors relating to potential change need to be identified.
  • The nature of the Planning process then needs to be considered – Whose idea is it? Who else is involved? How carefully has it been researched? What have been identified as potential success criteria?
  • Careful consideration needs to be given to what actually takes place in the (Action) implementation of the project, as the best laid plans rarely work out smoothly in practice.
  • The Reaction stage is the one at which formal assessment is usually carried out. Basically, what are the responses of all the stakeholders (pupils, teachers, parents, the wider community) to the project? This can be done by means of achievement or other tests but also by tapping into the attitudes of everyone involved by using questionnaires and interviews.
  • At each step of the way this information is drawn together and Evaluated in order to enable project leaders to decide upon appropriate courses of action.

The Cognitive Education Development Unit has been able to operationalise this approach by drawing up fourteen criteria from research and theory on school effectiveness and improvement, and requiring schools to gather evidence as to how these criteria are being met by introducing thinking skills and strategies into the curriculum and the teaching/learning process. The production of a portfolio of this evidence is then scrutinised at the CEDU and, if found to meet the criteria, is then followed up by a one-day visit to the school by one or more members of the CEDU team. During this visit representatives of all stakeholders are interviewed – pupils, teachers, LSAs, parents, members of the school’s governing body – in order to gather their thoughts and feelings about the school’s learning journey. Lessons are observed and a wide range of the children’s work is examined. Where recent Ofsted, Estyn or other inspections have taken place, the reports are scrutinised. However, it must be emphasised that this process should not be considered any kind of formal inspection; rather an opportunity to share the values, ethos and achievements of the school with information and interested outsiders.

As far as assessment is concerned, much of the information gathered is of a qualitative nature. If we take into account the key interactive elements of pupils, teachers and curriculum activities, we need to consider how we wish to measure learning outcomes. Certainly, the continuing high achievements of the pupils across a range of curriculum subjects must be seen as important alongside their attendance record, taking into account the ethnographic background of the community. However, if we consider what pupils learn in school to have potentially profound effects upon their attitudes to the purpose of learning across the lifespan and their confidence in their own ability to deal with problem situations as they arise, we clearly need to investigate more widely and deeply. At the CEDU we have developed a number of questionnaires and techniques for assessing pupils’ perceptions of their learning capabilities, their attitudes towards thinking, their metacognitive awareness and their views on the quality of mediated learning experiences provided in their lessons. Thinking schools are encouraged to draw upon these techniques as a means of researching wider outcomes of their cognitive curricular activities, as required for future accreditation as Advanced Thinking Schools.

By the same token, we should expect those involved in the teaching of thinking skills and strategies to be transformed themselves by the experience, particularly with regard to their confidence in providing differentiated learning activities and heightened expectations of all pupils’ learning capabilities as well as renewed satisfaction in the teaching process. At a more general level we would also expect a highly positive school ethos, high levels of inclusion and minimal levels of bullying or disruption. Each of these areas is open to objective investigation by means of other instruments developed at the CEDU.

Drawing all this data together is inevitably a complex task, as is its translation into accurate and meaningful reports. The accreditation reports provided by CEDU for schools are structured around the framework of the SPARE wheel. Accreditation is only allocated for a period of three years as much can happen over the ensuing period. Most schools, however, have applied for re-accreditation, whilst several have continued to move forward to achieving Advanced Thinking School status.

 

Information on any of the techniques mentioned above alongside permissions for use is available from the CEDU on request.

Reference: Burden, R.L. (2008) ‘Illuminative Evaluation’. In B. Kelly, L. Woolfson and J. Boyle (eds) Frameworks for Practice in Educational Psychology.(pp 218-234). London: Jessica Kingsley.