Assessment for Accountability – Taking the Biscuit

Thinking about assessment and accountability again. I adapted this from a letter I wrote to the then Ed Select Committee.

The problem of accountability

If we take it to be the case that teachers and schools need to be ‘held to account’, then we need to ask ourselves some questions.

Held to account for what?

The answer to this is  crucial. For a long time we were held to account for pupil ‘attainment’. Recently there has been the reasonable suggestion that there are many factors outside of our control which impact on attainment, and that progress might be a better measure of how good or bad we are. The measurement of progress nevertheless, remains a massive challenge, in spite of attempts to contextualise it, or to use national trends for comparison: baseline data is not reliable (it’s ludicrous to believe that you can use the behaviour of 4-year-olds to derive data that will hold teachers to account at the end of KS2 – and beyond!); pupils do not make standard amounts of progress; domains at the start and end of the progress measurements are different (GCSE art teachers, beware!); cohorts are different; significance is difficult with small groups, etc.

Whilst ‘progress’ seems at first fairer and a preferable measure to ‘attainment’, neither is sufficient for our purposes – not the way currently measured and not when matched against the aspirations of the National Curriculum.

Does the current system even serve the right purpose?

In the drive to measure ‘attainment’ and ‘progress’, I think we sometimes forget that we are using these things as proxies for the quality of the education being provided. We need to return to the drawing board for how we might ensure this happens. Currently we use an assessment system that cannot do this; the measurement is too narrow, subject to chance variables, and too much driven by fear of failure, leading to all the perverse incentives the assessment experts have been writing about for so many decades. A quality primary education is not ensured by testing the very few items that are currently measured at the end of KS2, any more than a quality factory is ensured by eating one of its biscuits.

I think schools and teachers do want to provide a good education to their pupils and that a climate of fear is unnecessary and counterproductive. Real accountability must involve a move away from looking only at outcomes and focus instead on quality input: we need well-educated teachers with excellent and maintained subject knowledge, quality text books produced by experts and thoroughly vetted by the profession, online materials and required reading, use of evidence and avoidance of fads. Quality training is essential, as is development and retention of teachers with expertise.

How can we ensure accountability where it counts?

Realistically, I don’t expect a quick move away from summative assessments for the purpose of accountability, in spite of all the arguments against. But I feel that we could address some of the issues that arise with the current system by generating and providing (to the DfE if need be) not less but more information:

  • Frequent, low-stakes tests help both teaching and learning – require/provide tests throughout the year, every year.
  • Fine grained, specific tests provide useful information – test what we want to know about – keep that data.
  • Assessing the same domain more than once and in different ways helps to reduce unreliability – do not rely on one single end of year test.
  • Testing earlier in the cycle gives useful feedback for teaching – do not wait until the end of the year or the end of the Key Stage.
  • Random selection from a broad range of criteria helps to reduce ‘teaching to the test’ – test knowledge in all curriculum areas without publishing a narrow list of criteria.
  • Use assessment experts and design assessments that test what we want pupils to know or do. Criteria need to be reasonable – not obscure and mystical as they have been recently.

If these aspects were applied to an assessment system throughout the primary phase, I believe we could enhance learning, improve accountability in what really matters and provide vast amounts of data.

We really need to make better use of technology at all stages; this is the only way in which we can feasibly make assessment serve multiple purposes. There would need to be a move away from the high stakes pass/fail system which is not fit for purpose, towards a timely monitoring and feedback system that could alert all stakeholders to issues and provide useful tools for intervention. Data collected from continuous low-stakes assessments provides a far more valid picture of teaching and learning.

Advertisements

Timings and Tides: the Chartered College of Teaching inaugural conference – Sheffield

I’ve followed the development of the Chartered College of Teaching with some interest and much scepticism. In this mode, I joined as a founder member and spent not an inconsiderable amount of money and time attending the inaugural conference in Sheffield. I’d have liked to attend the London conference, but they saw fit to hold it during the week when only half of us were actually on half-term and many of us could not attend.

Nevertheless, I went with an open mind. I’m aware that there are great enthusiasts out there who see this as a bright beacon of hope on our general plain of educational misery. I wanted to see if there was any basis for this. The answer is that I’m not sure; I’m still sceptical. This blog is my discussion of the conference itself and the College overall.

Why the profession needs a collective voice

I’m afraid I am unable to comment on the first two items on the agenda, as it was impossible to arrive on time, coming by train but I was in time to catch the talk by Professor Chris Husbands (Vice-Chancellor of Sheffield Hallam University). He drew well on his experience as a teacher and spoke convincingly, I thought, on notions of ownership and what matters. He sought to redirect the idea that ‘teachers matter’ towards ‘teaching matters’. I think he was making the point that we needed to focus less on individuals being the key to a successful education system and more towards a systematic improvement of the process. If so, then I would agree this is probably correct – we need to address education in this country, at a level that is more than just ‘holding teachers to account’. Nevertheless there were dissenting voices in the room, arguing (rightly) that teaching is dependent on the individuals, in terms of defending teachers’ well-being, and because teaching requires complex ‘on-the-hoof’ analysis and seems inextricably tied up with human interactions and relationships.

I question, too, the fundamental assumption of a collective voice. Whilst I hate the ‘pot-luck’ approach to education that the English seem unable (and unwilling) to challenge, I know that there are many voices and I worry that collective may turn out to be dominant. I’m unconvinced that the cult of evidence is going to prevent teachers being censured, yet again, by the opinionated but ill-informed, for doing the ‘wrong thing’. I know this is certainly the fear of the neo-trads, even though the tide seems currently in their favour and in fact it’s a cultural, not a political problem: fads and flavours of the month cut all ways, and always have. Which brings me to the next item.

Why we need evidence

Chaired by Ann Mroz, The panel was: Sinead Gaffney, Lisa Pettifer, Aimee Tinkler, John Tomsett and Professor Samantha Twiselton. John opened by proposing the need to weigh up the forces of authority and evidence, with the suggestion that we should not be afraid to swim against the tide if necessary. Well, at this point, I couldn’t agree more, although I suspected that not all tides were equal in his mind. There was much discussion about the need for an evidence-based approach, but I was prompted to tweet thus:

There was a lot said about the importance of an ‘evidence-based’ profession, the use of evidence and about teachers conducting research in their own environments; this is where I derive my  concern that we’re heading towards something more like a ‘cult of evidence’ than an informed profession that questions assumptions (I’m not alone, I imagine). To those without a scientific or a sceptical background, the use of ‘evidence’ as a holy grail, is as dangerous as it is essential. I felt that Sinead was something of a lone voice calling hard for the critical evaluation of evidence rather than the gullible application of a set of tools condoned by the EEF. Personally, I have found that most people are easily persuaded by rhetoric and quickly descend into ritual. Teachers generally are ignorant. (If you wish to rebut that, consider (my anecdotal evidence) that not a single member of the very large staff at my school had actually heard of the College of Teaching, as I was leaving on Friday).

It is very difficult to impress upon people that research evidence almost never says we should do something one way or the other. On the contrary, its power is in calling into question things for which there is very little or no evidence. Most educational research would be considered worthless by any scientific standards and much of it is contradictory. Almost none of it stands up to replication. See this timely article. Like Sinead, I have looked past the meta-analyses on the EEF toolkit and examined some of the original research. If you do the same, you’ll find much of it evaporates into thin air. Try it for ‘feedback’ and see what happens. Moreover, there should be serious doubts about encouraging widespread experimentation and research conducted by teachers. It’s difficult to obtain rigour in research, even under the best experimental conditions. Biology is notoriously tricky. If you add to that the ethical and social considerations of working with young children and then sharing those unreliable findings, we’re opening a massive can of worms (no biological pun intended). Whilst some on the panel were arguing that we needed to be able to judge whether the evidence was robust, audience members, without irony, were still calling for the application of instinct and John T reminded us that any consideration of evidence at all, was still ‘miles away’ in most institutions.

The other elephant-in-the-room is that there is probably far too great an acceptance of the way in which we measure effects in educational research. I don’t mean a statistical issue, but a logistical and a philosophical one. When we try to determine if a practice has an effect, how do we measure the effect, if the product is learning? It may be easy enough to determine if an intervention on multiplication facts has worked – simply test pupils to see if they know those facts. But what if the outcome is trickier to measure? I entered a discussion recently where it was argued that allowing pupils to  do practical science might not be important because the evidence showed that didactic methods trumped investigation! My question would be, ‘in what way?’ If the measure is filling in the answer boxes on a test paper (and believe me when I say I’m a strong advocate of that in appropriate ways), then perhaps teaching the pupils to do just that will produce a greater effect. Yet practical science is about being able to do practical science! Investigations should enable us to become better at investigations. I’m not alone in arguing for appropriate measures – yet most evidence is based on a very narrow set. It’s difficult to see a move away from this in an educational system that now expects secondary teachers to predict art grades from KS2 aggregated English and Maths scores!

Going beyond your comfort zone

Penny Mallory was extremely engaging and I was extremely discomfited by the implications of her speech. Penny overcame self-doubt and domestic adversity to become a champion rally driver. Her questions to the audience were, ‘Can anyone become “world class?”‘ and ‘What qualities does a “world class” person have?’. I was gratified to hear some of the answers along the lines of, ‘It depends what you mean’ and ‘Good genes’. I know what the motivational intention was: we limit ourselves; we need a growth mindset; we should take risks etc. I’m slightly, but not entirely, on board with the growth mindset philosophy. I believe it is true that we can play what Eric Berne’s patients used to refer to as the game of ‘Wooden Leg’ and I work hard to counter that with my pupils. However, I profoundly dislike the contemporary message of ‘social mobility’ and the new populism which the College also seems to be promoting. Winning depends on there being others who will lose. Climbing the greasy pole will require stepping on the competitors. It’s a toxic message in a ruthless climate which seeks to replace the greater aspiration of social justice. Aiming to be ‘world class’ as an individual is a very selfish pursuit which by necessity will always be limited to a few. Becoming world class as an organisation  (or as a country!) needs a different approach altogether – one that I feel we’ve departed from rapidly since the 1980s.

Why being brave is important

Tim O’Brien (Visiting Fellow in Psychology and Hyman Development -UCL Institute of Education) chaired this panel. It being after lunch, my note-taking had decreased and I had moved myself to the back of the room to exit, if need be, but it was interesting to consider ideas of bravery. Perhaps the College could be a force for good, recognising that the profession is currently driven more by fear than it should be.

I know that many pin their hopes on the College to remedy this. Tim’s an eminent psychologist who comes across as knowing his stuff. I was in one of the focus groups he led in the ‘grounded theory’ research he conducted when the College was deciding its remit, and he spoke of this, thanking those of us who were there. In the midst of all the concurrence on the need for bravery, however, I wished I could have had the opportunity to point out that there’s a reason for the fear in education; being brave comes at a cost. Do those advocating it, understand the risks they are asking teachers to take?

Networking

I networked just enough to find that most of the attendees were enthusiasts and that some of them at least, were waiting specifically for the last part of the day – ‘Improving Wellbeing in the Classroom’ – Professor Tanya Byron. She was a bona-fide TV celebrity, so to speak, and the audience seemed engaged. I left before the end – it was old territory for me.

So was it worth it? Well, I still feel I have done the right thing in joining and in attending. This is a novel development that may bring something good. At the very least, access to research is something that I’ve missed since finishing the Master’s. In the conference itself, I would have benefited enormously from a more structured approach to networking. This was left largely to us to do informally during the breaks. I knew that there were twitter contacts there I would have liked to meet, but it was not easy to discern who they were and my social ineptitude hindered me in approaching people ‘cold’, particularly if they were already talking in apparently established groups.

Ultimately, I’d make a plea to those who are sceptical, members or otherwise. Keep it up. To the enthusiasts, I’d say the same, alongside the request that you allow all manner of criticism. There was much enthusiasm evident among the attendees; this in itself can create a charismatic tide. Those who swim against it are always needed.