50 shades of uncertainty in educational research

Again Lisa Pettifer examines the problems faced by those trying to engage in research in schools

Over the Rainbow - Lisa Pettifer

Follows on from http://wp.me/s49ySc-abstract

Do we engage in educational research or don’t we?

Lots of people engage in educational research. Some of them make a living from it, even though they’re not teachers; some of them don’t, even though they are. It seems there are many tensions between the worlds of teachers, trained to teach (by the universities, for the most part, at least until recently) yet not engaged in research, and the worlds of researchers, engaged in gathering evidence about education, yet not involved, directly, in school teaching. At best, the university researcher offers an objective breath of fresh air to what might have become a stuffy room of ingrained practice; at worst, they are seen to function like a secret Ofsted, reporting to government agencies about what should be done in the world of education and how teachers should be doing it.

Researchers within universities may visit schools…

View original post 1,119 more words

Advertisements

Observing without grades: homework done

For good or for ill, my school has still been grading lessons when carrying out formal observations. Of course, we have been aware of what Ofsted have been saying on the matter, and also of what a number of campaign-bloggers have been arguing. Through much of the past 12 months, I and like-minded colleagues such as @Bterziyski have been conducting a quiet insurgency, devising toolkits to peer-observe without grades, messing with the minds of some of our teachers as we blankly refuse to use numbers or Ofsted dog-whistle words.

But so far we have failed to turn the whole ship around. My headteacher – for very sound reasons – wants to be able to point to a list and say, Here are my best teachers, or Here are the ones we need to work on. She absolutely understands that observation can and should be a development activity, but she needs it also be a tracking device. Before half-term I made my latest bid to SLT to consider the merits of LONG (Lesson Observation – Non Graded). After an amiable chat, I was set a homework. Below is my progress so far.

Appraisal at Our School

Appraisal must meet the needs of our school. It should also respect the reasonable expectations of teachers (as represented by their unions), whilst at the same time being mindful of government policy on Teaching Standards and Performance Related Pay.

Consequently, the Appraisal system at our school attempts to:

  • Develop the quality of teaching
  • Through this, improve the outcomes of students
  • Inform the PD plan
  • Provide evidence for the Headteacher to make decisions on staffing and PRP.

How we do this currently

Each teacher is assigned an appraiser. In many cases this is their HoF, or other immediate line manager; in all cases it is a colleague who works closely with them. At the beginning of the Autumn term, the teacher and appraiser together review the objectives set the previous year, using evidence from the teacher’s portfolio (CCPDP) – exam review, formal lesson observations and other evidence the teacher has chosen to present. The appraiser declares that the teacher has met the Teachers’ Standards (or not). The Head decides on PRP. Together the teacher and appraiser set 3 further objectives for the coming year, reviewing them on an interim basis in the Spring. They agree on the date and focus of an ‘appraisal lesson’, which is graded according to the school’s summative criteria. In the course of other internal reviews across the year, the teacher may be formally observed (and graded) a further two times. The Head maintains a spreadsheet on everyone’s graded lessons. The AHT (staff development) reviews all objectives, to discern common themes that can be addressed in the school PD plan.

Lesson Observation

“Ofsted does not award a grade for the quality of teaching for any individual lessons visited and it does not grade individual lessons. It does not expect schools to use the Ofsted evaluation schedule to grade teaching or individual lessons.” ( Ofsted clarification for schools, October 2014)

At our school, we have continued to grade individual lessons, using a template we devised in November 2011 to match the Ofsted inspection criteria.

Some numbers:

At our school, a mainscale teacher, teaching 25 lessons a week, will teach nearly 1000 lessons across the year; put another way, they will teach lessons lasting 50,000 minutes. Agreed with unions, we grade 1 appraisal lesson of 50 minutes, plus up to 2 more lasting 20-50 minutes. We say, on the evidence of this amount of observed time, we can grade the quality of individual teachers. Why might we be wrong?

  • Aside from teaching the classes, teachers spend an uncounted amount of time planning and marking, running after-school revision, intervention and other clubs. None of that is observed, and arguably none of it contributes to the teacher’s grade.
  • We know that ‘high-stakes’, ‘one-off’ lesson grading systems lead to cultures which are about proving, rather than improving; they encourage performance, not learning (Chris Watkins, IOE).
  • Teachers play it safe, covering ground they know their students are comfortable with.
  • Grading is unreliable. It is notoriously difficult to train observers so that they will agree on a grade.
  • Because graded lessons are poor predictors of student outcomes.
  • Because 63% of all lesson judgements are wrong (Strong et al, Journal of Teacher Education, 2011.)
  • If a lesson is graded ‘inadequate’ there is a 90% chance a second observer will grade it differently (Measures of Effective Teaching (MET) project, Bill & Melinda Gates Foundation, metproject.org)

Thanks here to @shaun_allison and others at the wonderful Class Teaching

However…

“Ofsted will usually expect to see evidence of the monitoring of teaching and learning and its link to teachers’ performance management and the Teachers’ Standards, but this should be the information that the school uses routinely and not additional evidence generated for inspection.” (Ofsted inspections – clarification for schools)

So we do need to have robust means of capturing the quality of teaching at our school.

More numbers:

Assume each teacher is formally observed for a total of 100 minutes, and receives feedback lasting the same amount. As this involves 2 (in fact sometimes 3) people, we can say that this consumes 400 minutes. If we say this happens to 80 members of staff each year, this amounts to a total of 533.3 hours per year. We could do a lot with that time. We might say that this was time well spent if:

  • It improved outcomes for students
  • It improved the practice of the observed teacher
  • It improved the practice of the observing teacher
  • It provided reliable evidence of the quality of teaching and learning across the school
  • It identified common development needs that could then be addressed by the in-house PD programme
  • It reliably identified weak teachers and strong ones, so we could perhaps know what we needed to do next and how
  • It contributed to staff wellbeing, because it was an element of staff development rather than merely appraisal
  • It was about Quality Improvement, not just Quality Assurance (Joanne Miles) The role of mindset in the move from graded to ungraded lesson observation cycles

 

 

How might we capture great teaching that leads to great learning?

“A formative teacher evaluation system – based on a continuous assessment and feedback rather than a high-stakes test – must incorporate a range of measures from different sources, using a variety of methods.” (Developing Teachers: Improving professional development for teachers, January 2015. The Sutton Trust.) Link here. It is crucial to avoid over-reliance on a single source of evidence (such as a graded lesson), as there can be “a lot of noise around a weak signal.”

The Sutton Trust report (compiled with the Bill and Melinda Gates Foundation from 80 leading teachers from across 6 Anglophone countries) identifies 6 approaches to teacher assessment. None of these should be relied upon in isolation; validity arises from their triangulation.

Moderate validity:

  • classroom observations by peers, principals or external evaluators
  • ‘value-added’ models (assessing gains in student achievement)
  • student ratings

More limited evidence:

  • principal (or headteacher) judgement
  • teacher self-reports
  • analysis of classroom artefacts and teacher portfolios

Our appraisal procedures may already bear elements of these approaches. We have:

  • Observations (graded for appraisal and internal review; peer observation for development purposes.)
  • Exam review (for individual teachers, using residuals.)
  • Headteacher judgement (based on provided evidence, for PRP)
  • Teacher self-reports and portfolios (in the appraisal booklet and CCPDP.)

I propose two changes.

  1. A new lesson observation protocol Great Teaching and Learning Toolkit
  2. Adapt our teacher portfolio, now to be the C…. Professional Development Portfolio (CPDP) CPDP Grid

Each change is designed so that we better identify the strengths and development needs of our teachers so that: they can benefit from a more strategic in-house PD plan; the Head can make staffing and PRP decisions based on more valid evidence; the overall quality of teaching improves and leads to better outcomes for students.

Lesson Observation Great Teaching and Learning Toolkit

“Successful teacher observations are primarily used as a formative process – framed as a development tool creating reflective and self-directed teacher learners as opposed to a high stakes evaluation or appraisal.” (Developing Teachers: Improving professional development for teachers, January 2015. The Sutton Trust. My italics.) The report claims that observation can be effective when it is collaborative and collegial; however, the research also concludes that challenge is needed in the process, so may involve school leaders or external experts. Overall, it emphasises the requirement to have strong observation protocols, including observers who are trained.

The new protocol has two parts. The first, Dialogue and Development, will be completed by the teacher and their observer. It records what happened, what went well and what could have been better. Crucially, it requires that next steps be identified, with some plan for their implementation. Therefore, this will not be a ‘cosy chat’; it is joint practice development (JPD): it will require two professional colleagues to think deeply about what took place in the lesson and how to develop their practice. They can be assisted in this by the second part of the protocol, the C…. Great Teaching and Learning Toolkit. When a colleague asks us what great teaching looks or feels like, it is fair that we should have some idea. Everything here has been derived from the advances in pedagogy we have made here in the last few years, from research conducted by colleagues and from the peer observation toolkits trialled by Development Coordinators. It absolutely is NOT a checklist against which to measure a single lesson, as it understands that much learning in not ‘visible’ and rather takes place over time.

This is a protocol which can be used formally (for appraisal and review purposes) and informally (peer observation between colleagues). There is no space for a grade; I believe that words such as ‘outstanding’ or ‘inadequate’ are also best avoided, as they tend to crowd out the more meaningful dialogue around development.

Teacher Portfolio CPDP Grid

We have a long tradition at our school of maintaining a portfolio and presenting it to the Head when making a threshold application. Though insufficiently robust as a single source, taken with lesson observations and exam review it may be a valid measure of the quality of teaching.

Why we should change the current portfolio

  • It is largely ignored by teachers not applying for UPR.
  • It inadequately reflects the degree to which a teacher is meeting the Standards at the level appropriate for that point in their career.
  • It is a compilation of achievements, or ‘jobs done’, but is not a reflection on future development needs;
  • It therefore fails to inform appraisal objectives for individual teachers; and
  • Taken together, they cannot be read for the purposes of PD planning for the school.

I therefore propose a C…. Professional Development Portfolio (CPDP), arranged around the 8 Teachers’ Standards, the wording of which is adapted to refer to the C…. Great Teaching and Learning Toolkit. Two further tabs have been added (on engagement in PD and on collaboration) in order to encourage JPD as a form of teacher development.

The CPDP Grid will act as the contents page of the portfolio. It will be completed as part of appraisal (in reality, mainly before the appraisal meeting) and provide the ‘baseline’ for the appraisal objectives to be set, and the ‘evidence of outcomes’ for the previous year’s objectives. [The Appraisal Booklet will be adapted to reflect this. Alternatively, the grid may simply appear as the first pages in the booklet.] In this way, the appraiser and teacher will have a clearer view of strengths and areas for development; the Head will be able to say, with more security, which teachers need more targeted support and which should gain PRP; and the AHT will be able to identify common themes and address these through the PD programme.

Next stages

Assuming there is agreement on the principles of this, we need to consult widely with students and staff to truly ascertain what we feel constitutes ‘great’ teaching and learning at our school. We also need to do serious work on observers and teachers, so that neither expects a grade (or codeword) to emerge from an observation. Then we need to take a deep breath, and watch as the whole thing unfolds.

Assessment Without Levels – The story so far

The Class Teaching team update us on their progress with Assessment without Levels

Class Teaching

In May 2014 we were awarded a DfE Innovation Fund to develop a method of assessing without levels in KS3.  We saw this as an opportunity to reclaim purposeful assessment and so came up with our ‘Growth & Thresholds Model’ which is outlined here.   The system has now been in operation with Y8 since September, so I thought it would be timely to share how it’s been going.

The whole process has required a shift in thinking for staff, parents and students.  Rather than focusing on a predetermined end-point and how to get there i.e. an end of key stage target level, we are focusing on their starting points and then how to move them on – without their progress being ‘capped’ by a target.  We do this by scaffolding their learning through four thresholds, towards excellence.  The idea is that all students will aim for excellence.  This principled approach to assessment…

View original post 1,389 more words

Abstract

Great post from Lisa Pettifer on some research pitfalls

Over the Rainbow - Lisa Pettifer

Heraclitus is often paraphrased, “No man ever steps in the same river twice, for it’s not the same river and he’s not the same man.” And like the river, the classes we teach will never be the same again, for having been taught, they are changed, and having changed, they require new teaching.

river

These are complex processes, interwoven with social and emotional erosion and deposition, with the transportation of other factors such as time of day, week, term or year, curriculum innovation, ability grouping, school ethos. Is the equipment of educational hydrometry sophisticated enough to gauge how well our efforts as teachers are causing our charges to discharge through the school system?

I am very much in favour of research, both as a tool for school improvement and a means of staff development. My school has previously operated the Teacher Learning Academy (introduced by the GTCE, later overseen by the…

View original post 1,104 more words

An ice-cold bucket for the #workloadchallenge

I know a teacher.  You’ll know her too, or someone very similar to her. She works hard; she does not have her work in balance with her life; she loves her job; she is bloody good at it. Know the one? Apparently the DfE doesn’t,  or they wouldn’t be insulting her with their disingenuous promises on workload.
How best to describe this teacher? In her 3rd year of teaching, she has no paid responsibilities beyond her classroom, she is not on UPR, she is mainscale in old money. As such, she teaches 25 of 30 lessons across Key Stages 3, 4 and 5. She has a form whom she cares for as individuals,  calling their mums, addressing their worries. She had parents’ evening last week, one of the 6 she will attend this year. She has a ‘data entry window’ to meet by Thursday – as she does every half term for every class she teaches. I’ve lost count of the number of GCSE revision sessions she helps run each week (it would be more, but for the staff development time that’s also on the school calendar. ) Speaking of which, she volunteers to lead development for other colleagues,  and if there is a whole-school drive to improve differentiation,  or literacy, or AfL… there she is at the vanguard of it. Naturally she is also a student teacher mentor. If I haven’t managed to plan my lesson properly, I can always look on the shared drive; there I will find a resource,  replete with lesson plan.
As I say,  I reckon she loves her job. I do wonder what she must have made of Nicky Morgan’s pre-election grab for the teacher vote. I suppose she will be relieved that there will be no in-term announcements of further curriculum change (and think to herself,  Was it not this very government that have been doing just that? ) She might also be surprised to receive sympathy from an Education department that scrapped the list of admin tasks that teachers could not be obliged to undertake. No more sudden moving of the Ofsted goalposts,  she hears them say. They, who have rewritten the handbook each year they have been in office. She would quite like to get on with planning the new A level syllabus,  but she is as we speak teaching the revised GCSE spec, which itself has only one year to run. The new National Curriculum is a bit of a bugger an’ all, but at least she can reinvent the assessment regime as she goes along.
Have you met this friend of mine? You might recognise her by the slightly frazzled grin she gives you when you ask her if she’s had a nice weekend. There you go: my belated recommendation to the Department. Give my colleagues their weekends back.