What the data says about selective schools and Ofsted
A few weeks ago, Ian Widdows replied to my Swarm blogpost asking if the Swarm could highlight Secondary Modern schools as a group, and Grammar Schools as a group. Ian is part of an association of Secondary Modern schools, and he feels there is some bias in Ofsted judgements against Secondary Modern schools. At the time, the Swarm didn’t include data on admission policies. I can’t include all the DfE data or the Swarm would be slow, but I have now added admission policy. Both Secondary Moderns and Grammars appear as a “virtual LA” in the LA dropdown list.
Here is a picture of the Secondary Moderns: the green discs are outstanding schools, yellow are good, red are grade 3, black grade 4 and white have no inspection grade.
You can see a broad range of grades for the 125 schools with DfE data that have secondary modern admission policies. There are some interesting features to the distribution though. For example, there are many Secondary Moderns with KS2 average point scores above the national average. That surprised me. Clearly the Secondary Moderns are a very diverse group. You also see many grade 3s in the Secondary Modern group (the red discs), and many of those are above the trendline of the Swarm, so on the face of it, there does seem to be some bias. What about the Grammar schools? Here they are:
Wow! What a lot of green! There are only 2 of the 164 Grammar Schools which are grade 3 or 4 (and 7 currently ungraded). This certainly deserved a closer look, so I fired up my stats software and got it to do some counting. Here are the frequencies of grades for each type of admission policy.
|Number||Grade 1||Grade 2||Grade 3||Grade 4||No Grade||Total|
Hmmm, that does look as if Grammars do better than others in terms of Ofsted judgements. Let’s convert to proportions so we can compare more easily.
|Number||Grade 1||Grade 2||Grade 3||Grade 4||No Grade||Total|
Of course, when you move to percentages, there may be rounding errors so the rows may not add precisely to 100%. There is clearly a much higher proportion of Grammars which are outstanding than for other admission policies. Secondary moderns seem to be similar to comprehensives. What would this look like graphically?
So, Grammars are overwhelmingly more likely to be graded Outstanding and far less likely to be grade 3 or below than other schools. You can also see how the Secondary Moderns are less likely to be graded outstanding. However, there doesn’t seem to be a bias against Secondary Moderns so much as a bias in favour of Grammars.
But couldn’t this be just random chance? After all there are a lot fewer Secondary Moderns or Grammars than Comprehensives aren’t there? If you are of the view that schools of all admission policies should have the same proportion of each Ofsted grade as a starting hypothesis, there is a statistical test you can do for this. Formally, you are testing that the proportion of each grade is independent of admission policy and the test is a chi-squared test. The result of doing such a test on the table above is that the chance of such a distribution occurring by chance is less than one in a thousand trillion. So, no, it isn’t random chance.
So Ofsted are biased then?
I don’t susbscribe to conspiracy theories about this, and I think you have to be careful interpreting the data. I’d say there were some genuine questions raised by this though.
Is it justifiable that Grammar schools are more often graded outstanding?
This is a value judgement. What are we expecting Ofsted to be grading? Much of what Ofsted does is based on national norms. Academically, Grammars are going to do well on attainment. It is also much easier to get expected and better than expected progress from KS2 on level 5 than lower levels, so the progress scales are easier for Grammars to hit too. It would be interesting to consider the school effect of having so many level 5s in each class compared to in Comprehensives.
Secondly, if you believe Michael Wilshaw’s statement that Grammars are “stuffed full of middle class kids” you might think that they also have a head start where behaviour and safety comes in too. Incidentally, while the proportion of disadvantaged children in Grammars is very low at 6%, it is also lower in Secondary Moderns (24%) than in Comprehensives (28%).
However to those of us in schools being judged by Ofsted, we want to be judged in our own context with the challenges facing us individually taken into account, not some arbitrary national norm. Ofsted teams do try to do this, but perhaps sheer concentration of very bright children in a school with very few of them being disadvantaged gets in the way.
Does this show Ofsted teams are biased?
Not proven in my view, despite the smoking gun of the data. For one thing, an Ofsted grade depends on more than data, despite all the moaning about how Ofsted inspections are data-driven. Also the current framework is all about national comparisons which will inevitably favour Grammars given the current indicators used. However the conclusion that the public will draw from this (that Grammars are all well run and super and Secondary Mods are less well led and poor) is not the reality. If Ofsted teams are not biased, and are just applying a biased framework, what then?
Shouldn’t the framework be altered for selective schools?
I say yes. The new indicator of Progress 8 may be more of a challenge to Grammar Schools, but the tricky thing is defining the baseline on which the progress is based. The glass ceiling of level 5 works in Grammar Schools’ favour, and as levels disappear from primaries, what will form the baseline for Progress 8? Far harder is to know what standards Grammars should be attaining in Behaviour & safety. Getting over the advantage enjoyed by them because of their social as well as academic selectivity is more difficult, as the Halo Effect is very powerful.
I’m sure that Secondary Moderns would welcome a new Ofsted Framework addressing these issues, while the rest of us might heave a big sigh at yet another change.
I used DfE data and Ofsted data to create the Swarm datafile, and only picked schools with 2013 data for the major indicators. This will be lower than the total number of schools in England as not all of them had data for all grades in the 2013 performance tables. Ofsted grades are correct as of 31 January 2014.