Does Ofsted have selective judgement?

What the data says about selective schools and Ofsted

A few weeks ago, Ian Widdows replied to my Swarm blogpost asking if the Swarm could highlight Secondary Modern schools as a group, and Grammar Schools as a group. Ian is part of an association of Secondary Modern schools, and he feels there is some bias in Ofsted judgements against Secondary Modern schools. At the time, the Swarm didn’t include data on admission policies. I can’t include all the DfE data or the Swarm would be slow, but I have now added admission policy. Both Secondary Moderns and Grammars appear as a “virtual LA” in the LA dropdown list.
Here is a picture of the Secondary Moderns: the green discs are outstanding schools, yellow are good, red are grade 3, black grade 4 and white have no inspection grade.

Secondary Modern Schools in The Swarm

Secondary Modern Schools in The Swarm

You can see a broad range of grades for the 125 schools with DfE data that have secondary modern admission policies. There are some interesting features to the distribution though. For example, there are many Secondary Moderns with KS2 average point scores above the national average. That surprised me. Clearly the Secondary Moderns are a very diverse group. You also see many grade 3s in the Secondary Modern group (the red discs), and many of those are above the trendline of the Swarm, so on the face of it, there does seem to be some bias. What about the Grammar schools? Here they are:

Grammar Schools in the Swarm

Grammar Schools in the Swarm

Wow! What a lot of green! There are only 2 of the 164 Grammar Schools which are grade 3 or 4 (and 7 currently ungraded). This certainly deserved a closer look, so I fired up my stats software and got it to do some counting. Here are the frequencies of grades for each type of admission policy.

Number of schools of each type of Admission Policy at each Ofsted grade (as at 31 Jan 2014)
Number Grade 1 Grade 2 Grade 3 Grade 4 No Grade Total
Comprehensive 493 1177 609 138 211 2628
Secondary Modern 16 68 30 7 4 125
Grammars 124 31 1 1 7 164

Hmmm, that does look as if Grammars do better than others in terms of Ofsted judgements.  Let’s convert to proportions so we can compare more easily.

Percentage of schools of each type of Admission Policy at each Ofsted grade (as at 31 Jan 2014)
Number Grade 1 Grade 2 Grade 3 Grade 4 No Grade Total
Comprehensive 18.8% 44.8% 23.2% 5.3% 8.0% 100%
Secondary Modern 12.8% 54.4% 24% 5.6% 3.2% 100%
Grammars 75.7% 19.9% 0.6% 0.6% 4.3% 100%

Of course, when you move to percentages, there may be rounding errors so the rows may not add precisely to 100%. There is clearly a much higher proportion of Grammars which are outstanding than for other admission policies. Secondary moderns seem to be similar to comprehensives. What would this look like graphically?

Ofsted grades by Admission Policy

Ofsted grades by Admission Policy (up to 31 January 2014)

So, Grammars are overwhelmingly more likely to be graded Outstanding and far less likely to be grade 3 or below than other schools. You can also see how the Secondary Moderns are less likely to be graded outstanding. However, there doesn’t seem to be a bias against Secondary Moderns so much as a bias in favour of Grammars.
But couldn’t this be just random chance? After all there are a lot fewer Secondary Moderns or Grammars than Comprehensives aren’t there? If you are of the view that schools of all admission policies should have the same proportion of each Ofsted grade as a starting hypothesis, there is a statistical test you can do for this. Formally, you are testing that the proportion of each grade is independent of admission policy and the test is a chi-squared test. The result of doing such a test on the table above is that the chance of such a distribution occurring by chance is less than one in a thousand trillion. So, no, it isn’t random chance.

So Ofsted are biased then?

I don’t susbscribe to conspiracy theories about this, and I think you have to be careful interpreting the data.  I’d say there were some genuine questions raised by this though.

Is it justifiable that Grammar schools are more often graded outstanding?

This is a value judgement.  What are we expecting Ofsted to be grading?  Much of what Ofsted does is based on national norms.  Academically, Grammars are going to do well on attainment.  It is also much easier to get expected and better than expected progress from KS2 on level 5 than lower levels, so the progress scales are easier for Grammars to hit too.  It would be interesting to consider the school effect of having so many level 5s in each class compared to in Comprehensives.

Secondly, if you believe Michael Wilshaw’s statement that Grammars are “stuffed full of middle class kids” you might think that they also have a head start where behaviour and safety comes in too.  Incidentally, while the proportion of disadvantaged children in Grammars is very low at 6%, it is also lower in Secondary Moderns (24%) than in Comprehensives (28%).

However to those of us in schools being judged by Ofsted, we want to be judged in our own context with the challenges facing us individually taken into account, not some arbitrary national norm.  Ofsted teams do try to do this, but perhaps sheer concentration of very bright children in a school with very few of them being disadvantaged gets in the way.

Does this show Ofsted teams are biased?

Not proven in my view, despite the smoking gun of the data.  For one thing, an Ofsted grade depends on more than data, despite all the moaning about how Ofsted inspections are data-driven.  Also the current framework is all about national comparisons which will inevitably favour Grammars given the current indicators used. However the conclusion that the public will draw from this (that Grammars are all well run and super and Secondary Mods are less well led and poor) is not the reality. If Ofsted teams are not biased, and are just applying a biased framework, what then?

Shouldn’t the framework be altered for selective schools?

I say yes.  The new indicator of Progress 8 may be more of a challenge to Grammar Schools, but the tricky thing is defining the baseline on which the progress is based.  The glass ceiling of level 5 works in Grammar Schools’ favour, and as levels disappear from primaries, what will form the baseline for Progress 8?  Far harder is to know what standards Grammars should be attaining in Behaviour & safety. Getting over the advantage enjoyed by them because of their social as well as academic selectivity is more difficult, as the Halo Effect is very powerful.

I’m sure that Secondary Moderns would welcome a new Ofsted Framework addressing these issues, while the rest of us might heave a big sigh at yet another change.

Note

I used DfE data and Ofsted data to create the Swarm datafile, and only picked schools with 2013 data for the major indicators. This will be lower than the total number of schools in England as not all of them had data for all grades in the 2013 performance tables. Ofsted grades are correct as of 31 January 2014.

Advertisements

5 thoughts on “Does Ofsted have selective judgement?

  1. This analysis is fascinating. It demonstrates, as far as I am concerned, a bias which cannot be denied. I would be most interested to hear an alternative explanation of these patterns. I think that this “smoking gun” is evidence of extreme bias – and whether that is on behalf of the work of the Ofsted teams (perhaps under the influence of the ‘Halo Effect’, ignorance of the nature of selection or just old fashioned snobbery?) or in the Framework which the teams are applying it is clear evidence beyond reasonable doubt of partiality and prejudice which rests at the door of Ofsted (it is their Framework after all and if it is flawed who’s job is it to address this?).
    Perhaps what is needed is not a new framework. Perhaps Progress 8 will be the huge step towards a more level playing field which surely all schools, and the communities which they serve, deserve. What I would expect, however, is for Ofsted to be self reflective and self evaluative in the way that school leaders are expected to be. Ofsted should be analysing their own data and addressing the questions that this raises.
    This is strong evidence for a need for Ofsted to “close the gap” between schools in different groups. It’s now time for Secondary Moderns (and all others who desire an inspectorate which makes meaningful judgements in order to promote improvement and share best practice) to call for some answers. Ofsted claim (in the “Who we are and what we do” section of their website) to be “independent and impartial”.
    The evidence, however, suggests otherwise.

    Like

  2. Very interested in this as I am involved in Comprehensive Future campaigning for an end to selection at 11, You mentioned the 125 secondary moderns, with 163 grammars I would think there should really be about 3 times that number of schools designated as secondary moderns. But in the past many secondary modern schools became officially designated as comprehensives – which in terms of intake they are not if they are in the 15 or so local authority areas with a strong element of selection. There are 36 LAs with grammars across the country. England is not really comprehensive, selection has a big effect on secondary education.

    Like

    • Yes, you make a good point. Perhaps someone with more local knowledge than I could repeat the analysis after re-classifying. One thing that surprised me though is the large proportion of secondary moderns with above average KS2 average points scores – 29% of secondary moderns have higher KS2 APS than the national average. Of course, this doesn’t really matter at a local level which is where selective systems have their impact. I wonder whether it is admission policy or simply prior attainment which is impacting the likely Ofsted grade – probably meat for another analysis.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s