Of course, like most data geeks, I love the performance tables. In 1994, the publication of data about school performance was a revolution, and like most revolutions was greeted with horror and satisfaction depending on your prejudices. In the decade since, there has been much debate about the performance table data but usually with more heat then light being shed. We’ve been through raw scores, value-added, contextual value-added, and now we’re focused on national averages before our next change to Attainment 8 and Progress 8. But can we use this sort of data for improvement?
How good is the “Similar Schools” feature!
From a statistical point of view, the government hasn’t covered itself in any glory. The eminent statistician Harvey Goldstein has been particularly forceful in letting us know how misleading they can be, casting serious doubt on the the stated purpose of allowing parents to choose a school wisely.
But like the curate’s egg there have been some good parts. For example, in the summer of 2013, the tables added extra information about the 55 school most similar to your own, so you can see how you compare. Of course, “similar” is a weasel-word, and the similarity is basically only in terms of KS2 average points scores, but it showed how the tables can be used in a way to support school improvement. The similar schools table highlighted schools performing significantly better than your school but within 75 miles, so you could pop over for tea and biscuits and find out what makes them so good. This is the first time I was thought that the performance tables might actually help me improve my school.
According to my calculations the two most average schools in England are Boroughbridge High School and The Lancaster School, at least in terms of their KS2 average points and their 5A*-C+EM results. Like all schools they will be unique in a hundred other ways, but if you’ve never seen the similar schools pages, here they are for Boroughbridge High and The Lancaster School.
The list of similar schools makes very clear one of the pervasive and astonishing features of the data in the performance tables – how very variable school performance is, even when you try to find “similar” schools.
No surprise to those of us working in schools of course. Every school is strongly influenced by its recent history, the quality of its staff, the quality of its leadership and a hundred other contingencies unlikely ever to make their way into performance tables.
However, it’s always worth finding out “who’s doing better than us, and why?”. The similar schools table was a start, but limiting to the nearest 55 KS2 scores is a problem. Fifty-five is a good number to display on a web page, but there are over 3000 secondary schools in England so 55 represents only 2% of them. Some fairly similar really great schools could be missed. Also, tabular data is a drag. Why can’t we explore visually?
With modern web tools such as d3.js, it’s not that hard to mash up a 2-D graphical display, and here’s my effort. I call it the Swarm, because while each school is going up or down each year, the group is likely to move much more slowly. Here’s the landing page of the Swarm.
How can you explore performance table data with the Swarm?
First go to the Swarm website. Each disc is a secondary school placed according to its KS2 average points score and its 5A*-C including English & Maths (at least at the start). It’s a BIG graph, so you might need to zoom out to view it in a small monitor.
Hovering is your friend. Hovering over a disc gives lots of information about the school. Hovering over one of the dotted red lines shows the national average figure.
Click a disc to see the Ofsted page for that school. Then you can click through to Ofsted reports or the data dashboard.
Try a search. Enter the first bit of your school name in the “search for a school name” box. See where you are!
Or how about highlighting all the schools in a local authority? Choose a local authority from the LA list to see where all the schools in that authority area are.
By Ofsted Grade? Change the highlight type from “By School Name” to “By Ofsted” Beware! The list of most recent Ofsted grades was as of August 2013, so it’s a little out of date now (the grades up to December 2013 are published in March 2014 and I plan to update). Also, because there are nearly 3000 schools on the chart, each one has to be tiny. I have to zoom in big time to see the detail I need. (I can’t make the discs bigger as they’d all overlap).
Change your variables. The Swarm starts off with 5A*-C including English and Maths on the y-axis and KS2 average point score on entry for the x-axis. As you can see, there is a strong link (*correlation) between these, so with great average KS2 points comes great expectations of 5A*-C+EM. Referring to a simple national average is such a poor way to go about things.
You can choose a variety of alternatives for the y-axis including English attainment/progress, Maths attainment/progress and best 8 value-added measure. All of these are shown overall, for pupil premium pupils and for other pupils. I’ve also produced a “gap” measure based on the difference between pupil premium and other pupils.
For every graph the national averages for the x- and y-axes are shown as red vertical and horizontal lines.
Some thoughts arising from just eye-balling the graphs
Can we stop just looking at national averages? All attainment measures are strongly correlated to KS2 average points scores. By focussing on national averages, we’re tyrannising some schools who have low KS2 inputs and letting others with high KS2 inputs off the hook. Which was “obvious”, but worth saying. Again. With reference to the actual data.
Progress measures still correlate to prior attainment. The proportion of students making expected progress in English and in Maths still shows some correlation to KS2 average points scores. So we have a progress measure which is actually clinging on to prior attainment. Ofsted know this, which is why they have KS2-4 transition tables in RAISE, and part of the reason why we have to have national comparators for each level of prior attainment in them.
Hmm, except for the best 8 value added measure. The best 8 average point measure shows much less variation with average KS2 points scores. Good job, because that will likely make it more fit for purpose when it transmogrifies to Progress 8.
Variation, variation, variation. Variation of all these indicators with the proportion of disadvantaged students in KS4 is also high, but the correlation is negative i.e. the more disadvantaged students you have, the lower the indicators in general. However, for schools above the national average in terms of the proportion of disadvantaged the variation is huge. For example, Bath Community Academy has 53% pupil premium in KS4, almost twice the national average and scores 18% on 5A*-C+EM. St Thomas More Catholic School in Haringey has slightly more disadvantaged pupils at 57% but scores 91% 5A*-C+EM. These are extremes with the full range in between. Clearly, the impact of pupil premium is still too variable, even for those school receiving large amounts of it.
Do schools with more disadvantage help the disadvantaged better? There is some indication that in schools with high proportions of disadvantaged, the disadvantaged do better (though there is huge variation). Contrariwise, in those schools there is some indication that the advantaged do slightly less well than in more typical schools. This is a little clearer for progress in Maths, but is still a mild trend in huge amounts of variation. We should probably focus on the huge variation.
The variation in the best 8 score is much more for the disadvantaged pupils than the others. This is partly to be expected – any minority comes with more variability statistically. However, this just adds to the general theme – schools ability to get the best out of disadvantaged children varies hugely, despite the best efforts of people like the Sutton Trust and the stiff accountability being thrown at pupil premium spending.
If your school isn’t in, it’s probably because there was no data for it in 2013 – perhaps an academy conversion or a very small free school. Let me know and I’ll figure out why. Sometimes there is data for some indicators but not others – in these cases your school will sometimes be visible.
I’ve chosen only 18 y-variables and 2 x-variables to keep the datafile small (it is still nearly a megabyte) but I’d be willing to add more if there is a useful indicator you’d like. I’d love to add per pupil funding as an x-variable, but this isn’t in the tables for academies (just over 50% of secondary schools) as they are not required to publish it.
I’d be keen to hear of people who have visited similar schools as a result of using the DfE similar schools feature or the Swarm.
I’d also appreciate feedback about the Swarm and how it can be improved.
Update 30/03/2014: The Swarm now includes primary school data. Because of the much larger number of primary schools, these come in regions e.g. London, Yorks & Humberside, rather than as a national swarm.
Update 26/01/2016: The Swarm has been updated to use the 2015 Performance Tables (published 21 January 2015) and most recent published Ofsted grades as of 31 December 2015.