Chart hit: or, how years of preparation made the ministerial resignation chart an overnight success…

Summer 2022. A much-covered chart hit that was suddenly everywhere again, revived by a binge-watchable boxset drama.

Not Kate Bush’s Running Up That Hill, given a new audience by Netflix’s Stranger Things, but the Institute for Government’s chart of ministerial resignations outside reshuffles. Popular during Theresa May’s premiership, but given new life by the spectacular collapse of Boris Johnson’s government.

The chart is archetypal of how we tried to do data at the Institute for Government (IfG). It built on earlier charts on different subjects and data gathered and refined over years, combined domain and data expertise, and benefited hugely — in data gathering and distribution — from working in the open.

When I arrived in 2013 to run the IfG’s Whitehall Monitor project (which analyses data about the size, shape and performance of government), our main focus was on data around the civil service. But colleagues had started developing a ‘ministerial database’ (in Microsoft Access, with data dragged through into Excel) to try to quantify and understand reshuffles and the disruption they caused by moving ministers.

We published our first analysis from the database shortly after David Cameron’s October 2013 reshuffle. It’s primitive in many ways: Whitehall Monitor updates between annual reports were published as PDF bulletins on a microsite rather than blogposts or explainers (on the web, but very much not of the web) and the charts leave a lot to be desired. (Compare the colours, axis lines, titles, text and even bar thickness of the prototype with the more developed product we used for years afterwards.)

Our first attempt to liveblog a reshuffle as it happened, with charts and insights from across the IfG, came in July 2014. We’ve done every one since, including Theresa May’s January 2018 reshuffle. That unfolded just as we were going to press on Whitehall Monitor 2018, which meant rewriting an entire chapter in a day while continuing the live analysis. Our work helped change the media narrative, from ‘nothing has changed’ (the press thought there’d been less movement than expected and focused on ministers who refused to move) to ‘six culture secretaries and six justice secretaries in less than eight years is quite a lot of disruption, actually’. Our ministerial data analysis could have been reshuffled out of our repertoire in early 2016 as we reviewed the project and had to prioritise, but feedback suggested our data-informed approach to reshuffles was useful and unique. Watching Evan Davis cite our analysis on Newsnight, seeing it pop up on the Guardian liveblog and all those lovely retweets suggested we made the right decision.

As 2018 wore on, there was a general sense that Theresa May was suffering more ministerial resignations outside of reshuffles than ever before. Ministerial resignations fell into that category of basic information you thought someone would have collated but hadn’t, a classic example of easy data that was hard to find. There were some useful sources — Butler’s British Political Facts, Wikipedia, the Commons Library, a journal article by Anthony King and Nicholas Allen — but none that were comprehensive, no canonical version of the truth.

So we set James, one of our research assistants, to work, compiling a list of all resignations outside reshuffles since 1979. I added some more, using our ministerial database for a new purpose: working out the dates of reshuffles and finding ministers who’d left office between them. We were worried we’d missed some, so we published the spreadsheet and invited comments (a major hat tip to Mr Memory on Twitter, among others). As it became obvious that the number of resignations under May was not, to coin a phrase, normal, Alasdair took the list back to 1900.

We started classifying the reasons for resignations (standards, accountability, disagreement, personal). While that’s not been fully analysed, it made it obvious that the nature as well as the number of resignations was not normal: May was experiencing more resignations due to political disagreement than any of her predecessors. Indeed, that was the story that came through most clearly in our first attempt to visualize the data, although the congestion on the chart makes the point about volume, too.

I think it was Julian — who built the original ministerial database — who first suggested that a step chart showing the cumulative ‘run rate’ would be the best way to show the sheer numbers. Luckily, we had a template for that — a similar chart showing the number of women in Cabinet over time. This was an area chart rather than line, but built on the same principle in Excel: use a date axis, and if a number changed on a particular day, insert a data point for the day before the change, giving you the straight line from the previous change and the step effect on the day with the change. This more faithfully illustrates the data than just using a line chart, as you can see the cumulative number at any given point. We also used a similar chart for government defeats in the Commons, which we set to music. (I started playing with a byelection ‘run rate’ per parliament chart as well, which included using Excel formulae to make it less fiddly to update. It’s possible I may also have used this chart type to illustrate certain rugby union matches.)

As May’s trickle of resignations turned into a steady stream, our chart started to get more attention. This was partly because the story was very clearly conveyed: more resignations in a shorter timeframe than any other prime minister.

But it wasn’t just the original that circulated: several news outlets published their own cover versions, all with full credit to the IfG for the data (like this one from BBC News). This was down to us publishing the data openly in a Google Sheet. Some journalists used it directly, others got in touch to ask questions. Some outlets changed the numbers slightly, and understandably so: a ‘ministerial resignation’ sounds like it should be easy to define, but isn’t (what if you announce in advance of a reshuffle that you’re stepping down? What if you use the opportunity of a reshuffle to very clearly quit? What about sackings? We could have done more to explore these challenges and make our own working even clearer on the spreadsheet). That our step chart was such a good graphical representation of the political tumult meant most followed the chart design, too, but with the subtle shifts of their organisation’s own style guides.

Making it onto Have I Got News For You was probably the highlight. ‘A chart on a comedy show?’ Indeed.

While our open approach on resignations worked well, it worked less well on other occasions: when we set up a Google Sheet to track MPs announcing they were stepping down before the 2019 election, the initially useful crowdsourcing quickly collapsed into various highly original comedians adding ‘Boris Johnson’ and others to the list. Choose your access settings wisely.

The resignation chart was already proving its worth under Boris Johnson. At various points — particularly the start of his premiership — the resignation run rate was ahead of May’s. But it really came into its own as the stream turned into a deluge, the skyscraping straight line showing how bent out of shape Johnson’s administration had become.

It’s never a good sign when an IfG chart has a timestamp. We expect it of some of our portfolio — the Cabinet moves chart, a staple of our reshuffle analysis, for example — but probably didn’t expect it of this one. That said, IfG charts are designed to make them as easy as possible to update quickly as new data is published, one of the benefits of having built up a set list of standard charts that have been refined over many years and data updates.

In Tim’s very capable hands, the chart went everywhere in its own right, again remixed by others, from Sky News to the Telegraph to The Economist. Again, the open nature of the data meant others could adapt it: Statista and PA added parliamentary private secretaries (the aides who are less the first rung on the ministerial ladder, more the ground beneath it), while the New Statesman went with the number of resignations in one go (again, here’s one we made earlier, which made a Telegraph headline). There were new jokes about R numbers and exponential growth. But there was also a consistent thread.

Now that the flood has abated — at least until the next PM takes over — what does the success of the ministerial resignation chart tell us about dataviz?

Bringing together domain knowledge — about ministers and their movements — with data/dataviz knowledge meant the IfG had real credibility. That dataviz knowledge was developed over many years, built on previous work that we improved iteratively over time. We tried different ways of visualizing the data and worked out what would work best.

Working in the open had huge benefits — it gave us better data and better distribution.

There’s nothing hugely sophisticated in what we used — Microsoft Excel and Google Sheets, both readily available — although our knowledge of how to use them had been honed over years.

Data which we thought might already exist, didn’t. Building our canonical list was the most time consuming part of the process (and needs maintenance), but paid off.

And while everyone intuited that something unusual was happening with resignations under Theresa May, bringing the facts together and the numbers to life was hugely powerful.

Let’s see how the next Prime Minister fares on the chart…

For more on how to do data visualization as a thinktank, seemy piece for Smart Thinking.