Opponent-adjusted metrics: Part 3, using opponent-adjusted margin to evaluate managerial changes in the 2012-13 Premier League season

Introduction

In the previous three posts, a mechanism for opponent-adjustment was first explained for opponent-adjusted record in college football, before being applied to margins of victory and further applied to find expected records for each team, given their average and standard deviation of margin of victory.

While simple opponent-adjusted margin of victory could be calculated for each team in the 2012-13 Premier League, the round-robin nature of the Premier League (in contrast to college football, whose opponent-adjusted margin of victory has previously been studied) means that the outcome of such a study would align with each team's goal difference. Instead, scenarios are required where the pool of teams faced differs. Because of this, teams which changed managers are considered. There is some overlap between the teams faces by each manager of a given club, however, depending upon the point in the season at which the change was made, the list of teams faced by each manager can vary quite significantly.

In this post, the Premier League is used and opponent-adjusted margins of victory calculated for the 5 teams which changed managers during the 2012-13 season. This allows the relative performance of each manager (the one who leaves and his replacement) to be compared on a level playing field.

Method

The calculation for average opponent-adjusted margin is as described in my previous post, on the 2012 college football season.

Teams included in the study are those to have changed manager after the first game of the 2012-13 season and before the last game. These are -- in chronological order of managerial change -- Chelsea (di Matteo was replaced by Benitez), QPR (Hughes/Redknapp), Southampton (Adkins/Pochettino), Reading (McDermott/Adkins) and Sunderland (O'Neill/di Canio). 

Each team is treated in the calculation as two separate teams, one pre-managerial change and one post-managerial change. However, the calculations are carried out separately for each of the 5 teams. So in each calculation, there are 21 "teams" involved: the team in question before the managerial change, the team in question after the managerial change and the other 19 Premier League teams.

This process gives average opponent-adjusted margins for each of the 10 managers.

Results

A graph showing the performance of each team before (left column) and after (right column) the change in managers is shown below:

As can be seen, the changes were made with differing degrees of success:

• Rafa Benitez achieved a marginally better performance than did Roberto di Matteo
• Harry Redknapp improved the QPR team from the level of performance under Mark Hughes. The team remained poor, however, despite a large January outlay on players
• Southampton's manager change was a success, with Pochettino taking what had been a below-average side under Adkins and turning it into a (nearly) average team
• Nigel Adkins again came out on the wrong side of this comparison at Reading, where the performance under him was worse than that under his predecessor, Brian McDermott
• Paolo di Canio enjoyed a good result against Newcastle United and secured Sunderland's survival, but in terms of opponent-adjusted margin, his performance was worse than that under Martin O'Neill. This is discussed in more detail below

Di Canio and O'Neill

To the observer, it seemed like Paolo di Canio enjoyed a successful run at the end of the season at Sunderland, improving performances. However, this metric calculates that performances under him were worse than those under O'Neill. Why is that?

• Each manager's performance was given as an average for the whole season. So although di Canio took over a side which had been struggling recently, these performances were not representative of the season as a whole
• di Canio beat Newcastle United, 3-0, at St James' Park. This result can be overstated by the observer, since it was a derby game, away from home. The calculation simply sees a 3-0 win against another below-average side: still a good result, but given no more or less weight than any other game played under di Canio
• di Canio's Sunderland were beaten 6-1 by Aston Villa, who were poor for most of the season. This counted strongly against di Canio
• di Canio is an interesting and divisive figure about whom much was written in the press. It is easy to confuse this attention with good performance. In fact, his record as manager was P7 W2 D2 L3 F:A 8:11.

Similar points could be made concerning the calculation for each of the other managerial changes.

Conclusion

This work gives a good picture of the relative performance of the teams considered, before and after mangerial changes. The results are easy to understand and present.

The main shortcoming here is the lack of a "form" element. This means that improvement or regression of teams during a season is not captured by this model. So if a team starts the season badly and faces one of the managers studied during this spell, before improving through the season and facing his replacement later on, each manager is judged to have faced a team of the same ability. Form is a very difficult problem to solve, but one which I will attempt to deal with in the future.