The finish line to the Oilers Hackathon contest is within sight. This is the first of three posts on the fourth and final question.
Conduct a predictive analysis of your choice on some dimension of potential value to the Oilers. The analysis must be testable in the upcoming season and judged on its difficulty, accuracy, clarity, and value.
The fourth question will be submitted online, but will take the form of a PDF executive summary. The summary will be limited to one page of 10 pt font text that outlines the problem solved, data used, methodology, and predicted results. The methodology and predicted results must be sufficiently clear that the judges can replicate and/or verify the results. In addition to the one page executive summary, the Entrant may include up to four pages of annotated visuals or charts further describing their method and results. The source of any outside data must be clearly identified and publicly available.
The fourth question will be judged on four dimensions:
difficulty/novelty – how difficult and new is the analysis attempted by the Entrant
accuracy – how accurate were the predictions made by the Entrant
clarity – how clear is the reasoning, methodology, and results/implications. This includes both the written portion and any visuals in the submission
value – how valuable is the findings in helping the Oilers achieve their goal of becoming a perennial contender.
This post walks through the initial analysis I did to support my question #4 response to the Oilers. Today I look at the league on a year by year basis. Next up will be a post looking team by team. The third post will be my actual response to question #4. I should have that all done and on the site by the middle of next week.
My response will have a prediction element , but I was not really out to create a predictive model. Instead, I spent some time working through a thought I have had in my mind for a few years.
Those teams that spend the most time on the power play or the fewest minutes killing penalties are the teams that are successful. Success is defined as making the playoffs.*
* The playoffs are a crap-shoot so I can’t define success as making the conference finals or winning the Stanley Cup. If a team makes the dance they have a chance.
If a team spent the entire game on the power play, it should win 100% of its games. I haven’t tried to figure out a precise expectation, but a team with a 15% success rate should expect to score 4-5 goals per game (15% success rate on a two minute power play translated to 7.5% success rate per minute). A team with a 25% success rate should expect 7-8 goals scored per game.
Conversely, if a team spent the entire game short handed it would probably never win a game.
Sticking with the rough math, I think the probability of scoring an even strength goal is roughly 3-5% per minute. Obviously, being on the power play increases your chances of scoring while greatly reducing the other team’s chances.
To test my idea, I made use of one of the spreadsheets the Oilers provided to me and looked at every team for every season starting in 1997. That gave me 15 seasons worth of data. I compiled a net power play number for each team (defined as the number of power plays less the number of penalty kills). I then ranked each team 1-30 (or 1-to lesser number for years prior to expansion). For the top 16 teams on this metric, I looked to see if they made the playoffs.
I have mostly forgotten my university stats classes, so I stayed away from correlations or statistical significance. Instead, I looked at a few other metrics and again ranked 1-30 to see how many of the top 16 made the playoffs. If the power play differential metric was more predictive than the other metrics, my idea would have some merit.
Here is the list of the stats I looked at:
- Power play differential (power plays less penalty kills) (“PPd”)
- Goal differential (“GF-GA”) or (“Gd”)
- Shot differential (“SF-SA”) or (“Sd”)
- Power play goal differential (power play goals less short handed goals) (“PPGd”)
- A couple different blended weightings of PPd and PPGd
- Power play percentage differential (Power play percentage less penalty kill percentage) (“PP%-(1-PK%”) or (“PP%d”)
That last one, PP%d, is a bit of a play on the PDO concept, discussed in my question 3 response. The idea is that some teams will deliver a PP% that the PP% against them (i.e. they bring to the table some combination of better than average PP or better than average PK).
With that as the background, here are the year by year details followed by the averages for each methodology (click on that first graph to enlarge).
Not surprisingly, goal differential was the gold standard, predicting 14.5 of 16 playoff teams on average. That number seems to have decreased post lockout, likely due to the shootout and the impact of some games being worth two points and some games being worth three points.
The column on the far right looks at team shot differential. Of the top 16 in the metric each season, on average 12 qualify for the postseason. Of the five special team metrics I computed, only one managed to beat the shot differential prediction.
Starting with my hypothesis on PPd (number of power plays less penalty kills), it predicted a little over 10 teams per season. I am not sure if that is statistically significant or not, but it is far and away the worst predictor of the seven I computed. My question has been answered: spending the most time on the power play or the fewest minutes killing penalties is not enough to get a ticket to the playoffs.
The third column represents the power play goal differential, with 11.75 of the top 16 teams in the metric making the playoffs. In my mind, this number would be somewhat driven by the PPd. A team could rate highly on PPGd through volume of attempts instead of a high PP% or PK%. I attempted to quantify this in the next two columns.
The first was a simple average of the PPd and PPGd rankings each season. For example, the Oilers in 2012 ranked 27th in PPd and 13th in PPGd, giving a simple average of 20. That ranked 22nd in the league. On average this metric had 11.4 of the top 16 making the playoffs over the 15 year period.
The next column put a greater weighting on the PPGd metric. That moved the average to 11.5 teams in the playoffs.
Overall, blending the two metrics together did not increase the number of top 16 teams in the playoffs. Adding some PPGd results to the PPd numbers increased the average. Adding PPd to the PPGd results lowered the average. PPGd is a better predictor than any of the combined metrics.
The last metric I computed happens to be the one that best predicted playoff teams outside of overall goal differential. The 15 year average for top 16 teams in power play percentage differential is 12.1 teams in the playoffs. The metric is just as predictive as shot differential despite ignoring all even strength play. Interesting.
The 2012 Oilers provide a cautionary tale in PP%d, ranking 3rd (CORRECTION: 6TH) in the metric while finishing 29th overall in the standings. The Oilers were dynamite on the power play last year, converting on 20.6% of their opportunities (league average around 17%). But they drew 10 less penalties than league average and took 25 more than average.
The Oilers result made me wonder if I could improve on the PP%d metric. The following charts are the result.
Could I eliminate or reduce the Oilers-like outliers if I combined the power play differential with the power play percentage differential? A simple 50/50 weighting of the two metrics came in below PP%d on its own. 75% of the PP%d ranking plus 25% of the PPd ranking did nudge up my playoff average to 12.3 teams from 12.1 teams via PP%d.
For future reference, I will call this 0.25PPd + 0.75PP%d the weighted power play differential, or wPP%d.
The Oilers went from 6th in PP%d to 9th in wPP%d. Colorado went from 12th to 17th, knocking them out of the top 16. It appears that the weighted metric will bump the occasional bubble team out of the top 16. In the 15 years, the wPP%d never predicted fewer playoff teams than PP%d. That is a good sign the weighted metric is an improvement.
wPP%d predicts an additional playoff participant every 5 years or so which is something I suppose. I suspect someone with better stats skills could fiddle with the relationship to increase the gap over PP%d. I did not attempt, but normalizing PP% and PK%’s for teams that experience a huge year over year gap might help. That adjustment would get rid of unsustainable spikes in shot percentage on the PP%d results in a given year.
Translating this analysis to Hackathon question #4 is tricky. I could predict that 12 of the top 16 in the wPP%d metric will make the playoffs this upcoming season. That prediction wouldn’t score terribly well on judging dimensions outlined by the Oilers. I need to bring something to the table that might drive a change in decision making.
My next post will take a look at these metrics at a team level. I will be looking for trends that might suggest a team is about to jump into or fall out of the playoffs. If so, that might be instructive and somewhat valuable to the Oilers.