Shutdowns and Meltdowns

Over at fangraphs, they recently proposed a fun statistic for keeping track of relief pitcher performances: shutdowns and meltdowns (actually, the idea was first proposed by tangotiger on The Book blog, but I'm going with the fangraphs version of the stat). We saw the full range of reliever performances last night, so I figured it would be a good opportunity to try it out.

The basic idea is to come up with something that serves the same purpose as saves and holds, but isn't so stupid. I don't think it is controversial, even among statistical traditionalists, to point out that a closer who comes in for the ninth inning with a three run lead and manages to convert the save isn't all that impressive. Or that the tensest moment in a game might come in the seventh inning, with a one run lead, the bases loaded, and one out. In that situation, a strikeout or double play ground ball can swing the entire game.

The shutdowns/meltdowns stat uses Win Probability Added (WPA) to quantify each relief pitcher's effect on the game. Win Probability is exactly what it sounds like -- just use the historical record of baseball games to calculate the probability of a team to win the game, given the base/out state. For example, at the start of the 4th inning yesterday, with zero outs and the Rangers up 1-0, the A's Win Probability was 38.3%. Then, Cliff Pennington grounded out to the pitcher and that probability fell to 35.4%, as precious opportunities slipped away. It's important to note that not all plate appearances are treated equally by Win Probability. The out that Pennington made to start the 4th reduced Oakland's chances by 2.9% but Ryan Sweeney's strickout to end the inning was only cost 1.4%. This leads us to Leverage Index (LI). The LI for a plate appearance is a measure of how important that PA is, relative to an average situation. Pennington's out hurt more because the Leverage Index is higher at the start of an inning than later on in the inning (because it is easier to start a rally if there are few outs). Of course, the real fun with Leverage Index comes in the late innings of close games, where the game result hinges on individual at-bats.

More discussion of WPA can be found here or here. In the AN stats primer, danmerqury describes it as:

A statistic that sums up a player's contributions to winning a single game. For example, before a batter's plate appearance, the chances of his team winning the game are calculated, and subtracted from the chances of winning after the batter's plate appearance. This change can be added up for one game or a whole season.

(Actually, I feel like I'm sort of invading danmerqury's territory with this whole fANpost).

Ok, after all that introduction, let's get to the fun part. A shutdown is defined as any relief appearance where the pitcher contributes positive Win Probability of at least 6%. A meltdown is defined as whenever the relief pitcher contributes Win Probability less than -6%. Six percent doesn't sound like a whole lot, but it was chosen so that the total number of shutdowns in the league over the past few years would more or less match the total number of saves plus holds.

Ok, now let's get to last night's game. I'll just go through the relief pitchers one-by-one to identify the best and worst performances.

• Jerry Blevins -- Blevins was brought in with two outs and nobody on in the sixth inning and the score tied 2-2. He got David Murphy to fly out, ending the inning and tallying 1.8% of Win Probability. Blevins remained in the game to start the seventh inning. At this point, the A's had a 3-2 lead thanks to a Patterson triple(!!). With a small lead in the late innings, Blevins Leverage Index was up to 1.91, so the situation is almost twice as important as an average one. Blevins did not help the situation by walking Smoak to lead off the inning. However, the Rangers sacrifice bunted (this play improved the A's Win Probability by 3.7% -- most sac bunts are counter productive!) and then hit into a fielder's choice, with Smoak out at third. At this point, Blevins was lifted so that Ziegler could face Andrus. Jerry's final line was 1.0 IP without allowing a run, but leaving a baserunner on first with two outs. In a close game, that makes for a total Win Probability contribution of 7.8% = SHUTDOWN (but a borderline one)
• Brad Ziegler -- Ziggy came in to face Andrus, but the seventh inning ended abruptly when Borbon was caught stealing. Win Probability doesn't assign contributions to defenders, so Brad gets the credit for that one. Things didn't go so smoothly in the eighth inning, though, as Ziegler hit Michael Young with a pitch and then gave up a home run to Josh Hamilton. Hamilton's HR swung the Win Probability by 51.6%, as Oakland went from a 1 run lead to a 1 run deficit. Ziggy was pulled after giving up a double to Kinsler, leaving the game with a blown save and Win Probability contribution of -47.2% = MELTDOWN
• Darren Oliver -- Meanwhile, on the Rangers side, Darren Oliver pitched a perfect top of the 8th but then ran into trouble in the 9th. With a 1 run Texas lead, Chavez led off the inning by reaching on an error from Andrus (but Win Probability assigns the blame to the pitcher) and then Rosales singled, advancing Chavez to 3rd but was thrown out trying to get to second. Oliver was pulled with 1 out in the ninth, runner on third, and his team hanging on to a 1 run lead (this adds up to an edge of your seat Leverage Index of 4.42). His Win Probability Added was -9.2% = MELTDOWN
• Neftali Feliz -- Feliz came in for Oliver and promptly hit a batter (Fox), then gave up three singles in a row. Over the span of four batters, the Rangers 1 run lead turned into a 1 run deficit. Feliz finished the inning, but his WPA was a ugly -45.5% = MELTDOWN
• Craig Breslow -- Breslow only faced one batter, getting David Murphy to fly out to end the eighth. There was a runner on second at the time, but this still only changed the Win Probability by 2.5%, so Breslow doesn't earn a shutdown.
• Andrew Bailey -- Bailey was brought in for the classic closer situation -- bottom of the ninth with a one run lead. And he blew it, walking Smoak to start the inning and eventually allowing him to score on an Andrus single. It could have been worse -- the Rangers could have gotten a walk-off win -- but Bailey's WPA of -28.1% was clearly a MELTDOWN
• Frank Francisco, Michael Wuertz -- The score remained tied at 5-5, as neither team scored in the 10th inning. Francisco allowed a couple of baserunners, but got out of the jam. These pitching situations with little margin of error are what shutdowns/meltdowns were created for, and both pitchers succeeded to the tune of +14.9% Win Probability = SHUTDOWN
• Chris Ray -- This is what happens in extra inning games: all of your good relief pitchers have already been used, so you throw Chris Ray and his 6.51 xFIP into a high leverage situation and he gives up a HR to Daric "I made 5 outs but still had a WPA of .584" Barton. Ray's WPA was -28.1% = MELTDOWN
• Tyson Ross -- Ross came in for the bottom of the 11th with a one run lead and gave it up on an RBI single from Julio Borbon, resulting in yet another blown save for Oakland. The A's had a 78.1% Win Probability at the start of the inning, but that fell to 50% by the end. This would have been a big old meltdown for Ross, but Geren was running out of relief pitchers, so Tyson stayed in for the 12th and 13th innings. He didn't allow any further runs to score, and Daric Barton came through with the game winning RBI. So the finally WPA tally for Ross was +8.7% = SHUTDOWN
• Darren O'Day -- Scoreless top of the 12th, +14.9% WPA = SHUTDOWN
• Dustin Nippert -- Gave up the winning run in the top of the 13th, -28.1% WPA = MELTDOWN

So there you have it -- one game, five shutdowns and six meltdowns, and the A's are tied for first! Just to wrap this up, let's take a look at the shutdowns and meltdowns for the Oakland bullpen so far this season (all stats available at fangraphs). The WPA column gives the total Win Probability contributed by each pitcher over all of their appearances. The gmLI column is the average Leverage Index for when the pitcher is brought into a game (i.e. the A's are in a sticky situation, so Geren turns to Michael Wuertz).