Update 1/2/2011 around 2:30 PM: my file-sharing host is down for maintenance, so all charts are unavailable. Will check back periodically- I'd expect them back tomorrow
This is my first attempt at a (real, stat-oriented) FanPost.
My last class in statistics was 30 years ago (and it was right after lunch, and the guy had a high-pitched whiny voice, so my attention even at the time wasn't all that great), so I have forgotten most of what I once knew as I think I've amply demonstrated to the stat-savvy in numerous threads. When it comes to excel- I don't, so my skillset and knowledge set wrt stats could be termed 'autodidacticism'. I have almost certainly misused some terms but just try to follow the process, and judge that.
Back in the 2009-2010 offseason, danmerqury wrote a piece The Siera Club where he introduced SIERA to AN. His piece had the virtue of presenting, in easy-to-read form, what was essentially a predictor matrix for 2010 for A's pitchers. Recently some commenter linked to that piece and I was astonished at how much nearly all the pitchers from 2009 who continued with the A's in 2010 outperformed the predictions. I got further interested in which of the stats did best or worst in the context of the A's, and what this might mean for 2011.
stole used dan's research as a basis for my own and hereby present it in modified form beyond the fold. If this is a no-no, kindly tell me and I won't do it again...
- How to read the above: the green column represents actual 2009 ERA's for the pitchers dan listed who also pitched for the A's in 2010.
- The next 4 columns show the 2009 FIP, xFIP, tERA (from the link: to get to the formula for tERA vs. tRA, scroll down and read the discussion), and SIERA dan either figured or provided for his chart. Since, in each case, the rationale for the stat's existence as presented by the inventor is to improve on the predictive value of "this year's ERA", I have called these stats predictors throughout this piece.
- The final column shows the actual ERA for 2010.
- If a cell is white, it means the ERA was predicted (for the columns except the final one) or was actually (for the final 2010 ERA column) within 0.50 runs of 2009 ERA.
- If the cell is orange, it means the 2010 ERA was either predicted to be or actually ended up at least 0.50 runs worse than his 2009 ERA.
- If a cell is blue, it means the 2010 ERA was either predicted to be or actually ended up at least 0.50 runs better than his 2009 ERA.
- If the value in a cell is bold, it means that the 2010 prediction and/or the 2010 result was at least + or - 1.00 from his 2009 ERA. funny from the New Yorker
Some things become apparent right away: as noted above, some pitchers outperformed just about every projection- by a lot. All the stats predicted that Cahill would actually pitch even worse in 2010 than he did in 2009. As we know, his results were far better than those predictions. All the fundamentals revealed by these stats showed that Gio's 'stuff'' was better than his ERA in 2009, thus predicted that he would improve in 2010, and he did: he outperformed by more than a half run over every stat on the board- even though every stat predicted he'd improve by more than a run over 2009. On the basis of his 2009, none of the predictors picked Breslow to be good in 2010. He was. On the flip side, none of the stats predicted Sheets' awful 2010 numbers, nor Wuertz's.
Most stat-savvy writers on AN agree that the lower-than-expected ERA's put up by A's pitchers is because of our great defense in 2010 (and not because of their 'stuff' per se), and also expect that the 2011 A's fielding will regress. PaulThomas even put a number on the expected effect of the A's fielding on the A's pitcher's actual ERA's versus their FIPs: he said he would expect a .3-.4 improvement in their ERA vs. their FIP. (Qualifier: I don't know if he meant that improvement should have been applied to their 2009 stats, or to their expected 2011 stats based on their 2010 stats...) Of course there is raging controversy over whether Cahill is doing something unusual or is just lucky. So, the next question should be: by how much did the 2009 predictor stats miss the actual 2010 stats for the A's pitchers?
So, next I found each of the values by which each pitcher's actual ERA varied from the expected ERA as established by 2009 ERA, FIP, xFIP, tERA and SIERA. I called these v-numbers (v for variance). If the 2010 ERA was lower than the predicted value, I called the resulting difference a positive number. If the 2010 ERA was worse than (underperformed) the prediction, I called it negative (the reason for reversing these stats is that if a 2010 ERA is better than the 2009 stats, that is a positive improvement, even though the number is lower).
- In this graph, what matters is the accuracy of the prediction.
- I've used a (readable) palette sort of like a rainbow: best is blue, worst is pink:
- blue = a prediction that came within + or - 0.49 runs of actual ERA
- green = between + or - 0.50 to 0.99 runs of actual ERA
- yellow: between + or - 1.00 to 1.49 runs of actual ERA (not so close!)
- orange: between + or - 1.50 to 1.99 runs of actual ERA (pretty far off?)
- pink: more than + or - 2 runs off from actual ERA (off>awful)
None of the models got within 1.50 runs of Cahill's actual ERA, and they all did just as poorly with Wuertz, too. The predictors did fairly well with both Anderson and Mazzaro, two very different pitchers. Another outlier is Ziegler: all of the models were excellent at predicting his 2010 pitching, unlike any other pitchers.
Look at the columns now. At a glance, it appears that the best predictor of 2010 performance is 2009 ERA (there's more blue and less yellow, orange and pink than any other column). However, we need to consider significance: the advanced stats, unlike ERA, are designed and tested against mounds of past statistical data to see if on average they indeed do outperform ERA as a predictor of next year's ERA- if they do not, they don't get published...
What is the fairness of just looking at misses for each pitcher, no matter how many innings or batters the pitcher faced? Is a miss on Wuertz as significant as the same-sized miss on Cahill? Not in my book...
So, I decided to measure 'significance' in order to properly weigh the variances. For 2009 ERA- treated here as a predictor (however poor)- also FIP and xFIP, I used a weight based on the number of innings pitched by each pitcher vs. the total number of innings pitched by this group of pitchers. For tERA and SIERA, which are both more based on plate appearances against each pitcher, I used a weight based on the number of total batters faced (PA) by each pitcher vs. the total number of batters faced by this group of pitchers.
I called the results either IP Sig or PA Sig and multiplied these results by the relevant 'variances' from the chart above, to arrive at a total number of runs per 9 innings each predictor varied from the actual number of earned runs allowed by this group of pitchers in 2010.
- the numbers 1.003 and .9999 above are just 'check-sums'- they show that "IP Sig" is slightly less accurate than "PA Sig"- by .0002 due to rounding issues, that's all.
- What do it mean? Well, looking at the numbers in yellow, the ERA of this group of pitchers improved by an average of .755 runs from 2009 to 2010. Thus, the yellow band shows the "net" variance -in runs allowed per 9 innings- of this group of pitchers taken as a whole.
- 2009 ERA missed by more than 3/4 of a run in predicting 2010 ERA. So, how did it do, when measuring A's pitchers vs. the 'advanced pitching stats'? Not too shabby, actually. Middle of the pack, actually. Much better than SIERA and xFIP, actually...
- Each of the advanced stat predictors also expected an ERA for the whole group of between .728 and .931 higher than the actual ERA. Compare that to PaulThomas' estimate that the A's defense might lower ERA by .3-.4 runs.
- Finally, when the total misses were summed without regard to whether they missed on the upside or the downside compared to the actual ERA's, we get the green numbers- here we can see that 2009 ERA actually was the least accurate, and SIERA did the best, which is a good thing, since this is exactly what the statisticians would expect...
Finally, for any who read all the above and still wish to rely on 'luck' as a significant factor in this variance of .755 in ERA, the sum total innings pitched by this group is 1196, or nearly 6 average seasons for a starter. Cahill's numbers could be luck. The group as a whole? Nah. So, What can we do with this miss-information? We can call the misses "defense"- whether or not they are actually due to pitching prowess or defensive prowess or park effects. The 2010 A's "defense" lowered the expected ERA's of this group of pitchers by about 3/4 of a run per 9 innings, no matter which metric one uses!
Will the 2011 A's "defense" regress? Let's assume so. Let's further assume that the 2011 A's "defense" will regress halfway back towards a league-average defense. This is erring on the cautious side, I think, but it also allows us to use the PaulThomas prognosticator of about -.3.50 to evaluate our A's pitchers' projected results vs. their 2010 FIP, xFIP and tERA.
Thus to the final chart (I have left Mazzaro, Sheets and Duke off the chart, because they are traded away, will be rehabbing all year or are currently not signed...)
- It should be self-evident how to read this chart:
- "Ave pred" is just the average of 2010 FIP, xFIP, tERA. Even though 2009 ERA performed just as well as any of these stats for 2010, i didn't include it in the averages because of the next stat- it would have skewed the "Def Adj".
- "Def Adj" is the effect of the A's "defense" where "defense" stands for everything not covered by the fielding-independent stats, and is on the low side of a regression halfway back to league-average.
- "2010 Est" is "Ave pred" - "Def Adj".
- the three numbers in hideous pink: Bailey and Cahill's 2010 ERA's are far away from the three predictive stats, Bailey's even more so than Cahill's, and I thought you'd want to know that. I didn't make a defensive adjustment for Ziegler because all the stats did a very good job predicting his 2010 ERA...
All in all, it doesn't look all that bad, certainly not as bad as a quick glance at the staff's advanced stats might suggest- and there's just not a huge fall-off from Anderson>Gio>Braden>Cahill. Good news!