Baseball Projections. How much do they tell us?

The biggest problem to me is that statistical analysis in any sense, can be used to proffer a wholly subjective argument that seems compelling. This goes double for emergent statistical tools. Too many political and entertainment junkies mistake these arcane tools as some final word in any projection of potential output going forward.

I realize this isn’t specifically on topic, but I’ll take this post as the latest salvo in the ongoing war between those enamored of their favorite analytical tools and those who would rather use a more qualitative approach.

It’s like a heated (but fun and harmless) exchange I had recently with another member here. My assertion was that, based on his on field performance and lack of notable progress from last year to this year, Gio shouldn’t just be handed a position in the rotation. I argued that, qualitatively and quantitatively, Gio still sucks. My fellow AN’er wanted to show that Gio had improved dramatically because a few select statistical contortions showed improvement or a likelihood to improve. Care was taken, however, to avoid those stats which might spell doom for Gio. For the record, I hope Gio turns it around and makes a fool out of me (but only if it’s for the A’s). No, I don’t want to continue the debate.

The point is this: maybe I’m not as smart about teh besbol as the stat-wizards. They clearly have a great love of the game. But I am smart enough to know when someone is using a statistical argument to prove a subjective point while acting like they “have the math on their side”. That’s some Karl Rove level chicanery. It’s all speculation, no matter how you dress it up.

I realize this isn’t specifically on topic, but I’ll take this post as the latest salvo in the ongoing war between those enamored of their favorite analytical tools and those who would rather use a more qualitative approach.

Unless I’m missing my guess, here, this is not one of the salvos in that war — the qualitative/quantitative war. This is taking a well deserved shot at the extra wide confidence interval producing quantitative models that attempt to project future skills progressions/declines. It does seem ridiculous had wide these models project their guesses.

I rec’d this post.

although I do like to still see a player before I make a judgement on him.

I guess this is one of those times where the stats and my lying eyes agree because I think Mr. Anderson is the best of this baseball bunch.

Does anyone else remember the baseball bunch with Johnny Bench or am I the only one old enough?

“huge” relative to what? and why is that bad? there is a wider range of plausible outcomes for pitchers/younger players/inconsistent players. I don’t understand what the problem is with those percentiles. Predicting what baseball players will do in the future seems pretty uncertain. So, why shouldn’t projections demonstrate the uncertainty?

The analogy to political polls does not make any sense. Political polls are saying with their margin of error that a 95% (or whatever) chance that their numbers are within a certain range of the “real” number, where the “real” number would be interviewing everyone. They are not projecting anything. No poll purports to predict what will happen in a year. That is completely beside the point of polls. It’s a completely different thing…

Since Stair’s median OPS is within Utley’s 80% confidence interval, statistically speaking, Pecota cannot be used to conclude that Utley is a better hitter than Stairs.

Projections are not a sampling of data.

by coming here

your argument only applies to projection systems. The complex stats that are used by us statheads on A’s nation most of the time (WAR, wOBA, FIP, tRA, etc.) are not projections, but simply means of classifying data. There’s no “margin of error” for WAR, because WAR is just a data set of already-completed events put through a formula to produce a number.
Sure there are problems with all these stats in their ability to show skill levels, but there’s no margin of error in WAR’s ability to show Wins Above Replacement.

the confidence level of the various sabermetric stats is 100%. Aside from the possibility that the person compiling them typed a wrong number into the spreadsheet, they are a perfect measure of what they seek to measure. They are no more uncertain than, say, batting average. You don’t ask what’s the margin of error on a guy’s batting average.

The question is what predictive power they have. In the list of ZiPS score, there’s no margin of error. That really is the correct ZiPS scores for those guys. How likely it is that the guy with the best ZiPS score will be the best pitcher next year is another question entirely.

I would also add that some of the stats are rather aggressively named. WAR, for instance, stands for “wins above replacement”. A guy’s WAR number is a completely reliable measurement of his skills as defined by the WAR formula. But if you proceed on the assumption that it represents that many actual wins over an actual replacement player, then you’re making an interpretation equal to your faith in the formula. The formula itself is based on analysis of past data, so it’s not just arbitrary or subjective, but even so it’s just measuring what numbers have correlated with wins in the past.

I don’t believe that Cahill is one of the five most talented pitchers on the A’s solely because of the ZiPS projection. It’s my opinion based on a combination of scouting opinions as conveyed by Sickels and Goldstein and BA, projections including ZiPS and PECOTA and CHONE, the judgement of the A’s, and my own eyes.

The fact that ZiPS thinks he’s ready to be one of the five best pitchers on the team based on his record up through the middle of 2009, means that there’s a strong argument that he should be in the rotation right now, no matter how bad he was in the first half of 2009.

Since ZiPS doesn’t provide confidence intervals or standard deviations, I choose not to worry about them, since it is the most respected mid-season projection that I’m aware of.

Since I assume that other posters don’t care about my amateur scouting or gut feeling on Cahill’s talent or readiness, I believe the best way for me to contribute to the discussion is to cite whatever evidence that I have access to.

If someone chooses to ignore evidence just because it doesn’t come with a disclaimer or confidence intervals that’s their decision.

That was easy. :-D

have basically nothing in common other than the fact that Nate Silver’s made money off of both of them.

The data involved are massively different. Poll aggregation (like what fivethirtyeight.com did during the 2008 election) is collecting data on future performance, i.e. a voter says “I’m definitely voting for Obama”. What baseball projections are usually doing is taking the past performance of a player, comparing it to players who had similar past performances, and using the continued performances of those players to predict the future performance.

When you get players who have a small number of comps like Giambi, such systems are going to give much less definite results because of a much smaller pool of data. PECOTA always chokes on Ichiro. Silver and the rest say it flat out, there’s no good comps out there for him. He is a truly unique player in the history of baseball.

I don’t directly provide the margin of error, but choose a different approach of having my projections available to any and all interested parties that want to evaluate how projections fare (and they are a lot of them floating around on the web). Simply put, the differences between ZiPS and PECOTA and CHONE are so minute and there are enough legitimate methods for determining accuracy that one can always make one of these systems “win” over another one.

Even if I go into that with the best of intentions, it still involves people, 99.999% of them not knowing me on a personal level, having to take my word for something and even if I don’t consciously pick the best method for making ZiPS look good, I’m also a human with a human ego and my brain will likely hint to me that an approach that makes me look better is better than an approach that makes me look worse.

I do post error ranges, but I present them slightly differently as a series of odds rather than percentiles (so BP would list the BA that the player has 10%, 20% and so on chance of achieving and I list what chance specific BA threshold have of being surpassed, etc.)

In-season projection margins are really tricky to do and present, simply because it’s not fixed. As the season advances, your sample size to base a projection off of increases, but the actual future play decreases, which results in the odd situation of rest-of-season projections being the most accurate in May (after game 161, you have the most information about a player to-date, but only 1 game left, so projections have a ridiculous error!).

For in-season, I do a relatively simple model, as I’ve noted. It has to be as simple as possible so that Appelman can have everything automatically update on FanGraphs, so it’s kind of using Ghetto Bayesian Inference on Excel.