Seth Smith, Splits, Sample Sizes, and You

There's going to be a lot of talk about Seth Smith's splits in the coming months. Home/away, righty/lefty, all of it. This post is a friendly reminder of what splits can do...and what they can't. There are two big problems with using splits, and they both stem from sabermetrics' favorite "you can't do that" refrain, sample size issues.

Let me start with an example. If a player hits 3 singles over 10 plate appearances in two games, does that mean that his true talent batting average is .300? No one would possibly suggest it. What if he batted .300 over 50 plate appearances? 100? 5000? Where is the sample size line where we can be reasonably confident in our assessment of his batting ability?

Let's say that John Q. Baseballer has an OBP of .350 in 250 career plate appearances. How close to his true skill level is that .350 figure? It turns out that we can actually calculate the uncertainty in the OBP figure by using a formula derived from the distribution of OBP talent in baseball. Over 250 plate appearances, the uncertainty in John's OBP comes out to 0.030. For the math-inclined, this uncertainty is expressed in standard deviations, but to put it more simply, we can say that John Q. Baseballer's OBP has a 68% chance of being within 0.030 of .350 (in other words, it lies between .320 and .380). If we double this uncertainty interval to 0.060, John's true OBP now has a 95% chance of being within the interval (between .290 and .410).

Clearly, 250 plate appearances don't tell us much. But if we quadruple the amount of plate appearances to 1000 (about two seasons worth), the uncertainty now falls in half to 0.015, which is a far more useful estimation of his talent. See the big problem here? Quadrupling the sample size cuts the uncertainty in half. To quote Tom Tango in The Book, "if you want twice as accurate a measurement, you need four times as much data". So splitting a player's body of work into small splits results in uncertainties that grow into huge analytical problems. But that's just the beginning.

The second problem arises from the first. Let's go back to John Q. Baseballer.

Home: .370 OBP
Away: .330 OBP

Let's assume he has two full seasons under his belt with 1000 total PAs, half at home and half away. That seems like it should be a big enough sample to conclude that he has giant home/away splits, right? Wrong. Let me cite those splits again, this time with the appropriate calculated uncertainties included.

Home: .370 ± .022 OBP
Away: .330 ± .021 OBP

That certainly changes things, doesn't it? Remember, this means that there's only a 68% chance that his true OBP at home is between .348 and .392. That's the difference between a league average player and an MVP. And there's a 32% chance that it's outside that range! In nearly every use of splits you'll see in baseball commentary, you have not just one fuzzy number, but two, and they both have uncertainty intervals that overlap. With 1000 PAs split into two equal groups, there's a 3% chance that John Q. Baseballer is actually better on the road than he is at home! It's a small chance, to be sure, but there's a much greater chance that he has a home/road split that's smaller than average. We just don't know.

But enough about hypotheticals. What about Seth Smith?

Seth Smith is widely regarded as a guy who can mash RHP, but flails against LHP. His numbers make for a perfect illustration.

vs RHP: .377 wOBA
vs LHP: .262 wOBA

That's the difference between Evan Longoria and Jeff Mathis at his "peak". It's absolutely nuts. But are we certain that Seth Smith's true platoon split is that large? Just like John Q. Baseballer, I'm going to cite his splits again, this time with uncertainties. In his career, Smith has had 1209 PAs against RHP, and only 239 PAs against LHP.

vs RHP: .377 ± .015 wOBA
vs LHP: .262 ± .030 wOBA

Now it's pretty obvious—we can say with an extremely high level of confidence that Seth Smith is much better against RHP than LHP, even after assuming a hefty dose of "number fuzz". But there's another tool in the sabermetrics toolbelt called regression to the mean, which allows us to refine our estimate a bit. We have one more piece of information to use—Seth Smith is a left-handed baseball player, and left-handed baseball players generally have a certain average platoon split. In essence, we can "fill in" the uncertainty with league average plate appearances to give a sharper estimate of Smith's true talents. I'm going to skip the math, as there's no way I could possibly make it readable, but after regressing to the average LH batter's platoon split, Seth's splits cited above look more like:

vs RHP: .359 wOBA
vs LHP: .292 wOBA

Well, that's not so bad at all. It's certainly a larger split than normal, but it's not the Jeff Mathisian debacle that his non-regressed splits say. If Oakland plays Seth Smith as a full time player, assuming that LHP make up 25% of the pitchers in the league, we can expect a .342 wOBA. Not too shabby.

• Uncertainty and regression equations courtesy of The Book, by Tom Tango. Specifically, the appendix, which is very aptly titled Don't Try This At Home. Subtitle, The Gory Details.
• Billy Beane has already said that Seth Smith won't be used as a platoon player (link 1, link 2). Guess we'll find out how accurate these estimates are.
• It's been a lot of fun, AN. So long.