A frequent criticism against Billy Beane is that he has drafted poorly, and a common defense is that the draft is pretty much a crapshoot. Here's a look at the numbers, to see if we can shed some light on the argument.
At the suggestion of iglew, I looked at Baseball Reference, which conveniently gave me both the number of players who made it to the majors at each draft position from 1965 to 2011, as well as their total WAR contributions:
In the first graph, we can see a downward trend in the rate at which the players make it to the majors. 87% of players from the first two positions make it to the MLB, and 77% of players from the next two do. The actual percentage would presumably be higher, since these include very recent drafts. It is clearly no crapshoot at these positions, but the trend line crosses 50% at about the end of the first round, so the scouts' performance isn't really something to write home about.
Next, we use WAR to get a sense of what kind of players we're getting. Here, the contributions of the first overall draft pick stands out, at 799.2 WAR to the next best 506.9 WAR of the second overall pick. This would suggest that it's really important to get that first pick, since they average 19.5 WAR to the second pick's 12.4 WAR.
However, note the numerous little peaks above the trend line in the bottom graph. Why do the 10th, 19th, 20th, 22nd, and 30th picks do so much better than expected? Turns out, those positions yielded guys like Mark McGwire (63.1 WAR), Roger Clemens (128.8 WAR), Mike Mussina (74.6 WAR), Craig Biggio (66.2 WAR), and Mike Schmidt (108.3 WAR). What happens if we consider these guys to be outliers, and remove the contributions of the best player from each position?
We can immediately see from the top graph that that a lot of the unusual peaks have disappeared (into the green bars), which means that they represented drafting truly remarkable players who were not recognized early. The second graph, which is the average WAR after discounting the best player of each position, is also telling: beyond the 6th overall pick, the average WAR contributions rarely exceed 5.
The peak representing the first overall pick highlights how important it is to get this slot. They are dramatically better players: 25 of the 41 first overall picks who made it to the majors exceeded 10 WAR, and 16 of them exceeded 20 WAR. Only 18 of 41 second overall picks exceeded 10 WAR, and only 9 exceeded 20 WAR.
Now, I think we can all agree that selecting a Clemens as the 19th pick is pure luck, so it seems fair to exclude these players from consideration, and the conclusion that I'm coming to is that beyond the first five overall picks, there doesn't seem to be an obvious, high-probability way to find star players.
As for the A's, their best picks seem to have been Reggie Jackson (2nd overall, 74.6 WAR) and McGwire (10th overall, 63.1 WAR). Their best 1st overall pick was Rick Monday in 1965, yielding 32.7 WAR, but didn't do so well in the 2nd overall picks (Mark Mulder, Ben Grieve, Pete Broberg). Since 1965, the A's only had ten chances at top 5 overall picks, and these are chances that you weren't supposed to miss, so this had to really hurt.
Thus, I believe today's problem lies with not getting many shots at all at the "can't-miss" slots, and not having gotten lucky with the other slots. I'm inclined to excuse Beane or any other manager for this. I think perhaps people grossly overestimate the advantages of Moneyball, particularly when it comes to drafting young players.