Friday, March 11, 2011

A Few Thoughts on Scouting Reports and Sabermetrics

If you've visited this site at all over the past month or so, odds are you've come here to view a scouting report. Of my ten most popular posts (by page hits) this year, seven have been a scouting report of one player or another. Clearly they're a big draw. Whether you're a regular reader of CiF (thank you) or have stumbled in here because you found your players scouting report via a Google search or followed a link from another site, you're still here because you hope to find information that can further your understanding of one player or another.

The ugly truth however is that no matter how good of a scout someone is, no matter how large the viewing sample size, projecting players using just your eyes is really, really tough. This is pretty obvious of course, and in all likelihood you've heard it before. Yet fans as well as teams place enormous amounts of responsibility - tens of millions of dollars worth on an annual basis - on the subjective observations of scouts. It's an incredible leap of faith when you think of it in that way.

Of course, a lot of the time scouts get it right. Regardless of how big of a crap-shoot the draft is (and make no mistake, it is a huge crap-shoot) the bulk of the MLB talent still comes from the first round and the rounds that follow closely. There's a reason for that, and it's because scouts generally know a high-end pro talent when they see one enough. Yet players, Hall-Of-Fame players even, constantly slip through the cracks falling into the middle rounds, or even the obscure depths of the deep rounds.

How is that every Major League team passed on the decades most dominant player (and one of the greatest right handed hitters of all time), Albert Pujols, for 13 rounds? How is it that team after team thought his defense was ok, but didn't think his bat would play? Really? Teams passed on Greg Maddux until the second round because they didn't think he was 'physical' enough. The fact that his stuff was better than they had seen in a long time seemed to not be enough to sway them. One scout observed that he felt Maddux lacked enough control... In fairness to that scout he only saw Maddux once and it's really tough to get a great feel of any player or how they'll develop even if you watched them for a full season.

But that's just the problem isn't it? We never actually see a player enough to know everything there is to know. Not even a team of scouts with the best training and the best organizations behind them will correctly identify every good player, or even most of them. It's just not that easy.

Now, not to undo all the work I've done over the past month or so, but understand that when you read my scouting reports it's important to remember that they're likely to be incorrect in one or more ways. It's also important to remember that just because one scout got it wrong (or even a fleet of them) it doesn't mean that scout did their job poorly or doesn't understand the game. Even the best players with the greatest talents fizzle out. Some times because they don't have the necessary work ethic, poor player development, and even more times, because of injuries.

To help try and mitigate some of that uncertainty, we've come to understand that numbers, when utilized by those who can properly parse them, can be a huge tool in increasing the accuracy of a players projection. There is a reason after all that so many teams have made such significant investments into their sabermetric teams - it's because it works.

Still, even the teams who do the best jobs of blending sabermetrics with good scouting - the Rays and Red Sox come to mind - are going to miss. A lot. That's just the nature of the game.

For my part of course, I try and blend the two. Doing so I feel helps in a couple ways. On the one hand, you're giving yourself an opportunity to spot discrepancies. Perhaps a players hit tool comes in way over expectations - with sabermetrics we have a way of determining whether that player was improperly evaluated or whether they simply had a fluky year and the initial valuation was wrong. Or maybe the player has simply improved and we need to re-evaluate them.

The second way sabermetrics blended with scouting can help is in spotting consistencies. Ideally what you'll have is a situation where your scouting report and your statistical analysis will line up. In that event you can start to feel even more confident of your evaluation process. That's key for teams when they're trying to determine whether or not they want to promote a player, or trade for someone elses player. It gives you a crack at finding someone who is under valued or trading away someone who might be over valued.

In the end it's important to understand what scouting reports are: snapshots in time of a players career. One game, or a few games in what is a career filled with hundreds or even thousands of games. A game or five which isn't representative of how their bodies or skills will develop with time. A scout can't predict when a great player will eat himself out of shape and sabermetrics (for the most part) can't project for his work ethic.

Instead when we scout a player we use our observations combined with a body of accumulated lifetime knowledge about how players develop, and make what amounts to a highly educated guess. When we use our sabermetic knowledge, we make try and use mountains of historical data to project how players "usually" develop. We can be really specific with that data - far more than most people think. You can break it down to how 18 yearolds who are 6'2" and weigh 180lbs develop as hitters. Or even more specific, there is enough data out there at this point that it's possible.

Even with all our data and all our observational genius, scouting and player projection are still far more art than science. And a very imperfect at that.

So know a few things when you read my reports:

1) I work really, really hard on them. They're all compiled after having watched those players and studied their performances closely.

2) I then cross check all my work with the observations of others. Then I double check my own work again.

3) Despite all of that, only about 30% of what I say will end up being correct. Maybe (probably?) not even that much. Another 30% will be able to be interpreted as right or wrong. The final 40%? Well it's likely to be completely wrong.

That's not because I'm bad at what I do. I'm arrogant enough to think I'm really good, actually. That's just the really unfortunate nature of the business.

So when you're reading my reports, know that the information you're getting on a player is really, really good. Know that the sabermetric evaluations and predictions are valuable and important. Just understand that a lot of it will eventually prove to be wrong because players development seldom follow our neatly planned designs.




Corey Ettinger is a Senior Writer for Baseball Digest as well as a proud contributor to both 612Sports.net, 312Sports.com, and 313sports.com. He also provides extensive analysis of the American League Central Division at his own blog, AL Central In Focus. Be sure to follow me on Twitter @Coreyettinger for the latest updates, random thoughts and general tomfoolery.

2 comments:

  1. I think this is very important, and is a point that often gets overlooked by both groups. Too often, people feel like it's scouting vs. sabermetrics, and the book Moneyball didn't help very much (often because the readers misunderstood the point of the book).

    Just like you're doing, the best way to judge a player is by using a combination of the statistics and scouting.

    ReplyDelete
  2. I'm not certain if the the book didn't help - I thought it was wonderfully written and incredibly thought forward. Like you said, I think people who didn't understand it at all - or worse, didn't read and still felt worthy of commenting on it - hurt.

    The thing is, when you get right down to it, the book Moneyball isn't necessarily about sabermetrics. It's about finding undervalued assets and investing in them. It's a wallstreet concept.

    If for example, people were suddenly placing too much emphasis on sabermetrics and not enough on scouting, then, theoretically, traditional scouting could become a moneyball asset.

    Moneyball wasn't about one facet of the game, or one set of numbers - it was about applying a very simple idea that has existed since the dawn of time, to baseball.

    ReplyDelete