Projections and Accuracy for hitters – a look back at 2016

When it comes to trying to determine who had the most accurate projections, there are a lot of things to consider. And, if you try to consider them all, you won’t have a post – you’ll have  a book.

You can use straight statistics, or you can bias-adjust. You can use whole stats. Rate stats. One-on-one comparisons for individual players. All stats. Just fantasy-relevant stats.

What I’ll do, here, is a fairly straightforward comparison between the numbers projected and the results. I’ll do some comparisons with and without bias adjustments, although part of your forecast skill should include not having large biases in the first place. And, for today, we’ll limit ourselves to hitters.

I’m going to be looking at average forecast errors for multiple statistics. If you’re going to do that, you really need to make sure that you are dealing with a consistent set of players. Having someone included in stat A but not stat B’s list can distort things badly. I eventually came up with a list of 256 players to form my focus group, if you will. These 256 were people that I thought, in the 2016 pre-season, would get at least 200 PA in 2016. But not just me – they only made the list if I, and Steamer, and Pecota, and Rotowire, all agreed they would get 200 PA, and then really did. We all made forecasts for these players, and they played enough to make a reasonable grade.

For projections, I have my own, of course, two varieties – the pure computer printout (“Autoclay”), and what I get from plugging those projections into lineups and depth charts and making my own judgments of playing time (“Clay”). I took these from copies I had on my computer from April 3; the latter were the ones I used for my own fantasy league drafts, last April 2 and 3. The fellows at FantasyPros.com helped me out by sharing copies they had made for Zips and Steamer (FanGraphs) last spring. I had downloaded stats from Pecota (Baseball Prospectus) and Rotowire, and I could run a variety of stats myself.

Now, the simplest, most naive forecast you could make, would be to simply use each player’s 2015 major league performance as his 2016 projection. That yields this

The numbers for PA through CS are simply the raw, average, absolute value of the differences, between forecast and reality, for all 256 players. The number for batting average shows the average difference, in hits, between projected and actual batting average, given real atbats. For example – suppose someone really went 100-for-400, a .250 batting average. The forecast was .293 – so the error is 17.2 hits (.293 minus .250, times 400). Similarly, the ISO number shows the difference in extra bases between actual and forecast isolated power. bbrate is the difference, in walks, between real and forecast BB/PA, for actual PA; so rate is the same thing for strikeouts. The “Sum” box is the simple sum of the 15 category boxes; “rates” sums just the four rate statistics; “roto” sums the R, HR, RBI, SB, and BA categories. So a 327 sum – that’s you’re naive baseline.

You can do a little better by taking 2013-15 average, instead of just 2015:

Your next step up in sophistication is to start adding some minor league stats into the ratings. When I run a simple 2013-15 DT for each player, and use that for my forecast – this will have no park factors, no adjustments to league average, and the PA will generally be projected into a 650-PA total, so they won’t project part-time PA at all – well, that gets me this:

which isn’t really very good at all, thanks to the way PA are thrown off, and all the counting stats along with it. Projecting to a 650_AP season is definitely NOT the way to go. But the rates stats are dramatically better. Lets improve on the previous by taking a simple average of the three years 2013-15, with one little added wrinkle – minor league stats only count for 50% of major league stats.

That figure, by itself, is comparable to most of the other stats I’ll run. That value is pretty much the primary input to the autoclay – the computer printout, without any further help from me:

Next was ZIPS, taking another step forward:

Followed by Rotowire’s projections;

Ahead of Rotowire is where I show up:

I’d like to say that’s where it ends, but I have to honestly report that still leaves PECOTA

And, on top, Steamer:

The differences really aren’t that big between them. For instance, if we take the best score on a player by player basis for the 256 player group, then among the top three the results are fairly even – Clay was best on 80, Steamer on 89, Pecota on 87. Clay beats Steamer 129-127 head to head; Pecota beats Clay 133-123; and Steamer beats Pecota 132-124.

{Note – I had this virtually finished this by about Valentine’s Day, and then – right before hitting the publish key – made one more check and discovered that the “real” stats I was using for validation were not the actual, real stats. Wiping out a lot of work and several erroneous conclusions. Then things went crazy at work, and I focused on getting stats for other leagues in, and before I know it its March – not just barely, but a full week in. Time sure seems to move a lot faster than it used to. May also have to do with being more committed to my day job than I used to be – I could never work on this during the day, the way I used to.]