As some of you know, we developed our own Player Rater methodology called Point Shares about a year ago. Since then, it’s been one big poontrain….zooming past Statgeek Station. Perhaps it’s because my hat and eyewear aren’t goofy enough?
Anyway, we’ve done some informal comparisons in the past w/ ESPN but – after having a prolonged, dorktastic debate – I decided to take it one step further. I created a test where I pitted our Point Shares against two other player raters: ESPN and RotoTimes.
The test went like this: Create 10 random 10 team leagues and calculate the standings using the real stats. Then do the same using the rating points for each of the services (so if you add up the ESPN HR player rater points for each team, the one with the most points gets a 10 in HR). The team point totals calculated by the player raters is compared to the point totals from using the real stats and we added up all the differences (so if the real points total was 70 and the player rater projected 68, it’s worth 2 points). The lower the point total the better.
Before we go into the results, it’s worth noting that this test puts our Point Shares at a disadvantage. Why? Our methodology factors in position and projected team standings so that you can use it to directly estimate the impact of a trade or draft pick. We adjust our ratings by crediting points against the average found at the player’s position instead of the average hitter. All stats equal, we know that a catcher will be much more valuable than a 1st baseman (why else would people draft Russ Martin and V-Mart in the 3rd round last year?). Geovany Soto and Justin Morneau both hit 23 HRs. While RotoTimes and ESPN ignore position and credit each with the same number of points, we have a 0.5 difference. This is saying that – assuming you fill the rest of the rosters up with average players – owning Soto will give you a 0.5 advantage in HR points vs. owning Morneau in a 10-team league.
But a test where you just add up rosters takes this out of the equation. You could conceivably be perfect just by creating a ‘rating point’ completely proportional to the real total. So 20 vs. 30 HRs could be worth .2 and .3 points or 2 and 3 points, etc. Will those numbers help determine the value of 20 vs. 30 HRs? Not really.
Okay, with that said, the results of the test are as follows (if you want to see the spreadsheet, click here – warning: it’s messy). Out of 100 teams, the total points off by the Player Raters was:
RotoTimes – 112
Razzball Point Shares – 143
ESPN – 152
Our Point Shares were the most accurate for 3 of the 10 leagues while RotoTimes claimed at least a share of 1st in the other 7 (ESPN tied them 3 times).
At a category level, I performed correlation tests between the ‘real’ standings and those of the player raters. 100% would be perfect.
As you can see, Point Shares held up pretty well to ESPN and RotoTimes despite taking on the extra burden of factoring in position and tying it to expected point gains/losses in the standings. The test did shine light on some improvements we can make on pitcher counting stats (W, SV, K) – note the higher correlation we have on ERA/WHIP because we can directly tie it to IP.
Final summary:
1) RotoTimes is the most accurate of the three. Combined with the facts that you can both customize the rankings based on league size, categories, roster size, etc. and they provide $ estimates, we’d say this is the best pure ranking tool.
2) Point Shares have proven to be in the same league as these other player raters. Given the extra utility of Point Shares because it directly estimates the impact on team points of player moves, we’d say our Point Shares are the most useful of the three.
3) ESPN, while not the ‘testwide leader’, did pretty well – especially in hitting stats. Their biggest pure weakness is ERA/WHIP as it would appear that they don’t factor in IP like Razzball (and, we assume, RotoTimes). But we’d say their biggest total weakness is that their player points mean ABSOLUTELY NOTHING. They are arbitrary numbers that could be used to rank players but not to estimate their value in the standings or worth at the draft table. They do a good job, though, at keeping it updated throughout the year…
