Don't be shellfish...Share on FacebookTweet about this on TwitterShare on Google+

As some of you know, we developed our own Player Rater methodology called Point Shares about a year ago. Since then, it’s been one big poontrain….zooming past Statgeek Station. Perhaps it’s because my hat and eyewear aren’t goofy enough?

Anyway, we’ve done some informal comparisons in the past w/ ESPN but – after having a prolonged, dorktastic debate – I decided to take it one step further.  I created a test where I pitted our Point Shares against two other player raters:  ESPN and RotoTimes.

The test went like this:  Create 10 random 10 team leagues and calculate the standings using the real stats.  Then do the same using the rating points for each of the services (so if you add up the ESPN HR player rater points for each team, the one with the most points gets a 10 in HR).  The team point totals calculated by the player raters is compared to the point totals from using the real stats and we added up all the differences (so if the real points total was 70 and the player rater projected 68, it’s worth 2 points).  The lower the point total the better.

Before we go into the results, it’s worth noting that this test puts our Point Shares at a disadvantage.  Why?  Our methodology factors in position and projected team standings so that you can use it to directly estimate the impact of a trade or draft pick.  We adjust our ratings by crediting points against the average found at the player’s position instead of the average hitter.  All stats equal, we know that a catcher will be much more valuable than a 1st baseman (why else would people draft Russ Martin and V-Mart in the 3rd round last year?).  Geovany Soto and Justin Morneau both hit 23 HRs.  While RotoTimes and ESPN ignore position and credit each with the same number of points, we have a 0.5 difference.  This is saying that – assuming you fill the rest of the rosters up with average players – owning Soto will give you a 0.5 advantage in HR points vs. owning Morneau in a 10-team league.

But a test where you just add up rosters takes this out of the equation.  You could conceivably be perfect just by creating a ‘rating point’ completely proportional to the real total.  So 20 vs. 30 HRs could be worth .2 and .3 points or 2 and 3 points, etc.  Will those numbers help determine the value of 20 vs. 30 HRs?  Not really.

Okay, with that said, the results of the test are as follows (if you want to see the spreadsheet, click here – warning: it’s messy).  Out of 100 teams, the total points off by the Player Raters was:

RotoTimes – 112
Razzball Point Shares – 143
ESPN – 152

Our Point Shares were the most accurate for 3 of the 10 leagues while RotoTimes claimed at least a share of 1st in the other 7 (ESPN tied them 3 times).

At a category level, I performed correlation tests between the ‘real’ standings and those of the player raters.  100% would be perfect.

As you can see, Point Shares held up pretty well to ESPN and RotoTimes despite taking on the extra burden of factoring in position and tying it to expected point gains/losses in the standings.  The test did shine light on some improvements we can make on pitcher counting stats (W, SV, K) – note the higher correlation we have on ERA/WHIP because we can directly tie it to IP.

Final summary:

1) RotoTimes is the most accurate of the three.  Combined with the facts that you can both customize the rankings based on league size, categories, roster size, etc. and they provide $ estimates, we’d say this is the best pure ranking tool.

2) Point Shares have proven to be in the same league as these other player raters.  Given the extra utility of Point Shares because it directly estimates the impact on team points of player moves, we’d say our Point Shares are the most useful of the three.

3) ESPN, while not the ‘testwide leader’, did pretty well – especially in hitting stats.  Their biggest pure weakness is ERA/WHIP as it would appear that they don’t factor in IP like Razzball (and, we assume, RotoTimes).  But we’d say their biggest total weakness is that their player points mean ABSOLUTELY NOTHING.  They are arbitrary numbers that could be used to rank players but not to estimate their value in the standings or worth at the draft table.  They do a good job, though, at keeping it updated throughout the year…

  1. Lou says:
    (link)

    NICE JOB!!!

  2. Brian Barnes says:
    (link)

    This is terrific, question I had looking at it was how point share is determined, per category, I will use wins as an example.

    Lidge 2Wins = 1.3 wins pt share
    Soria 2 Wins = 1.9 wins pt share
    Nathan 1 win = 1.3 wins pt share

    Have these been normalized some how b/c of innings pitched and if so why?

    I’m not criticizing just trying to understand as I’m attempting to come up with my own player rating.

    Also one other question, have you tried to apply this to past season to see how good of an indicator they would have been (say plug in the 06 stats and then look at the 07 results)?

  3. @Lou: Thanks!

    @Brian Barnes: We’ve got a few posts where we elaborate on the methodology (e.g., http://razzball.com/razzball-2008-player-rater/). .

    ERA and WHIP are normalized by innings. AVG by AB. The other stats are adjusted somewhat by position (including Starter vs. Reliever).

    We haven’t used past seasons as an indicator. There are various forecasters out there (Baseball Prospectus, CHONE, Hardball Times, etc) that are better than we’ll ever be at that. We’ll just create a 2009 Pre-Season Player Rater based on those stats. It’s the conversion of those projections to fantasy baseball impact where we add value.

    Good luck on the Player Rater creation. While we are partial to ours, our #1 goal here is to provide the best fantasy baseball info and entertain while we do it (so apologies for the dryness of these posts!). So if there’s something better out there, we’ll point to it (i.e., RotoTimes)

  4. @Brian Barnes: note: not sure where you got those point shares but Lidge and Soria have the same point share total (-0.8 for 10 team)….

  5. Nick says:
    (link)

    Rudy,

    Hats off to you for publishing results that don’t necessarily prove your system to be the best in all cases.

    So I’m troubled by the other systems having an advantage by not factoring in position. It would seem that one way to test it would be to find two players at two different positions and add up their total point shares, and compare them to two other players at the same positions with the same total point shares. Then calculate their actual stats and see who comes out ahead (or hopefully they come up equal). This would be really difficult to do over all 5 categories, but even looking at just one should show us something.

    For example…

    Chase Utley 33 HR 1.4 pts
    Jacoby Ellsbury 9 HR -1.3 pts
    Total #1 42 HR .1pts

    Grady Sizemore 33 HR 1.1 pts
    Brian Roberts 9 HR -1.0 pts
    Total #2 42 HR .1 pts

    Not sure how representative this example is, as I just randomly picked an OF and 2B and happened to get the same total, and I understand that there may be a rounding issue with point shares, but at least here the system works.

    If we were to take the same players on ESPN…

    Utley 2.7 + Ellsbury .15 = 2.85

    Roberts .15 + Sizemore 2.7 = 2.85

    Maybe I’m recreating what you actually did, but this method shows the difference between the two systems while at the same time showing how they agree.

    Not sure exactly what this proves, you’ve got to like the system that shows the value of position scarcity…

  6. Nick says:
    (link)

    For the sake of completion, here’s RotoTimes with the same example:

    Utley .90 HRpts
    Ellsbury -1.07 HRpts
    -.17

    Roberts -1.07 HRpts
    Sizemore .90 HRpts
    -.17

    Again, similar to ESPN’s “absolute” system. However, it’s interesting that the net value of 42 HRs is actually a negative when spread across two positions in this format, whereas point shares showed a small positive. Wonder what’s going on here…

  7. Nick says:
    (link)

    One more thing,

    A MAJOR advantage of the point shares system is that it can be applied to the projections of your choice! Assessing value at the end of the season is interesting, but it’s the beginning of the year when we have all of these projections and various rankings but no one bothers to combine the two! (save for BP’s PFM, which I still have trouble understanding, despite several email exchanges with Ben Murphy).

    That, I believe, is the true advantage of your system. And then comparing those projected points to ADPs and identifying values should make for a tremendous advantage when drafting.

  8. @Nick: Thanks for writing in. The test you proposed above should equal out for the absolute systems and not necessarily for Point Shares. I would imagine Point Shares will be close to equal. As for why Point Shares might be slightly positive compared to RotoTimes negative, I can’t say for sure. i don’t know if RotoTimes neagtive vs. positive scores mean anything. If they do, they’d be based on the average HRs for ANY position where Point Shares would be based on specific position(s).

    I imagine the other services could use projections to create a pre-season player rater. Whether they do or not is another matter :)

Comments are closed.