Don't be shellfish...Share on FacebookTweet about this on TwitterShare on Google+

This is part of an annual multi-part series designed to help Fantasy Baseball fans determine on what fantasy rankings and projections to rely.  The first part will cover Rankings.  The next parts will cover Projections.

Below are the ranking sources that are part of the test.  I have hyperlinked to the actual rankings wherever possible.  Some of these links, including ours, will override once 2014 rankings are published.

Rankings Source Ranking/$ Date Hit/Pitch
CBSSports.com (Nando DiFino) $ ? 69/31
CBSSports.com (Al Melchior) $ ? 67/33
CBSSports.com (Scott White) $ ? 66/34
ESPN (Matthew Berry) Ranking  3/27 69/31
ESPN (Tristan Cockcroft) Ranking  3/23 65/35
ESPN Top 300 (Staff) $  3/28 67/33
ESPN Custom Value Calculator $ ? 65/35
FakeTeams (Ray Guilfoyle) Ranking  3/24 75/25
FantasyPros.com (Rankings) Ranking 4/1 68/32
FantasyPros.com (Auction Values)  $ 4/1 68/32
Fantistics  $ 3/19 65/35
KFFL  $  ? 59/41
Razzball.com (Grey Albright)  Ranking 4/1 69/31
Razzball.com (Rudy Gamble) $ 4/1 62/38
Rotochamp.com $ ? 67/33
SI.com (Eric Mack) $ ? 69/31
USAToday.com (Steve Gardner)  Ranking 2/25 77/23

Methodology notes:

  • Rankings-only sources were converted to auction values based on converting each rank to the $ value my rankings assigned for that rank.
  • ESPN Custom Value Calculator, KFFL, Fantistics, and my rankings were set to the RCL league parameters.  For ESPN rankings, I chose 12 team vs 10 team.  Most/all the others just produce one set of mixed rankings that I assume is tailored to the mass market (which usually plays 10-12 team mixed).
  • I provided the effective hit/pitch $ split for each ranking source.  These averaged out at 68/32 whereas hitters represent only 59% of starting players (13/22).  The only source that had a true 59/41 split was KFFL rankings while my splits were the next closest at 62/38.  I will do some analyses later this pre-season to test what split worked best in the 2013 RCLs.  (My guess is that it doesn’t matter much)
  • Only players drafted in 1 or more 2013 Razzball Commenter Leagues were included in the tests. The RCL format is 5×5 12-team mixed league with C/1B/2B/SS/3B/5 OF/CI/MI/UTIL/9 P/3 Bench.  There were 64 leagues/768 teams in 2013.
  • Any drafted player not included in a rankings source or ranked/valued below replacement level (e.g., set less than $0) were converted to $0.  I did this because I feel any player that is truly below replacement level will likely be dropped early in the season before they can produce too much negative value.
  • The sum $ value of all drafted players ranged from $224 to $259 per team with the ‘Top 200′ rankings (ESPN/Berry, FakeTeams/Guilfoyle, and USAToday/Gardner) coming it at the bottom end.  For the ‘top 200′ rankings, this basically nets out to $0 estimates for any player valued at $0-$5 in the other rankings services.  Given that about 40-50% of these players end up providing $0 value, my guess is that these sources do not end up penalized by this.

Test #1 – Test Source’s Predicted Value of RCL Drafted Teams With Their Actual Point Totals

This is similar to the test I performed the last two years.  You add up the $ value, per source, for each of the 768 RCL teams’ hitters and pitchers and then correlate this against the standings points that team gained.  The one change for this year’s test is that I used a team’s ‘master standings’ points vs. the team’s standings points in their specific league.  ‘Master standings’ compares all 768 teams’ totals and prorates the points so the team that was best earns 12 points, second place is 11.98, yadda yadda yadda, the team that’s 768th gets 1 point.  I don’t think this changes things too much but it saved me some time in doing the calculations.

  Correlations of Projected Team Values By Rankings Source and Standings Points
Source Total Rank Pitching Rank Hitting Rank
Razzball Player Rater – End of Season Values 66.9%   60.0%   70.5%  
ESPN Player Rater – End of Season Values 66.5%   59.2%   68.5%  
Razzball (Grey) 26.9% 1 11.6% 2 37.1% 1
Razzball (Rudy) 14.0% 2 7.8% 3 28.7% 2
CBS (DiFino) 8.2% 3 6.3% 4 26.1% 3
Fantistics 5.6% 4 14.8% 1 18.3% 4
KFFL -3.9% 5 6.0% 6 14.7% 6
Rotochamp -6.9% 6 6.0% 5 13.4% 8
CBS (White) -9.8% 7 3.7% 9 14.0% 7
FakeTeams (Guilfoyle) -9.9% 8 4.3% 8 11.7% 12
CBS (Melchior) -11.9% 9 3.4% 10 12.3% 11
ESPN Custom Value Generator -11.9% 10 4.9% 7 11.3% 15
USAToday (Gardner) -12.5% 11 -2.0% 14 13.2% 9
FantasyPros (Rankings) -14.8% 12 -2.5% 15 16.9% 5
SI.com (Mack) -15.9% 13 0.7% 11 12.8% 10
ESPN (Berry) -18.8% 14 -1.5% 13 11.3% 14
FantasyPros (Auction Values) -21.9% 15 -1.3% 12 8.3% 16
ESPN (Cockcroft) -23.4% 16 -6.0% 17 11.4% 13
ESPN Top 300 -29.3% 17 -3.3% 16 2.1% 17
  • The end of season value of an RCL team’s draft has a high correlation (66.9% for Razzball Player Rater $, 66.5% based on ESPN Player Rater) to their final standings points.  The value of a team’s hitters is a better predictor of RCL success than its pitchers (70 to 60% in RPR, 68% to 59% in ESPN).  This means that when one looks at the average RCL team, you can ascribe about 2/3 of their success (or lack theirof) to the draft and 1/3 to other factors (including trades, FA pickups, streaming, etc.).
    • The values for Razzball Player Rater (End of Season Values) represent the ceiling for rankings success.  Even if one perfectly nailed the rankings, the most they’d achieve (based on my $ calculator) would be 66.9%.
    • In 2011, a team’s draft value explained 64% of a team’s success.  This dipped to 53% in 2012 – with the most notable factors being Mike Trout and RA Dickey.
    • While streaming pitchers and hitters is extremely prevalent in the RCL, this analysis shows that its impact is limited unless combined with a successful draft.
  • Negative values means that a rankings source actually was a worse predictor for a team’s success than just assigning 65 points to each team.  This is the main reason that I think ‘pre-season’ standings based on draft success are a useless exercise.
  • I put a little more stock in the ‘Hitting’ and ‘Pitching’ correlations vs. the ‘Total’.  Feels like perhaps the ‘Total’ correlations have a bit more noise.
  • One potential bias in this data that I’ve noted in previous years is that since this test is based on RCL performance, it is possible that this skews the results in favor of our rankings (the most competitive RCLers are likely the heaviest readers) and against ESPN (the least competitive RCLers may rely more on ‘default’ ADPs).  Therefore, I have introduced two new tests that should remove this bias and will save any commentary on specific rankings until the end.

Test #2 – Test Source’s Predicted $ Value Of Each Player Drafted in 50+% of RCLs Vs. Final Season $ Value As Measured By Razzball Player Rater

This is a fairly simple test.  I took the end of season values for all players drafted in 50% or more of RCL leagues and correlated that against the values of each ranking source.  I limited this to only players drafted in 50+% of leagues since the nature of this test gives equal weight to each player’s ranking/value and I wanted to focus on the players of consequence in the majority of leagues.

Source Total Rank Pitching Rank Hitting Rank
Razzball Player Rater – End of Season Values 100.0%   100.0%   100.0%  
ESPN Player Rater – End of Season Values 96.7%   97.7%   96.4%  
Average 42.7%   47.8%   40.2%  
Razzball (Grey) 46.0% 1 48.5% 6 45.5% 1
Fantistics 45.3% 2 51.3% 2 41.3% 5
Razzball (Rudy) 44.1% 3 46.2% 13 43.3% 2
CBS (DiFino) 44.0% 4 50.8% 3 40.6% 7
ESPN Custom Value Generator 43.7% 5 52.9% 1 39.1% 12
FantasyPros (Auction Values) 43.4% 6 46.4% 12 41.6% 4
Rotochamp 43.1% 7 43.6% 17 42.6% 3
FantasyPros (Rankings) 43.0% 8 47.6% 9 40.6% 6
ESPN (Cockcroft) 42.9% 9 47.2% 10 40.3% 8
ESPN Top 300 42.4% 10 48.3% 7 39.5% 10
ESPN (Berry) 42.4% 11 48.0% 8 39.4% 11
CBS (Melchior) 42.0% 12 47.0% 11 38.8% 13
CBS (White) 41.9% 13 50.0% 5 37.4% 16
KFFL 41.1% 14 43.8% 16 39.8% 9
FakeTeams (Guilfoyle) 41.0% 15 50.8% 4 37.2% 17
SI.com (Mack) 40.0% 16 44.4% 15 37.8% 15
USAToday (Gardner) 39.5% 17 46.0% 14 37.9% 14
  • The correlation %s are much higher in this test because it is a direct player to player comparison vs a less direct sum of a team’s players vs standings points.  Another way of looking at it is that while the first test has a ceiling of 66%, this test has a ceiling at 100%.
  • I was very surprised to see that the correlation for pitcher values is greater than that of hitter values.  This seems to call into question the assumption that hitters are more predictable than pitchers but there may be other factors at play.
  • The fact that the top 4 are the same between Test #1 and Test #2 (in slightly different orders) indicates to me that any pro-Razzball bias in Test #1 is minor.  The improved performance of ESPN rankings in Test #2 indicates that my concerns over some negative bias for ESPN rankings in Test #1 may have some validity.
  • Despite the fact that my Player Rater end of season values and ESPN Player Rater end of season values are highly correlated (96.7%), it is still possible, perhaps, that there’s a little bit of bias left for Razzball.  So Test #3 will remove Razzball-based player calculations altogether.

Test #3 – Test Source’s Predicted $ Value Of Each Player Drafted in 50+% of RCLs Vs. Final Season $ Value As Measured By ESPN Player Rater

Below is the same test as Test #2 except I compared the end of season values based on ESPN Player Rater vs. the values of each ranking source.

Source Total Rank Pitching Rank Hitting Rank
Razzball Player Rater – End of Season Values 96.7%   97.7%   96.4%  
ESPN Player Rater – End of Season Values 100.0%   100.0%   100.0%  
Razzball (Grey) 45.7% 1 46.9% 6 43.8% 1
Fantistics 44.8% 2 50.6% 2 39.7% 4
Razzball (Rudy) 43.7% 3 45.0% 12 42.9% 2
FantasyPros (Auction Values) 42.5% 4 44.9% 13 39.3% 5
FantasyPros (Rankings) 42.3% 5 45.7% 8 38.5% 7
Rotochamp 42.2% 6 42.4% 16 40.6% 3
ESPN (Cockcroft) 42.2% 7 45.3% 11 38.9% 6
ESPN Top 300 41.7% 8 46.4% 7 38.0% 8
ESPN (Berry) 41.6% 9 45.5% 9 37.5% 9
CBS (DiFino) 41.4% 10 47.2% 4 36.7% 11
ESPN Custom Value Generator 40.9% 11 52.0% 1 34.2% 17
CBS (White) 40.5% 12 47.4% 3 35.5% 15
CBS (Melchior) 40.5% 13 45.3% 10 35.7% 14
FakeTeams (Guilfoyle) 40.1% 14 47.1% 5 34.5% 16
USAToday (Gardner) 40.0% 15 43.8% 14 36.1% 13
SI.com (Mack) 39.5% 16 42.3% 17 36.5% 12
KFFL 39.0% 17 43.6% 15 36.9% 10
  • 3 of the top 4 remain unchanged with CBSSports.com’s Nando DiFino falling into the middle-of-the-pack in this test.
  • This test provides minor boosts to 3 of 4 ESPN Rankings sources (Berry, Cockcroft, Top 300) and hurts their Custom Value Generator.

Test #4 – Comparison Vs. Consensus (FantasyPros Rankings)

This test does not measure accuracy.  It just provides a sense of ‘uniqueness’ in each source’s rankings vs. the consensus (using FantasyPros’s aggregated rankings as the consensus).

  Correlation vs. FantasyPros Rankings
Source All Hitters Pitchers
FantasyPros (Rankings) 100% 100% 100%
FantasyPros (Auction Values) 97% 97% 96%
ESPN (Cockcroft) 96% 96% 96%
ESPN Top 300 96% 96% 96%
CBS (White) 96% 96% 97%
ESPN (Berry) 95% 95% 94%
Rotochamp 94% 95% 92%
USAToday (Gardner) 94% 95% 94%
SI.com (Mack) 94% 94% 91%
FakeTeams (Guilfoyle) 93% 94% 93%
Razzball (Grey) 93% 93% 93%
CBS (Melchior) 91% 89% 94%
CBS (DiFino) 91% 89% 92%
ESPN Custom Value Generator 90% 89% 91%
Fantistics 90% 92% 85%
Razzball (Rudy) 89% 90% 90%
KFFL 87% 89% 88%
  • Same as last year, KFFL was the most ‘unique’ rankings with my rankings coming in 2nd.

Conclusions

  • I can say with fairly strong conviction and mixed feelings (Grey and I are a tad competitive) that Grey had the best rankings.  He finished first on all three tests.  The 4th test shows that this was not driven by being very divergent from the consensus but just being a bit better.  Of the top 10 breakout hitters (based on Actual value minus FantasyPros value), Grey had one first place (Paul Goldschmidt) and three second places (Hunter Pence, Chris Davis, Jean Segura).  The only other source with more than 2 top 2′s was the ESPN Custom Value Generator with three first places (Pence, Chris Davis, Soriano) and one 2nd place (Brandon Moss).   Grey was stronger relative to others in Hitting finishing 1st on all three tests while finishing in the top 6 for pitching.
  • I call it a draw between my rankings and Fantistics for 2nd place.   Their superiority in pitching was larger than my superiority in hitting but I think the first test shows that predicting hitters is more important to team success than pitching.  We were both on the ‘unique’ end of the spectrum with Fantistics’ pitching being by far the most unique.
  • Last year’s winner (KFFL) performed better than average in Test #1 but fared poorly in Tests #2/#3.  Given they produce the most ‘unique’ rankings, this may indicate that they made a lot of bold calls and the ones that panned out were more important than the ones that did not.
  • In the battle of CBSSports.com rankings supremacy, my vote goes to Nando DiFino.  Not sure why ESPN’s Player Rater likes his values less than me – perhaps it is mad that Sam Walker did not mention it in Fantasyland.
  • It is hard to pick a winner out of the ESPN rankings.  The Custom Value Generator did very well in Tests #2/#3 for pitching but it tanked in hitting.   The other three rankings finished in the middle of the pack for both hitting/pitching.  I will give a slight nod to the Generator because 1) every reader here knows my penchant for machines, 2) it matched Grey for top 2 finishes among the top 10 breakout hitters, and 3) the other three ESPN rankings were much safer than the Generator (as seen in Test 4).
  • While I do not think there is a bias in these tests towards the source that only rank 200 players (ESPN/Berry, FakeTeams, USAToday), the fact all three perform in the bottom half indicates that there might be one.

Final Notes

  • The combined rankings uses in Tests #2-4 are available upon request to any source referenced in the above analysis.
  • I realize this is only a subset of all published fantasy baseball rankings.  If I did not include your rankings and it fits the following criteria, let me know (rudy@razzball.com) and I will keep it in mind for next year:  1) published top 200+ player rankings online, 2) it is free, 3) the page is active through Jan/Feb of following year, 4) no updates after opening day, and 5) your site’s readership is of some significant size.
  1. centerfield ballhawk says:
    (link)

    Any advice on the best way to convert personal rankings to auction values? Great job with the research.

    • @centerfield ballhawk: I’d go to my 2013 pre-season rankings (see Tools in menu) and cut/paste into excel. Then just cut/paste the $ right next to your ranked players. So whomever you have #1 gets the $ figure I have for the top-ranked player. This isn’t perfect but quick/easy.

  2. troy says:
    (link)

    first off wanted to say GREAT STUFF really is. also wanted to ask, i use rotowire software to help me draft. I will be using it for the first time this year for a auction draft. But its asking for % spent on batting and if i want to use inflation. Now im not sure what the norm would be here. I am just doing a standard yahoo draft. Tried doing some research but didnt have much luck, you have a input at all rudy?

    • @troy: For standard Yahoo, it should be somewhere between 151/109 and 170/90. I think it’s 11 hitters and 8 pitchers so 151/109 assumes you value each player the same. 170/90 tilts it to be the equivalent of 180/80 which is common in leagues with a more hitter-slanted ratio of 13/9 or 14/9 (depends on 1-2 catchers).

      I’d just go with 170/90 which is 65% hitting and 35% pitching.

    • Obmij76 says:
      (link)

      @troy: I see most experts recommend a near 70 on hitting, 30 on pitching

      • @Obmij76: not sure if they factor in that yahoo standard format has a higher % of pitchers in roster spots than other leagues.

        • troy says:
          (link)

          @Rudy Gamble: ah ok i see what ur talking about then. dont really need to put anything for inflation right? and thanks i appreciate that.

          • Andrew A says:
            (link)

            @troy:

            the draft software should calculate the inflation for you automatically

  3. Obmij76 says:
    (link)

    Thanks Rudy! Look forward to the projections analysis. Bit surprised Rotochamp finished in the middle of the pack. Also, did you look at some other rankings that weren’t included in the writeup? Guys like Behrens, Podhorzer, Funston, etc.

    • @Obmij76: re: Rotochamp, middle of the pack isn’t too bad. felt like this year that the ‘machines’ didn’t fare as well as last year. not sure why.

      i really like/respect the yahoo guys (behrens, funston, pianowski, evans) but it’s always a struggle finding top 200-300 rankings on their site. i only find top 100. talking w/ pianowski to get them in next year.

      also like/respect the fan/rotographs guys (eno sarris, podhorzer). talking w/ eno – i think their rankings got overwritten early so I couldn’t find them during my rankings gathering.

      given how well podhorzer did last year in some expert drafts, i imagine his rankings would’ve done well (he drafted better than me in our Razzball Commenter League but I beat him in the standings due to uber-streaming of hitters :))

  4. centerfield_ballhawk says:
    (link)

    Just to be clear, the auction values that will be published under the rankings section are your dollar values aligned to computer generated projections, and are unrelated to Grey’s position rankings and projections.

    • @centerfield_ballhawk: yes. grey will be doing auction values this year but that will be listed specifically under Grey.

      • centerfield_ballhawk says:
        (link)

        @Rudy Gamble: Awesome sauce! I feel like my draft preparation gets better and more fine tuned every year, and Razzball deserves a lot of the credit. Thanks again for all that you do.

  5. Jason says:
    (link)

    Thanks, this seems like allot of work you did. I love this kindof stuff. Regarding, hitters vrs pitchers easier to predict, Could it be the batting is skewed because of Braun and Tulo? I think if you could possibly work in minimum AB’s it might show differently.

    There really isn’t that much difference between the experts according to this, but by limiting to the top 200 you lose some of the sleepers no? Would be interesting to disregard own percentage. The draft advisers’ “performance” is more about helping us to find values. What if you flipped everything, and gave the most weight to pick 200? I might give this a shot just when i have some time.

    • @Jason: thanks. in my projections test, i do a playing time-neutral analysis where i use the actual playing time and then the projection source’s rate stats (e.g, HR per plate appearance). So that’ll isolate how much of this is driven by playing time.

      the 1st test, in particular, does value sleepers. if one ranking source had Marte at #100 and another at #200, the first ranking source will better predict a team’s end of season standings points. as noted in the post, grey did quite well in his ranking of some of the breakout players.

      fyi, an upcoming analysis i’m doing will show that, at least for hitting, the last 10 rounds or so of a snake draft has little impact on predicting team hitting (seems to have a bit more impact on pitching). stay tuned :)

  6. Count de Monetball says:
    (link)

    Thanks Rudy you’re a geek of the highest order sir!

  7. JoeCool says:
    (link)

    So Grey is God, we all knew that already…..

    Love crizzap like this, great read!

  8. Jason says:
    (link)

    What about a round per round “draft” between each expert. Who ever has the player ranked higher, will get that player. Then apply the actual result player ranks to see who’s ontop. I just tried this with DiFino verses Grey with only Shortstops (2013). It wasn’t pretty for Grey, but I didn’t take into account playing time.

    • @Jason: I’ve seen this done before and think it’s wonky. There’s no such thing as a two-person draft. You’d need to line up 12 ranking sources…but then there’s draft order bias so you’d need to design the test so that you’ve covered all the combinations which is roughly 480,000,000 combinations.

      Then there’s the artificiality where say a pitcher-friendly source like KFFL or mine end up with a SP-loaded first couple of rounds which would never realistically happen.

      Playing time, fwiw, has to be baked into the rankings. Usually, the pain is equally felt (e.g., Braun, Pujols, Tulo) but if a source overranks a player vs. the consensus b/c they were too bullish on playing time, that’s on them…

      • Jason says:
        (link)

        @Rudy Gamble: How do you keep your head from exploding?

        • @Jason: I avoid tests with 480,000,000 combinations and my ‘fro provides extra protection.

  9. Fish says:
    (link)

    YESSSSS!!!! The numbers are here!!!

    This is better than porn. Or mustaches. Or 1099 Forms. (In that order)

  10. Al koholic says:
    (link)

    rudy,thanks for all the hard work and stats,grey friended you just so he could copy in class,while in college didnt he?your awesome dude,keep up the great work and thanks again

    • Thx. Grey and I only shared the same class once. He was too drunk to copy my answers.

  11. Thorbs says:
    (link)

    Awesome read, Rudy!

  12. Yeshcheese says:
    (link)

    Awesome analysis! All hail Rudy, grey and the ever salivating hitter tron!

  13. Paul says:
    (link)

    Thanks for posting these Rudy, was hoping you would do this series again. A couple of observations.

    It seems in addition to strong facial hair Grey has a great gut. The interesting thing about his rankings is they don’t always equal what you would get if you ran them through an SGP valuation formula. They are close but it seems his gut bias leads him to rank players slightly differently regardless of what the advanced sabermetrics are saying. At least so far he appears to be beating the machines in this regard.

    Also thought I would throw out the idea of using ADP data as a potential data in the future. I consider ADP from multiple sites somewhat of a crowd sourced ranking system. They will certainly have bias towards the site where the draft takes place as late in drafts there is a lot of auto-picking going on but it would be interesting to see how a crowd sourced data site like an ADP from mockdraftcentral or NFBC compares.

    Also interesting to read your notes on the Hitting / Pitching splits. I reach Schecter’s book at your recommendation and it seems he is accurate in suggesting most sites follow a 69/31 split. Certainly there are small differences but anyone looking to draft on a value based model would seem well served to follow these splits.

    Great work as always!

    • Thanks Paul.

      Yes, Grey had a good gut in 2013. I ran Grey’s 2013 projections through my Player Rater and it correlates high but, of course, doesn’t perfectly match the rankings. The reality is that any valuation tool has a methodological foundation (like SGP) but the could choose a whole bunch of additional modifications/calculations. The projection analysis will go up in the next couple of weeks and providing a comparison point for Grey’s rankings vs projections.

      I used the RCL ADP in the 2011 test – http://razzball.com/review-of-2011-fantasy-baseball-player-rankings/. Performed in middle to bottom of pack. Since the ADP is an average draft pick across all leagues, there’s little reason to expect it to perform better than middle of the pack.

      Analysis on hit/pitch coming in next couple weeks as well. 69/31 a little too much of a skew in my eyes for 12-team mixed league snake drafts but it’s within a realistic range of possibilities.

  14. DaBulls says:
    (link)

    Any reason you did not include any of the forecast sets available on Fangraphs (ZiPS, Steamer, etc.?) By recollection, I believe you had included Steamer in prior years? I wouldn’t bother asking if I wasn’t a tremendous fan of the analysis…

    • @DaBulls: Those will be part of the next set of tests regarding projections. This test was about rankings.

  15. Fish says:
    (link)

    Dear Yahoo Fantasy Baseball,

    Please publish rankings that go past 100!!!! I mean, who plays in a league of only 5 people (that’s the largest size said rankings might be useful for)?

    • I’ve passed on similar sentiments to Scott Pianowski. Interested to see how we’d do against him, Behrens, and Funston.

Comments are closed.