LOGIN

This is part of an annual multi-part series designed to help Fantasy Baseball fans determine on what fantasy rankings and projections to rely.  The first part will cover Rankings.  The next parts will cover Projections.

Below are the ranking sources that are part of the test.  I have hyperlinked to the actual rankings wherever possible.  Some of these links, including ours, will override once 2014 rankings are published.

Rankings Source Ranking/$ Date Hit/Pitch
CBSSports.com (Nando DiFino) $ ? 69/31
CBSSports.com (Al Melchior) $ ? 67/33
CBSSports.com (Scott White) $ ? 66/34
ESPN (Matthew Berry) Ranking  3/27 69/31
ESPN (Tristan Cockcroft) Ranking  3/23 65/35
ESPN Top 300 (Staff) $  3/28 67/33
ESPN Custom Value Calculator $ ? 65/35
FakeTeams (Ray Guilfoyle) Ranking  3/24 75/25
FantasyPros.com (Rankings) Ranking 4/1 68/32
FantasyPros.com (Auction Values)  $ 4/1 68/32
Fantistics  $ 3/19 65/35
KFFL  $  ? 59/41
Razzball.com (Grey Albright)  Ranking 4/1 69/31
Razzball.com (Rudy Gamble) $ 4/1 62/38
Rotochamp.com $ ? 67/33
SI.com (Eric Mack) $ ? 69/31
USAToday.com (Steve Gardner)  Ranking 2/25 77/23

Methodology notes:

  • Rankings-only sources were converted to auction values based on converting each rank to the $ value my rankings assigned for that rank.
  • ESPN Custom Value Calculator, KFFL, Fantistics, and my rankings were set to the RCL league parameters.  For ESPN rankings, I chose 12 team vs 10 team.  Most/all the others just produce one set of mixed rankings that I assume is tailored to the mass market (which usually plays 10-12 team mixed).
  • I provided the effective hit/pitch $ split for each ranking source.  These averaged out at 68/32 whereas hitters represent only 59% of starting players (13/22).  The only source that had a true 59/41 split was KFFL rankings while my splits were the next closest at 62/38.  I will do some analyses later this pre-season to test what split worked best in the 2013 RCLs.  (My guess is that it doesn’t matter much)
  • Only players drafted in 1 or more 2013 Razzball Commenter Leagues were included in the tests. The RCL format is 5×5 12-team mixed league with C/1B/2B/SS/3B/5 OF/CI/MI/UTIL/9 P/3 Bench.  There were 64 leagues/768 teams in 2013.
  • Any drafted player not included in a rankings source or ranked/valued below replacement level (e.g., set less than $0) were converted to $0.  I did this because I feel any player that is truly below replacement level will likely be dropped early in the season before they can produce too much negative value.
  • The sum $ value of all drafted players ranged from $224 to $259 per team with the ‘Top 200’ rankings (ESPN/Berry, FakeTeams/Guilfoyle, and USAToday/Gardner) coming it at the bottom end.  For the ‘top 200’ rankings, this basically nets out to $0 estimates for any player valued at $0-$5 in the other rankings services.  Given that about 40-50% of these players end up providing $0 value, my guess is that these sources do not end up penalized by this.

Test #1 – Test Source’s Predicted Value of RCL Drafted Teams With Their Actual Point Totals

This is similar to the test I performed the last two years.  You add up the $ value, per source, for each of the 768 RCL teams’ hitters and pitchers and then correlate this against the standings points that team gained.  The one change for this year’s test is that I used a team’s ‘master standings’ points vs. the team’s standings points in their specific league.  ‘Master standings’ compares all 768 teams’ totals and prorates the points so the team that was best earns 12 points, second place is 11.98, yadda yadda yadda, the team that’s 768th gets 1 point.  I don’t think this changes things too much but it saved me some time in doing the calculations.

  Correlations of Projected Team Values By Rankings Source and Standings Points
Source Total Rank Pitching Rank Hitting Rank
Razzball Player Rater – End of Season Values 66.9%   60.0%   70.5%  
ESPN Player Rater – End of Season Values 66.5%   59.2%   68.5%  
Razzball (Grey) 26.9% 1 11.6% 2 37.1% 1
Razzball (Rudy) 14.0% 2 7.8% 3 28.7% 2
CBS (DiFino) 8.2% 3 6.3% 4 26.1% 3
Fantistics 5.6% 4 14.8% 1 18.3% 4
KFFL -3.9% 5 6.0% 6 14.7% 6
Rotochamp -6.9% 6 6.0% 5 13.4% 8
CBS (White) -9.8% 7 3.7% 9 14.0% 7
FakeTeams (Guilfoyle) -9.9% 8 4.3% 8 11.7% 12
CBS (Melchior) -11.9% 9 3.4% 10 12.3% 11
ESPN Custom Value Generator -11.9% 10 4.9% 7 11.3% 15
USAToday (Gardner) -12.5% 11 -2.0% 14 13.2% 9
FantasyPros (Rankings) -14.8% 12 -2.5% 15 16.9% 5
SI.com (Mack) -15.9% 13 0.7% 11 12.8% 10
ESPN (Berry) -18.8% 14 -1.5% 13 11.3% 14
FantasyPros (Auction Values) -21.9% 15 -1.3% 12 8.3% 16
ESPN (Cockcroft) -23.4% 16 -6.0% 17 11.4% 13
ESPN Top 300 -29.3% 17 -3.3% 16 2.1% 17
  • The end of season value of an RCL team’s draft has a high correlation (66.9% for Razzball Player Rater $, 66.5% based on ESPN Player Rater) to their final standings points.  The value of a team’s hitters is a better predictor of RCL success than its pitchers (70 to 60% in RPR, 68% to 59% in ESPN).  This means that when one looks at the average RCL team, you can ascribe about 2/3 of their success (or lack theirof) to the draft and 1/3 to other factors (including trades, FA pickups, streaming, etc.).
    • The values for Razzball Player Rater (End of Season Values) represent the ceiling for rankings success.  Even if one perfectly nailed the rankings, the most they’d achieve (based on my $ calculator) would be 66.9%.
    • In 2011, a team’s draft value explained 64% of a team’s success.  This dipped to 53% in 2012 – with the most notable factors being Mike Trout and RA Dickey.
    • While streaming pitchers and hitters is extremely prevalent in the RCL, this analysis shows that its impact is limited unless combined with a successful draft.
  • Negative values means that a rankings source actually was a worse predictor for a team’s success than just assigning 65 points to each team.  This is the main reason that I think ‘pre-season’ standings based on draft success are a useless exercise.
  • I put a little more stock in the ‘Hitting’ and ‘Pitching’ correlations vs. the ‘Total’.  Feels like perhaps the ‘Total’ correlations have a bit more noise.
  • One potential bias in this data that I’ve noted in previous years is that since this test is based on RCL performance, it is possible that this skews the results in favor of our rankings (the most competitive RCLers are likely the heaviest readers) and against ESPN (the least competitive RCLers may rely more on ‘default’ ADPs).  Therefore, I have introduced two new tests that should remove this bias and will save any commentary on specific rankings until the end.

Test #2 – Test Source’s Predicted $ Value Of Each Player Drafted in 50+% of RCLs Vs. Final Season $ Value As Measured By Razzball Player Rater

This is a fairly simple test.  I took the end of season values for all players drafted in 50% or more of RCL leagues and correlated that against the values of each ranking source.  I limited this to only players drafted in 50+% of leagues since the nature of this test gives equal weight to each player’s ranking/value and I wanted to focus on the players of consequence in the majority of leagues.

Source Total Rank Pitching Rank Hitting Rank
Razzball Player Rater – End of Season Values 100.0%   100.0%   100.0%  
ESPN Player Rater – End of Season Values 96.7%   97.7%   96.4%  
Average 42.7%   47.8%   40.2%  
Razzball (Grey) 46.0% 1 48.5% 6 45.5% 1
Fantistics 45.3% 2 51.3% 2 41.3% 5
Razzball (Rudy) 44.1% 3 46.2% 13 43.3% 2
CBS (DiFino) 44.0% 4 50.8% 3 40.6% 7
ESPN Custom Value Generator 43.7% 5 52.9% 1 39.1% 12
FantasyPros (Auction Values) 43.4% 6 46.4% 12 41.6% 4
Rotochamp 43.1% 7 43.6% 17 42.6% 3
FantasyPros (Rankings) 43.0% 8 47.6% 9 40.6% 6
ESPN (Cockcroft) 42.9% 9 47.2% 10 40.3% 8
ESPN Top 300 42.4% 10 48.3% 7 39.5% 10
ESPN (Berry) 42.4% 11 48.0% 8 39.4% 11
CBS (Melchior) 42.0% 12 47.0% 11 38.8% 13
CBS (White) 41.9% 13 50.0% 5 37.4% 16
KFFL 41.1% 14 43.8% 16 39.8% 9
FakeTeams (Guilfoyle) 41.0% 15 50.8% 4 37.2% 17
SI.com (Mack) 40.0% 16 44.4% 15 37.8% 15
USAToday (Gardner) 39.5% 17 46.0% 14 37.9% 14
  • The correlation %s are much higher in this test because it is a direct player to player comparison vs a less direct sum of a team’s players vs standings points.  Another way of looking at it is that while the first test has a ceiling of 66%, this test has a ceiling at 100%.
  • I was very surprised to see that the correlation for pitcher values is greater than that of hitter values.  This seems to call into question the assumption that hitters are more predictable than pitchers but there may be other factors at play.
  • The fact that the top 4 are the same between Test #1 and Test #2 (in slightly different orders) indicates to me that any pro-Razzball bias in Test #1 is minor.  The improved performance of ESPN rankings in Test #2 indicates that my concerns over some negative bias for ESPN rankings in Test #1 may have some validity.
  • Despite the fact that my Player Rater end of season values and ESPN Player Rater end of season values are highly correlated (96.7%), it is still possible, perhaps, that there’s a little bit of bias left for Razzball.  So Test #3 will remove Razzball-based player calculations altogether.

Test #3 – Test Source’s Predicted $ Value Of Each Player Drafted in 50+% of RCLs Vs. Final Season $ Value As Measured By ESPN Player Rater

Below is the same test as Test #2 except I compared the end of season values based on ESPN Player Rater vs. the values of each ranking source.

Source Total Rank Pitching Rank Hitting Rank
Razzball Player Rater – End of Season Values 96.7%   97.7%   96.4%  
ESPN Player Rater – End of Season Values 100.0%   100.0%   100.0%  
Razzball (Grey) 45.7% 1 46.9% 6 43.8% 1
Fantistics 44.8% 2 50.6% 2 39.7% 4
Razzball (Rudy) 43.7% 3 45.0% 12 42.9% 2
FantasyPros (Auction Values) 42.5% 4 44.9% 13 39.3% 5
FantasyPros (Rankings) 42.3% 5 45.7% 8 38.5% 7
Rotochamp 42.2% 6 42.4% 16 40.6% 3
ESPN (Cockcroft) 42.2% 7 45.3% 11 38.9% 6
ESPN Top 300 41.7% 8 46.4% 7 38.0% 8
ESPN (Berry) 41.6% 9 45.5% 9 37.5% 9
CBS (DiFino) 41.4% 10 47.2% 4 36.7% 11
ESPN Custom Value Generator 40.9% 11 52.0% 1 34.2% 17
CBS (White) 40.5% 12 47.4% 3 35.5% 15
CBS (Melchior) 40.5% 13 45.3% 10 35.7% 14
FakeTeams (Guilfoyle) 40.1% 14 47.1% 5 34.5% 16
USAToday (Gardner) 40.0% 15 43.8% 14 36.1% 13
SI.com (Mack) 39.5% 16 42.3% 17 36.5% 12
KFFL 39.0% 17 43.6% 15 36.9% 10
  • 3 of the top 4 remain unchanged with CBSSports.com’s Nando DiFino falling into the middle-of-the-pack in this test.
  • This test provides minor boosts to 3 of 4 ESPN Rankings sources (Berry, Cockcroft, Top 300) and hurts their Custom Value Generator.

Test #4 – Comparison Vs. Consensus (FantasyPros Rankings)

This test does not measure accuracy.  It just provides a sense of ‘uniqueness’ in each source’s rankings vs. the consensus (using FantasyPros’s aggregated rankings as the consensus).

  Correlation vs. FantasyPros Rankings
Source All Hitters Pitchers
FantasyPros (Rankings) 100% 100% 100%
FantasyPros (Auction Values) 97% 97% 96%
ESPN (Cockcroft) 96% 96% 96%
ESPN Top 300 96% 96% 96%
CBS (White) 96% 96% 97%
ESPN (Berry) 95% 95% 94%
Rotochamp 94% 95% 92%
USAToday (Gardner) 94% 95% 94%
SI.com (Mack) 94% 94% 91%
FakeTeams (Guilfoyle) 93% 94% 93%
Razzball (Grey) 93% 93% 93%
CBS (Melchior) 91% 89% 94%
CBS (DiFino) 91% 89% 92%
ESPN Custom Value Generator 90% 89% 91%
Fantistics 90% 92% 85%
Razzball (Rudy) 89% 90% 90%
KFFL 87% 89% 88%
  • Same as last year, KFFL was the most ‘unique’ rankings with my rankings coming in 2nd.

Conclusions

  • I can say with fairly strong conviction and mixed feelings (Grey and I are a tad competitive) that Grey had the best rankings.  He finished first on all three tests.  The 4th test shows that this was not driven by being very divergent from the consensus but just being a bit better.  Of the top 10 breakout hitters (based on Actual value minus FantasyPros value), Grey had one first place (Paul Goldschmidt) and three second places (Hunter Pence, Chris Davis, Jean Segura).  The only other source with more than 2 top 2’s was the ESPN Custom Value Generator with three first places (Pence, Chris Davis, Soriano) and one 2nd place (Brandon Moss).   Grey was stronger relative to others in Hitting finishing 1st on all three tests while finishing in the top 6 for pitching.
  • I call it a draw between my rankings and Fantistics for 2nd place.   Their superiority in pitching was larger than my superiority in hitting but I think the first test shows that predicting hitters is more important to team success than pitching.  We were both on the ‘unique’ end of the spectrum with Fantistics’ pitching being by far the most unique.
  • Last year’s winner (KFFL) performed better than average in Test #1 but fared poorly in Tests #2/#3.  Given they produce the most ‘unique’ rankings, this may indicate that they made a lot of bold calls and the ones that panned out were more important than the ones that did not.
  • In the battle of CBSSports.com rankings supremacy, my vote goes to Nando DiFino.  Not sure why ESPN’s Player Rater likes his values less than me – perhaps it is mad that Sam Walker did not mention it in Fantasyland.
  • It is hard to pick a winner out of the ESPN rankings.  The Custom Value Generator did very well in Tests #2/#3 for pitching but it tanked in hitting.   The other three rankings finished in the middle of the pack for both hitting/pitching.  I will give a slight nod to the Generator because 1) every reader here knows my penchant for machines, 2) it matched Grey for top 2 finishes among the top 10 breakout hitters, and 3) the other three ESPN rankings were much safer than the Generator (as seen in Test 4).
  • While I do not think there is a bias in these tests towards the source that only rank 200 players (ESPN/Berry, FakeTeams, USAToday), the fact all three perform in the bottom half indicates that there might be one.

Final Notes

  • The combined rankings uses in Tests #2-4 are available upon request to any source referenced in the above analysis.
  • I realize this is only a subset of all published fantasy baseball rankings.  If I did not include your rankings and it fits the following criteria, let me know ([email protected]) and I will keep it in mind for next year:  1) published top 200+ player rankings online, 2) it is free, 3) the page is active through Jan/Feb of following year, 4) no updates after opening day, and 5) your site’s readership is of some significant size.