Don't be shellfish...Share on FacebookTweet about this on TwitterShare on Google+

This is part of a two-part series designed to help Fantasy Baseball fans determine on what fantasy rankings and projections to rely.  The first part will cover Rankings.  The second part will cover Projections.  The methodology for the test relies on comparing Razzball Commenter League team drafts (576 teams in 2012 across 48 12-Team MLB leagues using ESPN’s default 5×5 format) and their end of season point totals.  Background on the methodology can be found here.

Everyone who participates in fantasy baseball snake drafts has their own approach to pre-draft rankings and analysis.  This approach usually results in a customized ranking of players that is a combination of trusted ranking sources, recent fantasy baseball news, and personal player preferences.

This is our annual attempt to help you determine which ranking sources are worth using.

The methodology behind testing the various ranking sources works as such:

  1. Convert every player for each source’s rankings to a $ figure using our Point Share dollar figures per draft pick – e.g., the #1 draft pick is usually ~$40, the next pick might be $38, etc.  (If the source produces $ figures, use those.)
  2. If the service provides an option to adjust league settings, set it to the Razzball Commenter League format (12 team MLB, C, 1B, 2B, SS, 3B, 5 OF, CI, MI, UTIL, 9P).
  3. Use these player $ figures to create team $ values for all Razzball Commenter League teams (576 teams across 48 leagues in 2012) based on their drafted roster.
  4. See how these team $ figures correlate with that team’s final standings points

In 2011, I found that a team’s draft was responsible for about 64% of its success.  In 2012, this number went down to 53.3% – meaning that in-season player moves had a greater impact on team standings.  Of course, this was a year where the top hitter (Mike Trout) started the year in the minors and the top-rated pitcher (RA Dickey) went undrafted.  If I credited Mike Trout’s stats to the 48 teams that received the most Trout AB during the year, this would change the % to 58.7%.  (Note:  Trout was drafted in 16 of 48 RCLs and 11 of those teams dropped him or traded him early so that he had more ABs on another team in the league.  Ouch.  Two more Trout stats:  1) The average RCL team with Trout had 84.3 points, the average team without Trout had 63.1 points.  2) According to RCL Commissioner VinWins, RCL teams with 100+ Trout ABs won 15 of the 48 leagues.)

So with 53% correlation as the proverbial ceiling for the ranking sources, let’s see how well the various ranking sources predicted success.

2012 Fantasy Baseball Rankings – Ranked By How Well They Predict Razzball Commenter League Success*
Rankings Source $ or Rankings Calc’d or Curated Avg Correlation Vs Other Sources Published Before March 1st Correlation of RCL Post-Draft Team Value with Total Team Standings Pts
2012 2011
End of Season Stats $ Calc 53.3% 63.7%
KFFL $ ? 90.2% N 15.6% (3.5%)
Last Player Picked $ Calc 92.7% N 15.0% (5.8%)
Razzball Point Shares $ Calc 91.5% N 14.7% 8.0%
Rotochamp $ Calc 92.3% N 13.4% 0.0%
FantasyPros.com Rank Mix 95.9% N 6.9% (3.9%)
USAToday Rank Curated 93.5% N 6.4% (3.9%)
Razzball – Grey Rank Curated 92.4% Y 6.1% 7.7%
ESPN-Matthew Berry Rank Curated 94.2% N 5.6% (9.3%)
HardballTimes Rank Curated 94.0% Y 4.9%
ESPN Custom Dollar Value Calc $ Calc 93.0% N 3.5%
CBSSports $ ? 92.4% ? 2.0% (8.0%)
ESPN-Tristan Rank Curated 94.8% N (Y but updated in March) 1.8%
SI.com $ Curated 92.6% Y (2.5%) (7.7%)
ESPN-Top 300 $ Curated 94.8% N (5.7%) (12.2%)

* The following players were removed from the rankings as significant changes happened to their value post-publication of some of the rankings:  Injuries (Carl Crawford, Chase Utley, Chris Carpenter, Ryan Madson, Joakim Soria), Increase in Value Because of Injuries (Sean Marshall, Greg Holland, Jonathon Broxton)

General Findings

  • The ranking sources did much better in 2012 vs. 2011 despite the major impact that Trout and Dickey had on team results.
  • While Grey and I did not dominate the rankings like we did last year, we still fared well with both of us finishing in the top half. Also, Grey finishing #1 for sources who posted their rankings prior to March 1st (which is very helpful for early drafts).  I’m planning on having Point Shares out in mid-Feb for 2013 with 1-2 updates in March to help all drafters.
  • Aside from ESPN’s custom value calculator, the sources that calculated the rankings vs. curated the rankings performed much better.  The only possible exception is KFFL which I suspect has some calculations behind it.
  • KFFL was the most unique in its player values vs the other ranking sources with a 90% average correlation.  They were also the most unique vs. FantasyPros’ aggregation (92.6%).  My Point Shares were the 2nd most unique (91.5% avg correlation vs. other sources, 93.5% with FPro).
  • Not surprisingly, FantasyPros’s aggregation of rankings were the least unique amongst the sources.  Honestly, I see little value in aggregated rankings.  How different is it from just using ADP?  That said, the safest play performed well this year – at least relative to the other curated ranking sources.
  • While KFFL’s rankings performed best, I caution using their $ values in 12-team auctions as they are too high.  While the average team value was $255 across all the sources (very close to $260), KFFL’s average was $304.
  • For those wondering why our friends at Yahoo! are not in this study, it is because Yahoo! only publishes a top 100 and the minimum needed for this study is Top 200.
  • ESPN’s Top 300 finished last for the 2nd straight year.  As with last year, I think it is possible that because our RCLs were hosted on ESPN, the ESPN Top 300 is very close to the default ESPN rankings which are used by bad/auto-drafted teams.
  1. Jacob says:
    (link)

    Could you go into further depth on how these correlation percent numbers translate to expected vs. actual performance? Did Grey predict 6.1% of the outcome? Or would his predictions have contributed 6.1% to a teams chance of winning? If that’s it, how is that calculated?

  2. I interpret it as Grey’s rankings predict 6.1% of a team’s success, about 47% in addition could be explained by the actual performance of a team’s drafted roster, and the other 47% is explained by all other factors including in-season moves/pickups.

    I explained the calculation in the post. It’s correlating the predicted worth of each team’s draft (using auction $) vs the team’s final standing points.

  3. drknss says:
    (link)

    Good work guys, as usual dominating those clowns at the Entertainment and Sports Paparazzi Network.

    What is your guys take on the high ranking of the top relievers in the player rater? Obviously Rodney had a fluke season, but the numbers that Chapman and Kimbrel put up are not that out of line from what we expected (for Chapman, at least what we would expect once he learned how to throw strikes).

    Do you see yourself ranking guys like Kimbrel and Chapman higher next year?

    • Thanks. I think Rodney vs. Kimbrel/Chapman are two different phenomenons. Rodney’s performance – from a Saves/ERA/WHIP standpoint were completely unforeseeable. His high ranking (#27) is driven by a high # of saves (48) and a ridiculous 0.60 ERA (2.67 XFIP). His K value – which is the most skill-dependent (or least luck-dependent) statistic was actually below average for a rosterable reliever in 12-team (-0.1 Point Shares). So I’d have a hard time ranking Rodney any higher than middle of the pack for relievers.

      Chapman and Kimbrel are a different story as they excel at K’s. Kimbrel’s 116 K’s (in 62 IP) and Chapman’s 122 K’s (in 72 IP) are absolutely incredible. The crazy K-rates were predictable at the start of the season for these two and foretold solid ERA/WHIP numbers. Chapman was not slated to be closer which depressed his value but still made him draftable (i had him at $7 for 12-team leagues mainly based on K’s). But the factor that put both players over the top of expectations were massive gains in control – Kimbrel went from a 3.74 BB/9 to 2.01 and Chapman went from 7.38 to 2.89. This made their ERA/WHIP contributions that much better.

      Given their phenomenal K-rates and improved control, I think Kimbrel and Chapman will likely end up in the top 40 for my 2013 Point Shares (Kimbrel was #29 last year and that assumed worse control). Just to give an idea from 2012, I had Kimbrel at $24 (finished at $27), Papelbon at $20 (finished at $19), and then Axford at $17 (finished at $11). I could see Kimbrel/Chapman at $25 and maybe Janzen or Motte next at $19.

  4. Chris says:
    (link)

    Surprised to see so many positive correlations this year, especially vs the results from 2011.

    But as you said in the opener (re: Trout, Dickey, etc), it wasn’t a normal season at all…

    • Agreed. With Trout, Dickey, and the closerpocalypse, I’d have thought correlations would be lower than 2011.

  5. Gibby says:
    (link)

    Good take on your answer to drknss. Gotta agree with your Kimbrel and Chapman take

    It’s interesting. This year I have been doing the Daily Fantasy sports instead of the seasonal fantasy sports and have been enjoying it. BUT I miss the season long fantasy teams. Well, except for those inconvenient injuries and jail time :)

Comments are closed.