Review of 2011 Fantasy Baseball Player Rankings

1. The test needs to consider all the fantasy baseball categories.  If you just focus on ERA, what about Wins, WHIP and K’s?  How would that help with Closers?  So you would need to figure out a way to test and properly weight each statistical category.
2. There is a BIG difference if your pre-season ranking screw up the value of a Top 3 round player vs a late round player.  So you would need to figure out a way to weight players based on their likely draft status.
3. If a player misses significant playing time (for a hitter, let’s say 30+ games), the timing and sequencing of these missed days as well as the availability of waiver/FA pickups plays a major role in terms of the player’s impact on a team’s success.  For instance, a player like Chipper Jones who might miss games throughout the season is harder to replace than Nelson Cruz who goes on 15 day DL trips.  If a player gets sent down before opening day (like Lonnie Chisenhall), it’s even easier to adjust for the loss in expected playing time.  So you would need to credit replacement level performance that varies depending on the timing/sequencing of missed playing time.
4. The timing and sequence of a player who underperforms can dramatically change how a player impacts one’s fantasy team.  Let’s say Player A and Player B had equal pre-season rankings and both underperform, hitting .220 with 15 HRs for the season.  If Player A goes .150 with 0 HR in April and Player B goes .240 with 3 HR, Player A will likely be dropped quicker and that team might find a replacement player who performs above replacement value.  So you would need to account for how quickly/slowly a player was replaced as well as estimate the impact of the replacement player.

Now, I can see an argument that particularly #3 and #4 come down to chance vs. a prognosticator’s skill.  But that’s not entirely true.  Some players are more injury-prone than others and they may be marked down accordingly (e.g., I knocked Kinsler’s playing time down about 50-75 PAs from my sources which took his value down about 10 spots in the rankings.  Shouldn’t I be credited/hurt because of that choice?).  And while there is some chance involved, I’d argue it’s still better to reflect the real impact these chance instances had on teams vs. find ways to remove them (like testing a rate stat like HR per AB instead of total HR).

So here is the test I put together that I think addresses these ‘near impossible to model’ variables:

1. Take the draft results by team from the 38 Razzball Commenter Leagues in 2011 (hosted on ESPN using ESPN’s default league formats for 12 team 5×5 MLB leagues).  This amounts to 456 teams’ worth of draft data.
2. Create team total values based on ‘expert’ rankings/$ totals/other arbitrary metric (like Point Shares or THT Z-Score)
3. See how these team totals correlate with each team’s final Total Standings Points

Since we’re testing based on a team’s pre-season aggregated player value and their end of year Total Standings Points, we are factoring in all the categories (#1).  The pre-season rankings provide a natural way to weight each player’s value/impact by matching them up against ranked auction dollar amounts (#2).  Since the end of year Total Standings Points reflects how fantasy baseball managers behaved when facing situations involved lower than expected playing time or performance, it accounts for these two last variables (#3 and #4).

Hope that makes sense to everyone!  Now on to the good stuff!

2011 Pre-Season Rankings Tested (click here for the aggregated results, links provided for rankings that are still posted):

• CBSSports Top 300
• ESPN – Matthew Berry Top 200
• ESPN – Top 300
• FantasyPros Aggregated Top 300
• FoxSports Top 300
• HardballTimes – Jeffrey Gross (never published)
• KFFL Top 200
• Last Player Picked (2011 Composite)
• Razzball – Grey’s Rankings
• Razzball – Rudy’s Point Shares – March 8
• Razzball – Rudy’s Point Shares – late March (forgot to publish – doh!)
• RCL ADP All (456 teams)
• RCL ADP Top 100 finishers
• RotoChamp top 300
• RotoExperts Top 300
• SI.com top 300
• USAToday Top 200

Here are some general notes before I start throwing data out:

• Most rankings do not provide $ estimates.  Since there is a larger gap in value between earlier draft picks vs. later (e.g, average gap between 10th and 11th best players greater than difference in 150th vs 151st best players), I converted all rankings into dollar estimates by providing the $ estimate for the corresponding rank from my Point Shares.  I used the published dollar estimates if they were available (vs forcing my $ estimates).  For HardballTimes and my Point Shares, I used the player scores since they estimate the value between picks (vs. just an arbitrary 1 between picks when done as rankings).
• I believe all these rankings were released between Feb 1 and March 30th with the majority in the March time frame.  To be fair, I gave everyone the same values for the following players whose value dropped significantly during Spring Training – even if they had not ranked them.  The values are based on my Point Share value as of late March (I put a Z-score equivalent to $6 for THT):
  • Adam Wainwright – $0 – Tommy John surgery
  • Chase Utley – $6 – Knee issues became more problematic
  • Kendry(s) Morales – $6 – Comeback plans hit a snag
• Most rankings do not specify the league format (# of teams, positions, etc).  Since the Razzball Commenter League format is about as standard as it gets (12 team, 5×5, ESPN roster format), I thought it was still fair to include in this test.  When possible, I used a source’s ’12 team MLB’ rankings/estimates.
• Some rankings (ESPN/Matthew Berry, KFFL, USAToday) ranked only 200 players.  I tested the impact of adding the next 100 players based on a composite of the other rankings and the correlation percentages decreased.  I didn’t feel this was fair so I kept it as just the top 200.
• Any player not in a source’s rankings who was drafted was valued at $0.
• I capped (or is it floored?) any estimates at the equivalent of $0 as only some sources such as Point Shares and Last Player Picked report negative dollar values for players.  This is probably for the best, anyway, as players with negative projected value before the draft are dropped like luxury good brand names in rap songs – early and often.
• In the previous post, I tested whether there was a clear bias in the RCL teams’ draft behavior.  The ADP of the RCL teams’ correlated closer to the ESPN Top 300 (96.7%) than either Grey’s rankings (92.6%) or my Point Shares (81.5%).  While I don’t have a second non-RCL sample to confirm, I would theorize the largest bias in the RCL teams’ behavior is driven by ESPN being the draft host and providing default rankings.

Here are the test results – see the following file for the raw results including the draft picks per team and the total standings points per team:

Chart #1:  Correlation % Between Team Final Standings’ Points & Team Drafted Player Projected Value By Rankings Source 

• While Chart #2 is a more straightforward test (e.g., compares rankings directly vs. running the results through 456 RCL teams), I think Chart #3 is the better test.
  • Example #1:  Grey’s rankings look close to the other systems in Chart #2 but are wildly unique when valuing RCL Teams.  If Grey’s rankings were as close as Chart #2 suggests, his team values wouldn’t have been so divergent from the rest of the sources (8% correlation is really low)
  • Example #2:  Point Shares should correlate highest with three other automated/quant-based systems (HardballTimes, RotoChamp, Last Player Picked).  This is not the case in Chart #2 but clearly the case in Chart #3.
• The four most unique rankings in Chart #3 finished in the top 4 in correlating to team success while the 5th (Matthew Berry) finished 2nd to last.  I don’t think this isn’t a coincidence – the most unique rankings should be furthest above/below the consensus rankings.
• It is odd how ESPN’s Top 300 could finish last and yet be so safe in terms of similarity to consensus rankings.   FantasyPro’s Aggregated Rankings finished middle of the pack so there appears to be some safety in consensus.  Seems most likely that ESPN just had some bad luck.
• The 4 quantitative-based solutions (HardballTimes, Point Shares, Rotochamp, and Last Player Picked) finish in the top 6 for uniqueness.

Two unsubstantiated theories:

• I have no idea how to prove this but here’s a theory for why so many rankings are below 0%….Most of the ‘curated’ rankings reflect conventional fantasy baseball thinking (e.g., don’t draft pitchers in the first 15 picks).  This conventional thinking either feeds or just mirrors what ends up as the default rankings within draft software.  Weaker players lean on the default rankings more than stronger players.  While success in a 12-team mixed league requires a lot of luck, there is enough skill during the draft (and in post-draft roster moves) that weaker players in aggregate will perform worse than stronger players.  If weaker players depend more on default rankings/ADPs than stronger players, a negative correlation would arise between team success and any ranking system similar to default rankings.  (If correct, this also means that if the RCL somehow used Grey’s rankings or my Point Shares as the default rankings, they’d automatically fare worse in a test like this.)
• The common advice of “Zag when others zig” seems true for both experts who publish rankings and fantasy baseball players who seek those rankings out.  If I was curating a rankings list, I’d follow Grey’s lead and take a lot of chances.  (We’ll see if Grey has similar success in 2012).  ‘Safe’ rankings have value only if people are drafting without default rankings (or with really bad default rankings).  If I were building a system from scratch to quantify player values, I would do my best to avoid using traditional rankings as a benchmark (I’d also advise against this because there’s a lot more learning curve than you think….I’d just use Point Shares for my draft and invest that time doing something that might get you laid like learning to play guitar.)

Other Notes:

• I plan on running a similar test at the end of 2012.  We’ve got about 10 more RCLs than last year (38 to 48) – more sample is always better.
• Since I got such a late start on this analysis, I’m going to hold off analyzing stat projections (Marcel, ZiPs, Steamer, etc.) until the offseason and have 2 years of data to reference.  I”ll only be including projections that are directly available for users.  Projections available for subscription will require direct permission from the publisher.
• The worksheet provided in Google Docs is considered public domain – feel free to create and publish complementary or contradictory analyses  with it.  Please just link to this post and note the data is available courtesy of Razzball.

Final Note:

• Huge thanks to our RCL Commissioner VinWins who compiled all the draft/team information and produces some great stats throughout the year such as this post summarizing the 2012 drafts.