We’d also probably have to repeat the test a number of times to account for a player’s value being somewhat league-dependent, but this seemingly would give you an average point value for a particular player. Seems pretty labor intensive, but maybe someone smarter than me could write a program that would do this.

I think we’ll have to agree to disagree on the LPP tests. I just don’t think being “realistic” compared to a normal draft applies when we know what the end stats will be. That said, there were some other issues with those tests that I haven’t quite figured out how to deal with. Anyway, I don’t want to run a test that hasn’t been agreed upon to be “fair,” so I’ll try to think of something else. Maybe Derek has some ideas…

The test I ran that avoids these biases is to formulate several rosters based on either 2008 stats or 2009 projections (using Mock Drafts). 2008 stats are easier because you don’t have to worry about which projection system is being used. A system that accurately valued players would do a good job at valuing these teams. But so would a system that is completely correlated to the stats – e.g., valuing 50 SBs as 5 pts and 5 SBs as 0.5 pts.

Here is an interesting test that would get rid of this issue…Take a league that has either been created based on 2008 final stats or mock drafted for 2009. Total the stats up so we have team standings as a baseline. We then take 20 or so players and trade them to every team for another player at that same position – e.g., McCann and Mauer traded, McCann and Russ Martin traded. After each trade, look at the point totals. Take the sum of the point totals for every trade variation. (Note: for OFs, SPs, and RPs, I’d swap with the 3rd OF, 3rd SP, and 2nd reliever to try and be as close to ‘average’ as possible).

This test would still have its issues but better than setting up unrealistic draft scenarios.

The best hypothetical test would have the methodologies dynamically recalculate after every draft selection. I’m not smart enough to program that…

i.e.
Raw value – Position Average
or
Raw value / Position Average

?

Finally, the test you ran with the Point Shares versus the other systems was a very interesting one, but I believe it answers a different question. Namely, “which valuation system will best project the standings in a typical league?” I believe however, that this is a different question from “which ranking system should I use to draft with?”

Anyway, I’ve likely alienated most of the readership with these comments, but I do believe we need to address these things if we are to create a test that best represents the differences between valuation systems.

I don’t believe an innings pitched minimum would be correct to enforce; let me explain why. When an average or replacement level is calculated from a given set of players, we must use the “best” set of players. If this optimal set includes more RP that one might think is “realistic,” that’s just the way it is. Just because it doesn’t seem realistic doesn’t mean that some of those middle relievers weren’t in fact more valuable than some commonly-rostered #5 starter types. If the argument is that when calculating the baseline (either avg or rep) one used more SP, then I think we should rethink that arbitrary inclusion of more SP when calculating said baselines. Of course, if you are tied to including more SP simply because that’s what most leagues do, we could just change the roster requirements to include 5 SP, or whatever. But again, I disagree with he masses on this point, and think that good middle relievers are undervalued. And isn’t an objective value system supposed to show us when the masses have made an incorrect assumption? In the LPP experiments, I didn’t think the problem was that the LPP teams punted Ks and Ws (if that’s the optimal strategy then that’s what you should do), but rather that since there were multiple teams using the same ranking system/strategy, the penalty for punting a category was less severe. In other words, it was hard to isolate the merits/shortcomings of a particular system because of the presence of that same system in the league. But if we only included one of each system, I think this problem would be minimized.

You first bring up the issue that players’ values change during a draft, according to both who you have taken for your team and the remaining available players. This is undeniably true, but a static valuation system like Point Shares or the Price Guide or Derek’s system by definition does not take this into account. In a draft what we really are after is marginal value – what is the difference in value between the best player available at a given position than the next best (or next few) players at that same position, as compared to the difference in value between the best players and next best players available at the other positions? A dynamic system (perhaps some sort of drafting software) might be able to make these moment-to-moment calculations, but this isn’t typically what we’re talking about when we’re talking about valuing players in a vacuum. In short, I believe that in evaluating the merits of a particular system we need to step back and let the system choose according to the values it has. Interfering and saying that since it just picked Sabathia then it shouldn’t choose a pitcher again, even if that’s who the system says is the most valuable, is defeating the purpose of the experiment.

The test isn’t perfect but I think it’s better than a snake draft test. If Derek has SGP/VORP/THTNS for it, I’ll tie it in and then share the analysis…

Do you have McCann higher than Wright? I know you have Scrappy Doo (Pedroia) higher than me….(he’s only #99 on CHONE)