We’d also probably have to repeat the test a number of times to account for a player’s value being somewhat league-dependent, but this seemingly would give you an average point value for a particular player. Seems pretty labor intensive, but maybe someone smarter than me could write a program that would do this.

I think we’ll have to agree to disagree on the LPP tests. I just don’t think being “realistic” compared to a normal draft applies when we know what the end stats will be. That said, there were some other issues with those tests that I haven’t quite figured out how to deal with. Anyway, I don’t want to run a test that hasn’t been agreed upon to be “fair,” so I’ll try to think of something else. Maybe Derek has some ideas…

The test I ran that avoids these biases is to formulate several rosters based on either 2008 stats or 2009 projections (using Mock Drafts). 2008 stats are easier because you don’t have to worry about which projection system is being used. A system that accurately valued players would do a good job at valuing these teams. But so would a system that is completely correlated to the stats – e.g., valuing 50 SBs as 5 pts and 5 SBs as 0.5 pts.

Here is an interesting test that would get rid of this issue…Take a league that has either been created based on 2008 final stats or mock drafted for 2009. Total the stats up so we have team standings as a baseline. We then take 20 or so players and trade them to every team for another player at that same position – e.g., McCann and Mauer traded, McCann and Russ Martin traded. After each trade, look at the point totals. Take the sum of the point totals for every trade variation. (Note: for OFs, SPs, and RPs, I’d swap with the 3rd OF, 3rd SP, and 2nd reliever to try and be as close to ‘average’ as possible).

This test would still have its issues but better than setting up unrealistic draft scenarios.

The best hypothetical test would have the methodologies dynamically recalculate after every draft selection. I’m not smart enough to program that…

i.e.
Raw value – Position Average
or
Raw value / Position Average

?

Finally, the test you ran with the Point Shares versus the other systems was a very interesting one, but I believe it answers a different question. Namely, “which valuation system will best project the standings in a typical league?” I believe however, that this is a different question from “which ranking system should I use to draft with?”

Anyway, I’ve likely alienated most of the readership with these comments, but I do believe we need to address these things if we are to create a test that best represents the differences between valuation systems.

I don’t believe an innings pitched minimum would be correct to enforce; let me explain why. When an average or replacement level is calculated from a given set of players, we must use the “best” set of players. If this optimal set includes more RP that one might think is “realistic,” that’s just the way it is. Just because it doesn’t seem realistic doesn’t mean that some of those middle relievers weren’t in fact more valuable than some commonly-rostered #5 starter types. If the argument is that when calculating the baseline (either avg or rep) one used more SP, then I think we should rethink that arbitrary inclusion of more SP when calculating said baselines. Of course, if you are tied to including more SP simply because that’s what most leagues do, we could just change the roster requirements to include 5 SP, or whatever. But again, I disagree with he masses on this point, and think that good middle relievers are undervalued. And isn’t an objective value system supposed to show us when the masses have made an incorrect assumption? In the LPP experiments, I didn’t think the problem was that the LPP teams punted Ks and Ws (if that’s the optimal strategy then that’s what you should do), but rather that since there were multiple teams using the same ranking system/strategy, the penalty for punting a category was less severe. In other words, it was hard to isolate the merits/shortcomings of a particular system because of the presence of that same system in the league. But if we only included one of each system, I think this problem would be minimized.

You first bring up the issue that players’ values change during a draft, according to both who you have taken for your team and the remaining available players. This is undeniably true, but a static valuation system like Point Shares or the Price Guide or Derek’s system by definition does not take this into account. In a draft what we really are after is marginal value – what is the difference in value between the best player available at a given position than the next best (or next few) players at that same position, as compared to the difference in value between the best players and next best players available at the other positions? A dynamic system (perhaps some sort of drafting software) might be able to make these moment-to-moment calculations, but this isn’t typically what we’re talking about when we’re talking about valuing players in a vacuum. In short, I believe that in evaluating the merits of a particular system we need to step back and let the system choose according to the values it has. Interfering and saying that since it just picked Sabathia then it shouldn’t choose a pitcher again, even if that’s who the system says is the most valuable, is defeating the purpose of the experiment.

The test isn’t perfect but I think it’s better than a snake draft test. If Derek has SGP/VORP/THTNS for it, I’ll tie it in and then share the analysis…

Do you have McCann higher than Wright? I know you have Scrappy Doo (Pedroia) higher than me….(he’s only #99 on CHONE)

Obviously, this isn’t exactly how ranking systems are used during a real draft, but I think it might serve to point out the differences between the different systems. I think we’d have to place some sort of conditions on the CI, MI, and especially DH slot, as we don’t want someone drafting a catcher as a DH. The other bug is that ideally I’d like to test 10 or 12 different systems against each other – maybe ESPN, RotoWorld’s rater, etc. I think Mays would probably agree to enter his Price Guide as well. But I don’t think that would get us to enough teams. So we could either make it a smaller league, or enter two or three teams from each system.

This is just an idea. If someone else has a better plan as to how we could compare the different valuation systems, I’m all ears…

Nick,
What would you propose?

I’m endlessly fascinated by the average vs. replacement baseline debate, but why don’t we just test it? We sort of tried to over at Last Player Picked, but I didn’t feel like there was a definite conclusion.

Couldn’t we just use 2008 stats and rankings based on whatever system (Derek’s SGP + replacement vs Rudy’s Point Shares) for common league settings and figure this out?

I have my doubts about the Forecasters Challenge sorting this out because in that contest it seems like we’re testing both projections and valuation systems simultaneously.

Anyway, I enjoy both of your writings very much. Thanks.

Also and maybe more importantly, it’s more solid evidence I can use when arguing to my friends when they say the Duke is a top pitcher.

To clarify, are you saying that you’ve understood me better here, or just understood some of the concepts better because of the way I laid them out?

If you ever have any trouble comprehending anything I write (or anything you hear anywhere else, for that matter), don’t ever hesitate to send me an e-mail. I’m happy to help out anyone who needs it.

as sometimes is the case , the discussion is more “prized” than the article .
comments i particularly enjoyed are : “stat fight !” , “magnificient bastard site ” , and , nadav’s “suite of measures” .

i must say that derek has made more “sense” to me , in this one thread , than i have ever understood before . perhaps i will return to his site and give it another go .

dicaprio , however , remains an enigma to me . i’m sure the connection is broken on my end . not precisely double-entendres .
more like tongue-in-cheek .

of course , my caveat ==> i am often mistaken .

still , this is good reading !! ….my thanks to ALL .

I don’t like Strikeouts as a hitting category. I’d replace that with AVG. And while TB will credit the otherwise ignored Doubles/Triples, it feels weak not having HR.

For pitching, this format begs teams to only start upper-echelon starters and stock up on relievers. The only stat where a mid-tier pitcher helps is in Wins. They hurt on Losses, useless on Saves/Holds, and may not be of great help for ERA and K/BB ratio. Would prefer a counting stat (K’s) over the K/BB ratio and remove Losses for WHIP (which is one of the better stats).

This basically means standard 5×5 with OBP and Holds thrown in. Kind of bleh for me….I say either do 5×5, a points league, or go the full sabermetric route….

Just my 2 cents…

@Clodbuster
The answer is yes. I actually did this last month, part of the rapidly increasing THTNS/Day (# of new stats made up by Hardball Times per day) stat Rudy mentioned
You can find the article here (or click my name on this post).
It’s called CAPS (Context Adjusted Pitcher Statistics). It adjusts not only a pitcher’s batting average, but all of his component skills and luck indicators (K, BB, GB, BABIP, HR/FB, etc).
I improved upon it here with individualized Road Park Factors (http://www.hardballtimes.com/main/fantasy/article/introducing-quality-of-opponent-adjustments-and-caps-for-pitchers/) and right now CAPS does this:
1) Home Park Factors
2) Road Park Factors
3) Quality of Opponent Factors
4) League Change Factors
5) Ground balls adjusted for league average line drive rate

Factoring all that in, you can get some pretty cool findings. You’ll have to read the article to see who I think is the biggest beneficiary for 2009, though

Unfortunately, I haven’t made public the entire database yet, although if you ever have any questions about a particular pitcher, please feel free to shoot me an e-mail and I’d be happy to help you out!

@Clodbuster
The answer is yes. I actually did this last month, part of the rapidly increasing THTNS/Day (# of new stats made up by Hardball Times per day) stat Rudy mentioned
You can find the article here: http://www.hardballtimes.com/main/fantasy/article/introducing-quality-of-opponent-adjustments-and-caps-for-pitchers/ (or click my name on this post).
It’s called CAPS (Context Adjusted Pitcher Statistics). It adjusts not only a pitcher’s batting average, but all of his component skills and luck indicators (K, BB, GB, BABIP, HR/FB, etc).
I improved upon it here with individualized Road Park Factors (http://www.hardballtimes.com/main/fantasy/article/introducing-quality-of-opponent-adjustments-and-caps-for-pitchers/) and right now CAPS does this:
1) Home Park Factors
2) Road Park Factors
3) Quality of Opponent Factors
4) League Change Factors
5) Ground balls adjusted for league average line drive rate

Factoring all that in, you can get some pretty cool findings. You’ll have to read the article to see who I think is the biggest beneficiary for 2009, though

Unfortunately, I haven’t made public the entire database yet, although if you ever have any questions about a particular pitcher, please feel free to shoot me an e-mail and I’d be happy to help you out!

As to FIP, yes, it is definitely better than ERA. It’s not as good as something like xFIP or LIPS ERA or QERA because it doesn’t normalize HR rate, but it is still better than ERA.

To clarify and summarize a little bit, we have component skills (K/9, BB/9, GB%, etc) and we have ERA estimators (FIP, LIPS, DIPS, QERA, etc). Those component skills are used to determine the ERA estimators, so it’s not really a matter of which is “better” since they sort of say the same thing. The ERA estimators give the final picture, but they’re not better per se.

Tim Lincecum, for example, had a 9.2 K/9 and 4.0 BB/9 in 2007 for a 3.63 FIP. Looking at this, we can say (and I did) that Lincecum has a chance for big improvement because he strikes out a lot of batters, posts a good FIP, yet walks a good amount of batters. If he can cut down on the walks (the easiest of the Big 3 skills to improve upon), he could have a big year. As we saw, he did just that (3.3 BB/9 in 2008).

So it’s more a matter of perspective than “this is better” in the case of component skills versus ERA estimators.

Results would probably be best for identifying elite set-up men and forecasting a pitcher who’d be moving to a new team/division, but I think they’d be relevant for ranking all starters if someone were to put in the time.

It’s helpful to know that say, Randy Johnson held batters to a .220 batting average last year but if we dig deeper, we find that those batters he faced, only managed a combined .225 avg. to begin with, making his performance less impressive. These #’s are fictional by the way.

Is there someone out there with software that keeps track of every batter’s Avg. at the time the pitcher faced him to come up with somewhat of a strength of schedule?

I followed your suggestion of trying (K-BB)/IP and found a few examples of guys who would be undervalued by relying on K/BB instead (based on 2008 stats):

Felix Hernandez vs. Kyle Lohse: Felix has a lower K/BB, but a higher (K-BB)/IP.

Tim Lincecum vs. Mike Mussina: Tim has a lower K/BB, but a higher (K-BB)/IP.

So basically, K/BB undervalues high-K, high-BB pitchers, relative to low-K, low-BB pitchers. Which is pretty much what you said in your post above. I now have a better picture of why relying too much on K/BB can be misleading.

Thanks!

And Derek, what’s wrong with FIP? Is it not better than ERA? Or are no ERA-type projections as important as the pitcher’s component skills?

I was under the impression FIP is helpful to predict for the next year, should I stop?

I followed the link you provided for Tangotiger’s discussion of GB/FB ratios, and I understand your/his point about “double the impact” better now. It looks like he was specifically talking about the use of the ratio in developing a composite stat that’s meant to provide a version of OPS with more predictive value. A parallel example for K/BB would be if you were generating a FIP-type stat using K/BB instead of properly-weighted versions of K% and BB%.

I’m still not 100% convinced that the extra insight provided by the K-BB RI stat is worth the extra work (at least for those of us who don’t already have spreadsheets programmed to calculate it). In your post about the stat, you mentioned the possibility of another post that would show how certain pitchers would be valued differently using K-BB RI instead of K/BB, but I wasn’t able to find a post like that in your archives. Do you happen to have any examples of where K-BB RI would give you a better picture of a pitcher’s value, while K/BB would lead you astray?

1) I’m not sure I fully understand your point here. Sure, use it as a suite, but if we’re going to combine K/BB with GB%, we’re still not getting an accurate view of the pitcher. Combining it with something else doesn’t eliminate the inherent inaccuracies with it. Using K% or GB%, as you said, is fine, because there’s nothing inherently flawed about those stats.

If you’re going to look at a suite, K/9, BB/9, and GB% are the three you want to look at.

Again, K/BB isn’t useless, just less accurate than something like K/BB RI or K/9 and BB/9 separately.

2) If I’m using a more complex stat like FIP or xFIP, I would be looking at other factors. K/BB RI isolates just Ks and BBs. FIP and xFIP deal with other factors like GB% and HR rate and such. Essentially, what we’re really trying to do is find ERA, so something like LIPS ERA would sort of be the end game in that regard. But if we’re curious how pitchers are doing it, we look at their component skills, which grow and decline at different rates and have important implications for future performance.

3) As to my argument against ratios, this article by Tom Tango at The Book Blog will be able to do it better or more succinctly (my attempts at being succinct for the purpose of this roundtable seem to have failed) than I: http://www.insidethebook.com/ee/index.php/site/comments/ratios_or_rates/

It deals with GB/FB, but the argument is the same, and the solution is generally the same: look at GB% and FB% separately, just as I like look at K/9 and BB/9 separately.

Finally, while the top and bottom guys do have high/low K/BBs, it’s because K/BB is still measuring two important stats. That will never change, so it’s hard to screw it up too bad. But it’s still flawed, and that was my point. Again, if you’re going for something quick, go with (K-BB)/IP instead. Subtracting is better than dividing in this instance.