When Homer Simpson heckles Ned Flanders' Pee-Wee football coaching ability, Marge scolds him, saying "It's easy to criticize!". Homer adds, "Fun, too." When Homer actually does become the coach, he is terrible at it, and the team promptly loses every game.
It is with this in mind that we begin our criticism of Puck Prospectus, a site for which we had such high hopes when it began. The site is certainly doing some very good work, but it's also pointing out some dangerously stupid things.
Here's an example of an egregious error in a writeup on shootouts - the author writes, "Additionally, the Sharks should give Malhotra [lifetime 1 for 2 shooter] a chance to prove whether he’s got the skill or not." This is an absurd claim for two reasons. First, teams practice shootouts, so in this case, the coaches likely (not certainly) have a much larger sample from which to draw. Second, a simple application of the binomial theorem and some playing around with numbers will show that 1 for 2 in shootouts has absolutely no significance. If we assume Dany Heatley's shootout % to be true (which it likely is not), after 2 attempts, he will have a goal 26% of the time. There is no reason to think Manny Malhotra is at all skilled at the shootout, any more than Marek Malik is the greatest shootout artist of all time.
Or how about this article on age and winning which confuses cause and effect, and makes little mention of players' roles on the team when factoring in average age?
Or the obsession of one writer with hockey players' heights and weights, taking to task teams for drafting big defensemen high in the 1st round during the 90s, as though there were any doubt left that these were not good moves?
What supposedly sets apart 'sabermetrics' from traditional methods of observation is rigor, and far too often Puck Prospectus lacks it. We hold out hope that these are merely preliminary observations, and that the writers are also finding their way.
Subscribe to:
Post Comments (Atom)
Yeah, I agree. I too had high hopes when PP first came out, and I've been disappointed with the large majority of stuff they've written.
ReplyDeletethey just don't seem to acknowledge variance enough, which makes everything they say suspect.
ReplyDeletebeyond that, i see that their articles are already appearing on ESPN. i think they are following the football outsiders model of how to nudge one's way into the mass media, and in this respect, they are clearly putting the cart before the horse.
"they just don't seem to acknowledge variance enough"
ReplyDeleteyep
There are a couple of guys doing good work there, I think.
ReplyDeleteThe problem is that too many of the writers at PPro don't understand the numbers, they just take Tom Awad's GVT numbers and others as gospel and make overly certain pronouncements based on them.
For example, in the Sens preview the writer states without qualification that Kuba is their best defensive defenceman and Campoli is better than Phillips or Volchenkov. He does so because GVT says so, which is because GVT doesn't take quality of opposition into account or zone start. Phillips and Volchenkov do a ton of heavy lifting in Ottawa, which is obvious to anyone familiar with the team.
If you don't understand the numbers you are using, you can do more harm than good, and PPro is in danger of than.
it was amusing when awad said in his writeup of GVT that g. desjardins had already done that work with QUALCOMP and QUALTEAM so he was just going to leave that aspect of it alone, like that isn't a huge portion of determining the value of a particular player.
ReplyDeletealso GVT doesn't seem to adjust for year-to-year variance in shooting percentage - campoli's not going to score 11 goals on 91 shots again as he did last season. i've forgotten exactly how GVT is compiled but i imagine it overvalues these goals, much as a statistic like VORP in baseball would overvalue a hitter whose BABIP is well above expected.
Actually, I should have said that the Sens preview used GVT projections - presumably that would regress Campoli's shooting percentage, although he's still projected to score 8 goals.
ReplyDeleteBut the defensive ratings and writeup were an insult to the intelligence of Senators fans. I appreciate the value of statistical analysis, but when I see Chris Campoli projected to be a more valuable defensive defenceman than Anton Volchenkov, I want to tell the writer to watch the Senators at least a couple of times before writing a preview on them. And when that's your content appearing on ESPN, I can't imagine the average fan thinks much of it.
If the statistical model is limited, that's one thing, but acknowledge those limitations.