Colts Draft Analysis Still Leaves Unanswered Questions

With the Colts season in shambles already halfway through the season, attention around the blogosphere has turned from what's actually going on on the field to how did it get this bad and how can it be fixed. It's something other NFL teams have done regularly starting in Week 9, but this is a new experience for us Colts fans. Over at 18to88.com yesterday, Ajit Kirpekar decided to find out for himself how the Colts have done draft-wise since Bill Polian took over control of the team back in 1998. He put a lot of time and effort into this, and he should be commended for getting off his rump, putting in some work, and coming out of it with conclusions.

However, he's unable to actually answer the question Colts fans want to know: Has Polian's draft record fallen off over the past five years?

Kirpekar concluded that, from 1998-2010, the Colts were the best drafting team in the NFL:

Despite all of these potential shortcomings, the evidence seems to clearly indicate that the Colts have had a remarkable run of success and that Polian’s draft performance has been spectacular, especially considering how often the Colts pick in the later rounds.

Anyone who has been following the NFL for some time will tell you that between 1998 and 2006, the Colts were far and away the best drafting team in the league. In fact, it isn't even close, and anyone who argues against that should have their head examined. Super Bowl XLI doesn't happen for the Colts without the draft class of 2006. For almost a decade it seemed as if Polian could do no wrong.

After that, however, the Colts have seemed to struggle, especially at the top end of the draft, finding the right players to keep the engine humming along, and I think we started seeing those signs last season, and are definitely seeing them this season, even with the absence of the greatest QB to ever lace them up. I was hoping Kirpekar's study would be able to answer that question, but unfortunately it cannot do that for a number of reasons, despite his conclusion that the Colts have done so.

My first hesitation came when he didn't specifically say what his criteria were, how many "points" each thing was worth, and whether anything was "subjective". He clarified with me that nothing was subjective, pulled mostly from Pro-football-reference.com, which is good. "Subjective" stats are ok in certain circumstances, but doing this study with every team in the NFL would add way too much error to it, as it would just be one person's opinion on how good players were, and it would be about players he/she wouldn't follow on a consistent basis.

When looking at the data presented, it looked like he was using more of a counting method, rather than a percentage method, when looking at "Games Started", which I assume was his primary statistic here. His lowest level counting stat was "Four-year Starter", which makes it very difficult for any player drafted after 2008 to qualify for, as they haven't been in the league that long. He talks about this problem, and his solution:

The other big point to mention is that the more recent draft choices were unable to earn as many points largely because they hadn’t played for very long. For example, I included the 2010 draft even though each pick could at most only start for 2 years and earn at most 1 pro bowl and all pro vote. For the recent draft picks, I awarded a point for having started at least 50 percent or more of their eligible career but awarded the same amount of points had they achieved 1 pro bowl or all pro. As a result, recent picks had fewer points than preceding picks, but since each team was affected equally, this did not bias the results.

He's right that team-by-team, the affect is equal. However, it still leads to the results being favorable the longer each draft class was in the league, which will obviously boost the Colts. Those numbers will be smaller, thus letting all those early years dwarf them in the results. When you're talking about 13 years, it's a big difference, and it doesn't make sense to just throw that data in there if it isn't large enough to stand on its own.

Finally, I asked him the important question...How do the Colts compare with the rest of the NFL between 2007 and 2010? That's what everyone wants to know. Here's what he said:

No I didn't compare the Colts from 07-10 for the same reason I didn't readjust the point system for recent draft picks, the sample size is too small. Sure, it seems like the data is cherry picking Polian's best work but frankly, I'd deal with a large sample of his work that includes the good and the bad, even if it feels like he's being commended for past work.

My biggest issue with the whole article comes to light here. If the sample size is too small to stand alone and be compared with everyone else, then it should not be lumped into the conclusion. While his conclusion is true that from 1998-2010 the Colts were the best drafting team in the NFL, the last four years are just kind of thrown in there, admittedly by him, without their ability to stand on their own. When you take those four years out, the conclusion is "Duh, of course the Colts were the best. Are you a Patriots homer or what?"

More from Kirpekar (emphasis mine):

Sure, I imagine the Colts 07-10 would appear poor, but honestly, if there were two things I wanted people to understand, it was that drafting late means you are less likely to find a great player and second, Polian has done a much better job (the best actually) over his career and so he deserves more recognition for this.

He also includes how he thinks 2007-10 would look, and it's completely opposite of what he concludes in his story. He's trying to play both sides here, and it is deceiving, unintentionally, to the reader. His best course of action, when presented with the small sample size (which is completely reasonable to conclude from 2007-10) would be to not include them in the study and just focus on 1998-2006. He also should have given more than one sentence to his findings, if that's what he wanted people to take from his study.

Evaluating a draft is extremely hard, given the fact that meaningful stats for several position are hard to come by. Stampede Blue did this back in April, and found a dropoff from 2007-09 (we didn't include 2010 due to sample size), but had nothing to compare to other teams with, so it's tough to say specifically the Colts have struggled as of late. I appreciated Kirpekar's effort in trying to shed light on the matter, but I think he focused too much on what we already knew: Namely, that Polian and the Colts Front Office kicked serious ass from 1998-2006, enough so that they are still better than everyone else when you include 2007-2010.