Thursday, August 16, 2012

The DOJ's Pardon Project

In recent weeks, the Department of Justice (DOJ) has called for a statistical study of presidential pardons (see formal announcement here). We have shared the announcement with our readers (here) with limited commentary. It now appears that ProPublica has done likewise (here). We would thus like to express our own hopes / expectations for the DOJ effort.

First, and foremost, it is hoped that the DOJ will report its findings in a manner that makes its effort an notable achievement in both government and the social sciences. Indeed, we hope that the report reaches the caliber of publications found in the professional journals, especially those of political science. This can be done the following ways:

The DOJ could begin by reviewing / understanding / incorporating / testing the conclusions and findings found in the vast literature on pardons. ProPublica's goals and ambitions were certainly quite narrow, but their claim to have performed "first systematic analysis of presidential pardons" is wildly ludicrous (see Epstein and King's exemplary commentary on such hyperbole and mis-use of the language here).

The DOJ appears to at least be willing to give lip service to the existence of the universe of pre-existing literature. The formal announcement, for example, calls for the creation and testing of a "conceptual" model (contra the non-parsimonious, barefoot empiricism routine). We also note that the DOJ recognizes the importance of the timing of applications. If there is any fact empirically documented in the literature, it is that most presidents have granted the largest number of their pardons in the fourth an final year of the term. It is also known that 1 out of every 2 pardons and commutations granted over the last 39 years has been granted in a single month, December. At this late date, it is an inexcusable critical error to wholly ignore such considerations in a multivariate complex while, at the same time, boldly spewing fantastic claims about the findings.

Related to the topic of time, is the fact that the DOJ will, apparently, examine the decision making process, as an actual process, with significant, distinct steps. The ProPublica "model," on the other hand, treats outcomes as the result of a kind of undifferentiated blob of activity, capped by a presidential signature.

The DOJ will, of course, enhance the probability of covering these bases if the guiding "advisors " selected by the Bureau of Justice Statistics and the U.S. Pardon Attorney truly are "knowledgeable" with respect to pardons. Advisors - however credentialed - with knowledge of math, or a general interest in criminal justice issues, would bring little to the table.  Washington insiders, with bureaucratic experience are not so likely to fill these gaps either, although we can think of one notable exception.

Given the small sample size, and the skews in outcomes (the vast majority of applications being denied or closed without action), it will be important for the DOJ to report some PRE (proportion of reduction in error) statistic, so readers will have a clear understanding of how the model(s) improve our understanding of outcomes over a mere guess. Everyone who knows anything about analysis of dichotomous dependent variables via discriminant analysis, logit, probit, etc. knows that missing data are, more often than not fatal. Presumably, the DOJ will reduce the breadth and certainty of its claims in direct proportion to the amount of missing data involved.

Finally, in testing hypotheses as critical and complex as the impact of race in this process, it will be absolutely necessary that the DOJ transparently test rival hypotheses and clear the decks of all reasonable concerns about multicollinearity. Hiding cross correlation matrices may be an option in the world of journalism, but it isn't when an effort is both serious and complex. The same would also be true with respect to measures of questionable validity.

1 comment:

Anonymous said...

The Justice Department can do all the studies it wants but with the election fast approaching the day of reconing for the Pardon Attorney might not be far off.

blogger templates | Make Money Online