Accuracy Equity, Predictive Parity & Equalised Odds
Predictive parity refers to making predictions that are independent of protected or sensitive attributes (eg. gender, ethnicity, etc.). In this context, it aims to ensure that the predicted probability of recidivism, for a specific risk score, is similar across different groups.[4]
Accuracy equity refers to the equal treatment of different groups in a dataset when considering the overall accuracy of a predictive model. In this context, it aims to ensure that predictions about recidivism are equally accurate across different groups.[4]
Equalised odds refers to similar error rates for different groups when making predictions. In this context, it aims to to make similar correct and incorrect predictions of recidivism for each group. Equalizing the odds means matching the true positive rates and false positive rates for different groups. [5]
COMPAS is ensuring similar prediction accuracy for Black and White defendants, and thereby being fair via achieving “accuracy equity”. ProPublica is criticising the algorithm’s inability to achieve “equalised odds” by achieving dissimilar outcomes when the algorithm is wrong (Different false positive rates for Black & White defendants). (explored in greater detail here)
NorthPointe argue that COMPAS demonstrates accuracy equity by achieving a similar “Positive Predictive Value” (PPV) for both groups of defendants. They argue that the variation in false positive rates (FPR) is a result of differences in “base rates” of recidivism for different groups. [4]
PPV, FPR & FNR
The positive predictive value (PPV or PV+) is the probability that a person with a positive prediction will re-offend.
A high PPV means that a positive prediction (”High risk”) will result in a positive outcome (rearrest) Achieving similar PPVs for different groups serves as a measure of fairness. Let’s see how that can be explained using True Positives (TP) and False Positives (FP).
A person with a positive prediction (likely to re-offend) can fit into one of the two possibilities : The prediction turns out to be true (True positive, TP) or false (False positive, FP). PPV measures probability of true positive predictions. It can be mathematically represented as follows :
ProPublica claims that COMPAS is biased, by exhibiting different False positive (FPR) & False negative rates (FNR) for White & Black offenders.
Visually, these can be represented in this way :
