Fairness Metrics¶
Supported metrics for model fairness checks are configurable. You can change the metrics for the selected project and/or model in Project Settings (pictured next) after signing in at app.truera.net.
TruEra supports a wide variety of fairness metrics for both classification and regression models. Selecting the appropriate root cause analyses to understand the sources of bias is discussed under Exploring TruEra Diagnostics in Fairness.
Note
Measurements are labeled TP (true positive), FP (false positive), TP (true negative), and FN (false negative).
Classification models¶
| Name | Range | Notes |
|---|---|---|
| Disparate impact ratio | [0, +inf) | Ratio of proportion predicted to be the y=1 class (i.e. (TP + FP) / (TP + FP + TN + FN)). |
| Statistical parity difference | [-1, 1] | Difference of proportion predicted to be the y=1 class (i.e. (TP + FP) / (TP + FP + TN + FN)). |
| True positive rate ratio | [0, +inf) | Ratio of true positive rates: TP / (TP + FN). |
| True positive rate difference | [-1, 1] | Difference of true positive rates: TP / (TP + FN). |
| False positive rate ratio | [0, +inf) | Ratio of false positive rates: FP / (FP + TN). |
| False positive rate difference | [-1, 1] | Difference of false positive rates: FP / (FP + TN). |
| True negative rate ratio | [0, +inf) | Ratio of true negative rates: TN / (FP + TN). |
| True negative rate difference | [-1, 1] | Difference of true negative rates: TN / (FP + TN). |
| False negative rate ratio | [0, +inf) | Ratio of false negative rates: FN / (TP + FN). |
| False negative rate difference | [-1, 1] | Difference of false negative rates: FN / (TP + FN). |
| Equality of opportunity ratio | [0, +inf) | Equivalent to true positive rate ratio when the y=1 class is favorable and true negative ratio otherwise. |
| Equality of opportunity difference | [-1, 1] | Equivalent to true positive rate difference when the y=1 class is favorable and true negative difference otherwise. |
| Predictive parity ratio | [0, +inf) | Ratio of predictive parity: TP / (TP + FP). Also referred to as "acceptance rate ratio". |
| Predictive parity difference | [-1, 1] | Difference of predictive parity: TP / (TP + FP). Also referred to as "acceptance rate difference". |
| Treatment equality difference | (-inf, +inf) | Difference of ratio of false positive rate to false negative rate. |
| Conditional acceptance difference | (-inf, +inf) | Difference of conditional acceptance: (TP + FN) / (TP + FP). |
| Conditional rejection difference | (-inf, +inf) | Difference of conditional rejection: (TN + FP) / (TN + FN). |
| Average odds difference | [-1, 1] | Difference of average of true positive rate and false positive rate. |
Choosing a fairness metric for classification models¶
Regression Models¶
| Name | Range | Notes |
|---|---|---|
| Mean score difference | (-inf, +inf) | Difference in average model score. |
| L1 error difference | (-inf, +inf) | Difference in average L1 error (i.e. mean absolute error). |
| L2 error difference | (-inf, +inf) | Difference in average L2 error (i.e. mean squared error). |
Click Next below to continue.