Fairness Metrics

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - Ported the following Huggingface `LambdaLR`-based schedulers: `ConstantLearningRateScheduler`, `ConstantWithWarmupLearningRateScheduler`, `CosineWithWarmupLearningRateScheduler`, `CosineHardRestartsWithWarmupLearningRateScheduler`.
+- Created the fairness module and added four fairness metrics: `Independence`, `Separation`, `Sufficiency`, and `DemographicParityWithoutGroundTruth`.
 - Added new `sub_token_mode` parameter to `pretrained_transformer_mismatched_embedder` class to support first sub-token embedding
 - Added a way to run a multi task model with a dataset reader as part of `allennlp predict`.
 - Added new `eval_mode` in `PretrainedTransformerEmbedder`. If it is set to `True`, the transformer is _always_ run in evaluation mode, which, e.g., disables dropout and does not update batch normalization statistics.

allennlp/common/testing/__init__.py

@@ -64,7 +64,13 @@ def assert_metrics_values(
     atol: float = 1e-05,
 ):
     for key in metrics:
-        assert_allclose(metrics[key], desired_values[key], rtol=rtol, atol=atol)
+        if isinstance(metrics[key], Dict) and isinstance(desired_values[key], Dict):
+            for subkey in metrics[key]:
+                assert_allclose(
+                    metrics[key][subkey], desired_values[key][subkey], rtol=rtol, atol=atol
+                )
+        else:
+            assert_allclose(metrics[key], desired_values[key], rtol=rtol, atol=atol)
 
 
 def global_distributed_metric(

allennlp/fairness/__init__.py

@@ -0,0 +1,14 @@
+"""
+This module contains tools to:
+
+1. measure the fairness of models according to multiple definitions of fairness
+2. measure bias amplification
+3. debias embeddings during training time and post-processing
+"""
+
+from allennlp.fairness.fairness_metrics import (
+    Independence,
+    Separation,
+    Sufficiency,
+    DemographicParityWithoutGroundTruth,
+)