Fairness Metrics

[ArjunSubramonian]

Changes proposed in this pull request:

• Created the fairness module and added four fairness metrics: Independence, Separation, Sufficiency, and Demographic Parity Without Ground Truth.

[AkshitaB]

@ArjunSubramonian I've left some comments. The main one is regarding the metrics working for different batches. When we run the trainer, we accumulate partial info for every batch, and compute the actual metric value for each epoch (and then reset). We want something similar here. Let me know if that makes sense, or if you would like to discuss it further.

[AkshitaB]

Minor comment: it'll be nice to have links to the papers too, similar to this in markdown format. Then, anyone reading the documentation can simply click on it.

[AkshitaB]

Also, we like semantic scholar links. 😄

[AkshitaB]

So, if I understand correctly, the protected_variable_labels will correspond to a single protected attribute, such as race, right? If we want to measure independence with different protected categories, say, race, gender, sexuality, then we setup a different Independence metric for each of them?

[ArjunSubramonian]

Yes, that's correct

[AkshitaB]

These metrics do not support the distributed setting currently. This is fine, but we should add a check similar to this so that users know.

You can also take a look at this and this for examples of metrics working in distributed settings.

[AkshitaB]

Also, this will compute the distributions correctly only for a single batch, right? We want that the distributions are computed based on all batches. (Again, you can take a look at other metrics for how this works). Ideally, the __call__ function accumulates the distributions in some way, and the get_metric() function computes the final kl divergence. (I'm not sure how efficient this will be, though).

[AkshitaB]

It might be helpful to add a small example of how the output can be interpreted in a real-world context. Eg. kl div of nearly 0 for some protected variable implies independence, which is useful in a certain context.

[epwalsh]

You could do format like this to make it look better on the docs site:

Suggested change

	Note: all tensors are expected to be on the same device.
	!!! Note
	All tensors are expected to be on the same device.

[ArjunSubramonian]

Woah, this is so cool :D I just changed it everywhere

[epwalsh]

That's interesting. Do you have a reference for that?

[ArjunSubramonian]

Added a link in the docstring!

[AkshitaB]

I think it's possible to vectorize this part if joint_counts_by_protected_variable_label and _protected_variable_label_counts are initialized as tensors instead of dicts (they can still be filled using the for loop like now) - with a bit of reshaping.

[ArjunSubramonian]

Great suggestion :) I just did this for all the metrics.