This unit contains one lab, two lectures, two groupwork worksheets, and assigned readings from foundational academic research in this field. By the end of the unit, students will be able to define algorithms' fairness goals and conduct mathematical analysis in R and Python to consider the trustworthiness of many common machine learning outputs.
This unit is built for Smith College's Statistical and Data Sciences program's Modeling for Machine Learning course but is adaptable to most undergraduate machine learning courses. Please reach out to Sarah Gillespie via Github issues for the answer key to the worksheet. The worksheet answer key is in a private repository.
Much of this unit is based on To Trust Or Not To Trust A Classifier's concepts and code and the fairness categories put forth in On Formalizing Fairness in Prediction with Machine Learning.
Jiang, H., & Hembise, C. (n.d.). Google/TrustScore: To trust or not to trust a classifier. A measure of uncertainty for any trained (possibly Black-box) classifier which is more effective than the classifier's own implied confidence (e.g. softmax probability for a neural network). GitHub. Retrieved December 22, 2021, from https://github.com/google/TrustScore
Jiang, H., Kim, B., Guan, M. Y., & Gupta, M. (2018, October 26). To trust or not to trust a classifier. arXiv.org. Retrieved December 22, 2021, from https://arxiv.org/abs/1805.11783
Gajane, P., & Pechenizkiy, M. (2018, May 28). On formalizing fairness in prediction with machine learning. arXiv.org. Retrieved December 22, 2021, from https://arxiv.org/abs/1710.03184
Papernot, N., & McDaniel, P. (2018, March 13). Deep K-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv.org. Retrieved December 22, 2021, from https://arxiv.org/abs/1803.04765v1
Interface to python. Interface to Python • reticulate. (n.d.). Retrieved December 22, 2021, from https://rstudio.github.io/reticulate/
Yahya, & Mortensen, P. (2021, May 8). Convergencewarning: Lbfgs failed to converge (status=1): Stop: Total no. of iterations reached limit. Stack Overflow. Retrieved December 22, 2021, from https://stackoverflow.com/a/62659927
This teachable unit is created from the references listed above, as well as my own work. It is offered under the Apache Licence.
Copyright 2021 Sarah Gillespie
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.