Skip to content

Latest commit

 

History

History
88 lines (68 loc) · 5.16 KB

README.md

File metadata and controls

88 lines (68 loc) · 5.16 KB
Raw

regression_uncertainty

overview image

Official implementation (PyTorch) of the paper:
How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?, 2023 [arXiv] [project].
Fredrik K. Gustafsson, Martin Danelljan, Thomas B. Schön.
We propose an extensive benchmark for testing the reliability of regression uncertainty estimation methods under real-world distribution shifts. It consists of 8 publicly available image-based regression datasets with different types of challenging distribution shifts. We use our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from OOD detection. We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark thus serves as a challenge to the research community.

If you find this work useful, please consider citing:

@inproceedings{gustafsson2023how,
  title={How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?},
  author={Gustafsson, Fredrik K and Danelljan, Martin and Sch{\"o}n, Thomas B},
  booktitle={Transactions on Machine Learning Research (TMLR)},
  year={2023}
}

Datasets:

Cells:

Cells-Tails:

Cells-Gap:

ChairAngle:

ChairAngle-Tails:

ChairAngle-Gap:

AssetWealth:

  • $ pip install wilds
  • Run AssetWealth/create_datasets.py to download the dataset (13 GB).

VentricularVolume:

  • Download EchoNet-Dynamic.zip from https://echonet.github.io/dynamic/index.html#dataset and extract.
  • Move the resulting EchoNet-Dynamic folder to regression_uncertainty/datasets.
  • Run VentricularVolume/create_datasets.py to generate the train, val and test splits.

BrainTumourPixels:

  • Download Task01_BrainTumor.tar from http://medicaldecathlon.com/ and extract.
  • Move the resulting Task01_BrainTumour folder to regression_uncertainty/datasets.
  • Run BrainTumourPixels/create_datasets.py to generate the train, val and test splits.

SkinLesionPixels:

  • Download the data from https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DBW86T (HAM10000_images_part_1.zip, HAM10000_images_part_2.zip, HAM10000_metadata, HAM10000_segmentations_lesion_tschandl.zip).
  • Create the folder regression_uncertainty/datasets/ham10000.
  • Move the files to regression_uncertainty/datasets/ham10000 and then extract.
  • Run SkinLesionPixels/create_datasets.py to generate the train, val and test splits.

HistologyNucleiPixels:

AerialBuildingPixels:

  • Download the data from https://project.inria.fr/aerialimagelabeling/ and extract.
  • Move the resulting AerialImageDataset folder to regression_uncertainty/datasets.
  • Run AerialBuildingPixels/create_datasets.py to generate the train, val and test splits.