How to add gaussian nosie images and uniform noise images to the test set to compute the AUROC against CIFAR10 when using CIFAR10 as in-distribution data?

[Emilia-Re]

Thank you for your great work and efforts for building this project.
When I run the code ,I want to know how to add additional ood data(like noise images) to the test set to compute the AUROC against the ID data. Besides, I didn't find the document of this project ,and it would be much better for me to understand and use this package if I have one.
Under the ./benchmark_imglist/cifar10 directory,there are many .txt files, what are these .txt files used for?
It would be highly appreciated if you could offer an answer.

[zjysteven]

Hi, those .txt files store and help index all relevant images. For example for CIFAR-10 (ID) v.s. CIFAR-100 (OOD), the corresponding txt files benchmark_imglist/cifar10/test_cifar10.txt and benchmark_imglist/cifar10/test_cifar100.txt store the relevant ID and OOD test images.

To test on a new dataset that is not currently encoded in OpenOOD, we encourage you to create a txt file similarly, and encode the dataset information in the DATA_INFO dictionary in openood/evaluation_api/datasets.py (see

OpenOOD/openood/evaluation_api/datasets.py

Lines 18 to 84 in be315e6

	'cifar10': {
	'num_classes': 10,
	'id': {
	'train': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/train_cifar10.txt'
	},
	'val': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/val_cifar10.txt'
	},
	'test': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/test_cifar10.txt'
	}
	},
	'csid': {
	'datasets': ['cifar10c'],
	'cinic10': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/val_cinic10.txt'
	},
	'cifar10c': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/test_cifar10c.txt'
	}
	},
	'ood': {
	'val': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/val_tin.txt'
	},
	'near': {
	'datasets': ['cifar100', 'tin'],
	'cifar100': {
	'data_dir': 'images_classic/',
	'imglist_path':
	'benchmark_imglist/cifar10/test_cifar100.txt'
	},
	'tin': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/test_tin.txt'
	}
	},
	'far': {
	'datasets': ['mnist', 'svhn', 'texture', 'places365'],
	'mnist': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/test_mnist.txt'
	},
	'svhn': {
	'data_dir': 'images_classic/',
	'imglist_path': 'benchmark_imglist/cifar10/test_svhn.txt'
	},
	'texture': {
	'data_dir': 'images_classic/',
	'imglist_path':
	'benchmark_imglist/cifar10/test_texture.txt'
	},
	'places365': {
	'data_dir': 'images_classic/',
	'imglist_path':
	'benchmark_imglist/cifar10/test_places365.txt'
	},
	}
	}
	},

for the example of CIFAR-10 as ID). After that, you should be able to evaluate on custom OOD datasets by using our new evaluator.