This is the official PyTorch implementation of our works: "Latent Space Regularization for Unsupervised Domain Adaptation in Semantic Segmentation" presented at the CVPR2021 Workshop on Autonomous Driving and "Road Scenes Segmentation Across Different Domains by Disentangling Latent Representations" submitted to Springer's The Visual Computer Journal.
In this paper we propose an effective Unsupervised Domain Adaptation (UDA) strategy, based on feature clustering, perpendicularity and norm alignment objectives to reach a regularized disposition of latent embeddings, while providing a semantically consistent domain alignment over the feature space.
We evaluate our framework in synthetic-to-real and real-to-real scenarios, in particular with GTA5, SYNTHIA, Cityscapes and Cross-City datasets.
The webpage of the paper is: here.
The CVPRW paper can be found on CVF'S Open Access.
This repository uses the following libraries:
- python (3.9.5)
- pytorch (1.9.0)
- torchvision (0.10.0)
- torchaudio (0.9.0)
- torchsummary (1.5.1)
- tensorboardX (2.4)
- tqdm (4.61.2)
- imageio (2.9.0)
- numpy (1.20.3)
- pillow (8.3.1)
- opencv-python (4.5.3.56)
A guide to install Pytorch can be found here.
For an easier installation of the other libraries, we provide you the requirement file to be run
pip install -r requirements.txtDownload the GTA5 dataset. The dataset splits (train.txt and val.txt) are provided in the datasets/splits/GTA5 folder.
Download the Cityscapes dataset. The dataset splits (train.txt, val.txt, trainplus.txt, test.txt) are provided in the datasets/splits/Cityscapes folder.
Download the SYNTHIA-RAND-CITYSCAPES dataset. The dataset splits (train.txt, val.txt) are provided in the datasets/splits/SYNTHIA folder.
Download the NTHU dataset. The dataset splits (train.txt, val.txt) for each city (Rome, Rio, Taipei, Tokyo) are provided in the datasets/splits/NTHU folder.
Such datasets and splits must be pointed to using the following command line arguments:
--data_root_path (path to dataset) --list_path (path to split lists) --target_root_path (path to target dataset) --target_list_path (path to split lists)
Download the official pytorch checkpoints pretrained on imagenet for the backbones you wish to use and place them into the pretrained_model folder:
GTA5 with ResNet101:
If you haven't already done so, download the ResNet101's weights pretrained on ImageNet from here and run:
python tools/train_source.py --backbone "resnet101" --dataset "cityscapes" --checkpoint_dir "./log/gta5_rn101_source" --data_root_path (path to dataset) --list_path (path to split lists) --target_root_path (path to dataset) --target_list_path (path to split lists) --target_base_size "1280,640" --target_crop_size "1280,640" --num_classes 19
Alternatively, you can download the GTA5 pretrained weights from here.
-
GTA5-to-Cityscapes with ResNet101:
python tools/train_UDA.py --backbone "resnet101" --pretrained_ckpt_file "gta5_rn101_source.pth" --source_dataset "gta5" --checkpoint_dir "./log/gta2city_rn101" --data_root_path (path to dataset) --list_path (path to split lists) --target_root_path (path to dataset) --target_list_path (path to split lists) --num_classes 19 --lambda_entropy .1 --IW_ratio .2 --lambda_norm 0.025 --lambda_ortho 0.2 --lambda_cluster 0.1
A checkpoint of the adapted model can be found here.
-
GTA5-to-Cityscapes (CS-full) with ResNet101:
python tools/train_UDA.py --backbone "resnet101" --pretrained_ckpt_file "gta5_rn101_source.pth" --source_dataset "gta5" --checkpoint_dir "./log/gta2cityfull_rn101" --data_root_path (path to dataset) --list_path (path to split lists) --target_root_path (path to dataset) --target_list_path (path to split lists) --num_classes 19 --lambda_entropy .1 --IW_ratio .2 --lambda_norm 0.025 --lambda_ortho 0.2 --lambda_cluster 0.1 --target_train_split "train_plus"
A checkpoint of the adapted model can be found here.
-
GTA5-to-Cityscapes with VGG-16:
python tools/train_UDA.py --backbone "vgg16" --pretrained_ckpt_file "gta5_vgg16_source.pth" --source_dataset "gta5" --checkpoint_dir "./log/gta2city_vgg16" --data_root_path (path to dataset) --list_path (path to split lists) --target_root_path (path to dataset) --target_list_path (path to split lists) --num_classes 19 --lambda_entropy .1 --IW_ratio .2 --lambda_norm 0.01 --lambda_ortho 0.4 --lambda_cluster 0.2
A checkpoint of the adapted model can be found here.
-
SYNTHIA-to-Cityscapes with ResNet-101:
python tools/train_UDA.py --backbone "resnet101" --pretrained_ckpt_file "synthia_rn101_source.pth" --source_dataset "synthia" --checkpoint_dir "./log/synthia2city_rn101" --data_root_path (path to dataset) --list_path (path to split lists) --target_root_path (path to dataset) --target_list_path (path to split lists) --num_classes 16 --lambda_entropy .1 --IW_ratio 0 --lambda_norm 0.1 --lambda_ortho 0.2 --lambda_cluster 0.05
A checkpoint of the adapted model can be found here.
-
Cityscapes-to-CrossCity with ResNet-101:
python tools/train_UDA.py --backbone "resnet101" --dataset "city" --pretrained_ckpt_file "city_rn101_source.pth" --source_dataset "cityscapes" --checkpoint_dir "./log/city2cc_rn101" --data_root_path (path to dataset) --list_path (path to split lists) --base_size "1280,640" --target_root_path (path to dataset) --target_list_path path to split lists) --num_classes 13 --lambda_entropy .1 --IW_ratio .2 --lambda_norm 0.025 --lambda_ortho 0.2 --lambda_cluster 0.1
A checkpoint of the model adapted to Rome can be found here.
GTA5-to-Cityscapes with ResNet101:
python tools/evaluate.py --backbone "resnet101" --source_dataset "gta5" --checkpoint_dir "./eval/gta2city_rn101" --pretrained_ckpt_file "./log/gta2city_rn101/gta52cityscapesbest.pth" --data_root_path (path to dataset) --list_path (path to dataset) --num_classes 19 --batch_size 1 --split "test" --data_loader_workers 4
If you use this repository, please consider to cite us
@article{barbato2021road,
title = {Road scenes segmentation across different domains by disentangling latent representations},
author = {Barbato, Francesco and Michieli, Umberto and Toldo, Marco and Zanuttigh, Pietro},
journal = {arXiv preprint arXiv:2108.03021},
year = {2021}
}
@InProceedings{barbato2021latent,
author = {Barbato, Francesco and Toldo, Marco and Michieli, Umberto and Zanuttigh, Pietro},
title = {Latent Space Regularization for Unsupervised Domain Adaptation in Semantic Segmentation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2021},
pages = {2835-2845}
}
The structure of this code is largely based on this and this repo.