Domain Adapt for Semantic Segmentation

Pytorch implementation domain adaption of semantic segmentation from the synthetic dataset (source domain) to the real dataset (target domain). Based on AdaptSegNet.

Contact: Yong-Xiang Lin (xiaosean5408 at gmail dot com)

Download

You can download the Paper and Pre-trained weight at here! Gated-AdaptSegNet-Material

Paper

Please cite our paper if you find it is useful for your research.

Yong-Xiang Lin, Daniel Stanley Tan, Wen-Huang Cheng, Kai-Lung Hua. “Adapting Semantic Segmentation of Urban Scenes via Mask-aware Gated Discriminator,” In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), Shanghai, China, Jul. 8 - 12, 2019. (Oral)

Yong-Xiang Lin, Daniel Tan, Wen-Huang Cheng, Yung-Yao Chen, Kai-Lung Hua. “Spatially-aware Domain Adaptation for Semantic Segmentatino of Urban Scenes,” In Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, September 22-25, 2019. (Oral)

Todo

• Add Spatially-aware Domain Adaptation training code
• Release the Gated/Spatial pretrain model
• Reproduce Gated/Spatial result.

Example Results

Model Architecture

Prerequisites

• Python 3
• NVIDIA GPU (10G up, I used Nvidia GTX 1080Ti) + CUDA cuDNN
• Pytorch(Vesion needs 0.4.1 up because I use spectral_norm)

Getting Started

Installation

You can choose pip install / conda install by yml

• Intall method 1. use pip

  • Install PyTorch and dependencies from http://pytorch.org
  • Install below librarys Note:

  dominate - visualizing on the web page, tenserflow - recording logs

  pip install dominate
  pip install scipy
  pip install matplotlib
  pip install pillow
  pip install pyyaml
  pip install opencv-python
  pip install tensorflow

• Intall method 2. use conda env

  conda env create -f environment.yml

  • On Windows

    activate Gated-AdaptSeg 

  • On Linux

    conda activate Gated-AdaptSeg 

  pip install dominate

• Clone this repo:

git clone https://github.com/xiaosean/Gated-AdaptSeg
cd Gated-AdaptSeg

Dataset

• Download the GTA5 Dataset as the source domain, and put it in the data/GTA5 folder

• Download the Cityscapes Dataset as the target domain, and put it in the data/Cityscapes folder

Testing

Note: This version classification only use ASPP Module [6, 12]，Similar as AdaptSegNet

python evaluate_cityscapes.py --restore-from ./Gated-GTA5-Cityscapes_250000.pth

python evaluate_cityscapes.py --spatial --restore-from ./Spatial-GTA5-Cityscapes_250000.pth

• Compute the IoU on Cityscapes (thanks to the code from VisDA Challenge)

python compute_iou.py ./data/Cityscapes/data/gtFine/val result/cityscapes

Training Examples

Firstly, Using FastPhotoStyle, transfer GTA5 dataset to Cityscapes photo style.

Second,

• Train the GTA5-to-Cityscapes model -
  • Config use => configs/default.yaml

python train.py

Note: if you want to reproduce ICIP version

python train.py --config ./configs/Spatial-ICIP-version.yaml

Note: if you have out of memory issue, probably use single-level training way, currently is multi-level training. You can check this paper to find out the difference between them. Learning to Adapt Structured Output Space for Semantic Segmentation

Visulization

Open ./check_output/index.html

Support Tensorboard

tensorboard --logdir=check_output\log --host=127.0.0.1

Model

Using model/gated_discriminator

Yong-Xiang Lin, Daniel Stanley Tan, Wen-Huang Cheng, Kai-Lung Hua. “Adapting Semantic Segmentation of Urban Scenes via Mask-aware Gated Discriminator,” In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), Shanghai, China, Jul. 8 - 12, 2019. (Oral)

Using model/spatial_prior_discriminator and download the spatial prior from GTA5.

Spatial prior download go to ECCV'18 cbst download the Spatial prior:Spatial prior brrowed from cbst

Yong-Xiang Lin, Daniel Tan, Wen-Huang Cheng, Yung-Yao Chen, Kai-Lung Hua. “Spatially-aware Domain Adaptation for Semantic Segmentatino of Urban Scenes,” In Proceedings of the IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, September 22-25, 2019. (Oral)

Acknowledgment

This code is heavily borrowed from AdaptSegNet. Visualize part is heavily borrowed from pix2pixHD. Spatial prior used cbst.

Especially, thanks these paper and code, Learning to Adapt Structured Output Space for Semantic Segmentation
Yi-Hsuan Tsai*, Wei-Chih Hung*, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang and Manmohan Chandraker
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (spotlight) (* indicates equal contribution).

Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
Yang Zou, Zhiding Yu, B.V.K. Vijaya Kumar, Jinsong Wangl; The European Conference on Computer Vision (ECCV), 2018, pp. 289-305
In ECCV 2018 paper

Note

The model and code are available for non-commercial research purposes only.

• May, 05, 2019: Add readme.txt
• Jul, 04, 2019: Add training code - Gated-AdaptSeg