Use pytorch to implement a Deep Neural Network for real time lane detection mainly based on the IEEE IV conference paper "Towards End-to-End Lane Detection: an Instance Segmentation Approach".You can refer to their paper for details https://arxiv.org/abs/1802.05591. This model consists of ENet encoder, ENet decoder for binary semantic segmentation and ENet decoder for instance semantic segmentation using discriminative loss function.
The main network architecture is:
First, download tusimple dataset here.
Then, run the following command to generate the training/ val/ test samples and the train.txt/ val.txt/ test.txt file.
Generate training set:
python tusimple_transform.py --src_dir path/to/your/unzipped/file --val False
Generate training/ val set:
python tusimple_transform.py --src_dir path/to/your/unzipped/file --val True
Generate training/ val/ test set:
python tusimple_transform.py --src_dir path/to/your/unzipped/file --val True --test True
path/to/your/unzipped/file should like this:
|--dataset
|----clips
|----label_data_0313.json
|----label_data_0531.json
|----label_data_0601.json
|----test_label.json
The environment for training and evaluation:
python=3.6
torch>=1.2
numpy=1.7
torchvision>=0.4.0
matplotlib
opencv-python
pandas
Using example folder with ENet:
python train.py --dataset ./data/training_data_example
Using tusimple folder with ENet/Focal loss:
python train.py --dataset path/to/tusimpledataset/training
Using tusimple folder with ENet/Cross Entropy loss:
python train.py --dataset path/to/tusimpledataset/training --loss_type CrossEntropyLoss
Using tusimple folder with DeepLabv3+:
python train.py --dataset path/to/tusimpledataset/training --model_type DeepLabv3+
If you want to change focal loss to cross entropy loss, do not forget to adjust the hyper-parameter of instance loss and binary loss in ./model/lanenet/train_lanenet.py
A pretrained trained model by myself is located in ./log (only trained in 25 epochs)
Test the model:
python test.py --img ./data/tusimple_test_image/0.jpg
The architecture of LaneNet is based on ENet, which is a very light model. That is why I can upload it to github. However, ENet is not the best model to detect lane and do instance segmentation. Now, the model support 2 decoder branch ENet, U-Net, DeepLabv3+ (update 2021.7.16).
Focal loss (update 2021.7.20) is also supported.
- E-Net Encoder and E-Net decoder
- U-Net Encoder and U-Net decoder
- Discriminative loss for instance segmentation
- DeepLabv3+ Encoder and DeepLabv3+ decoder (2021/7/16)
- Focal loss for binary branch (2021/7/22)
- [] ViT as encoder (2021/12/30)
[ ] Post Processing (will release 2021/7/26)[ ] Use other new encoder-decoder structure[ ] Add H-Net and merge the H-Net model to the main lanenet model
Future work will release soon.
The lanenet project refers to the following research and projects:
Neven, Davy, et al. "Towards end-to-end lane detection: an instance segmentation approach." 2018 IEEE intelligent vehicles symposium (IV). IEEE, 2018.
@inproceedings{neven2018towards,
title={Towards end-to-end lane detection: an instance segmentation approach},
author={Neven, Davy and De Brabandere, Bert and Georgoulis, Stamatios and Proesmans, Marc and Van Gool, Luc},
booktitle={2018 IEEE intelligent vehicles symposium (IV)},
pages={286--291},
year={2018},
organization={IEEE}
}
Paszke, Adam, et al. "Enet: A deep neural network architecture for real-time semantic segmentation." arXiv preprint arXiv:1606.02147 (2016).
@article{paszke2016enet,
title={Enet: A deep neural network architecture for real-time semantic segmentation},
author={Paszke, Adam and Chaurasia, Abhishek and Kim, Sangpil and Culurciello, Eugenio},
journal={arXiv preprint arXiv:1606.02147},
year={2016}
}
De Brabandere, Bert, Davy Neven, and Luc Van Gool. "Semantic instance segmentation with a discriminative loss function." arXiv preprint arXiv:1708.02551 (2017).
@article{de2017semantic,
title={Semantic instance segmentation with a discriminative loss function},
author={De Brabandere, Bert and Neven, Davy and Van Gool, Luc},
journal={arXiv preprint arXiv:1708.02551},
year={2017}
}
https://github.com/MaybeShewill-CV/lanenet-lane-detection
https://github.com/klintan/pytorch-lanenet
DeepLabv3+ Encoder and DeepLabv3+ decoder refer from https://github.com/YudeWang/deeplabv3plus-pytorch