FE-LSD

This repository contains the official PyTorch implementation of the paper: Detecting Line Segments in Motion-blurred Images with Events.

Introduction

FE-LSD is an event-enhanced line segment detection framework for motion-blurred images with thoughtful information fusion of both modalities and advanced wireframe parsing network. Extensive results on both synthetic and realistic datasets demonstrate the effectiveness of the proposed method for handling motion blurs in line segment detection.

Network Architecture

Results

FE-Wireframe Dataset

• Quantitative Comparisons

Method	sAP5	sAP10	sAP15	msAP	mAPJ	APH	FH	FPS
LSD	0.1	0.6	1.1	0.6	3.0	19.5	42.6	76.7
FBSD	0.2	0.4	0.9	0.5	2.9	24.9	47.0	21.7
L-CNN	3.4	5.1	6.2	4.9	7.0	22.7	38.8	28.8
HAWP	3.5	5.1	6.3	5.0	6.8	21.7	40.2	36.6
ULSD	3.5	5.3	6.8	5.2	7.5	20.2	40.3	39.7
LETR	2.8	5.0	6.5	4.8	7.3	21.9	41.9	4.2
L-CNN (Retrained)	40.6	45.8	48.2	44.8	45.6	70.5	71.1	10.6
HAWP (Retrained)	45.1	50.4	52.9	49.5	46.8	75.0	73.2	26.8
ULSD (Retrained)	47.0	52.7	55.2	51.7	48.8	72.2	73.7	32.2
LETR (Retrained)	24.7	34.7	39.7	33.1	25.4	66.1	71.5	3.9
FE-HAWP	48.7	53.9	56.2	53.0	49.4	77.1	75.1	22.2
FE-ULSD	50.9	56.5	58.8	55.4	51.1	75.3	75.9	24.2

• Qualitative Comparisons

FE-Blurframe Dataset

• Quantitative Comparisons

Method	sAP5	sAP10	sAP15	msAP	mAPJ	APH	FH	FPS
LSD	1.1	2.8	4.1	2.7	5.1	29.4	48.1	61.0
FBSD	0.9	1.9	2.7	1.8	5.1	34.2	53.2	15.9
L-CNN	7.5	11.5	13.7	10.9	12.4	27.9	45.2	29.7
HAWP	8.4	12.8	15.3	12.2	12.4	32.0	48.2	38.1
ULSD	6.8	10.8	13.0	10.2	11.8	26.7	45.6	40.6
LETR	7.1	13.0	16.8	12.3	12.1	30.2	51.1	3.6
L-CNN (Retrained)	34.0	40.3	43.0	39.1	40.3	66.0	67.1	17.7
HAWP (Retrained)	37.0	43.9	46.9	42.6	41.6	67.9	69.6	29.0
ULSD (Retrained)	42.0	47.8	50.4	46.7	48.5	67.0	69.3	32.2
LETR (Retrained)	22.6	33.8	38.8	31.7	23.2	57.7	65.4	3.3
FE-HAWP	47.5	53.0	55.4	52.0	50.9	74.0	73.9	19.3
FE-ULSD	47.3	52.9	55.2	51.8	52.2	72.9	73.7	19.7
FE-HAWP (Fine-tuned)	59.8	64.2	65.9	63.3	60.1	82.0	79.7	21.1
FE-ULSD (Fine-tuned)	59.3	63.8	65.7	62.9	61.0	77.8	77.1	21.6

• Qualitative Comparisons

Requirements

• torch>=1.6.0
• torchvision>=0.7.0
• CUDA>=10.1
• lh_tool, matplotlib, numpy, opencv_python, Pillow, scikit_learn, scipy, setuptools, tensorboardX, timm, torch, torchvision, tqdm, yacs,

Step-by-step installation

conda create --name FE-LSD python=3.8
conda activate FE-LSD

cd <FE-LSD-Path>
git clone https://github.com/lh9171338/FE-LSD.git
cd FE-LSD

pip install -r requirements.txt

python setup.py build_ext --inplace

Quickstart with the pretrained model

• There are pretrained models in Google drive and Baiduyun. Please download them and put in the model/ folder.
• Put your test data in the dataset/ folder and generate the test.json file.

python image2json.py --dataset_name <DATASET_NAME>

• The file structure is as follows:

|-- dataset
    |-- events
	|-- 000001.npz
	|-- ...
    |-- images-blur
	|-- 000001.png
	|-- ...
    |-- test.json

• Test with the pretrained model. The results are saved in the output/ folder.

python test.py --arch <ARCH> --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --save_image

Training & Testing

Data Preparation

• Download the dataset from Baiduyun.
• Unzip the dataset to the dataset/ folder.
• Convert event streams into synchronous frames using Event Spike Tensor (EST) representation.

python event2frame.py --dataset_name <DATASET_NAME> --representation EST

# for FE-Wireframe dataset
python event2frame.py --dataset_name FE-Wireframe --representation EST

# for FE-Blurframe dataset
python event2frame.py --dataset_name FE-Blurframe --representation EST -s 0.5

ln -s events-EST-10 events

Train

python train.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> [--gpu <GPU_ID>] # FE-HAWP
python train.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> [--gpu <GPU_ID>] # FE-ULSD

Test

python test.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --save_image --with_clear [--gpu <GPU_ID>] # FE-HAWP
python test.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --save_image --with_clear [--gpu <GPU_ID>] # FE-ULSD

Evaluation

To evaluate the mAP^J, sAP, and FPS

python test.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --evaluate [--gpu <GPU_ID>] # FE-HAWP
python test.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> --evaluate [--gpu <GPU_ID>] # FE-ULSD

To evaluate AP^H and F^H, MATLAB is required

cd metric
python eval_APH.py --arch FE-HAWP --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> # FE-HAWP
python eval_APH.py --arch FE-ULSD --dataset_name <DATASET_NAME> --model_name <MODEL_NAME> # FE-ULSD

Citation

@ARTICLE{10323537,
  author={Yu, Huai and Li, Hao and Yang, Wen and Yu, Lei and Xia, Gui-Song},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Detecting Line Segments in Motion-Blurred Images With Events}, 
  year={2023},
  pages={1-16},
  doi={10.1109/TPAMI.2023.3334877}
}