Skip to content

Latest commit

 

History

History
 
 

Matting

README.md

English | 简体中文

Matting

Image Matting is the technique of extracting foreground from an image by calculating its color and transparency. It is widely used in the film industry to replace background, image composition, and visual effects. Each pixel in the image will have a value that represents its foreground transparency, called Alpha. The set of all Alpha values in an image is called Alpha Matte. The part of the image covered by the mask can be extracted to complete foreground separation.

Update Notes

2021.11 Matting Project is released. [1] Support Matting models: DIM, MODNet. [2] Support model export and python deployment. [3] Support background replacement function. [4] Support human matting deployment in Android.

Contents

Installation

1. Install PaddlePaddle

Versions

  • PaddlePaddle >= 2.0.2

  • Python >= 3.7+

Due to the high computational cost of model, PaddleSeg is recommended for GPU version PaddlePaddle. CUDA 10.0 or later is recommended. See PaddlePaddle official website for the installation tutorial.

2. Download the PaddleSeg repository

git clone https://github.com/PaddlePaddle/PaddleSeg

3. Installation

cd PaddleSeg
pip install -e .
pip install scikit-image
cd contrib/Matting

Models

DIM-VGG16

MODNet performance on PPM-100.

Backbone SAD MSE Params(M) FLOPs(G) FPS Link
MobileNetV2 112.73 0.0098 6.5 15.7 67.5 model
ResNet50_vd 104.14 0.0090 92.2 151.6 28.6 model
HRNet_W18 77.96 0.0054 10.2 28.5 10.9 model

Note: The model input size is (512, 512) and the GPU is Tesla V100 32G.

Dataset preparation

Using MODNet's open source PPM-100 dataset as our demo dataset for the tutorial.

Organize the dataset into the following structure and place the dataset under the data directory.

PPM-100/
|--train/
|  |--fg/
|  |--alpha/
|
|--val/
|  |--fg/
|  |--alpha
|
|--train.txt
|
|--val.txt

The image name in the fg directory must be the same as the that in the alpha directory.

The contents of train.txt and val.txt are as follows:

train/fg/14299313536_ea3e61076c_o.jpg
train/fg/14429083354_23c8fddff5_o.jpg
train/fg/14559969490_d33552a324_o.jpg
...

You can download the organized PPM-100 dataset directly for subsequent tutorials.

If the full image is composited of foreground and background like the Composition-1k dataset used in Deep Image Matting, the dataset should be organized as follows:

Composition-1k/
|--bg/
|
|--train/
|  |--fg/
|  |--alpha/
|
|--val/
|  |--fg/
|  |--alpha/
|  |--trimap/ (if existing)
|
|--train.txt
|
|--val.txt

The contents of train.txt is as follows:

train/fg/fg1.jpg bg/bg1.jpg
train/fg/fg2.jpg bg/bg2.jpg
train/fg/fg3.jpg bg/bg3.jpg
...

The contents of val.txt is as follows. If trimap does not exist in dataset, the third column is not needed and the code will generate trimap automatically.

val/fg/fg1.jpg bg/bg1.jpg val/trimap/trimap1.jpg
val/fg/fg2.jpg bg/bg2.jpg val/trimap/trimap2.jpg
val/fg/fg3.jpg bg/bg3.jpg val/trimap/trimap3.jpg
...

Training, Evaluation and Prediction

Training

export CUDA_VISIBLE_DEVICES=0
python train.py \
       --config configs/modnet/modnet_mobilenetv2.yml \
       --do_eval \
       --use_vdl \
       --save_interval 5000 \
       --num_workers 5 \
       --save_dir output

note: Using --do_eval will affect training speed and increase memory consumption, turning on and off according to needs.

--num_workers Read data in multi-process mode. Speed up data preprocessing.

Run the following command to view more parameters.

python train.py --help

If you want to use multiple GPUs，please use python -m paddle.distributed.launch to run.

Evaluation

export CUDA_VISIBLE_DEVICES=0
python val.py \
       --config configs/modnet/modnet_mobilenetv2.yml \
       --model_path output/best_model/model.pdparams \
       --save_dir ./output/results \
       --save_results

--save_result The prediction results will be saved if turn on. If it is off, it will speed up the evaluation.

You can directly download the provided model for evaluation.

Run the following command to view more parameters.

python val.py --help

Prediction

export CUDA_VISIBLE_DEVICES=0
python predict.py \
    --config configs/modnet/modnet_mobilenetv2.yml \
    --model_path output/best_model/model.pdparams \
    --image_path data/PPM-100/val/fg/ \
    --save_dir ./output/results

If the model requires trimap information, pass the trimap path through '--trimap_path'.

You can directly download the provided model for evaluation.

Run the following command to view more parameters.

python predict.py --help

Background Replacement

export CUDA_VISIBLE_DEVICES=0
python bg_replace.py \
    --config configs/modnet/modnet_mobilenetv2.yml \
    --model_path output/best_model/model.pdparams \
    --image_path path/to/your/image \
    --bg_path path/to/your/background/image \
    --save_dir ./output/results

If the model requires trimap information, pass the trimap path through '--trimap_path'.

If --bg_path is not provided, green background is used。

You can directly download the provided model for background replacement.

Run the following command to view more parameters.

python bg_replace.py --help

Export and Deploy

Model Export

python export.py \
    --config configs/modnet/modnet_mobilenetv2.yml \
    --model_path output/best_model/model.pdparams \
    --save_dir output/export

If the model requires trimap information, --trimap is need.

Run the following command to view more parameters.

python export.py --help

Deploy

python deploy/python/infer.py \
    --config output/export/deploy.yaml \
    --image_path data/PPM-100/val/fg/ \
    --save_dir ouput/results

If the model requires trimap information, pass the trimap path through '--trimap_path'.

Run the following command to view more parameters.

python deploy/python/infer.py --help

Contributors

Thanks wuyefeilin, Qian bin, yzl19940819 for their contributons.