Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support LoveDA dataset #1028

Merged
merged 41 commits into from
Nov 24, 2021
Merged

Conversation

Junjue-Wang
Copy link
Contributor

@Junjue-Wang Junjue-Wang commented Nov 10, 2021

Old pr of LoveDA dataset is here: #1006.

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@Junjue-Wang
Copy link
Contributor Author

@mengzhangl I have already created a new pr with a new branch.

@codecov
Copy link

codecov bot commented Nov 10, 2021

Codecov Report

Attention: Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.64%. Comparing base (6b3e550) to head (bd9db45).
Report is 300 commits behind head on master.

Files Patch % Lines
mmseg/datasets/loveda.py 96.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1028      +/-   ##
==========================================
+ Coverage   89.55%   89.64%   +0.09%     
==========================================
  Files         119      120       +1     
  Lines        6626     6657      +31     
  Branches     1034     1037       +3     
==========================================
+ Hits         5934     5968      +34     
+ Misses        488      485       -3     
  Partials      204      204              
Flag Coverage Δ
unittests 89.64% <96.77%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@MengzhangLI
Copy link
Contributor

MengzhangLI commented Nov 10, 2021

Here is some results on validation and test set:

Val mIoU Test mIoU Model backbone Lr schd
50.28 47.51 DeepLabV3+ R-18-D8 80k
50.99 50.22 DeepLabV3+ R-50-D8 80k
51.47 49.60 DeepLabV3+ R-101-D8 80k
48.62 47.17 PSPNet R-18-D8 80k
50.46 50.32 PSPNet R-50-D8 80k
51.86 50.37 PSPNet R-101-D8 80k
49.3 47.57 HRNet FCN HRNetV2p-W18-Small 80k
50.87 49.25 HRNet FCN HRNetV2p-W18 80k
51.04 50.38 HRNet FCN HRNetV2p-W48 80k

FYI, the results of original paper:
image

TO DO:

  • Getting results of models above on test set.

  • Adding format-only in ./mmseg/datasets/loveda.py, enabling users of MMSegmentation could directly generating standard prediction output of test set for evaluation on official codaLab site.

@MengzhangLI MengzhangLI changed the title New LoveDA Pull Request [Feature] Support LoveDA dataset Nov 10, 2021
@Junjue-Wang
Copy link
Contributor Author

Junjue-Wang commented Nov 10, 2021

Here is some results on validation and test set:

Val mIoU Test mIoU Model backbone Lr schd
50.28 47.51 DeepLabV3+ R-18-D8 80k
50.99 50.22 DeepLabV3+ R-50-D8 80k
51.47 49.60 DeepLabV3+ R-101-D8 80k
48.62 47.17 PSPNet R-18-D8 80k
50.46 50.32 PSPNet R-50-D8 80k
51.86 50.37 PSPNet R-101-D8 80k
49.3 47.57 HRNet FCN HRNetV2p-W18-Small 80k
50.87 49.25 HRNet FCN HRNetV2p-W18 80k
51.04 50.38 HRNet FCN HRNetV2p-W48 80k
FYI, the results of original paper: image

TO DO:

  • Getting results of models above on test set.
  • Adding format-only in ./mmseg/datasets/loveda.py, enabling users of MMSegmentation could directly generating standard prediction output of test set for evaluation on official codaLab site.

Good job! This looks better than my test results, perhaps because of more data enhancement and iteration steps.

@MengzhangLI
Copy link
Contributor

MengzhangLI commented Nov 10, 2021

Here is some results on validation and test set:
Val mIoU Test mIoU Model backbone Lr schd
50.28 47.51 DeepLabV3+ R-18-D8 80k
50.99 50.22 DeepLabV3+ R-50-D8 80k
51.47 49.60 DeepLabV3+ R-101-D8 80k
48.62 47.17 PSPNet R-18-D8 80k
50.46 50.32 PSPNet R-50-D8 80k
51.86 50.37 PSPNet R-101-D8 80k
49.3 47.57 HRNet FCN HRNetV2p-W18-Small 80k
50.87 49.25 HRNet FCN HRNetV2p-W18 80k
51.04 50.38 HRNet FCN HRNetV2p-W48 80k
FYI, the results of original paper: image
TO DO:

  • Getting results of models above on test set.
  • Adding format-only in ./mmseg/datasets/loveda.py, enabling users of MMSegmentation could directly generating standard prediction output of test set for evaluation on official codaLab site.

Good job! This looks better than my test results, perhaps because of more data enhancement and iteration steps.

I just only test results of HRNet, PSPNet and DeepLabV3Plus which are showed in your table. If your time available, you could train transformer models such as SegFormer, SETR and Swin Transformer which may get better results I guess.

@MengzhangLI
Copy link
Contributor

MengzhangLI commented Nov 10, 2021

Hi, Junjue @Junjue-Wang
I think I've done most part of this pr.

Could you please:

(1) Checking out my README.md and results. For example, your test results could be generated by format-only command and be like:

python ./tools/test.py ./configs/pspnet/pspnet_r50-d8_512x512_80k_loveda.py ./work_dirs/temp/pspnet_r50-d8_512x512_80k_loveda/iter_80000.pth --format-only --eval-options "imgfile_prefix=./pspnet_test_results"

(2) Adding links on your official repo just like SETR and SegFormer so other people may use our codebase to set up their own benchmark, be like:

image

Best,

@MengzhangLI MengzhangLI self-assigned this Nov 10, 2021
@MengzhangLI MengzhangLI added the WIP Work in process label Nov 10, 2021
@MengzhangLI
Copy link
Contributor

@Junjue-Wang I am not sure whether this version is up-to-merge. Could you please check whether it could be run from (1) LoveDA data preparation to (2) inference using our provided model and config and getting (3) test results in your website?

Thank you so much.

If it works well, you could also use this codebase to train better benchmark for LoveDA.

Junjue-Wang and others added 2 commits November 16, 2021 20:21
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
Copy link
Collaborator

@Junjun2016 Junjun2016 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should add doc for preparing the results and upload it to the official evaluation server (both command and description).

@Junjun2016
Copy link
Collaborator

image

@MengzhangLI
Copy link
Contributor

MengzhangLI commented Nov 17, 2021

  • LoveDA Unittest coverage rate.
  • Add docs to advise how to submit prediction of LoveDA test set to server.

To do: Creating .sh shell file in ./tools to submit to server for various datasets.

@MengzhangLI
Copy link
Contributor

Here are inference process and results of website:

image

image

@Junjun2016 Junjun2016 requested a review from xvjiarui November 17, 2021 10:56
@MengzhangLI
Copy link
Contributor

Hi, @Junjue-Wang

I have uploaded a convert .py file because wget can not download dataset directly from google drive.

Could you testify my convert file? Thanks!

Here is file and doc.

@Junjue-Wang
Copy link
Contributor Author

Hi, @Junjue-Wang

I have uploaded a convert .py file because wget can not download dataset directly from google drive.

Could you testify my convert file? Thanks!

Here is file and doc.

@MengzhangLI Thank you for your suggestion and modification, we have uploaded the dataset to https://zenodo.org/record/5706578#.YZumXmBByUk. Now we can directly use wget to download the LoveDA dataset. So I think you can easily change the URL in the README.md.

@MengzhangLI
Copy link
Contributor

Hi, @Junjue-Wang
I have uploaded a convert .py file because wget can not download dataset directly from google drive.
Could you testify my convert file? Thanks!
Here is file and doc.

@MengzhangLI Thank you for your suggestion and modification, we have uploaded the dataset to https://zenodo.org/record/5706578#.YZumXmBByUk. Now we can directly use wget to download the LoveDA dataset. So I think you can easily change the URL in the README.md.

OK, got it. I will upload doc.

But it still needs dataset conversion. Can you have a try my .py file? Thanks!

@Junjue-Wang
Copy link
Contributor Author

Hi, @Junjue-Wang
I have uploaded a convert .py file because wget can not download dataset directly from google drive.
Could you testify my convert file? Thanks!
Here is file and doc.

@MengzhangLI Thank you for your suggestion and modification, we have uploaded the dataset to https://zenodo.org/record/5706578#.YZumXmBByUk. Now we can directly use wget to download the LoveDA dataset. So I think you can easily change the URL in the README.md.

OK, got it. I will upload doc.

But it still needs dataset conversion. Can you have a try my .py file? Thanks!

Okay, I'll try the conversion py later :).

@MengzhangLI
Copy link
Contributor

Hi, I have updated wget from zenodo in docs and also a convert dataset .py file.

@Junjue-Wang
Copy link
Contributor Author

I have already checked the pipeline for dataset preparation. It works well !

docs_zh-CN/inference.md Outdated Show resolved Hide resolved
@Junjun2016 Junjun2016 merged commit 2bd7f60 into open-mmlab:master Nov 24, 2021
bowenroom pushed a commit to bowenroom/mmsegmentation that referenced this pull request Feb 25, 2022
* update LoveDA dataset api

* revised lint errors in dataset_prepare.md

* revised lint errors in loveda.py

* revised lint errors in loveda.py

* revised lint errors in dataset_prepare.md

* revised lint errors in dataset_prepare.md

* checked with isort and yapf

* checked with isort and yapf

* checked with isort and yapf

* Revert "checked with isort and yapf"

This reverts commit 686a51d

* Revert "checked with isort and yapf"

This reverts commit b877e12.

* Revert "revised lint errors in dataset_prepare.md"

This reverts commit 2289e27

* Revert "checked with isort and yapf"

This reverts commit 159db2f

* Revert "checked with isort and yapf"

This reverts commit 159db2f

* add configs & fix bugs

* update new branch

* upload models&logs and add format-only

* change pretraied model path of HRNet

* fix the errors in dataset_prepare.md

* fix the errors in dataset_prepare.md and configs in loveda.py

* change the description in docs_zh-CN/dataset_prepare.md

* use init_cfg

* fix test converage

* adding pseudo loveda dataset

* adding pseudo loveda dataset

* adding pseudo loveda dataset

* adding pseudo loveda dataset

* adding pseudo loveda dataset

* adding pseudo loveda dataset

* Update docs/dataset_prepare.md

Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>

* Update docs_zh-CN/dataset_prepare.md

Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>

* Update docs_zh-CN/dataset_prepare.md

Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>

* Delete unused lines of unittest and Add docs

* add convert .py file

* add downloading links from zenodo

* move place of LoveDA and Cityscapes in doc

* move place of LoveDA and Cityscapes in doc

Co-authored-by: MengzhangLI <mcmong@pku.edu.cn>
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023
* map location to cpu when load checkpoint (open-mmlab#1007)

* [Enhancement] Support minus output feature index in mobilenet_v3 (open-mmlab#1005)

* fix typo in mobilenet_v3

* fix typo in mobilenet_v3

* use -1 to indicate output tensors from final stage

* support negative out_indices

* [Enhancement] inference speed and flops tools. (open-mmlab#986)

* add the function to test the dummy forward speed of models.

* add tools to test the flops and inference speed of multiple models.

* [Fix] Update pose tracking demo to be compatible with latest mmtrakcing (open-mmlab#1014)

* update mmtracking demo

* support both track_bboxes and track_results

* add docstring

* [Fix] fix skeleton_info of coco wholebody dataset (open-mmlab#1010)

* fix wholebody base dataset

* fix lint

* fix lint

Co-authored-by: ly015 <liyining0712@gmail.com>

* [Feature] Add ViPNAS models for wholebody keypoint detection (open-mmlab#1009)

* add configs

* add dark configs

* add checkpoint and readme

* update webcam demo

* fix model path in webcam demo

* fix unittest

* [Fix] Fix bbox label visualization (open-mmlab#1020)

* update model metafiles (open-mmlab#1001)

* update hourglass ae .md (open-mmlab#1027)

* [Feature] Add ViPNAS mbv3 (open-mmlab#1025)

* add vipnas mbv3

* test other variants

* submission for mmpose

* add unittest

* add readme

* update .yml

* fix lint

* rebase

* fix pytest

Co-authored-by: jin-s13 <jinsheng13@foxmail.com>

* [Enhancement] Set a random seed when the user does not set a seed (open-mmlab#1030)

* fix randseed

* fix lint

* fix import

* fix isort

* update yapf hook

* revert yapf version

* add cfg file for flops and speed test,  change the bulid_posenet to init_pose_model and fix some typo in cfg (open-mmlab#1028)

* [Enhancement] Add more functions for speed test tool (open-mmlab#1034)

* add batch size and device args in speed test script, and remove MMDataParallel warper

* add vipnas_mbv3 model

* fix dead link (open-mmlab#1038)

* Skip CI when some specific files were changed (open-mmlab#1041)

* update sigmas (open-mmlab#1040)

* add more configs, ckpts and logs for HRNet on PoseTrack18 (open-mmlab#1035)

* [Feature] Add PoseWarper dataset (open-mmlab#1006)

* add PoseWarper dataset and base class

* modify pipelines related to video

* add unittest for PoseWarper dataset

* add unittest for evaluation function in posetrack18-realted dataset, and add some annotations json files

* fix typo

* fix unittest CI failure

* fix typo

* add PoseWarper dataset and base class

* modify pipelines related to video

* add unittest for PoseWarper dataset

* add unittest for evaluation function in posetrack18-realted dataset, and add some annotations json files

* fix typo

* fix unittest CI failure

* fix typo

* modify some methods in the base class to improve code coverage rate

* recover some mistakenly-deleted notes

* remove test_dataset_info part for the new TopDownPoseTrack18VideoDataset class

* cancel uncompleted previous runs (open-mmlab#1053)

* [Doc] Add inference speed results (open-mmlab#1044)

* add docs related to inference speed results

* add corresponding Chinese docs and fix some typos

* add Chinese docs in readthedocs

* remove the massive table in readme

* minor modification to wording

Co-authored-by: ly015 <liyining0712@gmail.com>

* [Feature] Add PoseWarper detector model (open-mmlab#932)

* Add top down video detector module

* Add PoseWarper neck

* add function _freeze_stages

* fix typo

* modify PoseWarper detector and PoseWarperNeck

* fix typo

* modify posewarper detector and neck

* Delete top_down_video.py

change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown`

* fix spell typo

* modify detector and neck

* add unittest for detector and neck

* modify unittest for posewarper forward

* Add top down video detector module

* Add PoseWarper neck

* add function _freeze_stages

* fix typo

* modify PoseWarper detector and PoseWarperNeck

* fix typo

* modify posewarper detector and neck

* Delete top_down_video.py

change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown`

* fix spell typo

* modify detector and neck

* add unittest for detector and neck

* modify unittest for posewarper forward

* modify dependency on mmcv version in posewarper neck

* reduce memory cost in test

* modify flops tool for more flexible input format

* Add top down video detector module

* Add PoseWarper neck

* add function _freeze_stages

* fix typo

* modify PoseWarper detector and PoseWarperNeck

* fix typo

* modify posewarper detector and neck

* Delete top_down_video.py

change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown`

* fix spell typo

* modify detector and neck

* add unittest for detector and neck

* modify unittest for posewarper forward

* Add PoseWarper neck

* modify PoseWarper detector and PoseWarperNeck

* modify posewarper detector and neck

* Delete top_down_video.py

change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown`

* fix spell typo

* modify detector and neck

* add unittest for detector and neck

* modify unittest for posewarper forward

* modify dependency on mmcv version in posewarper neck

* reduce memory cost in test

* modify flops tool for more flexible input format

* modify the posewarper detector description

* modify some arguments and related fields

* modify default values for some args

* fix readthedoc bulid typo

* fix ignore path (open-mmlab#1059)

* [Doc]  Add related docs for PoseWarper (open-mmlab#1036)

* add related docs for PoseWarper

* add related readme docs for posewarper

* modify related args in posewarper stage2 config

* modify posewarper stage2 config path

* add description about val_boxes path for data preparation (open-mmlab#1060)

* bump version to v0.21.0 (open-mmlab#1061)

* [Feature] Add ViPNAS_Mbv3 wholebody model (open-mmlab#1055)

* add vipnas mbv3 coco_wholebody

* add vipnas mbv3 coco_wholebody md&yml

* fix lint

Co-authored-by: ly015 <liyining0712@gmail.com>

Co-authored-by: Lumin <30328525+luminxu@users.noreply.github.com>
Co-authored-by: zengwang430521 <zengwang430521@gmail.com>
Co-authored-by: Jas <jinsheng@sensetime.com>
Co-authored-by: jin-s13 <jinsheng13@foxmail.com>
Co-authored-by: Qikai Li <87690686+liqikai9@users.noreply.github.com>
Co-authored-by: QwQ2000 <396707050@qq.com>
wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023
* [Enhancement] inference speed and flops tools. (open-mmlab#986)

* add the function to test the dummy forward speed of models.

* add tools to test the flops and inference speed of multiple models.

* [Feature] Add ViPNAS models for wholebody keypoint detection (open-mmlab#1009)

* add configs

* add dark configs

* add checkpoint and readme

* update webcam demo

* fix model path in webcam demo

* fix unittest

* update model metafiles (open-mmlab#1001)

* [Feature] Add ViPNAS mbv3 (open-mmlab#1025)

* add vipnas mbv3

* test other variants

* submission for mmpose

* add unittest

* add readme

* update .yml

* fix lint

* rebase

* fix pytest

Co-authored-by: jin-s13 <jinsheng13@foxmail.com>

* add cfg file for flops and speed test,  change the bulid_posenet to init_pose_model and fix some typo in cfg (open-mmlab#1028)

* Skip CI when some specific files were changed (open-mmlab#1041)

* add voxelpose

* unit test

* unit test

* unit test

* add docs/ckpts

* del unnecessary comments

* correct typos in comments and docs

* Add or modify docs

* change variable names

* reduce memory cost in test

* get person_id

* rebase

* resolve comments

* rebase master

* rename cfg files

* fix typos in comments

Co-authored-by: zengwang430521 <zengwang430521@gmail.com>
Co-authored-by: Yining Li <liyining0712@gmail.com>
Co-authored-by: Lumin <30328525+luminxu@users.noreply.github.com>
Co-authored-by: jin-s13 <jinsheng13@foxmail.com>
Co-authored-by: Qikai Li <87690686+liqikai9@users.noreply.github.com>
Co-authored-by: QwQ2000 <396707050@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants