[Feature] Support video demo #1014

irvingzhang0512 · 2021-11-05T06:51:11Z

Motivation

Video Demo, use local video file or webcam as input, visualize predictions by imshow or generating local video file.

Use cases (Optional)

PS: Not sure if I should upload a demo video demo/demo.mp4.

python demo/video_demo.py ${VIDEO_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} [--device ${DEVICE_NAME}] [--palette-thr ${PALETTE}] \
    [--show] [--show-wait-time {SHOW_WAIT_TIME}] [--output-file {OUTPUT_FILE}] [--output-fps {OUTPUT_FPS}] \
    [--output-height {OUTPUT_HEIGHT}] [--output-width {OUTPUT_WIDTH}] [--opacity {OPACITY}]

Examples:

python demo/video_demo.py demo/demo.mp4 configs/cgnet/cgnet_680x680_60k_cityscapes.py \
    checkpoints/cgnet_680x680_60k_cityscapes_20201101_110253-4c0b2f2d.pth \
    --device cuda:0 --palette cityscapes --show

CLAassistant · 2021-11-05T06:51:15Z

All committers have signed the CLA.

codecov · 2021-11-05T08:46:38Z

Codecov Report

Merging #1014 (561beb5) into master (313189d) will increase coverage by 0.18%.
The diff coverage is n/a.

❗ Current head 561beb5 differs from pull request most recent head f561bb0. Consider uploading reports for the commit f561bb0 to get more accurate results

@@            Coverage Diff             @@
##           master    #1014      +/-   ##
==========================================
+ Coverage   89.57%   89.76%   +0.18%     
==========================================
  Files         120      119       -1     
  Lines        6717     6605     -112     
  Branches     1122     1029      -93     
==========================================
- Hits         6017     5929      -88     
+ Misses        496      475      -21     
+ Partials      204      201       -3

Flag	Coverage Δ
unittests	`89.76% <ø> (+0.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/core/evaluation/metrics.py	`90.42% <0.00%> (-4.26%)`	⬇️
mmseg/datasets/pipelines/transforms.py	`97.30% <0.00%> (-0.31%)`	⬇️
mmseg/apis/__init__.py	`100.00% <0.00%> (ø)`
mmseg/datasets/__init__.py	`100.00% <0.00%> (ø)`
mmseg/datasets/pipelines/__init__.py	`100.00% <0.00%> (ø)`
mmseg/datasets/loveda.py
mmseg/apis/train.py	`22.22% <0.00%> (+1.32%)`	⬆️
mmseg/models/backbones/mit.py	`91.33% <0.00%> (+9.58%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 313189d...f561bb0. Read the comment docs.

Junjun2016 · 2021-11-05T09:04:21Z

Hi @irvingzhang0512
Thanks for your contribution, nice PR!

PS: Not sure if I should upload a demo video demo/demo.mp4.

You could provide a link for us to download the demo.mp4,
We will download the file and upload it to OSS.
So users can download it from OSS and inference the demo.

docs/get_started.md

Junjun2016 · 2021-11-05T09:51:54Z

@irvingzhang0512
If the video is short, we could save it in the repository like https://github.com/open-mmlab/mmtracking/blob/master/demo/demo.mp4.

MengzhangLI · 2021-11-05T14:14:45Z

Hi, @irvingzhang0512 thanks for your nice pr!

Could you please upload demo.mp4 or just send this model to my e-mail (mcmong@pku.edu.cn)?

Best,

irvingzhang0512 · 2021-11-08T09:09:32Z

Sorry for the late response, I'm a little busy right now and I'll take care of this PR this weekend.

Junjun2016 · 2021-11-08T16:24:17Z

Sorry for the late response, I'm a little busy right now and I'll take care of this PR this weekend.

Thanks a lot.

RockeyCoss · 2021-11-15T08:26:13Z

Please merge the master branch into your branch, thank you.

Junjun2016 · 2021-11-24T15:35:58Z

Hi @irvingzhang0512
Could you please upload the video demo if your time is available?

MengzhangLI · 2021-11-24T15:35:59Z

Hi, @irvingzhang0512

Any progress? We can review this pr in the next few days.

Thanks.

MengzhangLI · 2021-12-01T05:40:48Z

Hi, sorry to bother you. @irvingzhang0512

Anything to update?

irvingzhang0512 · 2021-12-01T05:50:41Z

Sorry , I have worked overtime for a few weeks... I'll take care of this tonight...

irvingzhang0512 · 2021-12-01T06:01:44Z

BTW, I'm not familiar with image semgentation public dataset, is there any video suitable this demo?

MengzhangLI · 2021-12-01T06:22:33Z

Hoping these links may be useful to you ;)
They are all about Cityscapes dataset:

Original videos https://www.cityscapes-dataset.com/examples/#videos
Examples from youtube:
https://www.youtube.com/results?search_query=cityscapes+dataset
https://www.youtube.com/watch?v=lGOjchGdVQs

Best,

irvingzhang0512 · 2021-12-01T09:58:16Z

@MengzhangLI

This PR is ready for review.

Create a 10s long demo video based on Cityscapes test set berlin folder. Maybe we can upload this video to your server and add wget xxx in getting started docs.

demo.mp4

MengzhangLI · 2021-12-01T10:12:41Z

Hi, please fix the lint error. If our codebase is your upstream, you may

use git pull upstream master and pre-commit run --all-files.

docs/get_started.md

MengzhangLI · 2021-12-01T13:56:31Z

It looks good to me! Thanks for your nice pr.

Please fix lint version, typesetting and sentences in get_started.md above. Thanks!

Best,

docs/get_started.md

…deo_demo

* First commit for video demo * fix lint * update * fix * add wget in shell

* map location to cpu when load checkpoint (open-mmlab#1007) * [Enhancement] Support minus output feature index in mobilenet_v3 (open-mmlab#1005) * fix typo in mobilenet_v3 * fix typo in mobilenet_v3 * use -1 to indicate output tensors from final stage * support negative out_indices * [Enhancement] inference speed and flops tools. (open-mmlab#986) * add the function to test the dummy forward speed of models. * add tools to test the flops and inference speed of multiple models. * [Fix] Update pose tracking demo to be compatible with latest mmtrakcing (open-mmlab#1014) * update mmtracking demo * support both track_bboxes and track_results * add docstring * [Fix] fix skeleton_info of coco wholebody dataset (open-mmlab#1010) * fix wholebody base dataset * fix lint * fix lint Co-authored-by: ly015 <liyining0712@gmail.com> * [Feature] Add ViPNAS models for wholebody keypoint detection (open-mmlab#1009) * add configs * add dark configs * add checkpoint and readme * update webcam demo * fix model path in webcam demo * fix unittest * [Fix] Fix bbox label visualization (open-mmlab#1020) * update model metafiles (open-mmlab#1001) * update hourglass ae .md (open-mmlab#1027) * [Feature] Add ViPNAS mbv3 (open-mmlab#1025) * add vipnas mbv3 * test other variants * submission for mmpose * add unittest * add readme * update .yml * fix lint * rebase * fix pytest Co-authored-by: jin-s13 <jinsheng13@foxmail.com> * [Enhancement] Set a random seed when the user does not set a seed (open-mmlab#1030) * fix randseed * fix lint * fix import * fix isort * update yapf hook * revert yapf version * add cfg file for flops and speed test, change the bulid_posenet to init_pose_model and fix some typo in cfg (open-mmlab#1028) * [Enhancement] Add more functions for speed test tool (open-mmlab#1034) * add batch size and device args in speed test script, and remove MMDataParallel warper * add vipnas_mbv3 model * fix dead link (open-mmlab#1038) * Skip CI when some specific files were changed (open-mmlab#1041) * update sigmas (open-mmlab#1040) * add more configs, ckpts and logs for HRNet on PoseTrack18 (open-mmlab#1035) * [Feature] Add PoseWarper dataset (open-mmlab#1006) * add PoseWarper dataset and base class * modify pipelines related to video * add unittest for PoseWarper dataset * add unittest for evaluation function in posetrack18-realted dataset, and add some annotations json files * fix typo * fix unittest CI failure * fix typo * add PoseWarper dataset and base class * modify pipelines related to video * add unittest for PoseWarper dataset * add unittest for evaluation function in posetrack18-realted dataset, and add some annotations json files * fix typo * fix unittest CI failure * fix typo * modify some methods in the base class to improve code coverage rate * recover some mistakenly-deleted notes * remove test_dataset_info part for the new TopDownPoseTrack18VideoDataset class * cancel uncompleted previous runs (open-mmlab#1053) * [Doc] Add inference speed results (open-mmlab#1044) * add docs related to inference speed results * add corresponding Chinese docs and fix some typos * add Chinese docs in readthedocs * remove the massive table in readme * minor modification to wording Co-authored-by: ly015 <liyining0712@gmail.com> * [Feature] Add PoseWarper detector model (open-mmlab#932) * Add top down video detector module * Add PoseWarper neck * add function _freeze_stages * fix typo * modify PoseWarper detector and PoseWarperNeck * fix typo * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * Add top down video detector module * Add PoseWarper neck * add function _freeze_stages * fix typo * modify PoseWarper detector and PoseWarperNeck * fix typo * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * modify dependency on mmcv version in posewarper neck * reduce memory cost in test * modify flops tool for more flexible input format * Add top down video detector module * Add PoseWarper neck * add function _freeze_stages * fix typo * modify PoseWarper detector and PoseWarperNeck * fix typo * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * Add PoseWarper neck * modify PoseWarper detector and PoseWarperNeck * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * modify dependency on mmcv version in posewarper neck * reduce memory cost in test * modify flops tool for more flexible input format * modify the posewarper detector description * modify some arguments and related fields * modify default values for some args * fix readthedoc bulid typo * fix ignore path (open-mmlab#1059) * [Doc] Add related docs for PoseWarper (open-mmlab#1036) * add related docs for PoseWarper * add related readme docs for posewarper * modify related args in posewarper stage2 config * modify posewarper stage2 config path * add description about val_boxes path for data preparation (open-mmlab#1060) * bump version to v0.21.0 (open-mmlab#1061) * [Feature] Add ViPNAS_Mbv3 wholebody model (open-mmlab#1055) * add vipnas mbv3 coco_wholebody * add vipnas mbv3 coco_wholebody md&yml * fix lint Co-authored-by: ly015 <liyining0712@gmail.com> Co-authored-by: Lumin <30328525+luminxu@users.noreply.github.com> Co-authored-by: zengwang430521 <zengwang430521@gmail.com> Co-authored-by: Jas <jinsheng@sensetime.com> Co-authored-by: jin-s13 <jinsheng13@foxmail.com> Co-authored-by: Qikai Li <87690686+liqikai9@users.noreply.github.com> Co-authored-by: QwQ2000 <396707050@qq.com>

First commit for video demo

3745f7d

fix lint

561beb5

Junjun2016 reviewed Nov 5, 2021

View reviewed changes

docs/get_started.md Show resolved Hide resolved

update

9a5e8d5

Junjun2016 requested a review from MengzhangLI December 1, 2021 12:27

MengzhangLI reviewed Dec 1, 2021

View reviewed changes

docs/get_started.md Outdated Show resolved Hide resolved

MengzhangLI reviewed Dec 1, 2021

View reviewed changes

docs/get_started.md Show resolved Hide resolved

MengzhangLI added the WIP Work in process label Dec 1, 2021

irvingzhang0512 added 3 commits December 2, 2021 09:10

Merge branch 'master' of github.com:open-mmlab/mmsegmentation into vi…

93fd153

…deo_demo

fix

ccf1ea5

add wget in shell

f561bb0

irvingzhang0512 requested a review from MengzhangLI December 2, 2021 02:27

MengzhangLI approved these changes Dec 2, 2021

View reviewed changes

MengzhangLI added the Merging PR waited for merging label Dec 8, 2021

xvjiarui approved these changes Dec 9, 2021

View reviewed changes

xvjiarui merged commit dec5bf0 into open-mmlab:master Dec 9, 2021

MengzhangLI removed Merging PR waited for merging WIP Work in process labels Feb 13, 2022

bowenroom pushed a commit to bowenroom/mmsegmentation that referenced this pull request Feb 25, 2022

[Feature] Support video demo (open-mmlab#1014)

0d3ee4b

* First commit for video demo * fix lint * update * fix * add wget in shell

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support video demo #1014

[Feature] Support video demo #1014

irvingzhang0512 commented Nov 5, 2021

CLAassistant commented Nov 5, 2021 •

edited

Loading

codecov bot commented Nov 5, 2021 •

edited

Loading

Junjun2016 commented Nov 5, 2021

Junjun2016 commented Nov 5, 2021

MengzhangLI commented Nov 5, 2021

irvingzhang0512 commented Nov 8, 2021

Junjun2016 commented Nov 8, 2021

RockeyCoss commented Nov 15, 2021

Junjun2016 commented Nov 24, 2021

MengzhangLI commented Nov 24, 2021

MengzhangLI commented Dec 1, 2021

irvingzhang0512 commented Dec 1, 2021

irvingzhang0512 commented Dec 1, 2021

MengzhangLI commented Dec 1, 2021

irvingzhang0512 commented Dec 1, 2021

MengzhangLI commented Dec 1, 2021

MengzhangLI commented Dec 1, 2021

[Feature] Support video demo #1014

[Feature] Support video demo #1014

Conversation

irvingzhang0512 commented Nov 5, 2021

Motivation

Use cases (Optional)

CLAassistant commented Nov 5, 2021 • edited Loading

codecov bot commented Nov 5, 2021 • edited Loading

Codecov Report

Junjun2016 commented Nov 5, 2021

Junjun2016 commented Nov 5, 2021

MengzhangLI commented Nov 5, 2021

irvingzhang0512 commented Nov 8, 2021

Junjun2016 commented Nov 8, 2021

RockeyCoss commented Nov 15, 2021

Junjun2016 commented Nov 24, 2021

MengzhangLI commented Nov 24, 2021

MengzhangLI commented Dec 1, 2021

irvingzhang0512 commented Dec 1, 2021

irvingzhang0512 commented Dec 1, 2021

MengzhangLI commented Dec 1, 2021

irvingzhang0512 commented Dec 1, 2021

MengzhangLI commented Dec 1, 2021

MengzhangLI commented Dec 1, 2021

CLAassistant commented Nov 5, 2021 •

edited

Loading

codecov bot commented Nov 5, 2021 •

edited

Loading