diff --git a/projects/rtmo/README.md b/projects/rtmo/README.md index 18d9cd0ba3..62e9e66bc8 100644 --- a/projects/rtmo/README.md +++ b/projects/rtmo/README.md @@ -13,6 +13,16 @@ RTMO is a one-stage pose estimation model that achieves performance comparable t python demo/inferencer_demo.py $IMAGE --pose2d rtmo --vis-out-dir vis_results ``` +**rtmlib demo** + +[rtmlib](https://github.com/Tau-J/rtmlib/tree/main) provides simple and easy-to-use API for inference with RTMPose models. + +- Support OpenCV/ONNXRuntime/OpenVINO inference and does not require Pytorch or MMCV. +- Super user-friendly API for inference and visualization. +- Support both CPU and GPU inference. +- Automatically download onnx models from OpenMMLab model zoo. +- Support all series of RTMPose models (RTMPose, DWPose, RTMO, RTMW etc.) + ## 📜 Introduction Real-time multi-person pose estimation presents significant challenges in balancing speed and precision. While two-stage top-down methods slow down as the number of people in the image increases, existing one-stage methods often fail to simultaneously deliver high accuracy and real-time performance. This paper introduces RTMO, a one-stage pose estimation framework that seamlessly integrates coordinate classification by representing keypoints using dual 1-D heatmaps within the YOLO architecture, achieving accuracy comparable to top-down methods while maintaining high speed. We propose a dynamic coordinate classifier and a tailored loss function for heatmap learning, specifically designed to address the incompatibilities between coordinate classification and dense prediction models. RTMO outperforms state-of-the-art one-stage pose estimators, achieving 1.1% higher AP on COCO while operating about 9 times faster with the same backbone. Our largest model, RTMO-l, attains 74.8% AP on COCO val2017 and 141 FPS on a single V100 GPU, demonstrating its efficiency and accuracy. diff --git a/projects/rtmpose/README.md b/projects/rtmpose/README.md index 94bbc2ca2d..58f04f77b6 100644 --- a/projects/rtmpose/README.md +++ b/projects/rtmpose/README.md @@ -91,6 +91,7 @@ ______________________________________________________________________ | s | 72.2 | 200+ | 710+ | | m | 75.8 | 90+ | 430+ | | l | 76.5 | 50+ | 280+ | + | l-384 | 78.3 | - | 160+ | - 🛠️ **Easy to deploy** @@ -460,13 +461,24 @@ We also provide the ImageNet classification pre-trained weights of the CSPNeXt b ## 😎 Get Started [🔝](#-table-of-contents) -We provide two appoaches to try RTMPose: +We provide following appoaches to try RTMPose: - [Online RTMPose Demo](https://openxlab.org.cn/apps/detail/mmpose/RTMPose) - [Examples](https://github.com/open-mmlab/mmpose/tree/dev-1.x/projects/rtmpose/examples/onnxruntime) based on Python and ONNXRuntime (without mmcv) +- [rtmlib](https://github.com/Tau-J/rtmlib/tree/main) （without mmcv, pytorch） - MMPose demo scripts (based on Pytorch) - Pre-compiled MMDeploy SDK (Recommended, 6-10 times faster) +## rtmlib + +[rtmlib](https://github.com/Tau-J/rtmlib/tree/main) provides simple and easy-to-use API for inference with RTMPose models. + +- Support OpenCV/ONNXRuntime/OpenVINO inference and does not require Pytorch or MMCV. +- Super user-friendly API for inference and visualization. +- Support both CPU and GPU inference. +- Automatically download onnx models from OpenMMLab model zoo. +- Support all series of RTMPose models (RTMPose, DWPose, RTMO, RTMW etc.) + ### MMPose demo scripts MMPose provides demo scripts to conduct [inference with existing models](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html). diff --git a/projects/rtmpose/README_CN.md b/projects/rtmpose/README_CN.md index 6b33a20e9f..4438b807d3 100644 --- a/projects/rtmpose/README_CN.md +++ b/projects/rtmpose/README_CN.md @@ -87,6 +87,7 @@ ______________________________________________________________________ | s | 72.2 | 200+ | 710+ | | m | 75.8 | 90+ | 430+ | | l | 76.5 | 50+ | 280+ | + | l-384 | 78.3 | - | 160+ | - 🛠️ **易部署** @@ -451,13 +452,24 @@ RTMPose 是一个长期优化迭代的项目，致力于业务场景下的高性 ## 😎 快速尝试 [🔝](#-table-of-contents) -我们提供了两种途径来让用户尝试 RTMPose 模型： +我们提供了多种途径来让用户尝试 RTMPose 模型： - [在线 RTMPose Demo](https://openxlab.org.cn/apps/detail/mmpose/RTMPose) - [Examples](https://github.com/open-mmlab/mmpose/tree/dev-1.x/projects/rtmpose/examples/onnxruntime) 基于 Python 和 ONNXRuntime （无 MMCV 依赖） +- [rtmlib 推理库](https://github.com/Tau-J/rtmlib/tree/main) （无 MMCV、Pytorch 依赖） - MMPose demo 脚本 （基于 Pytorch） - MMDeploy SDK 预编译包 （推荐，速度提升6-10倍） +### rtmlib 推理库 + +[rtmlib](https://github.com/Tau-J/rtmlib/tree/main) 提供了开箱即用的 RTMPose 全系列官方及衍生模型的推理： + +- 无需安装 mmcv，mmengine，mmpose 等一系列训练库，无需 pytorch 环境，有 opencv 就能推理 +- 超级友好简洁的推理和可视化接口 +- 支持 CPU 和 GPU 推理 +- 自动下载和缓存 onnx 模型 +- 支持 RTMPose 全系列官方及衍生模型：RTMPose，DWPose，RTMO，RTMW etc. + ### MMPose demo 脚本 通过 MMPose 提供的 demo 脚本可以基于 Pytorch 快速进行[模型推理](https://mmpose.readthedocs.io/en/latest/user_guides/inference.html)和效果验证。 diff --git a/projects/rtmpose/examples/README.md b/projects/rtmpose/examples/README.md index 5846f039e7..4c9622ae31 100644 --- a/projects/rtmpose/examples/README.md +++ b/projects/rtmpose/examples/README.md @@ -16,3 +16,9 @@ This example shows how to run RTMPose inference with ONNXRuntime in Python. PoseTracker Android Demo Prototype based on mmdeploy. - [Original Repository](https://github.com/hanrui1sensetime/PoseTracker-Android-Prototype) + +### 4. rtmlib + +rtmlib is a super lightweight library to conduct pose estimation based on RTMPose models WITHOUT any dependencies like mmcv, mmpose, mmdet, etc. + +- [Original Repository](https://github.com/Tau-J/rtmlib/tree/main) diff --git a/projects/rtmpose/examples/rtmlib/README.md b/projects/rtmpose/examples/rtmlib/README.md new file mode 100644 index 0000000000..c34d8a6a85 --- /dev/null +++ b/projects/rtmpose/examples/rtmlib/README.md @@ -0,0 +1,124 @@ +# rtmlib + +![demo](https://github.com/Tau-J/rtmlib/assets/13503330/b7e8ce8b-3134-43cf-bba6-d81656897289) + +rtmlib is a super lightweight library to conduct pose estimation based on [RTMPose](https://github.com/open-mmlab/mmpose/tree/dev-1.x/projects/rtmpose) models **WITHOUT** any dependencies like mmcv, mmpose, mmdet, etc. + +Basically, rtmlib only requires these dependencies: + +- numpy +- opencv-python +- opencv-contrib-python +- onnxruntime + +Optionally, you can use other common backends like opencv, onnxruntime, openvino, tensorrt to accelerate the inference process. + +- For openvino users, please add the path `\envs\\Lib\site-packages\openvino\libs` into your environment path. + +## Installation + +- install from pypi: + +```shell +pip install rtmlib -i https://pypi.org/simple +``` + +- install from source code: + +```shell +git clone https://github.com/Tau-J/rtmlib.git +cd rtmlib + +pip install -r requirements.txt + +pip install -e . + +# [optional] +# pip install onnxruntime-gpu +# pip install openvino + +``` + +## Quick Start + +Run `webui.py`: + +```shell +# Please make sure you have installed gradio +# pip install gradio + +python webui.py +``` + +![image](https://github.com/Tau-J/rtmlib/assets/13503330/49ef11a1-a1b5-4a20-a2e1-d49f8be6a25d) + +Here is also a simple demo to show how to use rtmlib to conduct pose estimation on a single image. + +```python +import cv2 + +from rtmlib import Wholebody, draw_skeleton + +device = 'cpu' # cpu, cuda +backend = 'onnxruntime' # opencv, onnxruntime, openvino +img = cv2.imread('./demo.jpg') + +openpose_skeleton = False # True for openpose-style, False for mmpose-style + +wholebody = Wholebody(to_openpose=openpose_skeleton, + mode='balanced', # 'performance', 'lightweight', 'balanced'. Default: 'balanced' + backend=backend, device=device) + +keypoints, scores = wholebody(img) + +# visualize + +# if you want to use black background instead of original image, +# img_show = np.zeros(img_show.shape, dtype=np.uint8) + +img_show = draw_skeleton(img_show, keypoints, scores, kpt_thr=0.5) + + +cv2.imshow('img', img_show) +cv2.waitKey() +``` + +### Visualization + +| MMPose-style | OpenPose-style | +| :-------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------: | +| result | result | +| result | result | +| result | result | +| result | result | + +### Citation + +``` +@misc{rtmlib, + title={rtmlib}, + author={Jiang, Tao}, + year={2023}, + howpublished = {\url{https://github.com/Tau-J/rtmlib}}, +} + +@misc{jiang2023, + doi = {10.48550/ARXIV.2303.07399}, + url = {https://arxiv.org/abs/2303.07399}, + author = {Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai}, + keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences}, + title = {RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose}, + publisher = {arXiv}, + year = {2023}, + copyright = {Creative Commons Attribution 4.0 International} +} + +@misc{lu2023rtmo, + title={{RTMO}: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation}, + author={Peng Lu and Tao Jiang and Yining Li and Xiangtai Li and Kai Chen and Wenming Yang}, + year={2023}, + eprint={2312.07526}, + archivePrefix={arXiv}, + primaryClass={cs.CV} +} +```