Initial commit

Author: linh-gist

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "src/cpputils/fuzzylite"]
+	path = src/cpputils/fuzzylite
+	url = https://github.com/fuzzylite/fuzzylite.git
+[submodule "src/cpputils/pybind11"]
+	path = src/cpputils/pybind11
+	url = https://github.com/pybind/pybind11.git

README.md

@@ -1,2 +1,93 @@
-# VisualRFS
-This is the official Python and C++ implementation repository for a paper entitled "Visual multi-object tracking with re-identification and occlusion handling using labeled random finite sets", Pattern Recognition (https://arxiv.org/pdf/2407.08872).
+# Visual RFS
+
+This is the official Python and C++ implementation repository for a paper entitled "Visual Multi-Object Tracking with Re-Identification and Occlusion Handling using Labeled Random Finite Sets", *Pattern Recognition* ([arXiv 2405.18606](http://arxiv.org/abs/2405.18606)), ([MOTChallenge](https://motchallenge.net/method/MOT=8697&chl=5)).
+
+
+![](tracker_diagram.png)
+
+### Quick Overview
+- Exploiting object feature improves tracking performance of labeled random finite set filters (GLMB and LMB filters) by improving data association.
+- Using object deep feature is effective for re-ID tasks.
+- Fuzzy occlusion handling model improves tracking performance. 
+
+### Docker image
+Docker image to run VisualRFS codes can be found in [Docker Hub]().
+
+### Usage
+1. **Set Up Python Environment**
+    - Create a `conda` Python environment and activate it:
+        ```sh
+        conda create --name virtualenv python==3.7.16
+        conda activate virtualenv
+        ```
+    - Clone this repository recursively to have **fuzzylite**, and **pybind11**
+        ```sh
+        git clone --recursive https://github.com/linh-gist/visualrfs.git
+        ```
+
+2. **Install Packages**
+    - C++ Packages ([Make sure `Eigen 3.4.0` is installed](src/cpputils/README.md)): Navigate to the `src/cpputils` folder and run `python setup.py build develop`
+    - Python Packages: `pip install -r requirements.txt`
+
+3. **Configure Experiment Options**   
+Change the following options when initializing `GLMB` or `LMB` tracker
+    - C++ `GLMB(int width, int height, bool useFeat = true, bool useFuzzyPD = false)`
+    - C++ `LMB(int width, int height, bool useFeat = true, bool useFuzzyPD = false)`
+    - Python `LMB: def __init__(self, use_feat=True)`
+    - Python `GLMB:  def __init__(self, width, height, use_feat=True)`
+
+4. **Prepare Data**
+    - Datasets: 
+        - MOTChallenge datasets ([MOT16](https://motchallenge.net/data/MOT16/), [MOT17](https://motchallenge.net/data/MOT17/), [MOT20](https://motchallenge.net/data/MOT20/))
+        - You can run on your own dataset with your preference object detector
+    - Folder structure:
+        ```
+        |-- detection
+        |   |-- detector_fairmot256
+        |   |   |-- MOT16-01.npz
+        |   |   |-- MOT16-02.npz
+        |   |   |   ...
+        |   |   |-- MOT16-14.npz
+        |   |-- detector_fairmot256
+        |   |   |-- MOT20-01.npz
+        |   |   |-- ...
+        |   |   |-- MOT20-08.npz
+        |-- src
+        |   |-- cpputils
+        |   |-- joint_glmb
+        |   |-- joint_lmb
+        |   |-- tracking_utils
+        |-- requirements.txt
+        |-- README.md
+        ```
+    - OSPA2 is re-implemented in Python and following this paper, an example code is given in `ospa2.py`.
+        ```
+        @article{rezatofighi2020trustworthy,
+          title={How trustworthy are the existing performance evaluations for basic vision tasks?},
+          author={Tran Thien Dat Nguyen and Hamid Rezatofighi and Ba-Ngu Vo and Ba-Tuong Vo and Silvio Savarese and Ian Reid},
+          journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
+          year={2022}
+        }
+        ```
+
+5. **Run the Tracking Demo**
+   - Navigate to the [`joint_glmb`](src/joint_glmb) or [`joint_lmb`](src/joint_lmb) and run `python run_joint_glmb.py` OR `python run_joint_lmb.py`.
+
+
+
+### Contact
+Linh Ma (linh.mavan@gm.gist.ac.kr), Machine Learning & Vision Laboratory, GIST, South Korea
+
+### Citation
+If you find this project useful in your research, please consider citing by:
+
+```
+@article{linh2024inffus,
+      title={Visual Multi-Object Tracking with Re-Identification and Occlusion Handling using Labeled Random Finite Sets}, 
+      author={Linh~Van~Ma and Tran~Thien~Dat~Nguyen and Changbeom~Shim and Du~Yong~Kim and Namkoo~Ha and Moongu~Jeon},
+      journal={Pattern Recognition},
+      volume = {156},
+      year={2024},
+      publisher={Elsevier}
+}
+```

detection/detector_fairmot256/README

@@ -0,0 +1,42 @@
+Put detection of FairMOT with Re-Identification feature 256 dimension in this folder.
+
+@article{zhang2021fairmot,
+  title={Fairmot: On the fairness of detection and re-identification in multiple object tracking},
+  author={Zhang, Yifu and Wang, Chunyu and Wang, Xinggang and Zeng, Wenjun and Liu, Wenyu},
+  journal={International journal of computer vision},
+  volume={129},
+  pages={3069--3087},
+  year={2021},
+  publisher={Springer}
+}
+
+Detection files are Numpy files
+
+MOT16-01.npz  MOT16-04.npz  MOT16-07.npz  MOT16-10.npz  MOT16-13.npz
+MOT16-02.npz  MOT16-05.npz  MOT16-08.npz  MOT16-11.npz  MOT16-14.npz
+MOT16-03.npz  MOT16-06.npz  MOT16-09.npz  MOT16-12.npz
+
+The detection file stores information about the detected bounding boxes, including the coordinates
+(Top, Left, Bottom, Right) and the confidence score of each detection.
+It stores re-identification data linked to the corresponding bounding boxes.
+
+Below is an example of how this information is structured:
+dict_detect[str(self.frame_id) + '_det'] = np.copy(dets)
+dict_detect[str(self.frame_id) + '_feat'] = np.copy(id_feature)
+
+0000 = {str} '0_det'
+0001 = {str} '0_feat'
+0002 = {str} '1_det'
+0003 = {str} '1_feat'
+0004 = {str} '2_det'
+0005 = {str} '2_feat'
+0006 = {str} '3_det'
+0007 = {str} '3_feat'
+0008 = {str} '4_det'
+0009 = {str} '4_feat'
+0010 = {str} '5_det'
+0011 = {str} '5_feat'
+0012 = {str} '6_det'
+0013 = {str} '6_feat'
+0014 = {str} '7_det'
+0015 = {str} '7_feat'

detection/detector_gsdt/README

@@ -0,0 +1,40 @@
+Put detection of GSDT with Re-Identification feature 512 dimension in this folder.
+
+@inproceedings{wang2021joint,
+  title={Joint object detection and multi-object tracking with graph neural networks},
+  author={Wang, Yongxin and Kitani, Kris and Weng, Xinshuo},
+  booktitle={2021 IEEE international conference on robotics and automation (ICRA)},
+  pages={13708--13715},
+  year={2021},
+  organization={IEEE}
+}
+
+Detection files are Numpy files
+
+MOT20-01.npz  MOT20-03.npz  MOT20-05.npz  MOT20-07.npz
+MOT20-02.npz  MOT20-04.npz  MOT20-06.npz  MOT20-08.npz
+
+The detection file stores information about the detected bounding boxes, including the coordinates
+(Top, Left, Bottom, Right) and the confidence score of each detection.
+It stores re-identification data linked to the corresponding bounding boxes.
+
+Below is an example of how this information is structured:
+dict_detect[str(self.frame_id) + '_det'] = np.copy(dets)
+dict_detect[str(self.frame_id) + '_feat'] = np.copy(id_feature)
+
+0000 = {str} '0_det'
+0001 = {str} '0_feat'
+0002 = {str} '1_det'
+0003 = {str} '1_feat'
+0004 = {str} '2_det'
+0005 = {str} '2_feat'
+0006 = {str} '3_det'
+0007 = {str} '3_feat'
+0008 = {str} '4_det'
+0009 = {str} '4_feat'
+0010 = {str} '5_det'
+0011 = {str} '5_feat'
+0012 = {str} '6_det'
+0013 = {str} '6_feat'
+0014 = {str} '7_det'
+0015 = {str} '7_feat'

requirements.txt

@@ -0,0 +1,10 @@
+numpy==1.21.6
+h5py==3.1.0
+motmetrics==1.4.0
+pandas==1.3.5
+matplotlib==3.4.1
+packaging==24.0
+protobuf==4.24.4
+opencv-python==3.4.18.65
+lap==0.4.0
+openpyxl==3.0.10

src/cpputils/CMakeLists.txt

@@ -0,0 +1,31 @@
+cmake_minimum_required (VERSION 3.3)
+project (cpputils)
+
+add_subdirectory(pybind11)
+
+##### Murty
+#find_package (Eigen3 3.3 REQUIRED NO_MODULE)
+SET( EIGEN3_INCLUDE_DIR "/media/ubuntu/2715608D71CBF6FC/eigen-3.4.0" )
+IF( NOT EIGEN3_INCLUDE_DIR )
+    MESSAGE( FATAL_ERROR "Please point the environment variable EIGEN3_INCLUDE_DIR to the include directory of your Eigen3 installation.")
+ENDIF()
+INCLUDE_DIRECTORIES ( "${EIGEN3_INCLUDE_DIR}" )
+
+##### Fuzzy
+include_directories (./fuzzylite/fuzzylite)
+link_directories(./fuzzylite/fuzzylite/release/bin)
+
+##### OpenMP
+# find_package(OpenMP)
+# if (OPENMP_FOUND)
+#     set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${OpenMP_C_FLAGS}")
+#     set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
+#     set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${OpenMP_EXE_LINKER_FLAGS}")
+# endif()
+
+# Set the C++ standard to C++17
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED True)
+
+pybind11_add_module(cpputils src/main.cpp)
+target_link_libraries(cpputils PRIVATE  fuzzylite-static)

src/cpputils/README.md

@@ -0,0 +1,55 @@
+### Python wrapper for c++ implemented murty, gibbs, fuzzy, bbox overlap, esf, etc,.
+
+------------------------------------
+How to install this Python Wrapper
+1) Clone FuzzyLite: `git clone https://github.com/fuzzylite/fuzzylite.git`  
+   `cd fuzzylite`, `git checkout 5e8a1015e529ebba5a3d19430a05e51eef5f4a8b`  
+   You can also configure FuzzyLite with it latest code but need to configure `CMakeLists.txt`  
+2) Clone Pybind11: `git clone https://github.com/pybind/pybind11.git`
+3) _Build FuzzyLite_
+    1. Build your own FIS in Matlab, export to *.fis file, and copy to `tests` folder. You can also customize FIS in `build_engine()`, `fuzzy.hpp`. However, Matlab Fuzzy Logic Designer would be an easy tool to build and visualize your FIS. 
+    2. Build fuzzylite first to get static library in `./fuzzylite/release/bin`: 
+`cd fuzzylite` => `./build.sh` or  `build.bat` in Windows.
+4) Move to the root folder and run: `python setup.py build develop`
+
+------------------------------------
+Notes: 
+1) If you have a problem while building `fuzzylite` in Windows, e.g. _The C compiler is not able to compile a simple test program_, 
+2) run this command on your terminal `vcvarsall.bat x64`. Specify your Windows Platform by `x86_amd64`, `x64`, etc.
+3) Path example: `C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Auxiliary\Build\vcvarsall.bat x64`
+
+Install Eigen for Windows (after the following steps, add include directory `C:\eigen-3.4.0` for example.)
+1) Download Eigen 3.4.0 (NOT lower than this version) from it official website https://eigen.tuxfamily.org/ or [ZIP file here](https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip).
+2) `mkdir build_dir`
+3) `cd build_dir`
+4) `cmake ../`
+5) `make install`, this step does not require
+
+Install Eigen for Linux
+1) [install and use eigen3 on ubuntu 16.04](https://kezunlin.me/post/d97b21ee/) 
+2) `sudo apt-get install libeigen3-dev` libeigen3-dev is installed install to `/usr/include/eigen3/` and `/usr/lib/cmake/eigen3`.
+3) Thus, we must make a change in **CMakeLists.txt** `SET( EIGEN3_INCLUDE_DIR "/usr/local/include/eigen3" )` to `SET( EIGEN3_INCLUDE_DIR "/usr/include/eigen3/" )`.
+
+Murty Implementations
+1) [headers/murty.hpp] https://github.com/jonatanolofsson/mht/tree/master/murty 
+2) [headers/MurtyMiller.hpp] https://github.com/jonatanolofsson/MurtyAlgorithm
+3) https://github.com/fbaeuerlein/MurtyAlgorithm
+4) https://github.com/motrom/fastmurty
+5) https://github.com/gatagat/lap
+
+### Contact
+Linh Ma (linh.mavan@gm.gist.ac.kr), Machine Learning & Vision Laboratory, GIST, South Korea
+
+### Citation
+If you find this project useful in your research, please consider citing by:
+
+```
+@article{linh2024inffus,
+      title={Visual Multi-Object Tracking with Re-Identification and Occlusion Handling using Labeled Random Finite Sets}, 
+      author={Linh~Van~Ma and Tran~Thien~Dat~Nguyen and Changbeom~Shim and Du~Yong~Kim and Namkoo~Ha and Moongu~Jeon},
+      journal={Pattern Recognition},
+      volume = {156},
+      year={2024},
+      publisher={Elsevier}
+}
+```

src/cpputils/fuzzylite

@@ -0,0 +1,101 @@
+"""A setuptools based setup module.
+
+See:
+https://packaging.python.org/en/latest/distributing.html
+https://github.com/pypa/sampleproject
+"""
+
+# Always prefer setuptools over distutils
+from setuptools import setup, find_packages
+# To use a consistent encoding
+from codecs import open
+import os
+import re
+import sys
+import platform
+import subprocess
+from setuptools import setup, Extension
+from setuptools.command.build_ext import build_ext
+from distutils.version import LooseVersion
+
+here = os.path.abspath(os.path.dirname(__file__))
+
+
+class CMakeExtension(Extension):
+    def __init__(self, name, sourcedir=''):
+        Extension.__init__(self, name, sources=[])
+        self.sourcedir = os.path.abspath(sourcedir)
+
+
+class CMakeBuild(build_ext):
+    def run(self):
+        try:
+            out = subprocess.check_output(['cmake', '--version'])
+        except OSError:
+            raise RuntimeError("CMake must be installed to build the following extensions: " +
+                               ", ".join(e.name for e in self.extensions))
+
+        if platform.system() == "Windows":
+            cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)', out.decode()).group(1))
+            if cmake_version < '3.1.0':
+                raise RuntimeError("CMake >= 3.1.0 is required on Windows")
+
+        for ext in self.extensions:
+            self.build_extension(ext)
+
+    def build_extension(self, ext):
+        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
+        cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + extdir,
+                      '-DPYTHON_EXECUTABLE=' + sys.executable]
+
+        cfg = 'Release'
+        build_args = ['--config', cfg]
+
+        if platform.system() == "Windows":
+            cmake_args += ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}'.format(cfg.upper(), extdir)]
+            if sys.maxsize > 2 ** 32:
+                cmake_args += ['-A', 'x64']
+            build_args += ['--', '/m']
+        else:
+            cmake_args += ['-DCMAKE_BUILD_TYPE=' + cfg]
+            build_args += ['--', '-j2']
+
+        env = os.environ.copy()
+        env['CXXFLAGS'] = '{} -DVERSION_INFO=\\"{}\\"'.format(env.get('CXXFLAGS', ''),
+                                                              self.distribution.get_version())
+        if not os.path.exists(self.build_temp):
+            os.makedirs(self.build_temp)
+        subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
+        subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
+
+
+setup(
+    name='cpputils',
+    version='1.0.0',
+
+    description='build wrapper for murty, gibbs, fuzzy, bbox overlap, etc,.',
+
+    # The project's main homepage.
+    url='',
+
+    # Author details
+    author='Linh',
+    author_email='@gist.ac.kr',
+
+    # Choose your license
+    license='GPLv3',
+
+    # See https://pypi.python.org/pypi?%3Aaction=list_classifiers
+    classifiers=[
+        'Development Status :: 4 - Beta',
+        'Programming Language :: Python :: 3.8',
+    ],
+
+    # What does your project relate to?
+    keywords='murty, gibbs, fuzzy, bbox overlap, pybind11',
+    ext_modules=[CMakeExtension('cmake_example')],
+    cmdclass=dict(build_ext=CMakeBuild),
+    # You can just specify the packages manually here if your project is
+    # simple. Or you can use find_packages().
+    packages=find_packages(exclude=['contrib', 'docs', 'tests*']),
+)