Jia Huang1, Alvika Gautam1, Junghun Choi2 and Srikanth Saripalli1
1. Texas A&M University; 2. Hyundai Motor Company
- One 32 Channels Lidar: Velodyne Ultra Puck
- Three point grey RGB Cameras: TBA
- Inertial Navigation System (GPS/IMU): Vectornav VN-300 Dual Antenna GNSS/INS
- Ultra-Wideband sensors(UWB): Decawave (now Qorvo) modules (TREK 1000 RTLS evaluation kit with DW 1000 UWB chip configured as 4 Anchors and 6 Tags)
UWB_Dataset ├── UWB-gcart-camera-calibration --directory for the three front cameras calibration by using Kalibr ├── benchmarks --directory for object tracking benchmark by using Yolov5 and DeepSORT ├── decawave_ros --directory for TREK1000 UWB ros package containing ros node and messages ├── lidar2camera --directory for lidar to middle camera calibration by using SensorsCalibration toolbox ├── sensor_syn.py --script for sensor data approximate synchronization for future sensor fusion
The dataset contains 30,936 frames which come from the front three cameras with 10,312 frames each. Every 5-th frame was annotated manually and carefully in Computer Vision Annotation Tool (CVAT) for the front middle camera frames while the ones in between are interpolated automatically without adjustment. There are a total of 8,368 bounding boxes manually annotated for pedestrians and cyclists along with six tag IDs for those who carried tags, and 33,239 bounding boxes are interpolated in CVAT. These annotations are exported in MOT 1.1 format for the following multi-object tracking task evaluation for images. The annotations for the left and right cameras frames are in progress and will be updated on the Github gradually.
Annotations include raw images in the img1 folder, annotation tasks include the raw videos named output.mp4 used for annotation in CVAT.
rosbag_filename(subfolder_middle_annotation).zip
├── gt
├── gt.txt
└── labels.txt
├── img1
├── 000001.PNG
├── 000002.PNG
└── ......
Example of rosbag_filename(subfolder_middle_annotation).zip : 2023-02-04-16-04-35_0(1_middle_annotation)
rosbag_filename(subfolder_middle_task).zip
├── annotations.json
├── task.json
├── data
├── index.json
├── manifest.jsonl
└── output.mp4
Example of rosbag_filename(subfolder_middle_task).zip : 2023-02-04-16-04-35_0(1_middle_task)
Sequence 1_1: 2023-02-04-16-04-35_0.bag: (Annotation) (Task)
Sequence 1_2: 2023-02-04-16-04-35_0.bag: (Annotation) (Task)
Sequence 2: 2023-02-10-15-23-44_0.bag: (Annotation) (Task)
Sequence 3_1: 2023-02-10-15-26-14_1.bag:(Annotation) (Task)
Sequence 3_2: 2023-02-10-15-26-14_1.bag:(Annotation) (Task)
Sequence 4: 2023-02-10-15-28-44_2.bag: (Annotation) (Task)
Sequence 5_1: 2023-02-10-15-35-56_0.bag: (Annotation) (Task)
Sequence 5_2: 2023-02-10-15-35-56_0.bag: (Annotation) (Task)
Sequence 6: 2023-02-10-15-38-30_0.bag: (Annotation) (Task)
Sequence 7: 2023-02-12-13-59-09_0.bag: (Annotation) (Task)
Sequence 8: 2023-02-12-14-13-00_0.bag: (Annotation) (Task)
Sequence 9: 2023-02-12-14-14-40_0.bag: (Annotation) (Task)
Sequence 10:2023-02-12-14-20-14_0.bag: (Annotation) (Task)
Sequence 11:2023-02-12-14-24-09_0.bag: (Annotation) (Task)
Sequence 12:2023-02-12-14-25-15_0.bag: (Annotation) (Task)
Sequence 13:2023-02-12-14-31-29_0.bag: (Annotation) (Task)
Sequence 14:2023-02-17-15-40-11_0.bag: (Annotation) (Task)
Sequence 15:2023-02-17-15-42-27_0.bag: (Annotation) (Task)
Sequence 16:2023-02-17-15-44-23_0.bag: (Annotation) (Task)
Sequence 17:2023-02-17-16-00-23_0.bag: (Annotation) (Task)
Sequence 18_1:2023-02-17-16-53-27_0.bag: (Annotation) (Task)
Sequence 18_2:2023-02-17-16-53-27_0.bag: (Annotation) (Task)
Sequence 19:2023-02-17-16-54-38_0.bag: (Annotation) (Task)
Sequence 20:2023-02-17-17-01-15_0.bag: (Annotation) (Task)
Sequence 21:2023-02-17-17-03-49_0.bag: (Annotation) (Task)
The sensor suite cameras (left, middle and right on the autonomous shuttle roof rack) have a 30$^{\circ}$ overlapping field of view between adjacent cameras. The intrinsic and extrinsic camera parameters are estimated by the multiple camera calibration tool in the Kalibr toolbox. A 6 x 6 aprilgrid target with spacing of 0.03m is used. We utilize a pinhole projection model for our cameras, where a three-dimensional scene is projected onto an image plane through a perspective transform. The calibration details can be found in the folder UWB-gcart-camera-calibration.
To prepare data for LiDAR and middle camera calibration, the data across both sensors is first matched by finding the pairs with the closest time stamps between two sensors. Then the point cloud and image pairs are calibrated in SensorsCalibration toolbox to facilitate the sensor fusion process. The calibration details can be found in the folder lidar2camera.
| Case | HOTA | MOTA | MOTP | IDF1 | FP | FN | ID Sw. | Recall | Precision | Dets | GT Dets | IDs | GT IDs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 41.577 | 42.046 | 72.496 | 54.958 | 9144 | 11006 | 729 | 69.451 | 73.236 | 34165 | 36027 | 491 | 231 |
| 2 | 45.873 | 44.204 | 78.173 | 57.293 | 1607 | 2038 | 398 | 71.874 | 76.42 | 6815 | 7246 | 425 | 231 |
To reproduce the results, please refer to here
Data included in raw ROS bagfiles:
| Topic Name | Message Tpye | Message Descriptison |
|---|---|---|
| /Decawave | decawave_ros/uwb_distance | Distances between Anchors and Tags |
| /left/image_raw | sensor_msgs/Image | Images from the left camera |
| /middle/image_raw | sensor_msgs/Image | Images from the middele camera |
| /right/image_raw | sensor_msgs/Image | Images from the right camera |
| /vectornav/fix | sensor_msgs/NavSatFix | |
| /vectornav/gps | vn300/gps | GPS data from VectorNav-VN300 |
| /vectornav/imu | vn300/sensors | Imu data from VectorNav-VN300 |
| /vectornav/ins | vn300/ins | INS data from VectorNav-VN300 |
| /velodyne_points | sensor_msgs/PointCloud2 | PointCloud produced by the Velodyne Lidar |
The ROS bag information can be found on the google drive link. The following are the google drive links for the ROS Bag files.
Sequence 1: 2023-02-04-16-04-35_0.bag: (19.91GB)
Sequence 2: 2023-02-10-15-23-44_0.bag: (19.94GB)
Sequence 3: 2023-02-10-15-26-14_1.bag: (20.07GB)
Sequence 4: 2023-02-10-15-28-44_2.bag: (18.42GB)
Sequence 5: 2023-02-10-15-35-56_0.bag: (7.29GB)
Sequence 6: 2023-02-10-15-38-30_0.bag: (8.04GB)
Sequence 7: 2023-02-12-13-59-09_0.bag: (6.66GB)
Sequence 8: 2023-02-12-14-13-00_0.bag: (5.97GB)
Sequence 9: 2023-02-12-14-14-40_0.bag: (15.71GB)
Sequence 10:2023-02-12-14-20-14_0.bag: (5.69GB)
Sequence 11:2023-02-12-14-24-09_0.bag: (9.16GB)
Sequence 12:2023-02-12-14-25-15_0.bag: (8.66GB)
Sequence 13:2023-02-12-14-31-29_0.bag: (5.59GB)
Sequence 14:2023-02-17-15-40-11_0.bag: (4.26GB)
Sequence 15:2023-02-17-15-42-27_0.bag: (7.53GB)
Sequence 16:2023-02-17-15-44-23_0.bag: (16.47GB)
Sequence 17:2023-02-17-16-00-23_0.bag: (7.04GB)
Sequence 18:2023-02-17-16-53-27_0.bag: (3.93GB)
Sequence 19:2023-02-17-16-54-38_0.bag: (6.57GB)
Sequence 20:2023-02-17-17-01-15_0.bag: (6.57GB)
Sequence 21:2023-02-17-17-03-49_0.bag: (4.43GB)
- Add camera and UWB calibration results
- Annotate the left and right camera frames
- Annotate the lidar point clouds
@misc{huang2023wideview,
title={WiDEVIEW: An UltraWideBand and Vision Dataset for Deciphering Pedestrian-Vehicle Interactions},
author={Jia Huang and Alvika Gautam and Junghun Choi and Srikanth Saripalli},
year={2023},
eprint={2309.16057},
archivePrefix={arXiv},
primaryClass={cs.RO}
}
All datasets and code on this page are copyright by us and published under the GNU General Public License v2.0.