Possible ways to improve the accuracy of GT

[qiaozhijian]

I accumulate the 3D lane GT using pose provided by origin waymo dataset. I find there is a serious blur of accumulated 3D submap in the segment(segment-9568394837328971633_466_365_486_365_with_camera_labels).

Let's look at the first small section of this submap. There are even some outliers such as red points.

For easy understanding, the corresponding images are as follows

I think there are two ways to improve the quality of map:
First, use only the closer lane points. For example, 20m

Second, fit the 3D lanes again in the submap and project them to every frame.

[qiaozhijian]

Code is as follows.

import os
import json
import tensorflow._api.v2.compat.v1 as tf
import numpy as np
import open3d as o3d
from waymo_open_dataset import dataset_pb2 as open_dataset
import cv2

tf.enable_eager_execution()


class OpenLane(object):

    def __init__(self, dataset_dir):
        self.dataset_dir = dataset_dir

    def fetch_data(self, segment, timestamp):
        gt_json = os.path.join(self.dataset_dir, 'lane3d_1000/training', segment, '{}.json'.format(timestamp))
        with open(gt_json, 'r') as fp:
            gt_dict = json.load(fp)

        image_path = os.path.join(self.dataset_dir, 'images', gt_dict['file_path'])
        img = cv2.imread(image_path)

        lane_all = []
        for lane in gt_dict['lane_lines']:
            xyz = np.asarray(lane['xyz']).reshape(3, -1).T
            lane_all.append(xyz)
        lane_points = np.vstack(lane_all)

        return img, lane_points


def visualize_pcd(points):
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(points)

    o3d.visualization.draw_geometries([pcd])


def lanemap(filename, openLane_dir):
    dataset = tf.data.TFRecordDataset(filename, compression_type='')
    segment = filename.split('/')[-1].replace('.tfrecord', '')
    openLane = OpenLane(openLane_dir)

    world_frame = None
    lane_map = []
    for cnt, data in enumerate(dataset):
        frame = open_dataset.Frame()
        frame.ParseFromString(bytearray(data.numpy()))
        timestamp = "{:<018}".format(frame.timestamp_micros)  # openlane seems to use 18 digit

        image0 = next(c for c in frame.images if c.name == open_dataset.CameraName.FRONT)
        calibration = next(cc for cc in frame.context.camera_calibrations if cc.name == image0.name)
        ex0 = np.array(calibration.extrinsic.transform, dtype=np.float32).reshape(4, 4)
        vechile_pose = np.array(image0.pose.transform, dtype=np.float32).reshape(4, 4)
        cam0_pose = vechile_pose @ ex0

        img, lane_points_c = openLane.fetch_data(segment, timestamp)
        # lane_points_c = lane_points_c[lane_points_c[:,0]<20]

        if world_frame is None:
            world_frame = cam0_pose
        T_wc = np.linalg.inv(world_frame) @ cam0_pose
        lane_points_w = lane_points_c @ T_wc[:3, :3] + T_wc[:3, 3].reshape(1, 3)
        # visualize_pcd(lane_points_c)
        lane_map.append(lane_points_w)

    lane_map = np.vstack(lane_map)
    visualize_pcd(lane_map)


if __name__ == '__main__':
    # Replace this path with your own tfrecords.

    filename = '/media/qzj/Extreme SSD/datasets/waymo/raw_data/segment-9568394837328971633_466_365_486_365_with_camera_labels.tfrecord'
    openLane_dir = '/media/qzj/Extreme SSD/datasets/OpenLane/'
    lanemap(filename, openLane_dir)

[RicardLee]

@qiaozhijian thanks for your effort and advice on this dataset. We are trying to reproduce the problem and fix it. In fact, 3D lanes filter by projection, consistent with the your second way, have been applied. Due to the interpolation operation, some outlier points error may be caused.

[RicardLee]

Refer to the issue 31, it may causes this problem. In new openlane, this problem will be effectively improved.

[qiaozhijian]

Refer to the issue 31, it may causes this problem. In new openlane, this problem will be effectively improved.

Thanks, Ricard Lee. I agree with you that coordinate system may cause this problem. But better yet, check whether this problem is solved using new openlane referring to my code. I know producing GT, especially for far lane points, is very difficult and of a lot work. So, feel free!

[RicardLee]

@qiaozhijian Thansk , before release, we will check this problem again.

[qiaozhijian]

I'm sorry that I had a mistake in my code.
lane_points_w = lane_points_c @ T_wc[:3, :3] + T_wc[:3, 3].reshape(1, 3)
should be
lane_points_w = lane_points_c @ T_wc[:3, :3].T + T_wc[:3, 3].reshape(1, 3)