Add new OGR driver for OpenDRIVE (XODR) #9504

michikommader · 2024-03-19T13:57:15Z

What does this PR do?

This adds a read-only vector driver for the road description format ASAM OpenDRIVE®. OpenDRIVE is an open industry standard in the automotive domain of driving simulation, maintained by ASAM e.V. It is increasingly intersecting with the public and GIS domains which raises the need for better interoperability with open-source spatial tools.

This XODR driver is developed by German Aerospace Center (DLR) and licenced under MIT.
This XODR driver implementation uses libOpenDRIVE which is licenced under Apache 2.0.
This XODR driver is supposed to be used as plug-in.

I am am open to a fruitful discussion in order to make OpenDRIVE directly usable through GDAL. The context for this development has been introduced on FOSSGIS conference 2024, Geospatial World Forum 2024 and on FOSS4G Europe 2024.

What are related issues/pull requests?

Tasklist

Make sure newly added files are correctly formatted (cf pre-commit configuration)
Add test case(s)
Add documentation
~~Updated Python API documentation (swig/include/python/docs/)~~ I see no necessity there for adding a new driver
Clarify handling of custom docker/ubuntu-small/DockerfileXODR with GDAL maintainers: Which file location is better suited for future maintenance – the driver's source code directory or a separate repository?
Review
Adjust for comments
All CI builds and checks have passed

Environment

Provide environment details, if relevant:

OS: tested with Ubuntu 24.04, Windows 10
Compiler: GCC on Linux, GCC 13.1.0 x86_64 (MCF threads) MinGW-w64 MSVCRT on Windows

rouault

Nice work generally. See my comments.

I would also like this to be triggered more by our CI.
Can you add a libopendrive build in .github/workflows/ubuntu_20.04/Dockerfile.ci and .github/workflows/ubuntu_22.04/Dockerfile.ci

Do you know how your driver (and libopendrive) is robust to corrupted / hostile datasets? At the very minimum, I'd like to see a test with a "random file" that has xodr extension, but isn't valid XODR file. Ideally we'd want to stress-test it using OSSFuzz. Cf https://github.com/OSGeo/gdal/blob/master/fuzzers/README.TXT , https://github.com/OSGeo/gdal/blob/master/fuzzers/build.sh

autotest/ogr/ogr_xodr.py

doc/source/drivers/vector/xodr.rst

ogr/ogrsf_frmts/xodr/ogrxodrlayerroadmark.cpp

ogr/ogrsf_frmts/xodr/ogrxodrlayerroadobject.cpp

ogr/ogrsf_frmts/xodr/ogrxodrlayerroadsignal.cpp

rouault · 2024-03-19T15:47:51Z

Is the some packaging of libopendrive in Debian, conda-forge, etc ?

rouault · 2024-04-18T10:18:10Z

@michikommader Do you plan any further work on this any time soon ? I'm asking as I'm trying to keep the list of opened PRs low, and if something is not going to progress for some time, it is better to close the PR, and re-open it later when work restart

michikommader · 2024-04-18T13:31:47Z

@michikommader Do you plan any further work on this any time soon ?

Apologies for my inactivity. Yes, we already went through some of your valuable remarks internally but did not review and push changes yet. I plan to supply all the missing information in the first week of May.

michikommader · 2024-05-06T12:51:11Z

Is the some packaging of libopendrive in Debian, conda-forge, etc ?

Not yet. I myself am not maintainer of libOpenDRIVE and would rather see packaging responsibility on the maintainer side. We can separately discuss if our institute could take over this task. Also, ASAM e. V. as standardisation organisation behind OpenDRIVE could have an interest in that.

michikommader · 2024-05-15T20:56:10Z

Do you know how your driver (and libopendrive) is robust to corrupted / hostile datasets? At the very minimum, I'd like to see a test with a "random file" that has xodr extension, but isn't valid XODR file.

libOpenDRIVE forwards XML data loading directly to pugixml. According to pugixml's exception guarantees "most pugixml functions have a no-throw exception guarantee". For handling parsing errors the xml_parse_result would have to be inspected for its xml_parse_status. But, from GDAL driver side we do not have access to that. Unfortunately, libOpenDRIVE itself only uses simple printf() statements to report possible problems.

At least, libOpenDRIVE gives access to the parsed pugi::xml_document. Checking for the availability of "expected OpenDRIVE XML nodes" would allow basic validation if the parsing was successful. But, I don't see this as a very practicable way. If the dataset was corrupted "somewhere in the middle", pugixml would still have successfully parsed all previous content and made this accessible to the caller. For our use case of ensuring robustness in GDAL, I see this as a major deficit. The best solution would be to extend libOpenDRIVE to allow at least access to pugixml's xml_parse_status with which we could implement a somewhat meaningful error handling. I sent a pull request to the maintainer.

michikommader · 2024-05-30T16:02:49Z

I'll focus on satisfying cppcheck_2004 next days.

doc/source/development/building_from_source.rst

.github/workflows/ubuntu_20.04/Dockerfile.ci

.github/workflows/ubuntu_22.04/Dockerfile.ci

michikommader

@rouault In the current master branch you use ARROW_VERSION=16.0.0-1 but when testing, I have to set it to 16.1.0-1 in order to successfully build the Ubuntu 24.04 Docker image.

coveralls · 2024-05-31T12:59:35Z

coverage: 69.225% (+0.02%) from 69.21%
when pulling 4280011 on DLR-TS:libopendrive-pr
into da83066 on OSGeo:master.

rouault · 2024-05-31T12:59:37Z

I have to set it to 16.1.0-1 in order to successfully build the Ubuntu 24.04 Docker image.

yes, you've well done. The version has to be bumped regularly

rouault · 2024-05-31T15:26:43Z

@michikommader Do you mind rebasing on top of latest master? I've cherry-picked the change for ARROW_VERSION. And you'll have to do a few changes in your CMakeLists.txt file as well as the ogrxdrdrivercore.h due to the changes of #10068 that has just been merged. See 1cd120f as an example of the changes you'll have to do: adding NO_SHARED_SYMBOL_WITH_CORE, using #define FOO PLUGIN_SYMBOL_NAME(FOO) and removing CPL_DLL.

ogr/ogrsf_frmts/xodr/ogrxodrdrivercore.cpp

ogr/ogrsf_frmts/xodr/ogrxodrdatasource.cpp

rouault · 2024-06-02T17:22:49Z

ogr/ogrsf_frmts/xodr/ogrxodrlayerlaneborder.cpp

+        {
+            lineString.addPoint(laneVertex[0], laneVertex[1], laneVertex[2]);
+        }
+        OGRGeometry *geometry = lineString.MakeValid();


MakeValid() might return nullptr in case of failure, so this should be tested to avoid a nullptr dereference. Furthermore MakeValid() is only available or GEOS-enabled builds. So the choice is either to make GEOS a build requirement for the driver in ogr/ogrsf_frmts/CMakeLists.txt with ogr_dependent_driver(xodr OpenDRIVE "GDAL_USE_OPENDRIVE;GDAL_USE_GEOS"), or to only call MakeValid() if OGRGeometryFactory::haveGEOS() and use the original geometry then.
Is it common/expected that XODR datasets return non-valid geometries?

It should be rather uncommon that XODR datasets return non-valid geometries. But as we rely on libOpenDRIVE's internal linear approximation of parametric geometries, we should at least validate our derived Simple Feature version. I will make this more robust with your suggestions in mind. I see it reasonable to make GEOS dependency optional.

or to only call MakeValid() if OGRGeometryFactory::haveGEOS()

Does there exist any similarly convenient function to test if SFCGAL is available?

I will make this more robust with your suggestions in mind

I omit using MakeValid() now because handling of a potentially modified return geometry is not trivial. Instead, just a testing for IsValid() is performed. If that fails, the feature's geometry will stay unset and a warning is issued. I am not sure if allowing geometry-less features in such boundary cases is a reasonable approach? Theoretically, such cases should not occur.

Further, GEOS is a required dependency now because we need it for UnaryUnion().

Does there exist any similarly convenient function to test if SFCGAL is available?

I don't think so. But you could had a bool OGRGeometryFactory::haveSFCGAL() if needed next to haveGEOS() in ogr/ogrgeometryfactory.cpp)

Instead, just a testing for IsValid() is performed. If that fails, the feature's geometry will stay unset and a warning is issued

That seems a bit extreme. A lot (most) drivers will report invalid geometries, and let the responsibility to the user to fix them with MakeValid() (ogr2ogr has a -makevalid option for example), or discard them, or just process them as it. So if I were you, I would just report the geometry as obtained from libopendrive, ignoring if it is valid or not

ogr/ogrsf_frmts/xodr/ogrxodrlayerreferenceline.cpp

ogr/ogrsf_frmts/xodr/ogrxodrlayerroadmark.cpp

ogr/ogrsf_frmts/xodr/ogrxodrlayerroadsignal.cpp

doc/source/drivers/vector/xodr.rst

rouault · 2024-06-13T14:53:29Z

@michikommader Anything left on your side before merging this PR ? It looks good to me (the CI failures are unrelated)

michikommader · 2024-06-18T07:56:46Z

@michikommader Anything left on your side before merging this PR ? It looks good to me (the CI failures are unrelated)

I will today commit a small simplification regarding your last suggestions with MakeValid().

- Refactor file opening - Ensure m_ prefix for member variables - Simplify for loops - Use default member initializers - Simplify and optimise manual memory management using std::unique_ptr - Pass by const and reference where possible - Use SetGeometryDirectly on features to avoid memory leaks - Implement deferred driver loading capability as plugin, as per RFC 96 - Implement OGRGetNextFeatureThroughRaw - Move XODR driver config to official Dockerfiles - Add "test_ogrsf" compliance checking to Python tests - Improve documentation

- Pass std::string by const reference - Switch to postfix operators for non-primitive types - Avoid virtual function calling from subclass constructors

Mark GEOS as mandatory dependency

michikommader · 2024-06-19T10:01:36Z

@michikommader Anything left on your side before merging this PR ? It looks good to me (the CI failures are unrelated)

From my side, there is nothing more to add. The driver is well usable in practical applications.

As next steps, I will look into providing ready-to-use binary releases of the libOpenDRIVE dependency. If development of libOpenDRIVE progresses with new features, this XODR driver might require adjustments. I would prepare theses as pull requests in a similar manner here.

Thank you for your patience and guidance for making this driver part of GDAL!

rouault · 2024-06-24T13:18:47Z

I've refreshed the master Docker images, so "ghcr.io/osgeo/gdal:ubuntu-full-latest" and "ghcr.io/osgeo/gdal:alpine-normal-latest" contain the driver

dbaston · 2024-07-09T00:04:57Z

Should a FindOpenDrive.cmake file be added to avoid the following output in the cmake configure step?

CMake Warning at cmake/helpers/CheckDependentLibraries.cmake:149 (find_package):
  By not providing "FindOpenDrive.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "OpenDrive", but CMake did not find one.

  Could not find a package configuration file provided by "OpenDrive" with
  any of the following names:

    OpenDriveConfig.cmake
    opendrive-config.cmake

  Add the installation prefix of "OpenDrive" to CMAKE_PREFIX_PATH or set
  "OpenDrive_DIR" to a directory containing one of the above files.  If
  "OpenDrive" provides a separate development package or SDK, be sure it has
  been installed.
Call Stack (most recent call first):
  cmake/helpers/CheckDependentLibraries.cmake:807 (gdal_check_package)
  gdal.cmake:261 (include)
  CMakeLists.txt:240 (include)

https://github.com/OSGeo/gdal/actions/runs/9569105782/job/26380878179#step:13:361

rouault · 2024-07-09T09:05:35Z

Should a FindOpenDrive.cmake file be added to avoid the following output in the cmake configure step?

we want to avoid creating new Find cmake files when possible. Here it is not necessary as opendrive comes with CMake config files. Hence #10391 should fix it

rouault reviewed Mar 19, 2024

View reviewed changes