Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to specify port data type for known c++ types from Python #153

Merged
16 commits merged into from
Aug 9, 2022

Conversation

drobison00
Copy link
Contributor

TODO: still need to add additional unit tests, and update to support a 3rd tuple parameter indicating if the raw data type or shared pointer should be used as the underlying node data type.

This will make a number of non-breaking changes to the process of Ingress and Egress port creation via Python.

  • Implements @mdemoret-nv 's changes to remove the need for specifying Ingress and Egress port data types during construction, and instead shifts to a system of providing lambda functions that create the appropriate derived object and return a common un-templated base; this results in a much more flexible, cleaner, interface and eliminates any hard cap on the number of ports that can be created on a segment from Python.

  • Extends previous updates to support python bindings for ingress/egress port creation to support either simple strings, or for specifying a string + pybind11 registered type tuple.
    pipeline.make_segment("name", ["a", "b", "c"], [], init)
    pipleine.make_segment("name", [("a", my_type), ("b", another_type), "c"]

Internally, the "simple string" case, will result in Ingress/Egress nodes with PyHolder date types; which can be inefficient, and require multiple edge converters when connected to pure c++ (non-python) data types. With the newly supported syntax, providing a pybind11 registered type will result in an attempt to lookup the associated (wrapped) cpptype and create an Ingress or Egress port using that type.

  • The ability to "lookup" constructors for ingress and egress ports via cpptype is accomplished via a new static 'port_registry' class, and a PortUtil class. The port registry stores a map from std::type_index to PortUtil, and PortUtil provides a number of helper functions for creating, casting, and manipulating Ingress and Egress ports.

@drobison00 drobison00 added improvement Improvement to existing functionality non-breaking Non-breaking change architecture labels Jul 27, 2022
@drobison00 drobison00 requested review from a team as code owners July 27, 2022 07:04
@drobison00 drobison00 added this to the Multi-Segment Support milestone Jul 27, 2022
@drobison00 drobison00 changed the title [Draft] Add ability to specify port data type for known c++ types from Python Add ability to specify port data type for known c++ types from Python Jul 28, 2022
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great new feature and looking really good. Have a few suggestions and I think we can cleanup some pieces by using type information thats already there.

src/public/node/port_registry.cpp Show resolved Hide resolved
src/public/node/port_registry.cpp Show resolved Hide resolved
include/srf/segment/builder.hpp Outdated Show resolved Hide resolved
include/srf/segment/ports.hpp Outdated Show resolved Hide resolved
include/srf/segment/ports.hpp Outdated Show resolved Hide resolved
python/srf/_pysrf/include/pysrf/segment.hpp Outdated Show resolved Hide resolved
python/srf/core/common.cpp Outdated Show resolved Hide resolved
python/srf/_pysrf/src/segment.cpp Show resolved Hide resolved
include/srf/node/port_registry.hpp Show resolved Hide resolved
python/srf/_pysrf/src/pipeline.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once CI is passing this looks good to merge

@drobison00
Copy link
Contributor Author

@gpucibot merge

@ghost ghost merged commit 6794f46 into nv-morpheus:branch-22.08 Aug 9, 2022
ghost pushed a commit that referenced this pull request Sep 6, 2022
…tructor (#165)

If you create a source or a sink class that derives from `PythonSink`, `PythonSource` or `PythonNode` and the message type does not have a default constructor, you will run into a compilation error. This was due to the fact that the new Port auto registration added in PR #153 tries to compile ingress/egress ports for both `T` and `shared_ptr<T>`.

This PR adds a check to skip compiling a port with `T` if `T` does not have a default constructor.

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Devin Robison (https://github.com/drobison00)

URL: #165
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
architecture improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants