Modify Pybind LoDTensor API according to length-based LoD #11106
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some background can be found here.
Essentially, our level of detail info (LoD) is currently both implemented in C++ and exposed in Python using "offsets". We want to encapsulate the
offset
implementation detail in C++ and only exposelength
based interface in Python.What this PR does (See the main change in
pybind.cc
here):Previously, the
set_lod
andlod
API inpybind.cc
uses offset_based LoD.This PR modifies the LoDTensor API in
pybind.cc
so that the user on the python side can provide or obtain length-based LoD usingset_lod
andlod
, respectively.The
init
API is also modified so that user can create a LoDTensor object in Python and initialize it with a length-based LoD.A
has_valid_lod
API is added inpybind.cc
so that user can check if the LoD info in a tensor is valid and matches the its data dimension on the python side.Utility functions to create LoDTensor in the python side defined here is largely simplified using the updated
set_lod
andlod
in pybind. (See changes here)Over 30 tests that uses
set_lod
andlod
has been modified to reflect the updated LoDTensor API inpybind.cc
.