MXNet Extensions enhancements2 #19016

samskalicky · 2020-08-26T04:46:17Z

Description

This PR contains a few enhancements for MXNet extensions:

Refactors MXLoadLib to return the handle to the loaded library and expects the user (or language binding) to close the library later.
Refactors lib_api.h by moving the function definitions into lib_api.cc to simplify building extensions
Refactored relu_lib example to split into multiple files, separating CUDA code from CXX code.
Updates all the examples in example/extensions to use the new lib_api.cc build flow & removes Symbol tests (now that Symbol API is deprecated) and replaces them with Gluon tests

Other changes

Improves the instructions in config/linux_gpu.cmake for building for specific GPU architectures.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

…to external_ops

mxnet-bot · 2020-08-26T04:46:18Z

Hey @samskalicky , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [edge, clang, centos-cpu, sanity, unix-cpu, windows-gpu, website, windows-cpu, unix-gpu, centos-gpu, miscellaneous]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

samskalicky · 2020-08-26T06:59:33Z

@leezu for review of the cmake config file changes and @mseth10 and @rondogency for review of everything else

config/linux_gpu.cmake

example/extensions/lib_custom_op/test_transposerowsp.py

samskalicky · 2020-08-28T20:54:24Z

CMakeLists.txt

+add_library(customop_lib SHARED ${CMAKE_CURRENT_SOURCE_DIR}/example/extensions/lib_custom_op/gemm_lib.cc ${CMAKE_CURRENT_SOURCE_DIR}/src/lib_api.cc)
+add_library(transposecsr_lib SHARED ${CMAKE_CURRENT_SOURCE_DIR}/example/extensions/lib_custom_op/transposecsr_lib.cc ${CMAKE_CURRENT_SOURCE_DIR}/src/lib_api.cc)
+add_library(transposerowsp_lib SHARED ${CMAKE_CURRENT_SOURCE_DIR}/example/extensions/lib_custom_op/transposerowsp_lib.cc ${CMAKE_CURRENT_SOURCE_DIR}/src/lib_api.cc)
+add_library(subgraph_lib SHARED ${CMAKE_CURRENT_SOURCE_DIR}/example/extensions/lib_subgraph/subgraph_lib.cc ${CMAKE_CURRENT_SOURCE_DIR}/src/lib_api.cc)
+add_library(pass_lib SHARED ${CMAKE_CURRENT_SOURCE_DIR}/example/extensions/lib_pass/pass_lib.cc ${CMAKE_CURRENT_SOURCE_DIR}/src/lib_api.cc)
+


Offline discussion with @leezu, in another PR we should move this build code into CMakeLists.txt for each example and use add_subdirectory to include it and replace the current Makefiles so theres only 1 set of build steps for each example.

marcoabreu · 2020-08-29T19:16:23Z

src/initialize.cc

-LibraryInitializer::~LibraryInitializer() {
-  close_open_libs();
-}
+LibraryInitializer::~LibraryInitializer() = default;


Can we add an assertion which checks if all handles have been closed during shutdown of mxnet? That could allow to catch a possible leak.

This was removed since there are cases where (for external ops for example) an object is registered in an MXNet data structure like the operator registry and during shutdown the object is attempted to be destructed but its pointing to an object in the loaded library. This ended up causing a segfault. Without closing open handles, we let loaded libraries live longer than libmxnet.so and allow it to shutdown cleanly.

This is why we changed MXLoadLib to return the handle to the library and call dlclose on the handle in Python.

However, this still isnt an issue since on process exit the library will be closed by the OS anyway when it cleans up.

Right, but hence an assertion or some other debug log to make the user aware that there are still unclosed handles. The idea is to give a hint during shutdown. If somebody then sees hundreds of unclosed handles, that could be a strong indicator of something being wrong - I'm not talking about automatic closing of the handle, just a message

We cant check if handles are closed while libmxnet.so is closing. It would have to be checked elsewhere so the loaded libraries can live during libmxnet.so shutdown. If we put in a check it will print out to the user every time since the expectation is that the handles are closed after libmxnet.so

We're not talking about hundreds of handles, we're talking about one or maybe two libraries loaded by the user explicitly.

Okay, I was not aware that the goal was that these libraries have a longer lifecycle than libmxnet.so itself. Thanks for elaborating

mseth10

LGTM

rondogency · 2020-09-01T05:09:29Z

example/extensions/lib_custom_op/relu_lib.cc

+.setBackward(backwardGPU, "gpu");
+
+MXReturnValue initialize(int version) {
+  if (version >= 20000) {


are you sure about this? since gemm_lib is still 10700

Yes, we should update all the examples to 20000 on master. ill do that in the next PR.

discussed offline and we will change example corresponding to master

rondogency · 2020-09-01T05:17:04Z

example/extensions/lib_custom_op/test_transposecsr.py

@@ -54,25 +54,25 @@
 print("indices:", c.indices.asnumpy())
 print("indptr:", c.indptr.asnumpy())

-print("--------start symbolic compute--------")
+print("--------start Gluon compute--------")
 d = mx.sym.Variable('d')


should this sym also get deprecated then?

No, we still use Symbol class, but we removed all the usages like bind

rondogency · 2020-09-01T05:20:31Z

python/mxnet/library.py

+        self.handle = handle
+    def __del__(self):
+        libdl = ctypes.CDLL("libdl.so")
+        libdl.dlclose(self.handle)


so it is to elaborate libdl.so lives longer than libmxnet.so?

this is how we close the library we loaded before, by using libdl.so to call dlcose

rondogency · 2020-09-01T05:22:50Z

src/lib_api.cc

+/*!
+ * Copyright (c) 2019 by Contributors
+ * \file lib_api.cc
+ * \brief APIs to interact with libraries


can we say "extension API to load dynamic loaded custom libraries", and also say it will depend on lib_api.h (2-stop file instead of 1-stop header file)

Sure, lets work on what the new text should be and update it in another PR.

* initial commit * fixed c++17 downgrade * fixed stringstream * fixed cast * changed to use pointers for stringstream since not copyable * fixed includes * fixed makefile includes * skipped lint for malloc/free for passing across C ABI Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-220.us-west-2.compute.internal>

ZiyueHuang · 2020-10-13T09:56:36Z

Hi @samskalicky , it seems that now we need src/lib_api.cc to build the custom operator, how can a user access src/lib_api.cc if using pip install mxnet?

samskalicky · 2020-10-13T15:50:55Z

Hi @samskalicky , it seems that now we need src/lib_api.cc to build the custom operator, how can a user access src/lib_api.cc if using pip install mxnet?

Hi @ZiyueHuang the lib_api.cc file can be accessed by downloading directly from github:
https://raw.githubusercontent.com/apache/incubator-mxnet/1.8.0.rc1/src/lib_api.cc
or
https://raw.githubusercontent.com/apache/incubator-mxnet/master/src/lib_api.cc

ZiyueHuang · 2020-10-15T08:20:07Z

If the user install mxnet via pip, only the header files in mxnet/include and the python files in mxnet/python are downloaded. Then if the user wants to use the custom operator, the user must additionally download lib_api.cc of the specific mxnet version. I think it is more convenient if the custom operator only depends on the header file.

samskalicky · 2020-10-15T15:42:01Z

If the user install mxnet via pip, only the header files in mxnet/include and the python files in mxnet/python are downloaded. Then if the user wants to use the custom operator, the user must additionally download lib_api.cc of the specific mxnet version. I think it is more convenient if the custom operator only depends on the header file.

Hi @ZiyueHuang thanks for that feedback. Initially I was thinking the same way, we only had the lib_api.h file for MXNet versions 1.6 and 1.7. But in 1.8 we realized that the single lib_api.h approach severely limited how the user had to construct their custom library. Since at that time the lib_api.h actually included function definitions, not just declarations, it was causing duplicate symbol errors if the file was included in multiple places.

So in 1.8 we split the code in lib_api.h into lib_api.cc so that the header file only included declarations and all definitions were in lib_api.cc. This allows users more flexibility to organize their code in multiple files.

The pip wheel is organized as the easy path to install in order to run Python programs using MXNet. But any C/C++ compilation for MXNet requires cloning the whole repo. Building a library for custom ops only requires the lib_api.cc/h files, so its an improvement in that direction. Maybe we could consider bundling the lib_api.cc file in the pip wheel too.

samskalicky · 2020-10-21T18:41:01Z

@ZiyueHuang in #19393 i propose adding the lib_api.cc file into the pip wheel to help easily build extensions libraries without downloading from github: 3d28436
Will you review/approve that PR?

Ubuntu added 30 commits August 11, 2020 16:07

initial commit

4fea36d

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

6d5fce2

…to external_ops

split lib_api.h into lib_api.cc, updated examples for 2.0/gluon

3cea397

fixed licenses

ead2684

whitespace

51ce458

whitespace

1ec330b

modernize

bee854b

fix modernize

c36363e

fix modernize

53b4136

fix modernize

2953891

fixed move

0c0cceb

added lib_api.cc to CMakeLists.txt for example libs

7cbc99b

working example

6965cd7

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

db84377

…to external_ops

remove GLOBAL to fix protobuf issue

42d00d0

fixed library unload

2379eed

added test target

afa87a7

fixed sanity

b2f5a19

changed destructor to default

68a3733

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

f90d8ad

…to external_ops

added /LD option for customop_gpu_lib target

04e88fb

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

f4aaa84

…to external_ops

moved /LD inside the <>

b9f67ef

diff compile flags for relu_lib.cu and lib_api.cc

4b9a4dc

set CMAKE_VERBOSE_MAKEFILE for debug

4afe182

added -v to ninja

7a36a40

added /MT

8607847

another try

4165d02

Merge branch 'master' of https://github.com/apache/incubator-mxnet in…

55b441f

…to external_ops

changed /MT to -MT

f7de08e

Ubuntu added 2 commits August 26, 2020 00:27

changed nnvm to shared library

df85c38

backed out external ops changes

bc80960

samskalicky requested review from aaronmarkham, eric-haibin-lin, leezu and szha as code owners August 26, 2020 04:46

samskalicky mentioned this pull request Aug 26, 2020

MXNet external operators #18904

Closed

Ubuntu added 2 commits August 28, 2020 00:06

split relu example into separate files to test separate lib_api.h/cc

ab0cc43

sanity

8fd4e2d

samskalicky commented Aug 28, 2020

View reviewed changes

config/linux_gpu.cmake Outdated Show resolved Hide resolved

samskalicky commented Aug 28, 2020

View reviewed changes

example/extensions/lib_custom_op/test_transposerowsp.py Outdated Show resolved Hide resolved

addressed initial review items

2431c9e

leezu approved these changes Aug 28, 2020

View reviewed changes

samskalicky commented Aug 28, 2020

View reviewed changes

marcoabreu reviewed Aug 29, 2020

View reviewed changes

mseth10 approved these changes Aug 31, 2020

View reviewed changes

rondogency reviewed Sep 1, 2020

View reviewed changes

rondogency approved these changes Sep 1, 2020

View reviewed changes

samskalicky merged commit 8379740 into apache:master Sep 1, 2020

samskalicky mentioned this pull request Sep 1, 2020

[1.x] Backporting #19016 #19069

Merged

6 tasks

samskalicky mentioned this pull request Oct 21, 2020

More extensions fixes #19393

Merged

6 tasks

samskalicky mentioned this pull request Aug 12, 2021

[v1.9.x] stop closing opened libs #20523

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MXNet Extensions enhancements2 #19016

MXNet Extensions enhancements2 #19016

samskalicky commented Aug 26, 2020 •

edited

Loading

mxnet-bot commented Aug 26, 2020

samskalicky commented Aug 26, 2020

samskalicky Aug 28, 2020

marcoabreu Aug 29, 2020

samskalicky Aug 30, 2020

marcoabreu Aug 30, 2020

samskalicky Aug 30, 2020

marcoabreu Aug 31, 2020

mseth10 left a comment

rondogency Sep 1, 2020

samskalicky Sep 1, 2020

rondogency Sep 1, 2020

rondogency Sep 1, 2020

samskalicky Sep 1, 2020

rondogency Sep 1, 2020

samskalicky Sep 1, 2020

rondogency Sep 1, 2020

samskalicky Sep 1, 2020

ZiyueHuang commented Oct 13, 2020

samskalicky commented Oct 13, 2020 •

edited

Loading

ZiyueHuang commented Oct 15, 2020

samskalicky commented Oct 15, 2020

samskalicky commented Oct 21, 2020

MXNet Extensions enhancements2 #19016

MXNet Extensions enhancements2 #19016

Conversation

samskalicky commented Aug 26, 2020 • edited Loading

Description

Other changes

Checklist

Essentials

mxnet-bot commented Aug 26, 2020

samskalicky commented Aug 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mseth10 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZiyueHuang commented Oct 13, 2020

samskalicky commented Oct 13, 2020 • edited Loading

ZiyueHuang commented Oct 15, 2020

samskalicky commented Oct 15, 2020

samskalicky commented Oct 21, 2020

samskalicky commented Aug 26, 2020 •

edited

Loading

samskalicky commented Oct 13, 2020 •

edited

Loading