load 'tvm_runtime_op.so' error #3

zouzhene · 2019-10-21T11:44:49Z

when run
tvm_runtime_ops = load_library.load_op_library('tvm_runtime_op.so')
get
tensorflow.python.framework.errors_impl.NotFoundError: tvm_runtime_op.so: undefined symbol: _ZN3tvm7runtime6Module12LoadFromFileERKSsS3_

The text was updated successfully, but these errors were encountered:

tobegit3hub · 2019-10-22T02:47:10Z

It seems to be the problem of you dynamic library file. How do you build the tvm_runtime_op.so file? @zouzhene

junrushao · 2019-10-22T03:20:13Z

Doesn’t it look like some ABI issues?

zouzhene · 2019-10-22T03:26:48Z

@tobegit3hub
`#!/bin/bash
set -x
set -e
TF_CFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') )
TF_LFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') )

g++ -g -std=c++11 -shared tvm_runtime_kernels.cc tvm_runtime_ops.cc -o tvm_runtime_op.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2 -I${TVM_HOME}/include -I${TVM_HOME}/3rdparty/dmlc-core/include -I${TVM_HOME}/3rdparty/dlpack/include -I/usr/local/cuda/include -ltvm_runtime -L${TVM_HOME}/build -ldl -lpthread
`

Some warning like this:
In file included from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h:23, from tvm_runtime_kernels.cc:8: /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:77:0: warning: "LOG" redefined #define LOG(severity) _TF_LOG_##severity ^ In file included from /usr/tvm/3rdparty/dmlc-core/include/dmlc/io.h:15:0, from /usr/tvm/include/tvm/runtime/module.h:29, from tvm_runtime_kernels.cc:5: /usr/tvm/3rdparty/dmlc-core/include/dmlc/./logging.h:257:0: note: this is the location of the previous definition #define LOG(severity) LOG_##severity.stream() ^ In file included from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h:23, from tvm_runtime_kernels.cc:8: /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:89:0: warning: "VLOG" redefined #define VLOG(lvl) \ ^ In file included from /usr/tvm/3rdparty/dmlc-core/include/dmlc/io.h:15:0, from /usr/tvm/include/tvm/runtime/module.h:29, from tvm_runtime_kernels.cc:5: /usr/tvm/3rdparty/dmlc-core/include/dmlc/./logging.h:255:0: note: this is the location of the previous definition #define VLOG(x) LOG_INFO.stream() ^ In file included from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h:23, from tvm_runtime_kernels.cc:8: /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:97:0: warning: "CHECK" redefined #define CHECK(condition) \ ^ In file included from /usr/tvm/3rdparty/dmlc-core/include/dmlc/io.h:15:0, from /usr/tvm/include/tvm/runtime/module.h:29, from tvm_runtime_kernels.cc:5: /usr/tvm/3rdparty/dmlc-core/include/dmlc/./logging.h:205:0: note: this is the location of the previous definition #define CHECK(x) \ ^ In file included from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h:23, from tvm_runtime_kernels.cc:8: /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/default/logging.h:251:0: warning: "CHECK_EQ" redefined #define CHECK_EQ(val1, val2) CHECK_OP(Check_EQ, ==, val1, val2) ^ In file included from /usr/tvm/3rdparty/dmlc-core/include/dmlc/io.h:15:0, from /usr/tvm/include/tvm/runtime/module.h:29, from tvm_runtime_kernels.cc:5: /usr/tvm/3rdparty/dmlc-core/include/dmlc/./logging.h:213:0: note: this is the location of the previous definition #define CHECK_EQ(x, y) CHECK_BINARY_OP(_EQ, ==, x, y) ^ In file included from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/logging.h:25:0, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/lib/core/refcount.h:22, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/tensor_coding.h:21, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/resource_handle.h:19, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/allocator.h:24, from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/op_kernel.h:23, from tvm_runtime_kernels.cc:8:

zhuochenKIDD · 2019-10-28T12:15:17Z

@zouzhene
I guess you use pre-built tensorflow? What's your compiler version?

zouzhene · 2019-10-30T03:48:55Z

@zhuochenKIDD I use pre-built tensorflow ,version is tensorflow-gpu==1.15.0

VoVAllen · 2019-10-30T14:22:37Z

I met the same problem, with tensorflow 2.0.0

tensorflow.python.framework.errors_impl.NotFoundError: tvm_runtime_op.so: undefined symbol: _ZN3tvm7runtime6Module12LoadFromFileERKSsS3_

VoVAllen · 2019-10-30T15:41:48Z

Seems the linking error, which so should I link to get Module:LoadFromFile?

junrushao · 2019-10-30T16:21:20Z

On these issues, could you try “nm -g tvm_runtime_op.so” to see if similar symbols exist? If so, this would be indication of compiler ABI incompatibility

VoVAllen · 2019-10-30T16:47:59Z

I removed the --shared flag and compilation failed, saying

undefined reference to `tvm::runtime::Module::LoadFromFile(std::string const&, std::string const&)'

VoVAllen · 2019-10-30T16:48:57Z

It's not correctly linked

VoVAllen · 2019-10-30T16:52:11Z

I found this. I used the pip install version of tf 2.0. Should I change it to 1?

VoVAllen · 2019-10-30T17:45:41Z

Rebuilding tvm with cmake -DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 .. solves the problem. However there's another error for llvm's cxx11 abi.........................................

zhuochenKIDD · 2019-10-31T04:03:05Z

@VoVAllen Perhaps you need to build llvm from source, using -D_GLIBCXX_USE_CXX11_ABI=0 : apache/tvm#521 (comment)

I guess @tobegit3hub uses self-built TensorFlow with high version gcc(>=5), so ABI incompatibility issue did not occur.

I you use pre-built TensorFlow from pip install, tf is built by gcc 4.8.5, check your env's compiler version for ABI incompatibility.

VoVAllen · 2019-10-31T06:01:38Z

It's annoying. You have to rebuilt tensorflow or llvm/tvm to solve this. Both of them took long time to do so...

junrushao · 2019-11-01T21:24:49Z

TVM can be viewed as two parts:

libtvm.so: TVM compiler that does code generation, which may require LLVM for host-side code generation. The binary generated has stable pure C ABI, and guaranteed to be thread safe.
libtvm_runtime.so: TVM runtime that can load and run operators generated by TVM compiler, which does not depend on LLVM or anything else.

Especially, TVM runtime can be run in header-only mode, which means it could be potentially build together with TensorFlow to avoid any ABI issues. In this case, I would say that it could potentially avoid having to rebuild LLVM.

tobegit3hub · 2019-11-11T02:54:54Z

Sorry for the late response.

We have updated the code for the final Python API and the build script was updated to avoid conflict of re-loading tvm_runtime. Would you try the latest code now with the instructions in README? @zhuochenKIDD @VoVAllen

lsy643 · 2020-01-03T02:55:46Z

@tobegit3hub I have a similar issue here. In order to build correctly, I have upgrade cmake to 3.16, build with c++14 rather than c++11, and add include_directories(/usr/local/cuda/include). Besides, I am also using self-built tensorflow with version 1.14

And when I am running the addone example, I got this error tvm_dso_op.so: undefined symbol: _ZTIN10tensorflow8OpKernelE.

gmagogsfm · 2020-02-26T23:46:42Z

@lsy643

I am getting same error:
tvm_dso_op.so: undefined symbol: _ZTIN10tensorflow8OpKernelE

Have you, by any chance, found a fix already? Thanks!

lsy643 · 2020-03-08T03:07:42Z

@gmagogsfm

Yes, I rewrite the CMakelists.txt. I use target_link_libraries rather than target_link_options to link the Tensorflow Library.

By the way, directly compile TVM runtime source files into your project rather than link the prebuilt library

gmagogsfm · 2020-03-08T05:21:02Z

@lsy643

Thanks for the replay Siyuan. If you use target_link_libraries, does that mean you are not getting link flags from tf.sysconfig.get_link_flags? It seems a bit fragile when TF changes it link flags in the future.

Meanwhile, I did some investigation and found the issue is that target_link_options place -ltensorflow_framework in front of compile targets in g++ command, this effectively discards all symbols in tensorflow_framework.so*.

I fixed the issue by building tvm_dso_op.so using Bazel, which handles library dependency ordering correctly.

lsy643 · 2020-03-08T07:08:48Z

@gmagogsfm
Yes, you are right about the extra works need if target_link_libraries is used. And good to know that you build your project with Bazel

tobegit3hub · 2020-04-27T08:08:06Z

Thanks all for the usage. Now the patch is already merged into TVM upstream and you can build tvm with USE_TF_TVMDSOOP=ON to use this.

The issue will be closed and only maintain in tvm codebase.

tobegit3hub closed this as completed Apr 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load 'tvm_runtime_op.so' error #3

load 'tvm_runtime_op.so' error #3

zouzhene commented Oct 21, 2019

tobegit3hub commented Oct 22, 2019

junrushao commented Oct 22, 2019

zouzhene commented Oct 22, 2019

zhuochenKIDD commented Oct 28, 2019

zouzhene commented Oct 30, 2019

VoVAllen commented Oct 30, 2019 •

edited

Loading

VoVAllen commented Oct 30, 2019 •

edited

Loading

junrushao commented Oct 30, 2019

VoVAllen commented Oct 30, 2019 •

edited

Loading

VoVAllen commented Oct 30, 2019

VoVAllen commented Oct 30, 2019 •

edited

Loading

VoVAllen commented Oct 30, 2019

zhuochenKIDD commented Oct 31, 2019

VoVAllen commented Oct 31, 2019

junrushao commented Nov 1, 2019 •

edited

Loading

tobegit3hub commented Nov 11, 2019

lsy643 commented Jan 3, 2020

gmagogsfm commented Feb 26, 2020

lsy643 commented Mar 8, 2020

gmagogsfm commented Mar 8, 2020

lsy643 commented Mar 8, 2020

tobegit3hub commented Apr 27, 2020

load 'tvm_runtime_op.so' error #3

load 'tvm_runtime_op.so' error #3

Comments

zouzhene commented Oct 21, 2019

tobegit3hub commented Oct 22, 2019

junrushao commented Oct 22, 2019

zouzhene commented Oct 22, 2019

zhuochenKIDD commented Oct 28, 2019

zouzhene commented Oct 30, 2019

VoVAllen commented Oct 30, 2019 • edited Loading

VoVAllen commented Oct 30, 2019 • edited Loading

junrushao commented Oct 30, 2019

VoVAllen commented Oct 30, 2019 • edited Loading

VoVAllen commented Oct 30, 2019

VoVAllen commented Oct 30, 2019 • edited Loading

VoVAllen commented Oct 30, 2019

zhuochenKIDD commented Oct 31, 2019

VoVAllen commented Oct 31, 2019

junrushao commented Nov 1, 2019 • edited Loading

tobegit3hub commented Nov 11, 2019

lsy643 commented Jan 3, 2020

gmagogsfm commented Feb 26, 2020

lsy643 commented Mar 8, 2020

gmagogsfm commented Mar 8, 2020

lsy643 commented Mar 8, 2020

tobegit3hub commented Apr 27, 2020

VoVAllen commented Oct 30, 2019 •

edited

Loading

VoVAllen commented Oct 30, 2019 •

edited

Loading

VoVAllen commented Oct 30, 2019 •

edited

Loading

VoVAllen commented Oct 30, 2019 •

edited

Loading

junrushao commented Nov 1, 2019 •

edited

Loading