tensorflow-text not compatible with Conda -- NotFound error upon import

[VinuraD]

I am using tensorflow-gpu (2.4.1) with Python 3.7 and CUDA 10.1 on Ubuntu 20.04 with miniconda3. I install tensorflow-text with
pip install tensorflow-text==2.4.1 and it completes without error. When I try to import the module it throws the below error,
However, if I do not specify tensorflow-text version it installs tensorflow 2.5 (CPU) version and works on it. (removing the 2.4.1 GPU version)

!pip install tensorflow-text==2.4.1
!pip install tf-models-official==2.4

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  
import numpy as np
from official.nlp import optimization  

---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-5-9ea998addbe5> in <module>
      1 import tensorflow as tf
      2 import tensorflow_hub as hub
----> 3 import tensorflow_text as text  # Needed for loading universal-sentence-encoder-cmlm/multilingual-preprocess
      4 import numpy as np
      5 from official.nlp import optimization  # to create AdamW optimizer

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/__init__.py in <module>
     19 # pylint: disable=wildcard-import
     20 from tensorflow_text.python import keras
---> 21 from tensorflow_text.python import metrics
     22 from tensorflow_text.python.ops import *
     23 

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/python/metrics/__init__.py in <module>
     18 
     19 # pylint: disable=wildcard-import
---> 20 from tensorflow_text.python.metrics.text_similarity_metric_ops import *
     21 
     22 # Public symbols in the "tensorflow_text.metrics" package.

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/python/metrics/text_similarity_metric_ops.py in <module>
     26 from tensorflow.python.framework import load_library
     27 from tensorflow.python.platform import resource_loader
---> 28 gen_text_similarity_metric_ops = load_library.load_op_library(resource_loader.get_path_to_datafile('_text_similarity_metric_ops.so'))
     29 
     30 

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     55     RuntimeError: when unable to load the library or get the python wrappers.
     56   """
---> 57   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     58   try:
     59     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: /userdirs/test/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/python/metrics/_text_similarity_metric_ops.so: undefined symbol: _ZNK10tensorflow8OpKernel11TraceStringERKNS_15OpKernelContextEb

[broken]

The "undefined symbol" error means there is a disconnect in the ABI versions. TF Text is making a call to the TF library using an identifier that does not exist - almost always using incompatible library versions. I noticed that your miniconda environment is named "tfenvnightly" and you do not have a !pip install for tensorflow-gpu; are you certain you are not using a nightly version of TF?

I tried the versions you specified in a colab instance and didn't receive any errors.
https://colab.research.google.com/drive/1PS7czXVlH94qE69fRCblYlw3qom8FRRr?usp=sharing

[VinuraD]

@broken Sorry for the confusion. "nightly" is just the name for the virtual environment created as I first was going to install tf nighlty version which I did not install. I did not use pip to install tf-gpu. Used conda install -c anaconda tensorflow-gpu==2.4.1. The sequence I executed was,

conda install -c anaconda tensorflow-gpu==2.4.1 #gpu installed successfully
!pip install tensorflow-text==2.4.1
!pip install tf-models-official==2.4

[broken]

Thanks for the commands, I just tested this and got the same error. Doing a pip install tensorflow-gpu==2.4.1 fixed it.

It looks like anaconda rebuilt the library in a way that changed the symbol table to the TF library file. Personally, I always found it odd they have their own package install. No other Python environment does this, and it doesn't seem convenient to have to learn another package install command... The only reason had to be they were altered for the environment. My guess is some base library was altered for the environment setup to work correctly, which had the effect of having to change libs with it as a dependency and so forth, and once you start having your own personalized libraries, you always need them and the requirements branch out and you run into problems like this one for packages that have not been absorbed in your ecosystem. So while pip installing tf fixes this problem, I cannot guarantee that later tf will not have a problem with some library it depends on.

Anyway, you can see the differences here. I installed with your same procedure. Then executed the following commands:

(i644) $ md5sum libtensorflow_framework.so.2
e883a9f1822e5c59a2fd5422ca6edbab libtensorflow_framework.so.2
(i644) $ python -m pip -q install tensorflow-gpu==2.4.1
WARNING: You are using pip version 19.3.1; however, version 21.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(i644) $ md5sum libtensorflow_framework.so.2
0b1833111b2daf19fbc3c59df9cc6f53 libtensorflow_framework.so.2

[VinuraD]

@broken thanks for the answer. I saw somewhere before, that pip and conda libraries working together could make conflicts, I think it was your answer to a previous issue. Anyway, I will also try this and post here.

[VinuraD]

@broken This resolved the previous importing error. But it again falls back to CPU as tested by tf.test.is_gpu_available(), yet works on colab. I think it happens due to that cudatoolkits from conda are not compatible with that of pip installation of tensorflow-gpu. Does that mean tensorflow-text is entirely incompatible in a conda environment, for GPU support?

[broken]

Interesting; something like this is exactly what I was afraid of and why I mentioned that you may run into problems with TF and libs it depends on.

So yes, until Conda decides to add tf-text as a supported library, it is incompatible in a conda environment if you want GPU support.

[broken]

Thanks for the response and letting us know about this incompatibility. I updated the title of this issue to make it easier for others to find, and I'll have this noted in our documentation.

[hellodanylo]

I think the easiest long-term solution is to start building tensorflow-text in the conda-forge channel.

Installing Tensorflow from pip is not an easy solution for anyone who is already using conda, because of other transitive dependencies. For example, CUDA drivers from conda are not compatible with Tensorflow from pip. The same is probably true about other ABI-sensitive dependencies.

This issue should probably be raised in https://github.com/conda-forge/tensorflow-feedstock