Package recently broken on ubuntu 16.04 ?

[taynaud]

Hello,

On an ubuntu 16.04, when I install the latest tensorflow serving package, it does not start with:

~$ tensorflow_model_server --help
tensorflow_model_server: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by tensorflow_model_server)
tensorflow_model_server: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by tensorflow_model_server)

Idem when I really try to serve models.

My system is up to date.

It was working perfectly before an apt update + upgrade this week end (~03/17, not sure of exact date).

Current package state:

~(1)$ apt show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.6.0
Built-Using: Bazel
Priority: optional
Section: contrib/devel
Maintainer: TensorFlow Serving team
Installed-Size: inconnu
Homepage: https://github.com/tensorflow/serving
Download-Size: 88,8 MB
APT-Manual-Installed: yes
APT-Sources: http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages
Description: TensorFlow Serving ModelServer

~$ apt-cache show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.6.0
Architecture: all
Maintainer: TensorFlow Serving team
Priority: optional
Section: contrib/devel
Filename: pool/tensorflow-model-server/t/tensorflow-model-server/tensorflow-model-server_1.6.0_all.deb
Size: 88812004
SHA256: 04f7e0937ffd6554e04d95eb850a7740ad81ad3c39d594b4d213dc6b14d775d7
SHA1: 4bc7445661aeacc03fa373474c3be4ba86eee8dd
MD5sum: ceabe0a8fa3a3809015c5cd203714488
Description: TensorFlow Serving ModelServer
Description-md5: 9b7e03f5296f318009581d6e285e2f89
Homepage: https://github.com/tensorflow/serving
Built-Using: Bazel

~$ uname -a
Linux Lenovo 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Do you have any clue ? Is it an issue with the package or do I need another dependency ?

Best,

[sbadia]

Hi!

Same issue here with Ubuntu LTS 16.04, the 1.5.0 package is not listed anymore in the Release file, but you can download it via a wget.

root@0a96112b4fac:~# file 1.5.0/usr/bin/tensorflow_model_server 
1.5.0/usr/bin/tensorflow_model_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=d644ec743c5c224deea8fbad3378d9e0733769d4, not stripped
root@0a96112b4fac:~# file 1.6.0/usr/bin/tensorflow_model_server 
1.6.0/usr/bin/tensorflow_model_server: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=22568349c0aa5e934bb606ac117af06f4bc41be3, not stripped

• http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.5.0/t/tensorflow-model-server/tensorflow-model-server_1.5.0_all.deb
• http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.6.0/t/tensorflow-model-server/tensorflow-model-server_1.6.0_all.deb

[klockeph]

From how I understood several AskUbuntu and Stackoverflow threads, you have to enable the newest gcc version, by adding a certain repository to your apt-get package sources.

apt-get install -y software-properties-common # if not already installed
add-apt-repository ppa:ubuntu-toolchain-r/test -y

And then do the following commands (until it works, I had to do a dist-upgrade, but it could work earlier if I understood it correctly)

apt-get update
apt-get upgrade
apt-get dist-upgrade

Note that I usually don't use Ubuntu/apt-get and only did this to fix my docker.

Credits to https://askubuntu.com/a/582910

[lc0]

Seems like we would need to update https://www.tensorflow.org/serving/setup#installing_using_apt-get

[rsepassi]

Seeing the same. Here's a Travis build that uses the model server with error logs:
https://travis-ci.org/rsepassi/tensor2tensor/jobs/357041726#L2254

[nuchi]

I had the same problem, and it worked perfectly when I used the ubuntu:18.04 docker image instead of 16.04.

[dcyoung]

As @sbadia mentioned above, pulling the packages directly seemed to work. Here is an updated dockerfile that worked for me.

FROM ubuntu:16.04
# Install general packages
RUN apt-get update && apt-get install -y \
        curl \
        libcurl3-dev \
        unzip \
        wget \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Previous Installation of tensorflow-model-server (BROKEN RECENTLY)
#RUN echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list \
#    && curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add - \
#    && apt-get update && apt-get install tensorflow-model-server

# New installation of tensorflow-model-server
RUN TEMP_DEB="$(mktemp)" \
    && wget -O "$TEMP_DEB" 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.5.0/t/tensorflow-model-server/tensorflow-model-server_1.5.0_all.deb' \
    && dpkg -i "$TEMP_DEB" \
    && rm -f "$TEMP_DEB" 

# Download the trained model for serving
RUN wget SOME_HOSTED_URL/model-export.zip -O model_export.zip \
    && unzip model_export.zip -d model-export \
    && mv model-export/ /tmp \
    && rm model_export.zip

EXPOSE 9000

# Serve the model when the container starts
ENTRYPOINT ["tensorflow_model_server"]
CMD ["--port=9000", "--model_name=model", "--model_base_path=/tmp/model-export"]

[strategywise]

@dcyoung Does this update work on GPU enabled models?

[dcyoung]

@strategywise Whenever I create docker container intended for GPU support, I have them build tensorflow serving from source. I haven't tested the package in that scenario.

[chleibig]

While the solution suggested by @KillPinguin (thx!) above works fine (on ubuntu 16.04), it suffices to just update libstdc++6, as suggested by the actual error:

add-apt-repository ppa:ubuntu-toolchain-r/test -y
apt-get update
apt-get install libstdc++6

[cchung100m]

Hi @KillPinguin,

Thank you for the suggestion. I solve the same issue with update/upgrade command.
It's odd that the update/upgrade command didn't change my GLIBC/CXXAB version, upgrade my tensorflow-model-server version from 1.7.0 to 1.8.0 instead. I think maybe use tensorflow-model-server 1.8.0 directly is another workaround to this issue.

apt-get update
apt-get upgrade
apt-get dist-upgrade

System Information:

root@59dbb35082c6:/# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_DEBUG_MESSAGE_LENGTH

root@59dbb35082c6:/notebooks/downloads/installation_file# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep CXXAB
CXXABI_1.3
CXXABI_1.3.1
CXXABI_1.3.2
CXXABI_1.3.3
CXXABI_1.3.4
CXXABI_1.3.5
CXXABI_1.3.6
CXXABI_1.3.7
CXXABI_1.3.8
CXXABI_1.3.9
CXXABI_TM_1
CXXABI_FLOAT128

root@59dbb35082c6:/# apt show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.8.0
Built-Using: Bazel
Priority: optional
Section: contrib/devel
Maintainer: TensorFlow Serving team
Installed-Size: unknown
Homepage: https://github.com/tensorflow/serving
Download-Size: 98.0 MB
APT-Manual-Installed: yes
APT-Sources: http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages
Description: TensorFlow Serving ModelServer

root@59dbb35082c6:/# apt-cache show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.8.0
Architecture: all
Maintainer: TensorFlow Serving team
Priority: optional
Section: contrib/devel
Filename: pool/tensorflow-model-server/t/tensorflow-model-server/tensorflow-model-server_1.8.0_all.deb
Size: 97987044
SHA256: 3a3288a964402a5c7f1217ff26d3f971adfbd8a2a2206087f0ea61662cca1d6b
SHA1: 6e446cbdd6fe63c661899c2b14ebf8c0ab3a8df9
MD5sum: 79730a9f53b26e231e08e89b5fe664b9
Description: TensorFlow Serving ModelServer
Description-md5: 9b7e03f5296f318009581d6e285e2f89
Homepage: https://github.com/tensorflow/serving
Built-Using: Bazel