[BUG] hugectr not available in merlin-inference and merlin-tensorflow-training

[mattf]

merlin-inference -

$ docker run --rm -it nvcr.io/nvidia/merlin/merlin-inference:22.03 python -c 'import hugectr'

==========
== CUDA ==
==========

NVIDIA Release  (build )
CUDA Version 11.6.0.021

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'hugectr'

merlin-tensorflow-training -

$ docker run --rm -it nvcr.io/nvidia/merlin/merlin-tensorflow-training:22.03 python -c 'import hugectr'

================
== TensorFlow ==
================

NVIDIA Release 22.02-tf2 (build 32333867)
TensorFlow Version 2.7.0

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright 2017-2022 The TensorFlow Authors.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'hugectr'

[karlhigley]

I think HugeCTR should be available in the merlin-training and merlin-inference containers, and should not be available in the merlin-tensorflow-training container. Is that right, @albert17?

[mattf]

https://github.com/NVIDIA-Merlin/Merlin/blob/main/docker/README.md (as of fa1b8c4) says -training, -tensorflow-training, and -inference

i can only confirm it is in -training

[karlhigley]

I think what that's trying to say is "HugeCTR Tensorflow Embedding plugin," which doesn't actually mean that HugeCTR itself is available there, if I understand correctly. Maybe we can clarify this though?

cc @mikemckiernan @albert17

[mattf]

if that's the intention of the documentation, it should be clarified. the line break is unfortunate too.

this would leave the issue that HugeCTR is not available in there -inference container

[karlhigley]

The second part is a duplicate of #142, which has been addressed in the nightly images (see #135.)

[mattf]

$ docker run --rm -it nvcr.io/nvidia/merlin/merlin-inference:nightly python -c 'import hugectr' | tail -n3
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'hugectr'

is this an issue where "HugeCTR" in the merlin-inference container really means "HugeCTR Tensorflow Embedding plugin"

[karlhigley]

I don't think that's expected to work on the inference container; what it has is the Triton Inference Server HugeCTR Back-end for serving. (Correct me if I'm wrong here, @albert17.)

[viswa-nvidia]

@karlhigley , can we close this issue ? If not, please assign a priority