Add hadoop envs #565

EmmaQiaoCh · 2022-08-29T01:49:57Z

Update Merlin base version;

Add back hadoop envs;

Remove uninstallation of cmake;

Update Merlin base version Add back hadoop env

nvidia-merlin-bot · 2022-08-29T01:52:08Z

Click to view CI Results

GitHub pull request #565 of commit 92a23a400b3c4a75ac1893df48d207ec26f4edf4, no merge conflicts.
Running as SYSTEM
Setting status of 92a23a400b3c4a75ac1893df48d207ec26f4edf4 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/378/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/565/*:refs/remotes/origin/pr/565/* # timeout=10
 > git rev-parse 92a23a400b3c4a75ac1893df48d207ec26f4edf4^{commit} # timeout=10
Checking out Revision 92a23a400b3c4a75ac1893df48d207ec26f4edf4 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 92a23a400b3c4a75ac1893df48d207ec26f4edf4 # timeout=10
Commit message: "Remove uninstall cmake"
 > git rev-list --no-walk 5dce4218571df09c299b7328c999ade04d961df8 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins776726999630339532.sh
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py s      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
../../../.local/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

============ 2 passed, 1 skipped, 35 warnings in 115.30s (0:01:55) =============

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins7543201569672393842.sh

github-actions · 2022-08-29T01:52:08Z

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-565

bashimao · 2022-08-29T06:45:38Z

docker/dockerfile.ctr

@@ -46,8 +46,6 @@ RUN apt update -y --fix-missing && \
            libaio-dev libtbb-dev \
            clang-format \
       software-properties-common  && \
-    apt remove --purge cmake -y && \


I am fine with bringing back the standard cmake, but mind that we had a custom version, because some builds required a newer cmake version. Is that resolved now?

Is this still an issue?

Currently we have no cmake at all. So I suppose that is the reason why Emma removed it.

However, I wonder. We had a specific cmake version before because some project required a newer cmake than the standard ubuntu repos offer.

@EmmaQiaoCh Is that solved?

Hi sorry, I missed this...
I'm not sure which project required a newer cmake, but our build pipelines were passed when used the cmake-3.24.1 which installed in merlin-base. So I think it's fine.

bashimao · 2022-08-29T06:45:50Z

docker/dockerfile.ctr

+${HADOOP_HOME}/share/hadoop/mapreduce/*:\
+${HADOOP_HOME}/share/hadoop/yarn/*:\
+${HADOOP_HOME}/share/hadoop/yarn/lib/*
+


nvidia-merlin-bot · 2022-08-29T18:12:33Z

Click to view CI Results

GitHub pull request #565 of commit ac79ab71a87f8d127d2fc47bdac949a57c8bf3e8, no merge conflicts.
Running as SYSTEM
Setting status of ac79ab71a87f8d127d2fc47bdac949a57c8bf3e8 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/380/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/565/*:refs/remotes/origin/pr/565/* # timeout=10
 > git rev-parse ac79ab71a87f8d127d2fc47bdac949a57c8bf3e8^{commit} # timeout=10
Checking out Revision ac79ab71a87f8d127d2fc47bdac949a57c8bf3e8 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ac79ab71a87f8d127d2fc47bdac949a57c8bf3e8 # timeout=10
Commit message: "Merge branch 'main' into add_back_hadoop_env"
 > git rev-list --no-walk 84980a3d12b346626a4842994a4a1458895fbb1d # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins5860431214576233993.sh
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py s      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
../../../.local/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

============ 2 passed, 1 skipped, 35 warnings in 108.21s (0:01:48) =============

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins11074364364450922584.sh

Remove uninstall cmake

92a23a4

Update Merlin base version Add back hadoop env

EmmaQiaoCh changed the title ~~Remove uninstall cmake~~ Add hadoop envs Aug 29, 2022

bashimao requested review from bashimao, jperez999 and benfred August 29, 2022 06:46

bashimao added ci dependencies labels Aug 29, 2022

bashimao approved these changes Aug 29, 2022

View reviewed changes

Merge branch 'main' into add_back_hadoop_env

ac79ab7

jperez999 merged commit ba3a51f into NVIDIA-Merlin:main Aug 30, 2022

viswa-nvidia added this to the Merlin 22.09 milestone Sep 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hadoop envs #565

Add hadoop envs #565

EmmaQiaoCh commented Aug 29, 2022

nvidia-merlin-bot commented Aug 29, 2022

github-actions bot commented Aug 29, 2022

bashimao Aug 29, 2022

jperez999 Aug 29, 2022

bashimao Aug 30, 2022

EmmaQiaoCh Aug 30, 2022

bashimao Aug 29, 2022

nvidia-merlin-bot commented Aug 29, 2022

Add hadoop envs #565

Add hadoop envs #565

Conversation

EmmaQiaoCh commented Aug 29, 2022

nvidia-merlin-bot commented Aug 29, 2022

github-actions bot commented Aug 29, 2022

Documentation preview

bashimao Aug 29, 2022

Choose a reason for hiding this comment

jperez999 Aug 29, 2022

Choose a reason for hiding this comment

bashimao Aug 30, 2022

Choose a reason for hiding this comment

EmmaQiaoCh Aug 30, 2022

Choose a reason for hiding this comment

bashimao Aug 29, 2022

Choose a reason for hiding this comment

nvidia-merlin-bot commented Aug 29, 2022