Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixes support library lookups for torch #761

Merged
merged 2 commits into from
Dec 12, 2022

Conversation

jperez999
Copy link
Collaborator

New torch versions (pre release containers) are failing to import torch because they require support libraries that are not available. This libraries exist in the pytorch backend folder for triton. So we will leverage those to fix the import issues.

@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-761

@jperez999 jperez999 self-assigned this Dec 12, 2022
@jperez999 jperez999 added the ci label Dec 12, 2022
@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #761 of commit 1acb54c228d5507e0e989f797f5a03c633a19c01, no merge conflicts.
Running as SYSTEM
Setting status of 1acb54c228d5507e0e989f797f5a03c633a19c01 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/691/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_merlin/workspace
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/761/*:refs/remotes/origin/pr/761/* # timeout=10
 > git rev-parse 1acb54c228d5507e0e989f797f5a03c633a19c01^{commit} # timeout=10
Checking out Revision 1acb54c228d5507e0e989f797f5a03c633a19c01 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1acb54c228d5507e0e989f797f5a03c633a19c01 # timeout=10
Commit message: "Merge branch 'main' into fix-torch-dkr"
 > git rev-list --no-walk f56cc4f876c0ba9ccaf41bbcb10a3698fc0dde23 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins6136840233148843273.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.27,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.27,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.45,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='742671956'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-tp7_tq2g
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-tp7_tq2g
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 8e93b0394dce12a58e0434c567a363c9218b84a5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+30.g8e93b03) (2.22.0)
Collecting treelite==2.4.0
  Downloading treelite-2.4.0-py3-none-manylinux2014_x86_64.whl (852 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 852.7/852.7 kB 7.2 MB/s eta 0:00:00
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+30.g8e93b03) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+30.g8e93b03) (1.1.1)
Collecting treelite-runtime==2.4.0
  Downloading treelite_runtime-2.4.0-py3-none-manylinux2014_x86_64.whl (191 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 kB 158.5 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+30.g8e93b03) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+30.g8e93b03) (1.20.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2022.3.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (0.12.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.7.0)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 106.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+30.g8e93b03) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+30.g8e93b03-py3-none-any.whl size=100816 sha256=b032ab8de1b049a3029483c07719af0bce1b653962fcaf9cedc318c42be5de71
  Stored in directory: /tmp/pip-ephem-wheel-cache-fvf_ub32/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, treelite-runtime, treelite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: treelite-runtime
    Found existing installation: treelite-runtime 2.3.0
    Not uninstalling treelite-runtime at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite-runtime'. No files were found to uninstall.
  Attempting uninstall: treelite
    Found existing installation: treelite 2.3.0
    Not uninstalling treelite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'treelite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+30.g8e93b03 treelite-2.4.0 treelite-runtime-2.4.0
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-6i8oc4r8
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-6i8oc4r8
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 4c32ebd2ebb031e028a3efc950c54e7576c70406
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+63.g4c32ebd2) (0.3.0+12.g78ecddd)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.2 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2022.3.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (0.12.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+63.g4c32ebd2) (4.0.0)
Building wheels for collected packages: merlin-models, merlin-dataloader
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+63.g4c32ebd2-py3-none-any.whl size=353363 sha256=52bd0a0493f1c6923a1b94b637c5b5121637595b70f3f49377bf725d7e153fa2
  Stored in directory: /tmp/pip-ephem-wheel-cache-d6bi25xi/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=b218206ca2b8b0c14590e9aa1d84c7804b5c714cea70c4d21bcba679de3c2ad7
  Stored in directory: /tmp/pip-ephem-wheel-cache-d6bi25xi/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built merlin-models merlin-dataloader
Installing collected packages: merlin-dataloader, merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 merlin-models-0.9.0+63.g4c32ebd2
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-8kqfrt6m
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-8kqfrt6m
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 7d79d460d3c19852372a438ab398bcd8aeaf7028
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+17.g7d79d460) (0.3.0+12.g78ecddd)
Requirement already satisfied: merlin-dataloader>=0.0.2 in ./.tox/test-gpu/lib/python3.8/site-packages (from nvtabular==1.6.0+17.g7d79d460) (0.0.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+17.g7d79d460) (1.8.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2022.3.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (0.55.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+17.g7d79d460) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (0.12.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+17.g7d79d460) (4.0.0)
Building wheels for collected packages: nvtabular
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+17.g7d79d460-cp38-cp38-linux_x86_64.whl size=257532 sha256=a467bc30fbcd51c045c04bb7e12f6d976098c797809f30484605762fdbb484ec
  Stored in directory: /tmp/pip-ephem-wheel-cache-1vazj_7t/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+17.g7d79d460
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-0ep862w0
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-0ep862w0
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c2ef3f06ce2caf864d84bc6e92799c4d1392f74a
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+15.gc2ef3f0) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+15.gc2ef3f0) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+15.gc2ef3f0) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+15.gc2ef3f0) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+15.gc2ef3f0) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+15.gc2ef3f0) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+15.gc2ef3f0) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+15.gc2ef3f0) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+15.gc2ef3f0) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+15.gc2ef3f0) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+15.gc2ef3f0) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+15.gc2ef3f0) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+15.gc2ef3f0) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (0.12.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+15.gc2ef3f0) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+15.gc2ef3f0) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+15.gc2ef3f0) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+15.gc2ef3f0) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+15.gc2ef3f0) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+15.gc2ef3f0) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+15.gc2ef3f0) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+15.gc2ef3f0) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+15.gc2ef3f0) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+15.gc2ef3f0) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+15.gc2ef3f0) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+15.gc2ef3f0) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+15.gc2ef3f0) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+15.gc2ef3f0) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+15.gc2ef3f0-py3-none-any.whl size=119064 sha256=157d428672cd721ec8b2635891ce924df2145b44ac60cc27ba04862023ce71aa
  Stored in directory: /tmp/pip-ephem-wheel-cache-qswikvtt/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+15.gc2ef3f0
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 6 items / 1 skipped

tests/unit/test_version.py . [ 16%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 33%]
tests/unit/examples/test_scaling_criteo_merlin_models.py F [ 50%]
tests/unit/examples/test_scaling_criteo_optimize_notebook.py . [ 66%]
tests/unit/examples/test_z_legacy_notebooks.py .. [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
      tb1.execute()

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:32:


../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute
super().execute_cell(cell, index)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell
await self._check_raise_for_error(cell, cell_index, exec_reply)


self = <testbook.client.TestbookNotebookClient object at 0x7fe472a7a1f0>
cell = {'cell_type': 'code', 'execution_count': 4, 'id': '08cdbfcc', 'metadata': {'execution': {'iopub.status.busy': '2022-12...\n# for running this example on CPU, comment out the line below\nos.environ["TF_GPU_ALLOCATOR"] = "cuda_malloc_async"'}
cell_index = 11
exec_reply = {'buffers': [], 'content': {'ename': 'ModuleNotFoundError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'd7dacfe0-...e, 'engine': 'd7dacfe0-0ebd-4b48-8e25-ae61c0ba3f4e', 'started': '2022-12-12T16:07:13.118163Z', 'status': 'error'}, ...}

async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:
      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)

E nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E ------------------
E import os
E import nvtabular as nvt
E from nvtabular.ops import Rename, Filter, Dropna, LambdaOp, Categorify,
E TagAsUserFeatures, TagAsUserID, TagAsItemFeatures, TagAsItemID, AddMetadata
E
E from merlin.schema.tags import Tags
E
E import merlin.models.tf as mm
E from merlin.io.dataset import Dataset
E from merlin.datasets.ecommerce import transform_aliccp
E import tensorflow as tf
E
E # for running this example on CPU, comment out the line below
E os.environ["TF_GPU_ALLOCATOR"] = "cuda_malloc_async"
E ------------------
E
E �[0;31m---------------------------------------------------------------------------�[0m
E �[0;31mModuleNotFoundError�[0m Traceback (most recent call last)
E Cell �[0;32mIn [4], line 8�[0m
E �[1;32m 3�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mnvtabular�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mops�[39;00m �[38;5;28;01mimport�[39;00m Rename, Filter, Dropna, LambdaOp, Categorify,
E �[1;32m 4�[0m TagAsUserFeatures, TagAsUserID, TagAsItemFeatures, TagAsItemID, AddMetadata
E �[1;32m 6�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mschema�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtags�[39;00m �[38;5;28;01mimport�[39;00m Tags
E �[0;32m----> 8�[0m �[38;5;28;01mimport�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mmodels�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtf�[39;00m �[38;5;28;01mas�[39;00m �[38;5;21;01mmm�[39;00m
E �[1;32m 9�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mio�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mdataset�[39;00m �[38;5;28;01mimport�[39;00m Dataset
E �[1;32m 10�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mdatasets�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mecommerce�[39;00m �[38;5;28;01mimport�[39;00m transform_aliccp
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py:20�[0m
E �[1;32m 1�[0m �[38;5;66;03m#�[39;00m
E �[1;32m 2�[0m �[38;5;66;03m# Copyright (c) 2021, NVIDIA CORPORATION.�[39;00m
E �[1;32m 3�[0m �[38;5;66;03m#�[39;00m
E �[0;32m (...)�[0m
E �[1;32m 18�[0m
E �[1;32m 19�[0m �[38;5;66;03m# Must happen before any importing of tensorflow to curtail mem usage�[39;00m
E �[0;32m---> 20�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mdataloader�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtf_utils�[39;00m �[38;5;28;01mimport�[39;00m configure_tensorflow
E �[1;32m 21�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mmodels�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtf�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mcore�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mindex�[39;00m �[38;5;28;01mimport�[39;00m IndexBlock, TopKIndexBlock
E �[1;32m 22�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mmodels�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtf�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mcore�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtabular�[39;00m �[38;5;28;01mimport�[39;00m AsTabular, Filter, TabularBlock
E
E �[0;31mModuleNotFoundError�[0m: No module named 'merlin.dataloader'
E ModuleNotFoundError: No module named 'merlin.dataloader'

../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError
__________________________________ test_func ___________________________________

def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )
      tb1.execute()

tests/unit/examples/test_scaling_criteo_merlin_models.py:33:


../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute
super().execute_cell(cell, index)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell
await self._check_raise_for_error(cell, cell_index, exec_reply)


self = <testbook.client.TestbookNotebookClient object at 0x7fe5e2330310>
cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}
cell_index = 12
exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'b81a581b-e15a-4ff...e, 'engine': 'b81a581b-e15a-4ffe-bc68-f0757a20a741', 'started': '2022-12-12T16:07:23.716724Z', 'status': 'error'}, ...}

async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:
      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)

E nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E ------------------
E # Dask dashboard
E dashboard_port = "8787"
E
E # Deploy a Single-Machine Multi-GPU Cluster
E protocol = "tcp" # "tcp" or "ucx"
E if numba.cuda.is_available():
E NUM_GPUS = list(range(len(numba.cuda.gpus)))
E else:
E NUM_GPUS = []
E visible_devices = ",".join([str(n) for n in NUM_GPUS]) # Select devices to place workers
E device_limit_frac = 0.7 # Spill GPU-Worker memory to host at this limit.
E device_pool_frac = 0.8
E part_mem_frac = 0.15
E
E # Use total device size to calculate args.device_limit_frac
E device_size = device_mem_size(kind="total")
E device_limit = int(device_limit_frac * device_size)
E device_pool_size = int(device_pool_frac * device_size)
E part_size = int(part_mem_frac * device_size)
E
E # Check if any device memory is already occupied
E for dev in visible_devices.split(","):
E fmem = pynvml_mem_size(kind="free", index=int(dev))
E used = (device_size - fmem) / 1e9
E if used > 1.0:
E warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")
E
E cluster = None # (Optional) Specify existing scheduler port
E if cluster is None:
E cluster = LocalCUDACluster(
E protocol=protocol,
E n_workers=len(visible_devices.split(",")),
E CUDA_VISIBLE_DEVICES=visible_devices,
E device_memory_limit=device_limit,
E local_directory=dask_workdir,
E dashboard_address=":" + dashboard_port,
E rmm_pool_size=(device_pool_size // 256) * 256
E )
E
E # Create the distributed client
E client = Client(cluster)
E client
E ------------------
E
E �[0;31m---------------------------------------------------------------------------�[0m
E �[0;31mMemoryError�[0m Traceback (most recent call last)
E Cell �[0;32mIn [7], line 30�[0m
E �[1;32m 28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m
E �[1;32m 29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:
E �[0;32m---> 30�[0m cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m
E �[1;32m 31�[0m �[43m �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m
E �[1;32m 32�[0m �[43m �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m
E �[1;32m 33�[0m �[43m �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m
E �[1;32m 34�[0m �[43m �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m
E �[1;32m 35�[0m �[43m �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m
E �[1;32m 36�[0m �[43m �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m
E �[1;32m 37�[0m �[43m �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m
E �[1;32m 38�[0m �[43m �[49m�[43m)�[49m
E �[1;32m 40�[0m �[38;5;66;03m# Create the distributed client�[39;00m
E �[1;32m 41�[0m client �[38;5;241m=�[39m Client(cluster)
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m
E �[1;32m 365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES
E �[1;32m 366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)
E �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m
E �[1;32m 307�[0m �[38;5;28;01mreturn�[39;00m future
E �[1;32m 308�[0m �[38;5;28;01melse�[39;00m:
E �[0;32m--> 309�[0m �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m
E �[1;32m 310�[0m �[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m
�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m
�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m
E �[1;32m 311�[0m �[43m �[49m�[43m)�[49m
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m
E �[1;32m 374�[0m �[38;5;28;01mif�[39;00m error:
E �[1;32m 375�[0m typ, exc, tb �[38;5;241m=�[39m error
E �[0;32m--> 376�[0m �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)
E �[1;32m 377�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 378�[0m �[38;5;28;01mreturn�[39;00m result
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m
E �[1;32m 347�[0m future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)
E �[1;32m 348�[0m future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)
E �[0;32m--> 349�[0m result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future
E �[1;32m 350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
E �[1;32m 351�[0m error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
E
E File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m
E �[1;32m 759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
E �[1;32m 761�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m--> 762�[0m value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m
E �[1;32m 763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
E �[1;32m 764�[0m exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m
E �[1;32m 350�[0m �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:
E �[1;32m 351�[0m w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)
E �[0;32m--> 352�[0m �[38;5;28;01mawait�[39;00m w �[38;5;66;03m# for tornado gen.coroutine support�[39;00m
E �[1;32m 353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m
E �[1;32m 293�[0m �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(
E �[1;32m 294�[0m �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(
E �[1;32m 295�[0m �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout
E �[1;32m 296�[0m )
E �[1;32m 297�[0m )
E �[1;32m 298�[0m �[38;5;28;01melse�[39;00m:
E �[0;32m--> 299�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()
E �[1;32m 300�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning
E �[1;32m 301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m
E �[1;32m 344�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)
E �[1;32m 346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)
E �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()
E �[1;32m 348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:
E �[1;32m 349�[0m �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m
E �[1;32m 428�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 429�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m--> 430�[0m result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()
E �[1;32m 431�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
E �[1;32m 432�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m
E �[1;32m 683�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus
E �[1;32m 684�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m--> 685�[0m msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)
E �[1;32m 686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
E �[1;32m 687�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m
E �[1;32m 799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:
E �[1;32m 800�[0m logger�[38;5;241m.�[39merror(
E �[1;32m 801�[0m �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]
E �[1;32m 802�[0m )
E �[0;32m--> 803�[0m �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]
E �[1;32m 804�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 805�[0m �[38;5;28;01mreturn�[39;00m msg
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m
E �[1;32m 865�[0m �[38;5;124;03m"""�[39;00m
E �[1;32m 866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m
E �[1;32m 867�[0m �[38;5;124;03m"""�[39;00m
E �[1;32m 868�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m--> 869�[0m �[38;5;28;01mawait�[39;00m worker
E �[1;32m 870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:
E �[1;32m 871�[0m logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m
E �[1;32m 293�[0m �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(
E �[1;32m 294�[0m �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(
E �[1;32m 295�[0m �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout
E �[1;32m 296�[0m )
E �[1;32m 297�[0m )
E �[1;32m 298�[0m �[38;5;28;01melse�[39;00m:
E �[0;32m--> 299�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()
E �[1;32m 300�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning
E �[1;32m 301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m
E �[1;32m 1370�[0m �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:
E �[1;32m 1371�[0m logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))
E �[0;32m-> 1372�[0m �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]
E �[1;32m 1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()
E �[1;32m 1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m
E �[1;32m 3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):
E �[1;32m 3247�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m-> 3248�[0m result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)
E �[1;32m 3249�[0m �[38;5;28;01mif�[39;00m isawaitable(result):
E �[1;32m 3250�[0m result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m
E �[1;32m 74�[0m �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m
E �[1;32m 76�[0m pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m
E �[0;32m---> 78�[0m rmm�[38;5;241m.�[39mreinitialize(
E �[1;32m 79�[0m pool_allocator�[38;5;241m=�[39mpool_allocator,
E �[1;32m 80�[0m managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,
E �[1;32m 81�[0m initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,
E �[1;32m 82�[0m maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,
E �[1;32m 83�[0m logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,
E �[1;32m 84�[0m log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(
E �[1;32m 85�[0m worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory
E �[1;32m 86�[0m ),
E �[1;32m 87�[0m )
E �[1;32m 88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:
E �[1;32m 89�[0m �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m
E �[1;32m 32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(
E �[1;32m 33�[0m pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,
E �[1;32m 34�[0m managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,
E �[0;32m (...)�[0m
E �[1;32m 39�[0m log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,
E �[1;32m 40�[0m ):
E �[1;32m 41�[0m �[38;5;124;03m"""�[39;00m
E �[1;32m 42�[0m �[38;5;124;03m Finalizes and then initializes RMM using the options passed. Using memory�[39;00m
E �[1;32m 43�[0m �[38;5;124;03m from a previous initialization of RMM is undefined behavior and should be�[39;00m
E �[0;32m (...)�[0m
E �[1;32m 83�[0m �[38;5;124;03m corresponding to each device.�[39;00m
E �[1;32m 84�[0m �[38;5;124;03m """�[39;00m
E �[0;32m---> 85�[0m rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(
E �[1;32m 86�[0m pool_allocator�[38;5;241m=�[39mpool_allocator,
E �[1;32m 87�[0m managed_memory�[38;5;241m=�[39mmanaged_memory,
E �[1;32m 88�[0m initial_pool_size�[38;5;241m=�[39minitial_pool_size,
E �[1;32m 89�[0m maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,
E �[1;32m 90�[0m devices�[38;5;241m=�[39mdevices,
E �[1;32m 91�[0m logging�[38;5;241m=�[39mlogging,
E �[1;32m 92�[0m log_file_name�[38;5;241m=�[39mlog_file_name,
E �[1;32m 93�[0m )
E
E File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m
E
E File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m
E
E File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m
E
E �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
E MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError
----------------------------- Captured stderr call -----------------------------
2022-12-12 16:07:26,587 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:26,603 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:26,643 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:26,653 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:27,115 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:27,115 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:29,750 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-bsq5h2yb', purging
2022-12-12 16:07:29,751 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:30,014 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:30,015 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:32,837 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-9io1jiy0', purging
2022-12-12 16:07:32,838 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:32,854 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:32,856 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:32,930 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:33,383 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,384 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,399 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,400 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,442 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,443 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,503 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:33,504 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:36,172 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-33c6wp8i', purging
2022-12-12 16:07:36,172 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-ftfhcbe6', purging
2022-12-12 16:07:36,173 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-s31evpjm', purging
2022-12-12 16:07:36,173 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-78amrp9a', purging
2022-12-12 16:07:36,173 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-12-12 16:07:36,451 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors
yield
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
2022-12-12 16:07:36,452 - distributed.nanny - ERROR - Failed to start worker
Traceback (most recent call last):
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run
await worker
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _
await self.start()
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start
raise plugins_exceptions[0]
File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add
result = plugin.setup(worker=self)
File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup
rmm.reinitialize(
File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize
rmm.mr._initialize(
File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize
File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit
MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 57 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
=============================== warnings summary ===============================
../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33
/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
DASK_VERSION = LooseVersion(dask.version)

.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings
/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)

tests/unit/examples/test_z_legacy_notebooks.py::test_movielens_example
tests/unit/examples/test_z_legacy_notebooks.py::test_movielens_example
/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/tools/data_gen.py:126: DeprecationWarning: np.long is a deprecated alias for np.compat.long. To silence this warning, use np.compat.long by itself. In the likely event your code does not need to work on Python 2 you can use the builtin int for which np.compat.long is itself an alias. Doing this will not modify any behaviour and is safe. When replacing np.long, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
col_size + 1, dtype=np.long, min_val=col.multi_min, max_val=col.multi_max

tests/unit/examples/test_z_legacy_notebooks.py: 12 warnings
/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/tools/data_gen.py:140: DeprecationWarning: np.long is a deprecated alias for np.compat.long. To silence this warning, use np.compat.long by itself. In the likely event your code does not need to work on Python 2 you can use the builtin int for which np.compat.long is itself an alias. Doing this will not modify any behaviour and is safe. When replacing np.long, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
col_size, dtype=np.long, min_val=col.min_val, max_val=col.cardinality

tests/unit/examples/test_z_legacy_notebooks.py::test_movielens_example
/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/loader/init.py:19: DeprecationWarning: The nvtabular.loader module has moved to a new repository, at https://github.com/NVIDIA-Merlin/dataloader . Support for importing from nvtabular.loader is deprecated, and will be removed in a future version. Please update your imports to refer to merlinloader.
warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 360 168 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 98 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 16 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 26 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 120 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 4 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 11 81%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 96 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 37 39%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 4 81%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 347 129 63%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 624 336 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 12 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 11 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 65 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/loader/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/loader/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/loader/loader_base.py 418 81 81%
.tox/test-gpu/lib/python3.8/site-packages/merlin/loader/tensorflow.py 112 38 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/loader/torch.py 66 33 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 69 68 1%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 20 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 40 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 229 71 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 7 91%

TOTAL 5006 1827 64%

=========================== short test summary info ============================
SKIPPED [1] tests/unit/examples/test_scaling_criteo_merlin_models_hugectr.py:7: could not import 'hugectr': No module named 'hugectr'
======= 2 failed, 4 passed, 1 skipped, 50 warnings in 116.68s (0:01:56) ========
/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.9913.440715' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
data._warn(str(exc))
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[workspace] $ /bin/bash /tmp/jenkins12795574362483418985.sh

@jperez999 jperez999 merged commit 1282e7f into NVIDIA-Merlin:main Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants