Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dataloader unit testing to container run #728

Merged
merged 3 commits into from
Nov 15, 2022

Conversation

jperez999
Copy link
Collaborator

Adds an entry to run unit tests from dataloader repo in the ci script.

@jperez999 jperez999 self-assigned this Nov 10, 2022
@jperez999 jperez999 added chore Infrastructure update ci labels Nov 10, 2022
@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-728

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 849f075de9684b3964ad458cd95d8fc236f1dffe, no merge conflicts.
Running as SYSTEM
Setting status of 849f075de9684b3964ad458cd95d8fc236f1dffe to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/606/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 849f075de9684b3964ad458cd95d8fc236f1dffe^{commit} # timeout=10
Checking out Revision 849f075de9684b3964ad458cd95d8fc236f1dffe (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 849f075de9684b3964ad458cd95d8fc236f1dffe # timeout=10
Commit message: "add dataloader unit testing to container run"
 > git rev-list --no-walk 906b99b16854a5c2b372841544cca76f1b5297b0 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins2325272242778180785.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.6,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.6,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.43,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='576737619'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-3w8kshm3
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-3w8kshm3
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.4.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 28.1 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92222 sha256=338938ef257bbea01f7fbb5ce1b0b2c66205d1dfdc9d2ca11886e079f1a5f759
  Stored in directory: /tmp/pip-ephem-wheel-cache-0h1zjiqa/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-4gimmj94
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-4gimmj94
Resolved https://github.com/NVIDIA-Merlin/models.git to commit 237ab090cdfdd533c36f0992cc42c14afb2052a0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+38.g237ab090) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (21.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (65.4.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.0.0)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+38.g237ab090-py3-none-any.whl size=364668 sha256=b084533bd056b0101ed8839af9fc63507a73f865e6813501161572c59918de30
Stored in directory: /tmp/pip-ephem-wheel-cache-jcaaoeov/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+38.g237ab090

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-0um63k0m
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-0um63k0m
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.4.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Building wheels for collected packages: nvtabular
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257746 sha256=efd1a797271a5578e172a9817fbffe3f5395d731a336f06ac9782bcf5dc662ff
Stored in directory: /tmp/pip-ephem-wheel-cache-qhj729an/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-bny2his5
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-bny2his5
Resolved https://github.com/NVIDIA-Merlin/core.git to commit 1d9d3542725badd7c337bd2a7e47c4327ebb684c
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (3.19.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (6.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (65.4.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+10.g1d9d354) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.0.0)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+10.g1d9d354-py3-none-any.whl size=118620 sha256=bc9f61ddff5043b3269f596a471b18217954a594eba17150d58fc83af3bd477b
Stored in directory: /tmp/pip-ephem-wheel-cache-jvffc7ph/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+10.g1d9d354

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py . [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py F [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )
        tb1.execute()
        assert os.path.isfile("/tmp/output/criteo/train/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/valid/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/workflow/metadata.json")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "03-Training-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["INPUT_DATA_DIR"] = "/tmp/output/criteo/"
            """
        )
        tb2.execute()
        metrics = tb2.ref("eval_metrics")
      assert set(metrics.keys()) == set(
            [
                "auc",
                "binary_accuracy",
                "loss",
                "precision",
                "recall",
                "regularization_loss",
            ]
        )

E AssertionError: assert {'auc', 'bina...'recall', ...} == {'auc', 'bina...ization_loss'}
E Extra items in the left set:
E 'loss_batch'
E Use -v to get more diff

tests/unit/examples/test_scaling_criteo_merlin_models.py:54: AssertionError
----------------------------- Captured stderr call -----------------------------
2022-11-10 18:13:36,955 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-10 18:13:36,970 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-10 18:13:36,972 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-10 18:13:37,052 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
2022-11-10 18:13:56.599914: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:14:00.737937: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0
2022-11-10 18:14:00.738045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:14:00.738802: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1
2022-11-10 18:14:00.738854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:14:00.739516: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 2
2022-11-10 18:14:00.739565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:14:00.740182: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 3
2022-11-10 18:14:00.740234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 17 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 41 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 158 54%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 346 161 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 29 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 81 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 154 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 72 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 162 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 347 54%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 110 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 24 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 10 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 53 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 57 13 77%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 21 54%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 126 26 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 20 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 5 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 17 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 119 89 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 234 13%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%

TOTAL 14559 6842 53%

=================== 1 failed, 2 passed in 284.14s (0:04:44) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.15330.002296' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
data._warn(str(exc))
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins4964466855268989702.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 5ceab62055b7170c70ffe61989d72fa35b798a2e, no merge conflicts.
Running as SYSTEM
Setting status of 5ceab62055b7170c70ffe61989d72fa35b798a2e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/607/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 5ceab62055b7170c70ffe61989d72fa35b798a2e^{commit} # timeout=10
Checking out Revision 5ceab62055b7170c70ffe61989d72fa35b798a2e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk 849f075de9684b3964ad458cd95d8fc236f1dffe # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins8493249506107839320.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.6,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.6,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.43,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3385773356'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-3bx7gxyb
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-3bx7gxyb
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 84.2 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92222 sha256=309c7679530ef67b78df69371d6e1cd6d730e39b2e8a35365e82baeccb3978b6
  Stored in directory: /tmp/pip-ephem-wheel-cache-idtmjyd0/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-_cyeks75
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-_cyeks75
Resolved https://github.com/NVIDIA-Merlin/models.git to commit 237ab090cdfdd533c36f0992cc42c14afb2052a0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+38.g237ab090) (0.3.0+12.g78ecddd)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (21.3)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (8.1.3)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.1)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+38.g237ab090-py3-none-any.whl size=364668 sha256=6d407f9d3f955b2e033c046ff260a8e7ddad332eda30067f65eec38ca7b5d713
Stored in directory: /tmp/pip-ephem-wheel-cache-b8lp6d6m/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+38.g237ab090

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-1fuz_8s_
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-1fuz_8s_
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Building wheels for collected packages: nvtabular
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257746 sha256=973e5fc63127394f4f686f8be0488bc51aa0a5c1601aa6cdba4c4bc06138dc4a
Stored in directory: /tmp/pip-ephem-wheel-cache-0n8ai1lp/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-gbtfidjm
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-gbtfidjm
Resolved https://github.com/NVIDIA-Merlin/core.git to commit 1d9d3542725badd7c337bd2a7e47c4327ebb684c
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (21.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (8.1.3)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+10.g1d9d354) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.1)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+10.g1d9d354-py3-none-any.whl size=118620 sha256=cbab24720ea5a2849c822d2c8089f46067a01d981b7dbf5bf61ffb9971a3567a
Stored in directory: /tmp/pip-ephem-wheel-cache-psgx6249/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+10.g1d9d354

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py . [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

self = <testbook.client.TestbookNotebookClient object at 0x7eff2ec9fa60>
msg_id = '01defb8c-6a323a220f783fb689fa91bd_22276_33'
cell = {'id': '5927737c', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T18:20:56.983653Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
timeout = 120
task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>
task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>

async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:
          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:


obj = <coroutine object ZMQSocketChannel.get_msg at 0x7f000c9086c0>

async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:
          result = await obj

../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:


self = <jupyter_client.channels.ZMQSocketChannel object at 0x7f000c9616d0>
timeout = 120000.0

async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:
      raise Empty

E _queue.Empty

/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty

During handling of the above exception, another exception occurred:

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"
      tb2.inject(
            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:


../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject
cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)
../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell
cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell
exec_reply = await self.task_poll_for_reply
../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply
await self._async_handle_timeout(timeout, cell)


self = <testbook.client.TestbookNotebookClient object at 0x7eff2ec9fa60>
timeout = 120
cell = {'id': '5927737c', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T18:20:56.983653Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:
      raise CellTimeoutError.error_from_timeout_and_cell(
            "Cell execution timed out", timeout, cell
        )

E nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.
E The message was: Cell execution timed out.
E Here is a preview of the cell contents:
E -------------------
E ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']
E ...
E ['response = run_ensemble_on_tritonserver(', ' "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']
E -------------------

../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError
----------------------------- Captured stdout call -----------------------------
Signal (2) received.
Signal (2) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
----------------------------- Captured stderr call -----------------------------
2022-11-10 18:18:57.018609: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:19:01.124941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:19:01.125684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:19:01.126259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:19:01.126852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
WARNING clustering 446 points to 32 centroids: please provide at least 1248 training points
2022-11-10 18:20:50.373743: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:20:54.471563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:20:54.472302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:20:54.472941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:20:54.473551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
I1110 18:20:57.302001 23631 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7efd96000000' with size 268435456
I1110 18:20:57.302778 23631 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1110 18:20:57.311105 23631 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1
I1110 18:20:57.311191 23631 model_lifecycle.cc:459] loading: 0_queryfeast:1
I1110 18:20:57.311241 23631 model_lifecycle.cc:459] loading: 2_queryfaiss:1
I1110 18:20:57.311278 23631 model_lifecycle.cc:459] loading: 3_queryfeast:1
I1110 18:20:57.311315 23631 model_lifecycle.cc:459] loading: 4_unrollfeatures:1
I1110 18:20:57.311360 23631 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1
I1110 18:20:57.311419 23631 model_lifecycle.cc:459] loading: 6_softmaxsampling:1
I1110 18:20:57.598617 23631 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow
I1110 18:20:57.598659 23631 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10
I1110 18:20:57.598666 23631 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10
I1110 18:20:57.598671 23631 tensorflow.cc:2576] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}
I1110 18:20:57.598713 23631 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)
2022-11-10 18:20:57.599350: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:20:57.603858: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:20:57.603895: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:20:57.604001: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:20:58.016541: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4892 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:20:58.051190: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-11-10 18:20:58.053276: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:20:58.105225: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:20:58.129431: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 530091 microseconds.
I1110 18:20:58.140086 23631 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)
2022-11-10 18:20:58.140642: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:20:58.144081: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:20:58.144109: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:20:58.146057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4892 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:20:58.153451: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:20:58.207843: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:20:58.233815: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 93177 microseconds.
I1110 18:20:58.234119 23631 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:21:29.577577 23631 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)
2022-11-10 18:21:29.578668: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:21:29.597126: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:21:29.597167: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:21:29.599239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4892 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:21:29.628239: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:21:29.761743: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:21:29.814080: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 235421 microseconds.
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:21:37.669742 23631 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)
I1110 18:21:44.104868 23631 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)
I1110 18:21:44.105634 23631 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1
I1110 18:21:50.425133 23631 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)
I1110 18:21:50.428313 23631 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1
I1110 18:21:56.797430 23631 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)
I1110 18:21:56.797727 23631 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1
I1110 18:22:03.115730 23631 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)
I1110 18:22:03.116276 23631 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1
2022-11-10 18:22:03.117747: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:22:03.136141: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:22:03.136194: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:22:03.138286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4892 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:22:03.163203: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:22:03.316250: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:22:03.371430: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 253696 microseconds.
I1110 18:22:03.371654 23631 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)
I1110 18:22:03.371876 23631 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1
I1110 18:22:09.316399 23631 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1
I1110 18:22:09.319339 23631 model_lifecycle.cc:459] loading: ensemble_model:1
I1110 18:22:09.319641 23631 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1
I1110 18:22:09.319755 23631 server.cc:561]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1110 18:22:09.319856 23631 server.cc:588]
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 18:22:09.319971 23631 server.cc:631]
+---------------------------+---------+--------+
| Model | Version | Status |
+---------------------------+---------+--------+
| 0_queryfeast | 1 | READY |
| 1_predicttensorflowtriton | 1 | READY |
| 2_queryfaiss | 1 | READY |
| 3_queryfeast | 1 | READY |
| 4_unrollfeatures | 1 | READY |
| 5_predicttensorflowtriton | 1 | READY |
| 6_softmaxsampling | 1 | READY |
| ensemble_model | 1 | READY |
+---------------------------+---------+--------+

I1110 18:22:09.383012 23631 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB
I1110 18:22:09.383861 23631 tritonserver.cc:2214]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.25.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0] | /tmp/examples/poc_ensemble |
| model_control_mode | MODE_NONE |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 18:22:09.385227 23631 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001
I1110 18:22:09.385429 23631 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000
I1110 18:22:09.426326 23631 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
W1110 18:22:10.408696 23631 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 18:22:11.408891 23631 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 18:22:12.428906 23631 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
I1110 18:22:45.823960 23631 server.cc:262] Waiting for in-flight requests to complete.
I1110 18:22:45.823983 23631 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences
I1110 18:22:45.824203 23631 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1
I1110 18:22:45.824293 23631 server.cc:293] All models are stopped, unloading models
I1110 18:22:45.824292 23631 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 18:22:45.824308 23631 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests
I1110 18:22:45.824543 23631 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 18:22:45.824538 23631 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 18:22:45.824877 23631 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 18:22:45.840490 23631 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1
I1110 18:22:45.848513 23631 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1
I1110 18:22:46.824432 23631 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:22:47.800119 23631 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1
I1110 18:22:47.824599 23631 server.cc:300] Timeout 28: Found 4 live models and 0 in-flight non-inference requests
I1110 18:22:47.885164 23631 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1
I1110 18:22:47.951668 23631 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1
I1110 18:22:48.824723 23631 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:49.824831 23631 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:50.824944 23631 server.cc:300] Timeout 25: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:51.825059 23631 server.cc:300] Timeout 24: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:52.825197 23631 server.cc:300] Timeout 23: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:53.825332 23631 server.cc:300] Timeout 22: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:54.825470 23631 server.cc:300] Timeout 21: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:55.825604 23631 server.cc:300] Timeout 20: Found 2 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:22:56.825746 23631 server.cc:300] Timeout 19: Found 2 live models and 0 in-flight non-inference requests
I1110 18:22:56.985403 23631 model_lifecycle.cc:578] successfully unloaded '3_queryfeast' version 1
I1110 18:22:57.651234 23631 model_lifecycle.cc:578] successfully unloaded '0_queryfeast' version 1
I1110 18:22:57.826338 23631 server.cc:300] Timeout 18: Found 0 live models and 0 in-flight non-inference requests
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
------------------------------ Captured log call -------------------------------
ERROR traitlets:client.py:795 Timeout waiting for execute reply (120s).

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 15 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 24 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 136 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 93 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 346 157 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 20 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 78 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 151 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 71 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 154 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 337 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 108 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 22 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 8 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 52 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 57 13 77%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 20 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 126 26 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 18 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py 65 19 71%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 4 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 12 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 119 67 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 210 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py 22 20 9%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py 113 14 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py 37 29 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py 10 6 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py 32 13 59%

TOTAL 14838 6798 54%

=================== 1 failed, 2 passed in 384.10s (0:06:24) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins17959128222437562271.sh

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 5ceab62055b7170c70ffe61989d72fa35b798a2e, no merge conflicts.
Running as SYSTEM
Setting status of 5ceab62055b7170c70ffe61989d72fa35b798a2e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/608/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 5ceab62055b7170c70ffe61989d72fa35b798a2e^{commit} # timeout=10
Checking out Revision 5ceab62055b7170c70ffe61989d72fa35b798a2e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins12789615546673320212.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.6,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.6,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.43,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3659735015'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-nl3kaxm2
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-nl3kaxm2
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 63.3 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92222 sha256=a83aef353d5f1b4944b668f00a9e8ee56f80152295c6dbecd5fbb11168eea585
  Stored in directory: /tmp/pip-ephem-wheel-cache-u9zjnrs9/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-antx060m
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-antx060m
Resolved https://github.com/NVIDIA-Merlin/models.git to commit 237ab090cdfdd533c36f0992cc42c14afb2052a0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+38.g237ab090) (0.3.0+12.g78ecddd)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.4.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.4)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.1)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+38.g237ab090-py3-none-any.whl size=364668 sha256=c153066d898c33da2c86cccfbe1749b8e9085af5211e9dba6601b3abaed5a441
Stored in directory: /tmp/pip-ephem-wheel-cache-7xgb27cs/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+38.g237ab090

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-nohkd8nc
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-nohkd8nc
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Building wheels for collected packages: nvtabular
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257746 sha256=a2006de937f69e75002224416821c8a82581bd620471502a278e13a828e5ebc7
Stored in directory: /tmp/pip-ephem-wheel-cache-qdfjhc05/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-4jkbxqjx
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-4jkbxqjx
Resolved https://github.com/NVIDIA-Merlin/core.git to commit 1d9d3542725badd7c337bd2a7e47c4327ebb684c
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.4.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.4)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+10.g1d9d354) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.1)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+10.g1d9d354-py3-none-any.whl size=118620 sha256=62db3b10529fceca0ae3a07c0afba4b89eb844c974413c09e7cfd0fb660508ae
Stored in directory: /tmp/pip-ephem-wheel-cache-gnjmek2l/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+10.g1d9d354

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py . [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

self = <testbook.client.TestbookNotebookClient object at 0x7fed93e4ba60>
msg_id = '281ffe16-3e80af9a9f54ace3562f301a_27855_33'
cell = {'id': '55b32ae0', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T18:35:19.922806Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
timeout = 120
task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>
task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>

async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:
          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:


obj = <coroutine object ZMQSocketChannel.get_msg at 0x7fee71b07240>

async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:
          result = await obj

../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:


self = <jupyter_client.channels.ZMQSocketChannel object at 0x7fee71b63a60>
timeout = 120000.0

async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:
      raise Empty

E _queue.Empty

/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty

During handling of the above exception, another exception occurred:

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"
      tb2.inject(
            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:


../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject
cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)
../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell
cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell
exec_reply = await self.task_poll_for_reply
../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply
await self._async_handle_timeout(timeout, cell)


self = <testbook.client.TestbookNotebookClient object at 0x7fed93e4ba60>
timeout = 120
cell = {'id': '55b32ae0', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T18:35:19.922806Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:
      raise CellTimeoutError.error_from_timeout_and_cell(
            "Cell execution timed out", timeout, cell
        )

E nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.
E The message was: Cell execution timed out.
E Here is a preview of the cell contents:
E -------------------
E ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']
E ...
E ['response = run_ensemble_on_tritonserver(', ' "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']
E -------------------

../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError
----------------------------- Captured stdout call -----------------------------
Signal (2) received.
Signal (2) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
----------------------------- Captured stderr call -----------------------------
2022-11-10 18:33:25.853661: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:33:29.918161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:33:29.918915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:33:29.919534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:33:29.920160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
WARNING clustering 461 points to 32 centroids: please provide at least 1248 training points
2022-11-10 18:35:13.355287: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:35:17.422048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:35:17.422818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:35:17.423452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:35:17.424071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
I1110 18:35:20.261763 28427 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fcdd4000000' with size 268435456
I1110 18:35:20.262488 28427 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1110 18:35:20.270630 28427 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1
I1110 18:35:20.270690 28427 model_lifecycle.cc:459] loading: 2_queryfaiss:1
I1110 18:35:20.270739 28427 model_lifecycle.cc:459] loading: 0_queryfeast:1
I1110 18:35:20.270787 28427 model_lifecycle.cc:459] loading: 3_queryfeast:1
I1110 18:35:20.270859 28427 model_lifecycle.cc:459] loading: 4_unrollfeatures:1
I1110 18:35:20.270923 28427 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1
I1110 18:35:20.270985 28427 model_lifecycle.cc:459] loading: 6_softmaxsampling:1
I1110 18:35:20.544011 28427 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow
I1110 18:35:20.544060 28427 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10
I1110 18:35:20.544068 28427 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10
I1110 18:35:20.544074 28427 tensorflow.cc:2576] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}
I1110 18:35:20.544115 28427 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)
2022-11-10 18:35:20.544742: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:35:20.549255: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:35:20.549287: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:35:20.549380: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:35:20.961319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:35:20.998056: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-11-10 18:35:21.000144: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:35:21.053001: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:35:21.077560: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 532829 microseconds.
I1110 18:35:21.087586 28427 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)
2022-11-10 18:35:21.088106: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:35:21.091472: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:35:21.091498: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:35:21.093419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:35:21.100803: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:35:21.153740: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:35:21.178376: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 90275 microseconds.
I1110 18:35:21.178678 28427 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:36:00.301825 28427 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)
2022-11-10 18:36:00.302415: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:36:00.323304: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:36:00.323378: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:36:00.325833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:36:00.354125: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:36:00.487779: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:36:00.538764: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 236354 microseconds.
I1110 18:36:00.555516 28427 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)
I1110 18:36:06.946414 28427 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)
I1110 18:36:06.947012 28427 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1
I1110 18:36:13.294909 28427 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)
I1110 18:36:13.295227 28427 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1
I1110 18:36:19.661680 28427 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)
I1110 18:36:19.664138 28427 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1
I1110 18:36:25.621175 28427 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)
I1110 18:36:25.621494 28427 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1
I1110 18:36:31.546274 28427 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)
I1110 18:36:31.546597 28427 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1
2022-11-10 18:36:31.546850: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:36:31.556317: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:36:31.556355: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:36:31.558296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:36:31.581331: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:36:31.727739: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:36:31.781467: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 234622 microseconds.
I1110 18:36:31.781787 28427 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1
I1110 18:36:31.784707 28427 model_lifecycle.cc:459] loading: ensemble_model:1
I1110 18:36:31.785246 28427 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1
I1110 18:36:31.785389 28427 server.cc:561]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1110 18:36:31.785489 28427 server.cc:588]
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 18:36:31.785604 28427 server.cc:631]
+---------------------------+---------+--------+
| Model | Version | Status |
+---------------------------+---------+--------+
| 0_queryfeast | 1 | READY |
| 1_predicttensorflowtriton | 1 | READY |
| 2_queryfaiss | 1 | READY |
| 3_queryfeast | 1 | READY |
| 4_unrollfeatures | 1 | READY |
| 5_predicttensorflowtriton | 1 | READY |
| 6_softmaxsampling | 1 | READY |
| ensemble_model | 1 | READY |
+---------------------------+---------+--------+

I1110 18:36:31.847148 28427 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB
I1110 18:36:31.848029 28427 tritonserver.cc:2214]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.25.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0] | /tmp/examples/poc_ensemble |
| model_control_mode | MODE_NONE |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 18:36:31.849441 28427 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001
I1110 18:36:31.849961 28427 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000
I1110 18:36:31.891168 28427 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
W1110 18:36:32.870245 28427 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 18:36:33.870456 28427 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 18:36:34.898508 28427 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
I1110 18:37:16.793735 28427 server.cc:262] Waiting for in-flight requests to complete.
I1110 18:37:16.793782 28427 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences
I1110 18:37:16.794021 28427 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1
I1110 18:37:16.794124 28427 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 18:37:16.794359 28427 server.cc:293] All models are stopped, unloading models
I1110 18:37:16.794411 28427 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 18:37:16.794423 28427 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests
I1110 18:37:16.794501 28427 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 18:37:16.794703 28427 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 18:37:16.809272 28427 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1
I1110 18:37:16.816626 28427 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1
I1110 18:37:17.794599 28427 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:37:18.759993 28427 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1
I1110 18:37:18.794874 28427 server.cc:300] Timeout 28: Found 4 live models and 0 in-flight non-inference requests
I1110 18:37:18.845328 28427 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1
I1110 18:37:18.907581 28427 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1
I1110 18:37:19.795015 28427 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests
I1110 18:37:20.795544 28427 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests
I1110 18:37:21.795782 28427 server.cc:300] Timeout 25: Found 2 live models and 0 in-flight non-inference requests
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
I1110 18:37:22.796129 28427 server.cc:300] Timeout 24: Found 2 live models and 0 in-flight non-inference requests
------------------------------ Captured log call -------------------------------
ERROR traitlets:client.py:795 Timeout waiting for execute reply (120s).

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 15 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 24 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 136 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 346 161 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 29 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 81 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 151 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 71 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 154 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 337 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 108 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 22 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 8 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 52 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 57 13 77%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 20 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 126 26 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 18 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py 65 19 71%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 4 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 12 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 119 67 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 210 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py 22 20 9%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py 113 14 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py 37 29 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py 10 6 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py 32 13 59%

TOTAL 14838 6818 54%

=================== 1 failed, 2 passed in 372.45s (0:06:12) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins1983813372720849947.sh

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 5ceab62055b7170c70ffe61989d72fa35b798a2e, no merge conflicts.
Running as SYSTEM
Setting status of 5ceab62055b7170c70ffe61989d72fa35b798a2e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/609/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 5ceab62055b7170c70ffe61989d72fa35b798a2e^{commit} # timeout=10
Checking out Revision 5ceab62055b7170c70ffe61989d72fa35b798a2e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins16283888847078766267.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.6,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.6,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.43,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='334195191'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-pghl5qwa
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-pghl5qwa
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 21.6 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92222 sha256=1f807966327a82ceb7a01c895a94b94764b2e30311474fcf97363a4543973814
  Stored in directory: /tmp/pip-ephem-wheel-cache-zgusnmua/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-v5sq6isy
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-v5sq6isy
Resolved https://github.com/NVIDIA-Merlin/models.git to commit 237ab090cdfdd533c36f0992cc42c14afb2052a0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+38.g237ab090) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.1)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+38.g237ab090-py3-none-any.whl size=364668 sha256=fa48edf1887fc69b07dbbcf52664edee4234eb4a939d31a5852452b18390e87c
Stored in directory: /tmp/pip-ephem-wheel-cache-_o0le13b/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+38.g237ab090

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-8v93v1yx
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-8v93v1yx
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Building wheels for collected packages: nvtabular
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257746 sha256=2e247374c63534184319a9bea16d390c6bdd5e9e9604cbfddc6de7bf9abcacb1
Stored in directory: /tmp/pip-ephem-wheel-cache-s0ut8k/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-1ggpr8x6
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-1ggpr8x6
Resolved https://github.com/NVIDIA-Merlin/core.git to commit 1d9d3542725badd7c337bd2a7e47c4327ebb684c
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (7.0.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+10.g1d9d354) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.1)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+10.g1d9d354-py3-none-any.whl size=118620 sha256=ee1fa9ea7d602b649755e57bbb39fcc459d024d3b3b0f1504fa31f08bcb424ff
Stored in directory: /tmp/pip-ephem-wheel-cache-z383jlf8/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+10.g1d9d354

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py . [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

self = <testbook.client.TestbookNotebookClient object at 0x7f0788336b50>
msg_id = 'a5c0d4e1-5dbdfc843bc7873a1679b09f_30572_32'
cell = {'id': '2febb351', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T18:47:26.913668Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
timeout = 120
task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>
task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>

async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:
          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:


obj = <coroutine object ZMQSocketChannel.get_msg at 0x7f0865fa32c0>

async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:
          result = await obj

../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:


self = <jupyter_client.channels.ZMQSocketChannel object at 0x7f0866009250>
timeout = 120000.0

async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:
      raise Empty

E _queue.Empty

/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty

During handling of the above exception, another exception occurred:

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"
      tb2.inject(
            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:


../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject
cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)
../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell
cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell
exec_reply = await self.task_poll_for_reply
../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply
await self._async_handle_timeout(timeout, cell)


self = <testbook.client.TestbookNotebookClient object at 0x7f0788336b50>
timeout = 120
cell = {'id': '2febb351', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T18:47:26.913668Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:
      raise CellTimeoutError.error_from_timeout_and_cell(
            "Cell execution timed out", timeout, cell
        )

E nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.
E The message was: Cell execution timed out.
E Here is a preview of the cell contents:
E -------------------
E ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']
E ...
E ['response = run_ensemble_on_tritonserver(', ' "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']
E -------------------

../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError
----------------------------- Captured stdout call -----------------------------
Signal (2) received.
Signal (2) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
----------------------------- Captured stderr call -----------------------------
2022-11-10 18:45:30.729024: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:45:34.828056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:45:34.828800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:45:34.829424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:45:34.830013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
WARNING clustering 454 points to 32 centroids: please provide at least 1248 training points
2022-11-10 18:47:20.354658: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:47:24.388642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:47:24.389371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:47:24.390032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:47:24.390662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
I1110 18:47:27.239518 31135 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe5c6000000' with size 268435456
I1110 18:47:27.240284 31135 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1110 18:47:27.248349 31135 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1
I1110 18:47:27.248430 31135 model_lifecycle.cc:459] loading: 0_queryfeast:1
I1110 18:47:27.248467 31135 model_lifecycle.cc:459] loading: 2_queryfaiss:1
I1110 18:47:27.248498 31135 model_lifecycle.cc:459] loading: 3_queryfeast:1
I1110 18:47:27.248533 31135 model_lifecycle.cc:459] loading: 4_unrollfeatures:1
I1110 18:47:27.248587 31135 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1
I1110 18:47:27.248620 31135 model_lifecycle.cc:459] loading: 6_softmaxsampling:1
I1110 18:47:27.530673 31135 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow
I1110 18:47:27.530714 31135 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10
I1110 18:47:27.530721 31135 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10
I1110 18:47:27.530727 31135 tensorflow.cc:2576] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}
I1110 18:47:27.530769 31135 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)
2022-11-10 18:47:27.531402: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:27.536004: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:47:27.536043: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:27.536139: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:47:27.950071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:47:27.986946: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-11-10 18:47:27.989079: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:47:28.040779: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:28.065995: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 534604 microseconds.
I1110 18:47:28.076434 31135 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)
2022-11-10 18:47:28.076933: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:28.080534: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:47:28.080561: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:28.082562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:47:28.089895: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:47:28.146748: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:28.175695: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 98765 microseconds.
I1110 18:47:28.176042 31135 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:47:59.462407 31135 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)
2022-11-10 18:47:59.463760: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:59.490663: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:47:59.490720: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:59.492849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:47:59.522689: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:47:59.663934: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:47:59.721672: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 257924 microseconds.
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:48:07.565863 31135 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)
I1110 18:48:13.898806 31135 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)
I1110 18:48:13.899112 31135 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1
I1110 18:48:20.328121 31135 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)
I1110 18:48:20.328300 31135 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1
I1110 18:48:26.695407 31135 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)
I1110 18:48:26.696262 31135 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1
I1110 18:48:32.652929 31135 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)
2022-11-10 18:48:32.654826: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
I1110 18:48:32.656270 31135 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1
2022-11-10 18:48:32.673046: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 18:48:32.673088: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:48:32.675196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:48:32.699145: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 18:48:32.848979: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 18:48:32.898203: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 243389 microseconds.
I1110 18:48:32.898401 31135 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)
I1110 18:48:32.898626 31135 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1
I1110 18:48:38.800957 31135 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1
I1110 18:48:38.803881 31135 model_lifecycle.cc:459] loading: ensemble_model:1
I1110 18:48:38.804473 31135 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1
I1110 18:48:38.804618 31135 server.cc:561]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1110 18:48:38.804722 31135 server.cc:588]
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 18:48:38.804845 31135 server.cc:631]
+---------------------------+---------+--------+
| Model | Version | Status |
+---------------------------+---------+--------+
| 0_queryfeast | 1 | READY |
| 1_predicttensorflowtriton | 1 | READY |
| 2_queryfaiss | 1 | READY |
| 3_queryfeast | 1 | READY |
| 4_unrollfeatures | 1 | READY |
| 5_predicttensorflowtriton | 1 | READY |
| 6_softmaxsampling | 1 | READY |
| ensemble_model | 1 | READY |
+---------------------------+---------+--------+

I1110 18:48:38.867738 31135 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB
I1110 18:48:38.868600 31135 tritonserver.cc:2214]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.25.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0] | /tmp/examples/poc_ensemble |
| model_control_mode | MODE_NONE |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 18:48:38.870022 31135 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001
I1110 18:48:38.870386 31135 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000
I1110 18:48:38.911385 31135 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
W1110 18:48:39.885992 31135 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 18:48:40.886209 31135 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 18:48:41.904300 31135 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
I1110 18:49:22.784497 31135 server.cc:262] Waiting for in-flight requests to complete.
I1110 18:49:22.784532 31135 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences
I1110 18:49:22.784794 31135 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1
I1110 18:49:22.784817 31135 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 18:49:22.784889 31135 server.cc:293] All models are stopped, unloading models
I1110 18:49:22.784925 31135 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests
I1110 18:49:22.785045 31135 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 18:49:22.785167 31135 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 18:49:22.785198 31135 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 18:49:22.795005 31135 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1
I1110 18:49:22.811304 31135 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1
I1110 18:49:23.786076 31135 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 18:49:24.744295 31135 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1
I1110 18:49:24.786311 31135 server.cc:300] Timeout 28: Found 4 live models and 0 in-flight non-inference requests
I1110 18:49:24.921806 31135 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1
I1110 18:49:24.949922 31135 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1
I1110 18:49:25.786447 31135 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests
I1110 18:49:26.786588 31135 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests
I1110 18:49:27.787141 31135 server.cc:300] Timeout 25: Found 2 live models and 0 in-flight non-inference requests
I1110 18:49:28.787347 31135 server.cc:300] Timeout 24: Found 2 live models and 0 in-flight non-inference requests
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
I1110 18:49:29.787670 31135 server.cc:300] Timeout 23: Found 2 live models and 0 in-flight non-inference requests
------------------------------ Captured log call -------------------------------
ERROR traitlets:client.py:795 Timeout waiting for execute reply (120s).

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 15 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 24 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 136 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 346 157 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 20 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 78 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 151 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 71 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 154 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 337 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 108 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 22 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 8 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 52 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 57 13 77%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 20 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 126 26 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 18 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py 65 19 71%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 4 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 12 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 119 67 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 210 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py 22 20 9%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py 113 14 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py 37 29 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py 10 6 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py 32 13 59%

TOTAL 14838 6802 54%

=================== 1 failed, 2 passed in 375.70s (0:06:15) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.31894.949661' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
data._warn(str(exc))
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins4443799173301996471.sh

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 5ceab62055b7170c70ffe61989d72fa35b798a2e, no merge conflicts.
Running as SYSTEM
Setting status of 5ceab62055b7170c70ffe61989d72fa35b798a2e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/610/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 5ceab62055b7170c70ffe61989d72fa35b798a2e^{commit} # timeout=10
Checking out Revision 5ceab62055b7170c70ffe61989d72fa35b798a2e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins6703181717019639915.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.6,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.6,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.43,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1260225754'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-gnmih83l
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-gnmih83l
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 2.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92222 sha256=4911db9a00bb0df9fc5841ccc3a1cd627e76e8c751f825b6cf5e54cbe9390ace
  Stored in directory: /tmp/pip-ephem-wheel-cache-0zmadzxd/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-12vpc1hp
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-12vpc1hp
Resolved https://github.com/NVIDIA-Merlin/models.git to commit 237ab090cdfdd533c36f0992cc42c14afb2052a0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+38.g237ab090) (0.3.0+12.g78ecddd)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.4.1)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.4.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+38.g237ab090) (6.0.1)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+38.g237ab090-py3-none-any.whl size=364668 sha256=83bb8e88594ceda0609788813083688a2d77241706f0ce842fd4355e14cce3ca
Stored in directory: /tmp/pip-ephem-wheel-cache-8st2vmbq/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+38.g237ab090

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-3mjpq_5a
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-3mjpq_5a
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Building wheels for collected packages: nvtabular
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257746 sha256=05a718d729957874f681a9446212dba489d488b91eb7d15999202143e668c1f9
Stored in directory: /tmp/pip-ephem-wheel-cache-_rxi5cus/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-kzadn1q9
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-kzadn1q9
Resolved https://github.com/NVIDIA-Merlin/core.git to commit 1d9d3542725badd7c337bd2a7e47c4327ebb684c
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+10.g1d9d354) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+10.g1d9d354) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.4.1)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.4.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+10.g1d9d354) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+10.g1d9d354) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+10.g1d9d354) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+10.g1d9d354) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+10.g1d9d354) (6.0.1)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+10.g1d9d354-py3-none-any.whl size=118620 sha256=89fb9cfd06ac32b92058aaf253d97b7fc3f3a3edbeae70631562f31fd31f9b40
Stored in directory: /tmp/pip-ephem-wheel-cache-qlfb3q_b/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+10.g1d9d354

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py . [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

self = <testbook.client.TestbookNotebookClient object at 0x7fc55936cca0>
msg_id = '5da679a2-8d7621a93e6b968243d22b78_4751_32'
cell = {'id': 'df5c0ce6', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T19:00:04.561626Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
timeout = 120
task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>
task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>

async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:
          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:


obj = <coroutine object ZMQSocketChannel.get_msg at 0x7fc559318740>

async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:
          result = await obj

../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:


self = <jupyter_client.channels.ZMQSocketChannel object at 0x7fc55937d490>
timeout = 120000.0

async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:
      raise Empty

E _queue.Empty

/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty

During handling of the above exception, another exception occurred:

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"
      tb2.inject(
            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:


../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject
cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)
../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell
cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell
exec_reply = await self.task_poll_for_reply
../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply
await self._async_handle_timeout(timeout, cell)


self = <testbook.client.TestbookNotebookClient object at 0x7fc55936cca0>
timeout = 120
cell = {'id': 'df5c0ce6', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-10T19:00:04.561626Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:
      raise CellTimeoutError.error_from_timeout_and_cell(
            "Cell execution timed out", timeout, cell
        )

E nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.
E The message was: Cell execution timed out.
E Here is a preview of the cell contents:
E -------------------
E ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']
E ...
E ['response = run_ensemble_on_tritonserver(', ' "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']
E -------------------

../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError
----------------------------- Captured stdout call -----------------------------
Signal (2) received.
Signal (2) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
----------------------------- Captured stderr call -----------------------------
2022-11-10 18:58:01.676819: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 18:58:07.385127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 18:58:07.385865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 18:58:07.386424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 18:58:07.387018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
WARNING clustering 444 points to 32 centroids: please provide at least 1248 training points
2022-11-10 18:59:56.356716: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 19:00:01.932690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 19:00:01.936666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-10 19:00:01.938610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-10 19:00:01.940392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
I1110 19:00:04.915702 9304 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7ff8e4000000' with size 268435456
I1110 19:00:04.919950 9304 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1110 19:00:05.020226 9304 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1
I1110 19:00:05.020355 9304 model_lifecycle.cc:459] loading: 0_queryfeast:1
I1110 19:00:05.020424 9304 model_lifecycle.cc:459] loading: 2_queryfaiss:1
I1110 19:00:05.020514 9304 model_lifecycle.cc:459] loading: 3_queryfeast:1
I1110 19:00:05.020637 9304 model_lifecycle.cc:459] loading: 4_unrollfeatures:1
I1110 19:00:05.020715 9304 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1
I1110 19:00:05.020784 9304 model_lifecycle.cc:459] loading: 6_softmaxsampling:1
I1110 19:00:05.309260 9304 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow
I1110 19:00:05.309303 9304 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10
I1110 19:00:05.309311 9304 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10
I1110 19:00:05.309317 9304 tensorflow.cc:2576] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}
I1110 19:00:05.309360 9304 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)
2022-11-10 19:00:05.310026: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:05.314671: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 19:00:05.314704: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:05.314798: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-10 19:00:05.878268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 19:00:05.917224: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-11-10 19:00:05.919296: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 19:00:05.996370: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:06.022036: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 712022 microseconds.
I1110 19:00:06.100746 9304 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)
2022-11-10 19:00:06.101417: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:06.106668: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 19:00:06.106705: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:06.109195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 19:00:06.116644: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 19:00:06.171049: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:06.195102: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 93693 microseconds.
I1110 19:00:06.196250 9304 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 19:00:29.673301 9304 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)
2022-11-10 19:00:29.674628: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:29.692489: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 19:00:29.692535: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:29.694673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 19:00:29.724948: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 19:00:29.866565: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:00:29.963939: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 289322 microseconds.
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 19:00:45.956973 9304 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)
I1110 19:00:52.468841 9304 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)
I1110 19:00:52.470024 9304 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1
I1110 19:00:58.896219 9304 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)
I1110 19:00:58.896481 9304 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1
I1110 19:01:05.262743 9304 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)
2022-11-10 19:01:05.263488: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
I1110 19:01:05.264272 9304 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1
2022-11-10 19:01:05.282515: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-10 19:01:05.282581: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:01:05.285465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-10 19:01:05.327285: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-10 19:01:05.547306: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-10 19:01:05.608667: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 345192 microseconds.
I1110 19:01:05.608827 9304 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)
I1110 19:01:05.621558 9304 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1
I1110 19:01:11.615365 9304 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)
I1110 19:01:11.621127 9304 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1
I1110 19:01:17.683506 9304 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1
I1110 19:01:17.686461 9304 model_lifecycle.cc:459] loading: ensemble_model:1
I1110 19:01:17.686772 9304 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1
I1110 19:01:17.686892 9304 server.cc:561]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1110 19:01:17.686997 9304 server.cc:588]
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 19:01:17.687116 9304 server.cc:631]
+---------------------------+---------+--------+
| Model | Version | Status |
+---------------------------+---------+--------+
| 0_queryfeast | 1 | READY |
| 1_predicttensorflowtriton | 1 | READY |
| 2_queryfaiss | 1 | READY |
| 3_queryfeast | 1 | READY |
| 4_unrollfeatures | 1 | READY |
| 5_predicttensorflowtriton | 1 | READY |
| 6_softmaxsampling | 1 | READY |
| ensemble_model | 1 | READY |
+---------------------------+---------+--------+

I1110 19:01:17.750240 9304 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB
I1110 19:01:17.751113 9304 tritonserver.cc:2214]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.25.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0] | /tmp/examples/poc_ensemble |
| model_control_mode | MODE_NONE |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1110 19:01:17.752509 9304 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001
I1110 19:01:17.753020 9304 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000
I1110 19:01:17.794252 9304 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
W1110 19:01:18.798605 9304 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 19:01:19.798813 9304 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1110 19:01:20.818944 9304 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
I1110 19:01:55.611916 9304 server.cc:262] Waiting for in-flight requests to complete.
I1110 19:01:55.611956 9304 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences
I1110 19:01:55.612210 9304 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1
I1110 19:01:55.612371 9304 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 19:01:55.612420 9304 server.cc:293] All models are stopped, unloading models
I1110 19:01:55.612447 9304 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests
I1110 19:01:55.612513 9304 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 19:01:55.612589 9304 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1110 19:01:55.612902 9304 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1110 19:01:55.627562 9304 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1
I1110 19:01:55.634204 9304 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1
I1110 19:01:56.612594 9304 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1110 19:01:57.612744 9304 server.cc:300] Timeout 28: Found 5 live models and 0 in-flight non-inference requests
I1110 19:01:57.630507 9304 model_lifecycle.cc:578] successfully unloaded '3_queryfeast' version 1
I1110 19:01:57.704420 9304 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1
I1110 19:01:57.738944 9304 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1
I1110 19:01:57.763415 9304 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1
I1110 19:01:58.612884 9304 server.cc:300] Timeout 27: Found 1 live models and 0 in-flight non-inference requests
I1110 19:01:59.613025 9304 server.cc:300] Timeout 26: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:00.613161 9304 server.cc:300] Timeout 25: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:01.613299 9304 server.cc:300] Timeout 24: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:02.613434 9304 server.cc:300] Timeout 23: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:03.613569 9304 server.cc:300] Timeout 22: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:04.613962 9304 server.cc:300] Timeout 21: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:05.614347 9304 server.cc:300] Timeout 20: Found 1 live models and 0 in-flight non-inference requests
I1110 19:02:06.614489 9304 server.cc:300] Timeout 19: Found 1 live models and 0 in-flight non-inference requests
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
------------------------------ Captured log call -------------------------------
ERROR traitlets:client.py:795 Timeout waiting for execute reply (120s).
--------------------------- Captured stderr teardown ---------------------------
I1110 19:02:07.614829 9304 server.cc:300] Timeout 18: Found 1 live models and 0 in-flight non-inference requests

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 15 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 24 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 136 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 346 157 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 20 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 78 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 151 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 71 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 154 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 337 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 108 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 22 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 8 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 52 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 57 13 77%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 20 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 126 26 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 18 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py 65 19 71%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 4 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 12 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 119 67 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 210 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py 22 20 9%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py 113 14 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py 37 29 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py 10 6 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py 32 13 59%

TOTAL 14838 6802 54%

=================== 1 failed, 2 passed in 384.04s (0:06:24) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.13887.061952' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object
data._warn(str(exc))
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins5571904076221786080.sh

@oliverholworthy
Copy link
Member

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 5ceab62055b7170c70ffe61989d72fa35b798a2e, no merge conflicts.
Running as SYSTEM
Setting status of 5ceab62055b7170c70ffe61989d72fa35b798a2e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/611/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 5ceab62055b7170c70ffe61989d72fa35b798a2e^{commit} # timeout=10
Checking out Revision 5ceab62055b7170c70ffe61989d72fa35b798a2e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk 5ceab62055b7170c70ffe61989d72fa35b798a2e # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins11186724317157810341.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.7,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.7,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.43,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3470585611'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-y7pt3vkl
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-y7pt3vkl
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.5.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 20.6 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92222 sha256=d4561c51bfd75b45d3b7046ec222ab4ce656ca4dfa33ea07f1d7d2cd6cc4ee42
  Stored in directory: /tmp/pip-ephem-wheel-cache-q7xa48gf/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-gwa2_3xc
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-gwa2_3xc
Resolved https://github.com/NVIDIA-Merlin/models.git to commit d88c54027425473130c34945983487e8bcf57930
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+40.gd88c5402) (0.3.0+12.g78ecddd)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (65.5.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (4.0.0)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+40.gd88c5402-py3-none-any.whl size=364675 sha256=241773d441a4a32e7e7e458978a1bf7f0f55f695b015c16e27c91d7b2c4daf0b
Stored in directory: /tmp/pip-ephem-wheel-cache-q52skhgp/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+40.gd88c5402

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-u8jldrsb
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-u8jldrsb
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting merlin-dataloader>=0.0.2
Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.5 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.5.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257746 sha256=a4c6c61c492301294512e2830aa4527460f05553beef38c140d65e81438d4aad
Stored in directory: /tmp/pip-ephem-wheel-cache-tcd4cl2g/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Building wheel for merlin-dataloader (pyproject.toml): started
Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29205 sha256=678281a8f64774dc5ec2a7f7e944406086917d6ff017f3db8b96fb57bad9f314
Stored in directory: /tmp/pip-ephem-wheel-cache-tcd4cl2g/wheels/d5/ce/8c/31476c01e0b5a2278110fe2092bdd911efb0e5b83d0d3550ca
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-_ba89km2
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-_ba89km2
Resolved https://github.com/NVIDIA-Merlin/core.git to commit eb606d54fa2ddcbb7e4d0e6501ab2eb418c7fba9
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (65.5.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+11.geb606d5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.0.0)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+11.geb606d5-py3-none-any.whl size=118637 sha256=b62f3051979a9a8df06584167dd6875f50d2ae26429943c81f9a4983750f1839
Stored in directory: /tmp/pip-ephem-wheel-cache-o94uyyep/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+11.geb606d5

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py . [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

self = <testbook.client.TestbookNotebookClient object at 0x7fee5627cbe0>
msg_id = 'ea38e743-5a70c5c491de20bba77bf767_12159_32'
cell = {'id': '5c32f87d', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-11T14:18:46.812256Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
timeout = 120
task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>
task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>

async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:
          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:


obj = <coroutine object ZMQSocketChannel.get_msg at 0x7fee54eab9c0>

async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:
          result = await obj

../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:


self = <jupyter_client.channels.ZMQSocketChannel object at 0x7fef33f562b0>
timeout = 120000.0

async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:
      raise Empty

E _queue.Empty

/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty

During handling of the above exception, another exception occurred:

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"
      tb2.inject(
            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:


../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject
cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)
../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell
cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell
exec_reply = await self.task_poll_for_reply
../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply
await self._async_handle_timeout(timeout, cell)


self = <testbook.client.TestbookNotebookClient object at 0x7fee5627cbe0>
timeout = 120
cell = {'id': '5c32f87d', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-11T14:18:46.812256Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:
      raise CellTimeoutError.error_from_timeout_and_cell(
            "Cell execution timed out", timeout, cell
        )

E nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.
E The message was: Cell execution timed out.
E Here is a preview of the cell contents:
E -------------------
E ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']
E ...
E ['response = run_ensemble_on_tritonserver(', ' "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']
E -------------------

../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError
----------------------------- Captured stdout call -----------------------------
Signal (2) received.
Signal (2) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
Signal (15) received.
----------------------------- Captured stderr call -----------------------------
2022-11-11 14:16:52.279339: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-11 14:16:56.399356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-11 14:16:56.400133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-11 14:16:56.400754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-11 14:16:56.401366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
WARNING clustering 469 points to 32 centroids: please provide at least 1248 training points
2022-11-11 14:18:40.239631: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-11 14:18:44.316473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-11 14:18:44.317254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-11 14:18:44.317826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-11 14:18:44.318393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
I1111 14:18:47.198808 12723 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe7c6000000' with size 268435456
I1111 14:18:47.200166 12723 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1111 14:18:47.213575 12723 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1
I1111 14:18:47.213633 12723 model_lifecycle.cc:459] loading: 0_queryfeast:1
I1111 14:18:47.213675 12723 model_lifecycle.cc:459] loading: 2_queryfaiss:1
I1111 14:18:47.213719 12723 model_lifecycle.cc:459] loading: 3_queryfeast:1
I1111 14:18:47.213791 12723 model_lifecycle.cc:459] loading: 4_unrollfeatures:1
I1111 14:18:47.213824 12723 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1
I1111 14:18:47.213855 12723 model_lifecycle.cc:459] loading: 6_softmaxsampling:1
I1111 14:18:48.128160 12723 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow
I1111 14:18:48.128200 12723 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10
I1111 14:18:48.128210 12723 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10
I1111 14:18:48.128216 12723 tensorflow.cc:2576] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}
I1111 14:18:48.128260 12723 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)
2022-11-11 14:18:48.131988: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:18:48.137952: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-11 14:18:48.138657: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:18:48.140859: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-11 14:18:49.250177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-11 14:18:49.290928: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-11-11 14:18:49.295844: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-11 14:18:49.422535: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:18:49.446449: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 1314994 microseconds.
I1111 14:18:49.458960 12723 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)
2022-11-11 14:18:49.459705: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:18:49.462906: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-11 14:18:49.462930: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:18:49.464891: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-11 14:18:49.472144: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-11 14:18:49.529271: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:18:49.552105: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 92403 microseconds.
I1111 14:18:49.552862 12723 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1111 14:19:28.690067 12723 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)
2022-11-11 14:19:28.691533: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:19:28.703545: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-11 14:19:28.703580: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:19:28.705796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-11 14:19:28.735647: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-11 14:19:28.878064: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:19:28.931548: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 240020 microseconds.
I1111 14:19:28.949040 12723 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)
I1111 14:19:35.336585 12723 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)
I1111 14:19:35.340182 12723 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1
I1111 14:19:41.738558 12723 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)
I1111 14:19:41.739576 12723 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1
I1111 14:19:48.061157 12723 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)
I1111 14:19:48.061459 12723 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1
I1111 14:19:53.965372 12723 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)
I1111 14:19:53.968288 12723 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1
I1111 14:19:59.944508 12723 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)
I1111 14:19:59.944909 12723 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1
2022-11-11 14:19:59.945125: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:19:59.954581: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }
2022-11-11 14:19:59.954616: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:19:59.956577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-11 14:19:59.980058: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.
2022-11-11 14:20:00.128852: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel
2022-11-11 14:20:00.190430: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 245313 microseconds.
I1111 14:20:00.190760 12723 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1
I1111 14:20:00.193686 12723 model_lifecycle.cc:459] loading: ensemble_model:1
I1111 14:20:00.194238 12723 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1
I1111 14:20:00.194374 12723 server.cc:561]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1111 14:20:00.194470 12723 server.cc:588]
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1111 14:20:00.195154 12723 server.cc:631]
+---------------------------+---------+--------+
| Model | Version | Status |
+---------------------------+---------+--------+
| 0_queryfeast | 1 | READY |
| 1_predicttensorflowtriton | 1 | READY |
| 2_queryfaiss | 1 | READY |
| 3_queryfeast | 1 | READY |
| 4_unrollfeatures | 1 | READY |
| 5_predicttensorflowtriton | 1 | READY |
| 6_softmaxsampling | 1 | READY |
| ensemble_model | 1 | READY |
+---------------------------+---------+--------+

I1111 14:20:00.256252 12723 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB
I1111 14:20:00.257078 12723 tritonserver.cc:2214]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.25.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0] | /tmp/examples/poc_ensemble |
| model_control_mode | MODE_NONE |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1111 14:20:00.278899 12723 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001
I1111 14:20:00.280149 12723 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000
I1111 14:20:00.321731 12723 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002
W1111 14:20:01.280051 12723 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1111 14:20:02.280257 12723 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
W1111 14:20:03.298760 12723 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0
I1111 14:20:33.704575 12723 server.cc:262] Waiting for in-flight requests to complete.
I1111 14:20:33.704595 12723 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences
I1111 14:20:33.704793 12723 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1
I1111 14:20:33.704864 12723 server.cc:293] All models are stopped, unloading models
I1111 14:20:33.704879 12723 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests
I1111 14:20:33.704888 12723 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1111 14:20:33.705051 12723 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1111 14:20:33.705136 12723 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1111 14:20:33.705195 12723 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state
I1111 14:20:33.717211 12723 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1
I1111 14:20:33.727834 12723 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1
I1111 14:20:34.704979 12723 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1111 14:20:35.577163 12723 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1
I1111 14:20:35.705150 12723 server.cc:300] Timeout 28: Found 4 live models and 0 in-flight non-inference requests
I1111 14:20:35.801653 12723 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1
I1111 14:20:35.849997 12723 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1
I1111 14:20:36.705276 12723 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:37.705419 12723 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:38.705558 12723 server.cc:300] Timeout 25: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:39.705706 12723 server.cc:300] Timeout 24: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:40.705841 12723 server.cc:300] Timeout 23: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:41.705973 12723 server.cc:300] Timeout 22: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:42.706104 12723 server.cc:300] Timeout 21: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:43.706235 12723 server.cc:300] Timeout 20: Found 2 live models and 0 in-flight non-inference requests
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
I1111 14:20:44.706378 12723 server.cc:300] Timeout 19: Found 2 live models and 0 in-flight non-inference requests
I1111 14:20:44.954936 12723 model_lifecycle.cc:578] successfully unloaded '3_queryfeast' version 1
I1111 14:20:45.472317 12723 model_lifecycle.cc:578] successfully unloaded '0_queryfeast' version 1
I1111 14:20:45.706503 12723 server.cc:300] Timeout 18: Found 0 live models and 0 in-flight non-inference requests
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
------------------------------ Captured log call -------------------------------
ERROR traitlets:client.py:795 Timeout waiting for execute reply (120s).

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 15 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 24 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 136 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 347 161 54%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 29 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 81 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 151 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 71 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 154 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 337 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 108 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 22 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 10 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 52 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 57 13 77%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 20 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 126 26 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 18 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py 65 19 71%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 4 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 12 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 119 67 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 210 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py 22 20 9%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py 113 14 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py 37 29 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py 10 6 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py 32 13 59%

TOTAL 14839 6820 54%

=================== 1 failed, 2 passed in 377.60s (0:06:17) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins4033361495660827429.sh

@oliverholworthy
Copy link
Member

notebook test is timing out. Increasing limit from 2 mins to 3 mins seems to work #729 not sure why it's got slower though (and maybe worth a follow-up to check. might be the code, but might also be the environment?). Perhaps we were close to the limit before.

Based on this PR, it does seem like we're now consistently not able to run within the previous time limit.

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 67901f953a1d6647005791af1e02f6f50da95dfe, no merge conflicts.
Running as SYSTEM
Setting status of 67901f953a1d6647005791af1e02f6f50da95dfe to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/613/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 67901f953a1d6647005791af1e02f6f50da95dfe^{commit} # timeout=10
Checking out Revision 67901f953a1d6647005791af1e02f6f50da95dfe (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67901f953a1d6647005791af1e02f6f50da95dfe # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk bae3ccb8c0509ce844a280228943b5e1c73c49bb # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins5690764768634219605.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.8,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.8,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='9051771'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-taakvslu
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-taakvslu
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit f23adcb079ec74d15f74c492fbd6a3af19742f5c
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+12.gf23adcb) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+12.gf23adcb) (2.22.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (65.5.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 106.6 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+12.gf23adcb) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+12.gf23adcb-py3-none-any.whl size=92208 sha256=1967bd5f952070f4f930870fbc59406e87c8463fc49f7bb1fc51040682e16aad
  Stored in directory: /tmp/pip-ephem-wheel-cache-qgksyfyo/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+12.gf23adcb

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-iimiyh73
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-iimiyh73
Resolved https://github.com/NVIDIA-Merlin/models.git to commit d88c54027425473130c34945983487e8bcf57930
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+40.gd88c5402) (0.3.0+12.g78ecddd)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (65.5.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+40.gd88c5402) (4.0.0)
Building wheels for collected packages: merlin-models
Building wheel for merlin-models (pyproject.toml): started
Building wheel for merlin-models (pyproject.toml): finished with status 'done'
Created wheel for merlin-models: filename=merlin_models-0.9.0+40.gd88c5402-py3-none-any.whl size=364689 sha256=4ac1054a36dad9ea681b9b76085d7520f6e61a3b22eda828bfe75e7f1a2678a1
Stored in directory: /tmp/pip-ephem-wheel-cache-fzrfe2ry/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
Attempting uninstall: merlin-models
Found existing installation: merlin-models 0.7.0+11.g280956aa4
Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+40.gd88c5402

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-3mykjb1m
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-3mykjb1m
Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Collecting merlin-dataloader>=0.0.2
Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.4 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.5.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
Building wheel for nvtabular (pyproject.toml): started
Building wheel for nvtabular (pyproject.toml): finished with status 'done'
Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257596 sha256=911c4e1fdfcac5e2e61c9a684dc89354edd6a78554d86c066a337f27672ef801
Stored in directory: /tmp/pip-ephem-wheel-cache-ur1qop4m/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Building wheel for merlin-dataloader (pyproject.toml): started
Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29203 sha256=f3416b0a6b70a4fa4ebbc60a830533943523f4c730b5763df301fcad75a74e38
Stored in directory: /tmp/pip-ephem-wheel-cache-ur1qop4m/wheels/76/ef/ed/cb880e3ef5192ec5940e26fd9442247b569fb0cf8602f97137
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
Attempting uninstall: nvtabular
Found existing installation: nvtabular 1.1.1
Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+4.gba4c1415

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-h2rwb0_1
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-h2rwb0_1
Resolved https://github.com/NVIDIA-Merlin/core.git to commit eb606d54fa2ddcbb7e4d0e6501ab2eb418c7fba9
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (65.5.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+11.geb606d5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.0.0)
Building wheels for collected packages: merlin-core
Building wheel for merlin-core (pyproject.toml): started
Building wheel for merlin-core (pyproject.toml): finished with status 'done'
Created wheel for merlin-core: filename=merlin_core-0.8.0+11.geb606d5-py3-none-any.whl size=118619 sha256=8338c9ae90ac94eaa11e951d017374ab7346a71b1c38fd3ba5dd4cb5fb2790f1
Stored in directory: /tmp/pip-ephem-wheel-cache-mvfucuw3/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
Attempting uninstall: merlin-core
Found existing installation: merlin-core 0.3.0+12.g78ecddd
Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+11.geb606d5

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py F [100%]

=================================== FAILURES ===================================
__________________________________ test_func ___________________________________

self = <testbook.client.TestbookNotebookClient object at 0x7ff3cc5dca30>
cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55

def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)

../../../.local/lib/python3.8/site-packages/testbook/client.py:133:


args = (<testbook.client.TestbookNotebookClient object at 0x7ff3cc5dca30>, {'id': '9aaaa68f', 'cell_type': 'code', 'metadata'...on\x1b[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'"]}]}, 55)
kwargs = {}

def wrapped(*args, **kwargs):
  return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:


coro = <coroutine object NotebookClient.async_execute_cell at 0x7ff3cb2c2740>

def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()
  return loop.run_until_complete(coro)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:


self = <_UnixSelectorEventLoop running=False closed=False debug=False>
future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...rException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'\n')>

def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')
  return future.result()

/usr/lib/python3.8/asyncio/base_events.py:616:


self = <testbook.client.TestbookNotebookClient object at 0x7ff3cc5dca30>
cell = {'id': '9aaaa68f', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-13T20:03:29.395842Z',...ception\x1b[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'"]}]}
cell_index = 55, execution_count = None, store_history = True

async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count
  await self._check_raise_for_error(cell, cell_index, exec_reply)

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:


self = <testbook.client.TestbookNotebookClient object at 0x7ff3cc5dca30>
cell = {'id': '9aaaa68f', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-13T20:03:29.395842Z',...ception\x1b[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'"]}]}
cell_index = 55
exec_reply = {'buffers': [], 'content': {'ename': 'InferenceServerException', 'engine_info': {'engine_id': -1, 'engine_uuid': 'f2c1...e, 'engine': 'f2c191ec-9008-4a10-94b4-1b63ce132e38', 'started': '2022-11-13T20:03:29.396152Z', 'status': 'error'}, ...}

async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:
      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)

E nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E ------------------
E
E import shutil
E from merlin.core.dispatch import get_lib
E from merlin.models.loader.tf_utils import configure_tensorflow
E configure_tensorflow()
E df_lib = get_lib()
E batch = df_lib.read_parquet(
E os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
E num_rows=1,
E columns=["user_id_raw"],
E )
E from merlin.systems.triton.utils import run_ensemble_on_tritonserver
E response = run_ensemble_on_tritonserver(
E "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"
E )
E response = [x.tolist()[0] for x in response["ordered_ids"]]
E shutil.rmtree("/tmp/examples/", ignore_errors=True)
E
E ------------------
E
E �[0;31m---------------------------------------------------------------------------�[0m
E �[0;31mInferenceServerException�[0m Traceback (most recent call last)
E Cell �[0;32mIn [32], line 12�[0m
E �[1;32m 6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(
E �[1;32m 7�[0m os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),
E �[1;32m 8�[0m num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,
E �[1;32m 9�[0m columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],
E �[1;32m 10�[0m )
E �[1;32m 11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver
E �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m
E �[1;32m 13�[0m �[43m �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m
E �[1;32m 14�[0m �[43m)�[49m
E �[1;32m 15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]
E �[1;32m 16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:139�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m
E �[1;32m 137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
E �[1;32m 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:
E �[0;32m--> 139�[0m response �[38;5;241m=�[39m �[43msend_triton_request�[49m�[43m(�[49m
E �[1;32m 140�[0m �[43m �[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43moutput_columns�[49m�[43m,�[49m�[43m �[49m�[43mclient�[49m�[38;5;241;43m=�[39;49m�[43mclient�[49m�[43m,�[49m�[43m �[49m�[43mtriton_model�[49m�[38;5;241;43m=�[39;49m�[43mmodel_name�[49m
E �[1;32m 141�[0m �[43m �[49m�[43m)�[49m
E �[1;32m 143�[0m �[38;5;28;01mreturn�[39;00m response
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:192�[0m, in �[0;36msend_triton_request�[0;34m(schema, df, outputs_list, client, endpoint, request_id, triton_model)�[0m
E �[1;32m 190�[0m outputs �[38;5;241m=�[39m [grpcclient�[38;5;241m.�[39mInferRequestedOutput(col) �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list]
E �[1;32m 191�[0m �[38;5;28;01mwith�[39;00m client:
E �[0;32m--> 192�[0m response �[38;5;241m=�[39m �[43mclient�[49m�[38;5;241;43m.�[39;49m�[43minfer�[49m�[43m(�[49m�[43mtriton_model�[49m�[43m,�[49m�[43m �[49m�[43minputs�[49m�[43m,�[49m�[43m �[49m�[43mrequest_id�[49m�[38;5;241;43m=�[39;49m�[43mrequest_id�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[38;5;241;43m=�[39;49m�[43moutputs�[49m�[43m)�[49m
E �[1;32m 194�[0m results �[38;5;241m=�[39m {}
E �[1;32m 195�[0m �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list:
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322�[0m, in �[0;36mInferenceServerClient.infer�[0;34m(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)�[0m
E �[1;32m 1320�[0m �[38;5;28;01mreturn�[39;00m result
E �[1;32m 1321�[0m �[38;5;28;01mexcept�[39;00m grpc�[38;5;241m.�[39mRpcError �[38;5;28;01mas�[39;00m rpc_error:
E �[0;32m-> 1322�[0m �[43mraise_error_grpc�[49m�[43m(�[49m�[43mrpc_error�[49m�[43m)�[49m
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62�[0m, in �[0;36mraise_error_grpc�[0;34m(rpc_error)�[0m
E �[1;32m 61�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mraise_error_grpc�[39m(rpc_error):
E �[0;32m---> 62�[0m �[38;5;28;01mraise�[39;00m get_error_grpc(rpc_error) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m
E
E �[0;31mInferenceServerException�[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'
E InferenceServerException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'

../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

During handling of the above exception, another exception occurred:

def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"
      tb2.inject(
            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:


../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject
cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)


self = <testbook.client.TestbookNotebookClient object at 0x7ff3cc5dca30>
cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55

def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:
          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))

E testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:
E ------------------
E
E import shutil
E from merlin.core.dispatch import get_lib
E from merlin.models.loader.tf_utils import configure_tensorflow
E configure_tensorflow()
E df_lib = get_lib()
E batch = df_lib.read_parquet(
E os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
E num_rows=1,
E columns=["user_id_raw"],
E )
E from merlin.systems.triton.utils import run_ensemble_on_tritonserver
E response = run_ensemble_on_tritonserver(
E "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs, "ensemble_model"
E )
E response = [x.tolist()[0] for x in response["ordered_ids"]]
E shutil.rmtree("/tmp/examples/", ignore_errors=True)
E
E ------------------
E
E �[0;31m---------------------------------------------------------------------------�[0m
E �[0;31mInferenceServerException�[0m Traceback (most recent call last)
E Cell �[0;32mIn [32], line 12�[0m
E �[1;32m 6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(
E �[1;32m 7�[0m os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),
E �[1;32m 8�[0m num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,
E �[1;32m 9�[0m columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],
E �[1;32m 10�[0m )
E �[1;32m 11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver
E �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m
E �[1;32m 13�[0m �[43m �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m
E �[1;32m 14�[0m �[43m)�[49m
E �[1;32m 15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]
E �[1;32m 16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:139�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m
E �[1;32m 137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
E �[1;32m 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:
E �[0;32m--> 139�[0m response �[38;5;241m=�[39m �[43msend_triton_request�[49m�[43m(�[49m
E �[1;32m 140�[0m �[43m �[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43moutput_columns�[49m�[43m,�[49m�[43m �[49m�[43mclient�[49m�[38;5;241;43m=�[39;49m�[43mclient�[49m�[43m,�[49m�[43m �[49m�[43mtriton_model�[49m�[38;5;241;43m=�[39;49m�[43mmodel_name�[49m
E �[1;32m 141�[0m �[43m �[49m�[43m)�[49m
E �[1;32m 143�[0m �[38;5;28;01mreturn�[39;00m response
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:192�[0m, in �[0;36msend_triton_request�[0;34m(schema, df, outputs_list, client, endpoint, request_id, triton_model)�[0m
E �[1;32m 190�[0m outputs �[38;5;241m=�[39m [grpcclient�[38;5;241m.�[39mInferRequestedOutput(col) �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list]
E �[1;32m 191�[0m �[38;5;28;01mwith�[39;00m client:
E �[0;32m--> 192�[0m response �[38;5;241m=�[39m �[43mclient�[49m�[38;5;241;43m.�[39;49m�[43minfer�[49m�[43m(�[49m�[43mtriton_model�[49m�[43m,�[49m�[43m �[49m�[43minputs�[49m�[43m,�[49m�[43m �[49m�[43mrequest_id�[49m�[38;5;241;43m=�[39;49m�[43mrequest_id�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[38;5;241;43m=�[39;49m�[43moutputs�[49m�[43m)�[49m
E �[1;32m 194�[0m results �[38;5;241m=�[39m {}
E �[1;32m 195�[0m �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m outputs_list:
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322�[0m, in �[0;36mInferenceServerClient.infer�[0;34m(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)�[0m
E �[1;32m 1320�[0m �[38;5;28;01mreturn�[39;00m result
E �[1;32m 1321�[0m �[38;5;28;01mexcept�[39;00m grpc�[38;5;241m.�[39mRpcError �[38;5;28;01mas�[39;00m rpc_error:
E �[0;32m-> 1322�[0m �[43mraise_error_grpc�[49m�[43m(�[49m�[43mrpc_error�[49m�[43m)�[49m
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62�[0m, in �[0;36mraise_error_grpc�[0;34m(rpc_error)�[0m
E �[1;32m 61�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mraise_error_grpc�[39m(rpc_error):
E �[0;32m---> 62�[0m �[38;5;28;01mraise�[39;00m get_error_grpc(rpc_error) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m
E
E �[0;31mInferenceServerException�[0m: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'
E InferenceServerException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'ordered_ids' for model 'ensemble_model'

../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError
----------------------------- Captured stderr call -----------------------------
2022-11-13 20:01:37.861526: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-13 20:01:41.980676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-13 20:01:41.981383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-13 20:01:41.982010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-13 20:01:41.982607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown
h.close()
File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close
self.stream.close()
File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close
self.watch_fd_thread.join()
AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'
WARNING clustering 458 points to 32 centroids: please provide at least 1248 training points
2022-11-13 20:03:22.710741: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-13 20:03:26.855009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory: -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0
2022-11-13 20:03:26.855779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory: -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0
2022-11-13 20:03:26.856376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory: -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0
2022-11-13 20:03:26.856986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory: -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0
I1113 20:03:29.724891 5889 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f0626000000' with size 268435456
I1113 20:03:29.725620 5889 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1113 20:03:29.733866 5889 model_lifecycle.cc:459] loading: 2_queryfaiss:1
I1113 20:03:29.733929 5889 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1
I1113 20:03:29.733992 5889 model_lifecycle.cc:459] loading: 0_queryfeast:1
I1113 20:03:29.734038 5889 model_lifecycle.cc:459] loading: 3_queryfeast:1
I1113 20:03:29.734106 5889 model_lifecycle.cc:459] loading: 4_unrollfeatures:1
I1113 20:03:29.734142 5889 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1
I1113 20:03:29.734183 5889 model_lifecycle.cc:459] loading: 6_softmaxsampling:1
1113 20:03:30.789878 5928 pb_stub.cc:1016] Non-graceful termination detected.
__________________________________ test_func ___________________________________

def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )
      tb1.execute()

tests/unit/examples/test_scaling_criteo_merlin_models.py:33:


../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute
super().execute_cell(cell, index)
../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped
return just_run(coro(*args, **kwargs))
../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run
return loop.run_until_complete(coro)
/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
return future.result()
../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell
await self._check_raise_for_error(cell, cell_index, exec_reply)


self = <testbook.client.TestbookNotebookClient object at 0x7ff4a987f670>
cell = {'cell_type': 'code', 'execution_count': 10, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.s...gine="parquet", part_size=part_size)\nvalid_dataset = nvt.Dataset(valid_paths, engine="parquet", part_size=part_size)'}
cell_index = 18
exec_reply = {'buffers': [], 'content': {'ename': 'RuntimeError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'a719dd4e-676a-48...e, 'engine': 'a719dd4e-676a-4806-9c33-37ff91bf549d', 'started': '2022-11-13T20:03:45.039620Z', 'status': 'error'}, ...}

async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:
      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)

E nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E ------------------
E train_dataset = nvt.Dataset(train_paths, engine="parquet", part_size=part_size)
E valid_dataset = nvt.Dataset(valid_paths, engine="parquet", part_size=part_size)
E ------------------
E
E �[0;31m---------------------------------------------------------------------------�[0m
E �[0;31mRuntimeError�[0m Traceback (most recent call last)
E Cell �[0;32mIn [10], line 1�[0m
E �[0;32m----> 1�[0m train_dataset �[38;5;241m=�[39m �[43mnvt�[49m�[38;5;241;43m.�[39;49m�[43mDataset�[49m�[43m(�[49m�[43mtrain_paths�[49m�[43m,�[49m�[43m �[49m�[43mengine�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43mparquet�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mpart_size�[49m�[38;5;241;43m=�[39;49m�[43mpart_size�[49m�[43m)�[49m
E �[1;32m 2�[0m valid_dataset �[38;5;241m=�[39m nvt�[38;5;241m.�[39mDataset(valid_paths, engine�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124mparquet�[39m�[38;5;124m"�[39m, part_size�[38;5;241m=�[39mpart_size)
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:308�[0m, in �[0;36mDataset.init__�[0;34m(self, path_or_source, engine, npartitions, part_size, part_mem_fraction, storage_options, dtypes, client, cpu, base_dataset, schema, **kwargs)�[0m
E �[1;32m 306�[0m �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(engine, �[38;5;28mstr�[39m):
E �[1;32m 307�[0m �[38;5;28;01mif�[39;00m engine �[38;5;241m==�[39m �[38;5;124m"�[39m�[38;5;124mparquet�[39m�[38;5;124m"�[39m:
E �[0;32m--> 308�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mengine �[38;5;241m=�[39m �[43mParquetDatasetEngine�[49m�[43m(�[49m
E �[1;32m 309�[0m �[43m �[49m�[43mpaths�[49m�[43m,�[49m�[43m �[49m�[43mpart_size�[49m�[43m,�[49m�[43m �[49m�[43mstorage_options�[49m�[38;5;241;43m=�[39;49m�[43mstorage_options�[49m�[43m,�[49m�[43m �[49m�[43mcpu�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mcpu�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m
E �[1;32m 310�[0m �[43m �[49m�[43m)�[49m
E �[1;32m 311�[0m �[38;5;28;01melif�[39;00m engine �[38;5;241m==�[39m �[38;5;124m"�[39m�[38;5;124mcsv�[39m�[38;5;124m"�[39m:
E �[1;32m 312�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mengine �[38;5;241m=�[39m CSVDatasetEngine(
E �[1;32m 313�[0m paths, part_size, storage_options�[38;5;241m=�[39mstorage_options, cpu�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mcpu, �[38;5;241m�[39m�[38;5;241m�[39mkwargs
E �[1;32m 314�[0m )
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py:311�[0m, in �[0;36mParquetDatasetEngine.init__�[0;34m(self, paths, part_size, storage_options, row_groups_per_part, legacy, batch_size, cpu, **kwargs)�[0m
E �[1;32m 308�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mdataset_kwargs �[38;5;241m=�[39m �[38;5;28mself�[39m�[38;5;241m.�[39mread_parquet_kwargs�[38;5;241m.�[39mpop(�[38;5;124m"�[39m�[38;5;124mdataset�[39m�[38;5;124m"�[39m, {})
E �[1;32m 310�[0m �[38;5;28;01mif�[39;00m row_groups_per_part �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:
E �[0;32m--> 311�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_real_meta, rg_byte_size_0 �[38;5;241m=�[39m �[43mrun_on_worker�[49m�[43m(�[49m
E �[1;32m 312�[0m �[43m �[49m�[43m_sample_row_group�[49m�[43m,�[49m
E �[1;32m 313�[0m �[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_path0�[49m�[43m,�[49m
E �[1;32m 314�[0m �[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mfs�[49m�[43m,�[49m
E �[1;32m 315�[0m �[43m �[49m�[43mcpu�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mcpu�[49m�[43m,�[49m
E �[1;32m 316�[0m �[43m �[49m�[43mmemory_usage�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43;01mTrue�[39;49;00m�[43m,�[49m
E �[1;32m 317�[0m �[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mread_parquet_kwargs�[49m�[43m,�[49m
E �[1;32m 318�[0m �[43m �[49m�[43m)�[49m
E �[1;32m 319�[0m row_groups_per_part �[38;5;241m=�[39m �[38;5;28mself�[39m�[38;5;241m.�[39mpart_size �[38;5;241m/�[39m rg_byte_size_0
E �[1;32m 320�[0m �[38;5;28;01mif�[39;00m row_groups_per_part �[38;5;241m<�[39m �[38;5;241m1.0�[39m:
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py:484�[0m, in �[0;36mrun_on_worker�[0;34m(func, args, **kwargs)�[0m
E �[1;32m 479�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mrun_on_worker�[39m(func, �[38;5;241m
�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
E �[1;32m 480�[0m �[38;5;66;03m# Run a function on a Dask worker using delayed�[39;00m
E �[1;32m 481�[0m �[38;5;66;03m# execution (if a Dask client is detected)�[39;00m
E �[1;32m 482�[0m �[38;5;28;01mif�[39;00m global_dask_client():
E �[1;32m 483�[0m �[38;5;66;03m# There is a specified or global Dask client. Use it�[39;00m
E �[0;32m--> 484�[0m �[38;5;28;01mreturn�[39;00m �[43mdask�[49m�[38;5;241;43m.�[39;49m�[43mdelayed�[49m�[43m(�[49m�[43mfunc�[49m�[43m)�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mcompute�[49m�[43m(�[49m�[43m)�[49m
E �[1;32m 485�[0m �[38;5;66;03m# No Dask client - Use simple function call�[39;00m
E �[1;32m 486�[0m �[38;5;28;01mreturn�[39;00m func(�[38;5;241m
�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)
E
E File �[0;32m~/.local/lib/python3.8/site-packages/dask/base.py:292�[0m, in �[0;36mDaskMethodsMixin.compute�[0;34m(self, **kwargs)�[0m
E �[1;32m 268�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mcompute�[39m(�[38;5;28mself�[39m, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
E �[1;32m 269�[0m �[38;5;124;03m"""Compute this dask collection�[39;00m
E �[1;32m 270�[0m
E �[1;32m 271�[0m �[38;5;124;03m This turns a lazy Dask collection into its in-memory equivalent.�[39;00m
E �[0;32m (...)�[0m
E �[1;32m 290�[0m �[38;5;124;03m dask.base.compute�[39;00m
E �[1;32m 291�[0m �[38;5;124;03m """�[39;00m
E �[0;32m--> 292�[0m (result,) �[38;5;241m=�[39m �[43mcompute�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[43m,�[49m�[43m �[49m�[43mtraverse�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43;01mFalse�[39;49;00m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m
E �[1;32m 293�[0m �[38;5;28;01mreturn�[39;00m result
E
E File �[0;32m~/.local/lib/python3.8/site-packages/dask/base.py:575�[0m, in �[0;36mcompute�[0;34m(traverse, optimize_graph, scheduler, get, *args, **kwargs)�[0m
E �[1;32m 572�[0m keys�[38;5;241m.�[39mappend(x�[38;5;241m.�[39m__dask_keys())
E �[1;32m 573�[0m postcomputes�[38;5;241m.�[39mappend(x�[38;5;241m.�[39m__dask_postcompute
())
E �[0;32m--> 575�[0m results �[38;5;241m=�[39m �[43mschedule�[49m�[43m(�[49m�[43mdsk�[49m�[43m,�[49m�[43m �[49m�[43mkeys�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m
E �[1;32m 576�[0m �[38;5;28;01mreturn�[39;00m repack([f(r, �[38;5;241m�[39ma) �[38;5;28;01mfor�[39;00m r, (f, a) �[38;5;129;01min�[39;00m �[38;5;28mzip�[39m(results, postcomputes)])
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/client.py:3015�[0m, in �[0;36mClient.get�[0;34m(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)�[0m
E �[1;32m 3013�[0m should_rejoin �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m
E �[1;32m 3014�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m-> 3015�[0m results �[38;5;241m=�[39m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgather�[49m�[43m(�[49m�[43mpacked�[49m�[43m,�[49m�[43m �[49m�[43masynchronous�[49m�[38;5;241;43m=�[39;49m�[43masynchronous�[49m�[43m,�[49m�[43m �[49m�[43mdirect�[49m�[38;5;241;43m=�[39;49m�[43mdirect�[49m�[43m)�[49m
E �[1;32m 3016�[0m �[38;5;28;01mfinally�[39;00m:
E �[1;32m 3017�[0m �[38;5;28;01mfor�[39;00m f �[38;5;129;01min�[39;00m futures�[38;5;241m.�[39mvalues():
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/client.py:2167�[0m, in �[0;36mClient.gather�[0;34m(self, futures, errors, direct, asynchronous)�[0m
E �[1;32m 2165�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 2166�[0m local_worker �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
E �[0;32m-> 2167�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m
E �[1;32m 2168�[0m �[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_gather�[49m�[43m,�[49m
E �[1;32m 2169�[0m �[43m �[49m�[43mfutures�[49m�[43m,�[49m
E �[1;32m 2170�[0m �[43m �[49m�[43merrors�[49m�[38;5;241;43m=�[39;49m�[43merrors�[49m�[43m,�[49m
E �[1;32m 2171�[0m �[43m �[49m�[43mdirect�[49m�[38;5;241;43m=�[39;49m�[43mdirect�[49m�[43m,�[49m
E �[1;32m 2172�[0m �[43m �[49m�[43mlocal_worker�[49m�[38;5;241;43m=�[39;49m�[43mlocal_worker�[49m�[43m,�[49m
E �[1;32m 2173�[0m �[43m �[49m�[43masynchronous�[49m�[38;5;241;43m=�[39;49m�[43masynchronous�[49m�[43m,�[49m
E �[1;32m 2174�[0m �[43m�[49m�[43m)�[49m
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m
E �[1;32m 307�[0m �[38;5;28;01mreturn�[39;00m future
E �[1;32m 308�[0m �[38;5;28;01melse�[39;00m:
E �[0;32m--> 309�[0m �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m
E �[1;32m 310�[0m �[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m
�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m
�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m
E �[1;32m 311�[0m �[43m �[49m�[43m)�[49m
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, args, **kwargs)�[0m
E �[1;32m 374�[0m �[38;5;28;01mif�[39;00m error:
E �[1;32m 375�[0m typ, exc, tb �[38;5;241m=�[39m error
E �[0;32m--> 376�[0m �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)
E �[1;32m 377�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 378�[0m �[38;5;28;01mreturn�[39;00m result
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m
E �[1;32m 347�[0m future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)
E �[1;32m 348�[0m future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)
E �[0;32m--> 349�[0m result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future
E �[1;32m 350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
E �[1;32m 351�[0m error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
E
E File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m
E �[1;32m 759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
E �[1;32m 761�[0m �[38;5;28;01mtry�[39;00m:
E �[0;32m--> 762�[0m value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m
E �[1;32m 763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
E �[1;32m 764�[0m exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
E
E File �[0;32m~/.local/lib/python3.8/site-packages/distributed/client.py:2030�[0m, in �[0;36mClient._gather�[0;34m(self, futures, errors, direct, local_worker)�[0m
E �[1;32m 2028�[0m exc �[38;5;241m=�[39m CancelledError(key)
E �[1;32m 2029�[0m �[38;5;28;01melse�[39;00m:
E �[0;32m-> 2030�[0m �[38;5;28;01mraise�[39;00m exception�[38;5;241m.�[39mwith_traceback(traceback)
E �[1;32m 2031�[0m �[38;5;28;01mraise�[39;00m exc
E �[1;32m 2032�[0m �[38;5;28;01mif�[39;00m errors �[38;5;241m==�[39m �[38;5;124m"�[39m�[38;5;124mskip�[39m�[38;5;124m"�[39m:
E
E File �[0;32m~/.local/lib/python3.8/site-packages/dask/utils.py:39�[0m, in �[0;36mapply�[0;34m()�[0m
E �[1;32m 37�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mapply�[39m(func, args, kwargs�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m):
E �[1;32m 38�[0m �[38;5;28;01mif�[39;00m kwargs:
E �[0;32m---> 39�[0m �[38;5;28;01mreturn�[39;00m func(�[38;5;241m
�[39margs, �[38;5;241m
�[39m�[38;5;241m�[39mkwargs)
E �[1;32m 40�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 41�[0m �[38;5;28;01mreturn�[39;00m func(�[38;5;241m
�[39margs)
E
E File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py:1210�[0m, in �[0;36m_sample_row_group�[0;34m()�[0m
E �[1;32m 1206�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 1207�[0m �[38;5;28;01mif�[39;00m cudf�[38;5;241m.�[39mutils�[38;5;241m.�[39mioutils�[38;5;241m.�[39m_is_local_filesystem(fs):
E �[1;32m 1208�[0m �[38;5;66;03m# Allow cudf to open the file if this is a local file�[39;00m
E �[1;32m 1209�[0m �[38;5;66;03m# system (can be significantly faster in this case)�[39;00m
E �[0;32m-> 1210�[0m _df �[38;5;241m=�[39m cudf�[38;5;241m.�[39mio�[38;5;241m.�[39mread_parquet(path, row_groups�[38;5;241m=�[39m�[38;5;241m0�[39m, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)
E �[1;32m 1211�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 1212�[0m _df �[38;5;241m=�[39m _optimized_read_remote(path, �[38;5;241m0�[39m, �[38;5;28;01mNone�[39;00m, fs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36minner�[0;34m()�[0m
E �[1;32m 98�[0m �[38;5;129m@wraps�[39m(func)
E �[1;32m 99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
E �[1;32m 100�[0m libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
E �[0;32m--> 101�[0m result �[38;5;241m=�[39m func(�[38;5;241m
�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)
E �[1;32m 102�[0m libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
E �[1;32m 103�[0m �[38;5;28;01mreturn�[39;00m result
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:470�[0m, in �[0;36mread_parquet�[0;34m()�[0m
E �[1;32m 463�[0m �[38;5;28;01mif�[39;00m filters �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:
E �[1;32m 464�[0m warnings�[38;5;241m.�[39mwarn(
E �[1;32m 465�[0m �[38;5;124m"�[39m�[38;5;124mParquet row-group filtering is only supported with �[39m�[38;5;124m"�[39m
E �[1;32m 466�[0m �[38;5;124m"�[39m�[38;5;124m'�[39m�[38;5;124mengine=cudf�[39m�[38;5;124m'�[39m�[38;5;124m. Use pandas or pyarrow API directly �[39m�[38;5;124m"�[39m
E �[1;32m 467�[0m �[38;5;124m"�[39m�[38;5;124mfor full CPU-based filtering functionality.�[39m�[38;5;124m"�[39m
E �[1;32m 468�[0m )
E �[0;32m--> 470�[0m �[38;5;28;01mreturn�[39;00m _parquet_to_frame(
E �[1;32m 471�[0m filepaths_or_buffers,
E �[1;32m 472�[0m engine,
E �[1;32m 473�[0m �[38;5;241m�[39margs,
E �[1;32m 474�[0m columns�[38;5;241m=�[39mcolumns,
E �[1;32m 475�[0m row_groups�[38;5;241m=�[39mrow_groups,
E �[1;32m 476�[0m skiprows�[38;5;241m=�[39mskiprows,
E �[1;32m 477�[0m num_rows�[38;5;241m=�[39mnum_rows,
E �[1;32m 478�[0m strings_to_categorical�[38;5;241m=�[39mstrings_to_categorical,
E �[1;32m 479�[0m use_pandas_metadata�[38;5;241m=�[39muse_pandas_metadata,
E �[1;32m 480�[0m partition_keys�[38;5;241m=�[39mpartition_keys,
E �[1;32m 481�[0m partition_categories�[38;5;241m=�[39mpartition_categories,
E �[1;32m 482�[0m �[38;5;241m
�[39m�[38;5;241m�[39mkwargs,
E �[1;32m 483�[0m )
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36minner�[0;34m()�[0m
E �[1;32m 98�[0m �[38;5;129m@wraps�[39m(func)
E �[1;32m 99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m
�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
E �[1;32m 100�[0m libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
E �[0;32m--> 101�[0m result �[38;5;241m=�[39m func(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)
E �[1;32m 102�[0m libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
E �[1;32m 103�[0m �[38;5;28;01mreturn�[39;00m result
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:499�[0m, in �[0;36m_parquet_to_frame�[0;34m()�[0m
E �[1;32m 486�[0m �[38;5;129m@_cudf_nvtx_annotate�[39m
E �[1;32m 487�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_parquet_to_frame�[39m(
E �[1;32m 488�[0m paths_or_buffers,
E �[0;32m (...)�[0m
E �[1;32m 496�[0m �[38;5;66;03m# If this is not a partitioned read, only need�[39;00m
E �[1;32m 497�[0m �[38;5;66;03m# one call to _read_parquet�[39;00m
E �[1;32m 498�[0m �[38;5;28;01mif�[39;00m �[38;5;129;01mnot�[39;00m partition_keys:
E �[0;32m--> 499�[0m �[38;5;28;01mreturn�[39;00m _read_parquet(
E �[1;32m 500�[0m paths_or_buffers,
E �[1;32m 501�[0m �[38;5;241m
�[39margs,
E �[1;32m 502�[0m row_groups�[38;5;241m=�[39mrow_groups,
E �[1;32m 503�[0m �[38;5;241m�[39m�[38;5;241m�[39mkwargs,
E �[1;32m 504�[0m )
E �[1;32m 506�[0m �[38;5;66;03m# For partitioned data, we need a distinct read for each�[39;00m
E �[1;32m 507�[0m �[38;5;66;03m# unique set of partition keys. Therefore, we start by�[39;00m
E �[1;32m 508�[0m �[38;5;66;03m# aggregating all paths with matching keys using a dict�[39;00m
E �[1;32m 509�[0m plan �[38;5;241m=�[39m {}
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36minner�[0;34m()�[0m
E �[1;32m 98�[0m �[38;5;129m@wraps�[39m(func)
E �[1;32m 99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
E �[1;32m 100�[0m libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
E �[0;32m--> 101�[0m result �[38;5;241m=�[39m func(�[38;5;241m
�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)
E �[1;32m 102�[0m libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
E �[1;32m 103�[0m �[38;5;28;01mreturn�[39;00m result
E
E File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:576�[0m, in �[0;36m_read_parquet�[0;34m()�[0m
E �[1;32m 560�[0m �[38;5;129m@_cudf_nvtx_annotate�[39m
E �[1;32m 561�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_read_parquet�[39m(
E �[1;32m 562�[0m filepaths_or_buffers,
E �[0;32m (...)�[0m
E �[1;32m 573�[0m �[38;5;66;03m# Simple helper function to dispatch between�[39;00m
E �[1;32m 574�[0m �[38;5;66;03m# cudf and pyarrow to read parquet data�[39;00m
E �[1;32m 575�[0m �[38;5;28;01mif�[39;00m engine �[38;5;241m==�[39m �[38;5;124m"�[39m�[38;5;124mcudf�[39m�[38;5;124m"�[39m:
E �[0;32m--> 576�[0m �[38;5;28;01mreturn�[39;00m libparquet�[38;5;241m.�[39mread_parquet(
E �[1;32m 577�[0m filepaths_or_buffers,
E �[1;32m 578�[0m columns�[38;5;241m=�[39mcolumns,
E �[1;32m 579�[0m row_groups�[38;5;241m=�[39mrow_groups,
E �[1;32m 580�[0m skiprows�[38;5;241m=�[39mskiprows,
E �[1;32m 581�[0m num_rows�[38;5;241m=�[39mnum_rows,
E �[1;32m 582�[0m strings_to_categorical�[38;5;241m=�[39mstrings_to_categorical,
E �[1;32m 583�[0m use_pandas_metadata�[38;5;241m=�[39muse_pandas_metadata,
E �[1;32m 584�[0m )
E �[1;32m 585�[0m �[38;5;28;01melse�[39;00m:
E �[1;32m 586�[0m �[38;5;28;01mreturn�[39;00m cudf�[38;5;241m.�[39mDataFrame�[38;5;241m.�[39mfrom_arrow(
E �[1;32m 587�[0m pq�[38;5;241m.�[39mParquetDataset(filepaths_or_buffers)�[38;5;241m.�[39mread_pandas(
E �[1;32m 588�[0m columns�[38;5;241m=�[39mcolumns, �[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m*�[39mkwargs
E �[1;32m 589�[0m )
E �[1;32m 590�[0m )
E
E File �[0;32mcudf/_lib/parquet.pyx:113�[0m, in �[0;36mcudf._lib.parquet.read_parquet�[0;34m()�[0m
E
E File �[0;32mcudf/_lib/parquet.pyx:173�[0m, in �[0;36mcudf._lib.parquet.read_parquet�[0;34m()�[0m
E
E �[0;31mRuntimeError�[0m: CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory
E RuntimeError: CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory

../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError
----------------------------- Captured stderr call -----------------------------
2022-11-13 20:03:44,269 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-13 20:03:44,285 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-13 20:03:44,294 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-13 20:03:44,296 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2022-11-13 20:03:46,160 - distributed.worker - WARNING - Compute Failed
Key: _sample_row_group-13262d08-5b73-464e-8bbb-d31496fbcf62
Function: execute_task
args: ((<function apply at 0x7f14fa9301f0>, <function _sample_row_group at 0x7f144e2a2940>, ['/tmp/input/criteo/day_0.parquet', <fsspec.implementations.local.LocalFileSystem object at 0x7f14f05c9d60>], (<class 'dict'>, [['cpu', False], ['memory_usage', True]])))
kwargs: {}
Exception: "RuntimeError('CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 36 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 217 39%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 17 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 43 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 173 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 4 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 27 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 107 80 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 347 164 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 345 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 29 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 81 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 17 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 114 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 15 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 111 54%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 114 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 161 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 84 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 16 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 162 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 753 352 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 110 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 24 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 10 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 33 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 41 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 55 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%

TOTAL 13122 6391 51%

=================== 2 failed, 1 passed in 150.46s (0:02:30) ====================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)
___________________________________ summary ____________________________________
ERROR: test-gpu: commands failed
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins8425161798457689183.sh

@karlhigley
Copy link
Contributor

On the plus side, they're no longer timing out. On the other side, one of these looks like a legit error happening in the notebook. I don't think it's likely to be coming from the dataloader though. (The other one is an OOM, which is probably just multiple test runs happening in parallel and using too much total memory.)

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #728 of commit 67901f953a1d6647005791af1e02f6f50da95dfe, no merge conflicts.
Running as SYSTEM
Setting status of 67901f953a1d6647005791af1e02f6f50da95dfe to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/623/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/728/*:refs/remotes/origin/pr/728/* # timeout=10
 > git rev-parse 67901f953a1d6647005791af1e02f6f50da95dfe^{commit} # timeout=10
Checking out Revision 67901f953a1d6647005791af1e02f6f50da95dfe (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67901f953a1d6647005791af1e02f6f50da95dfe # timeout=10
Commit message: "Merge branch 'main' into add-dl-unit-tests"
 > git rev-list --no-walk 625779d30baf7907aef279658001679884965bb1 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins15653656885872434109.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.9,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.9,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1765701542'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-fkrfq5pm
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-fkrfq5pm
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 5c6746f3ca00891a9096f8123218ca3242b794dd
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+14.g5c6746f) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+14.g5c6746f) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+14.g5c6746f) (0.3.0+12.g78ecddd)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.7.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 45.4 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+14.g5c6746f) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+14.g5c6746f-py3-none-any.whl size=94014 sha256=51e5803c82151132a4b5552cf62d7c9edcef503070c6eda7af06e75cec9b419a
  Stored in directory: /tmp/pip-ephem-wheel-cache-w0wc0nde/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+14.g5c6746f
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-_rktx3l2
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-_rktx3l2
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit ceed7c5b99cdb58e99f626bc796e09df7c71a0a4
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+44.gceed7c5b) (0.3.0+12.g78ecddd)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.7.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+44.gceed7c5b) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+44.gceed7c5b-py3-none-any.whl size=364958 sha256=983287e6e38d90227830bd67edd4aee05e0255d23e96fde8eb44e9b259484d6f
  Stored in directory: /tmp/pip-ephem-wheel-cache-shhwhyec/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+44.gceed7c5b
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-kg0wggel
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-kg0wggel
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit ba4c14159a8e858c8998d4158a4376e65a8fa266
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (0.0.2)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+4.gba4c1415) (0.3.0+12.g78ecddd)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+4.gba4c1415) (1.8.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.19.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+4.gba4c1415) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.7.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+4.gba4c1415) (4.0.0)
Building wheels for collected packages: nvtabular
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+4.gba4c1415-cp38-cp38-linux_x86_64.whl size=257596 sha256=6ef78f2996381532bb854634627f33f6ea517258c1d93c50798c7ac1b9560ec9
  Stored in directory: /tmp/pip-ephem-wheel-cache-eg7ie0vq/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc
Successfully built nvtabular
Installing collected packages: nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed nvtabular-1.6.0+4.gba4c1415
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-wpwjrimv
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-wpwjrimv
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit eb606d54fa2ddcbb7e4d0e6501ab2eb418c7fba9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+11.geb606d5) (3.19.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+11.geb606d5) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.7.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+11.geb606d5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+11.geb606d5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+11.geb606d5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+11.geb606d5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+11.geb606d5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+11.geb606d5-py3-none-any.whl size=118619 sha256=fa22d59515d15b922fa0cd8f0cf62fa0aaffeb6350267db776a842ba20f4a431
  Stored in directory: /tmp/pip-ephem-wheel-cache-5gd3muhf/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+11.geb606d5
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 3 items

tests/unit/test_version.py . [ 33%]
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py . [ 66%]
tests/unit/examples/test_scaling_criteo_merlin_models.py . [100%]

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py 10 4 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py 356 213 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py 99 45 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py 195 108 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py 5 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py 122 15 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py 55 31 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py 141 29 79%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py 99 24 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py 344 136 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py 17 1 94%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py 22 1 95%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py 12 2 83%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py 21 11 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py 101 26 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py 141 102 28%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/booking/dataset.py 127 100 21%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py 45 37 18%
.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py 147 56 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py 57 38 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py 181 97 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py 61 38 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py 21 2 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py 347 161 54%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py 37 8 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py 127 108 15%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py 45 35 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py 603 344 43%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py 80 29 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py 190 81 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py 18 5 72%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py 14 5 64%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py 62 16 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py 15 1 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py 379 113 70%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py 21 17 19%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py 57 27 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py 40 15 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py 70 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py 44 28 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py 49 8 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py 99 75 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py 108 55 49%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py 117 58 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py 173 127 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py 169 89 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py 30 4 87%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py 29 7 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py 46 31 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py 35 12 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py 115 99 14%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py 241 107 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py 244 108 56%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py 426 151 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py 175 121 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py 104 69 34%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py 50 19 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py 280 71 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/distributed/backend.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py 60 36 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py 39 4 90%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py 458 154 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py 268 148 45%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py 9 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py 13 1 92%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py 115 57 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py 198 82 59%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py 756 340 55%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py 16 6 62%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py 67 43 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py 78 44 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py 10 2 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py 123 90 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py 91 51 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py 147 107 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py 9 2 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py 78 41 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py 37 22 41%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py 27 17 37%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py 98 63 36%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py 207 108 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py 68 22 68%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py 7 1 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py 59 33 44%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py 35 19 46%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py 73 31 58%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py 101 54 47%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py 63 14 78%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py 111 77 31%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py 435 346 20%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py 43 28 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py 17 6 65%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py 282 212 25%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py 158 75 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py 7 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py 85 12 86%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py 69 48 30%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py 34 22 35%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py 209 141 33%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py 3 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py 38 18 53%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py 26 19 27%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py 10 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py 31 8 74%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py 118 90 24%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py 27 24 11%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py 101 31 69%
.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py 90 39 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py 2 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py 20 4 80%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py 306 7 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py 190 34 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py 209 52 75%
.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py 82 6 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py 6 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py 4 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/dictarray.py 122 49 60%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py 46 20 57%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py 23 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py 26 1 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py 24 8 67%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py 77 3 96%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py 118 19 84%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py 79 7 91%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py 45 28 38%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py 51 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py 67 18 73%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py 50 1 98%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py 65 19 71%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py 11 2 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py 1 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py 11 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py 54 4 93%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py 140 47 66%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py 16 8 50%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py 49 12 76%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py 143 74 48%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py 268 210 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py 0 0 100%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py 72 13 82%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py 22 20 9%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py 113 14 88%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py 37 29 22%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py 10 6 40%
.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py 32 13 59%

TOTAL 14943 6877 54%

======================== 3 passed in 406.62s (0:06:46) =========================
/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
___________________________________ summary ____________________________________
test-gpu: commands succeeded
congratulations :)
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_merlin] $ /bin/bash /tmp/jenkins9890551578521404205.sh

@jperez999 jperez999 merged commit cf54f6a into NVIDIA-Merlin:main Nov 15, 2022
jperez999 added a commit that referenced this pull request Nov 16, 2022
Co-authored-by: Karl Higley <kmhigley@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Infrastructure update ci
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants