Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create standard Merlin dtypes in the merlin.dtypes module #170

Merged
merged 48 commits into from
Jan 13, 2023
Merged

Create standard Merlin dtypes in the merlin.dtypes module #170

merged 48 commits into from
Jan 13, 2023

Conversation

karlhigley
Copy link
Contributor

@karlhigley karlhigley commented Nov 20, 2022 •
edited
Loading

Original Merlin Dtypes proposal slides

This PR creates a merlin.dtype module that provides translations between a standard set of Merlin dtype descriptors and the dtype descriptors from external frameworks that we commonly use or interact with: Numpy, Pandas, cuDF, Tensorflow, Torch, and Triton. It also provides a mechanism for registering new sets of dtypes with mappings to Merlin dtypes. External frameworks are treated as optional and the relevant dtype mappings are only loaded/registered if the necessary packages are available (in order to avoid creating hard dependencies on e.g. deep learning frameworks or inference servers in Merlin Core.)

This has several benefits over using Numpy dtypes:

  • We can directly represent unknown dtypes with a non-None value without defaulting them to object (like Numpy does)
  • We can capture information in our dtypes that's used in Pandas/cuDF extension dtypes
  • We're not forced to do lossy translations between external frameworks with Numpy as a constraining intermediary

It also lets us limit the set of dtype translation tables required to one, since we can use these Merlin dtypes as a Rosetta stone for translating between any pair of frameworks that each provide a Merlin dtype mapping.

There are related PRs adjusting downstream libraries to use these dtypes:

Some of the code this PR lays the groundwork for replacing:

@karlhigley karlhigley added the enhancement New feature or request label Nov 20, 2022
@karlhigley karlhigley self-assigned this Nov 20, 2022
@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit 8612b749e3505ed1f29f89d149246045ff9c6c2b, no merge conflicts.
Running as SYSTEM
Setting status of 8612b749e3505ed1f29f89d149246045ff9c6c2b to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/302/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse 8612b749e3505ed1f29f89d149246045ff9c6c2b^{commit} # timeout=10
Checking out Revision 8612b749e3505ed1f29f89d149246045ff9c6c2b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8612b749e3505ed1f29f89d149246045ff9c6c2b # timeout=10
Commit message: "Prototype: `merlin.dtypes` module"
 > git rev-list --no-walk f48634c2e2a4f3f6139125a62666569e5a86a3e2 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins7303117623640608503.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.8.0+12.g8612b74.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.13,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.13,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.8.0+12.g8612b74,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='4036843137'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 401 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ..............                          [ 18%]

tests/unit/io/test_io.py ............................................... [ 30%]

................................................................         [ 46%]

tests/unit/schema/test_column_schemas.py .......................         [ 52%]

tests/unit/schema/test_schema.py .............                           [ 55%]

tests/unit/schema/test_schema_io.py .................................... [ 64%]

........................................................................ [ 82%]

........................................................                 [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 34741 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43603 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44193 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43979 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33943 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43759 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    212    40%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     141     68    52%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     46      0   100%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     190     18    91%

merlin/schema/schema.py                     209     33    84%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4711   1468    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 401 passed, 1 skipped, 25 warnings in 66.24s (0:01:06) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins10255857105331112861.sh

@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/core/review/pr-170

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit d4c0bbe0f297eb5545350405ecfdb7d904e00757, no merge conflicts.
Running as SYSTEM
Setting status of d4c0bbe0f297eb5545350405ecfdb7d904e00757 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/304/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse d4c0bbe0f297eb5545350405ecfdb7d904e00757^{commit} # timeout=10
Checking out Revision d4c0bbe0f297eb5545350405ecfdb7d904e00757 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d4c0bbe0f297eb5545350405ecfdb7d904e00757 # timeout=10
Commit message: "Incorporate Merlin dtypes into `ColumnSchema`"
 > git rev-list --no-walk 4ed1f991e3613a3c6ad4e7cd4203dd8e07b03983 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins6751400296538677198.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.8.0+13.gd4c0bbe.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.13,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.13,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.8.0+13.gd4c0bbe,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@e5b7351deb9e4885c4038aa0bbc9f146d8477a0e,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3180480230'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 401 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ..............                          [ 18%]

tests/unit/io/test_io.py ............................................... [ 30%]

................................................................         [ 46%]

tests/unit/schema/test_column_schemas.py .......................         [ 52%]

tests/unit/schema/test_schema.py .............                           [ 55%]

tests/unit/schema/test_schema_io.py .................................... [ 64%]

........................................................................ [ 82%]

........................................................                 [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44105 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44985 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43455 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38879 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33409 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 46689 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    212    40%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     141     68    52%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     61      3    95%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     181     11    94%

merlin/schema/schema.py                     211     33    84%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4719   1464    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 401 passed, 1 skipped, 25 warnings in 65.39s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins16788087018977261025.sh

@karlhigley karlhigley mentioned this pull request Nov 21, 2022
Merged
angmc
angmc reviewed Nov 22, 2022
View reviewed changes
merlin/dtype/__init__.py Outdated
int64 = DType("int64", ElementType.Int, 64, signed=True)
uint32 = DType("uint32", ElementType.UInt, 32)
uint64 = DType("uint64", ElementType.UInt, 64)
float32 = DType("float32", ElementType.Float, 32, signed=True)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would we have to include integers and floats at lower bits like 16?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's actually required in order to make the NVT tests pass. The first commit on this branch was a rough sketch, but I'm fleshing it out as I test it with the other Merlin repos.

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit 35c27d895be0ae4772a4d21c7b8df3fb353be4d9, no merge conflicts.
Running as SYSTEM
Setting status of 35c27d895be0ae4772a4d21c7b8df3fb353be4d9 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/308/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse 35c27d895be0ae4772a4d21c7b8df3fb353be4d9^{commit} # timeout=10
Checking out Revision 35c27d895be0ae4772a4d21c7b8df3fb353be4d9 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 35c27d895be0ae4772a4d21c7b8df3fb353be4d9 # timeout=10
Commit message: "Extract dtypes and mappings from `merlin.dtype` module's init"
 > git rev-list --no-walk 8aeed38a7350b13d161e9295d6a47df505177dcc # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins13142435459942551623.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.8.0+14.g35c27d8.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.14,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.14,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.8.0+14.g35c27d8,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3591952188'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 401 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py .........F....                          [ 18%]

tests/unit/io/test_io.py ............................................... [ 30%]

................................................................         [ 46%]

tests/unit/schema/test_column_schemas.py .......................         [ 52%]

tests/unit/schema/test_schema.py .............                           [ 55%]

tests/unit/schema/test_schema_io.py .................................... [ 64%]

........................................................................ [ 82%]

........................................................                 [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=================================== FAILURES ===================================

________________ test_dtypes_know_if_theyre_ragged[shape0-True] ________________


shape = (None, 10), is_ragged = True


@pytest.mark.parametrize(
    "shape, is_ragged",
    [((None, 10), True), ((10, None), True), ((10, 10), False), ((10,), False), (None, False)],
)
def test_dtypes_know_if_theyre_ragged(shape, is_ragged):
    merlin_type = dtype(int, shape)





  assert merlin_type.is_ragged == is_ragged





E       AssertionError: assert False == True

E        +  where False = DType(name='int64', elemtype=<ElementType.Int: 'int'>, elemsize=64, signed=True, shape=(None, 10)).is_ragged


tests/unit/dtypes/test_module.py:93: AssertionError

=============================== warnings summary ===============================

tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 34931 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38323 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 41495 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 40605 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 39639 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 37441 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    212    40%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     145     71    51%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     21      4    81%

merlin/dtype/dtypes.py                       40      0   100%

merlin/dtype/mappings.py                     23      1    96%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     181     11    94%

merlin/schema/schema.py                     211     33    84%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4746   1469    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

======= 1 failed, 400 passed, 1 skipped, 25 warnings in 66.39s (0:01:06) =======

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins2056519556964715211.sh

@karlhigley karlhigley force-pushed the feature/merlin-dtypes branch 2 times, most recently from 5ad1de6 to 8b1d3b9 Compare November 22, 2022 16:05
@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit 5ad1de6e14e175322c9ee2a9ba291759d352d130, no merge conflicts.
Running as SYSTEM
Setting status of 5ad1de6e14e175322c9ee2a9ba291759d352d130 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/309/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse 5ad1de6e14e175322c9ee2a9ba291759d352d130^{commit} # timeout=10
 > git rev-parse origin/5ad1de6e14e175322c9ee2a9ba291759d352d130^{commit} # timeout=10
 > git rev-parse 5ad1de6e14e175322c9ee2a9ba291759d352d130^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_core] $ /bin/bash /tmp/jenkins2869037470862343249.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit 8b1d3b93e4bc137211630266da30ae7a44310705, no merge conflicts.
Running as SYSTEM
Setting status of 8b1d3b93e4bc137211630266da30ae7a44310705 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/310/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse 8b1d3b93e4bc137211630266da30ae7a44310705^{commit} # timeout=10
Checking out Revision 8b1d3b93e4bc137211630266da30ae7a44310705 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8b1d3b93e4bc137211630266da30ae7a44310705 # timeout=10
Commit message: "Extract dtypes and mappings from `merlin.dtype` module's init"
 > git rev-list --no-walk 35c27d895be0ae4772a4d21c7b8df3fb353be4d9 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins6449837873153555086.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.8.0+14.g8b1d3b9.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.14,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.14,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.8.0+14.g8b1d3b9,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1953082168'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 401 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ..............                          [ 18%]

tests/unit/io/test_io.py ............................................... [ 30%]

................................................................         [ 46%]

tests/unit/schema/test_column_schemas.py .......................         [ 52%]

tests/unit/schema/test_schema.py .............                           [ 55%]

tests/unit/schema/test_schema_io.py .................................... [ 64%]

........................................................................ [ 82%]

........................................................                 [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42975 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 45585 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 36997 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 45541 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38965 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 34115 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    212    40%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     145     71    51%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     17      1    94%

merlin/dtype/base.py                         29      3    90%

merlin/dtype/dtypes.py                       17      0   100%

merlin/dtype/mappings.py                      7      0   100%

merlin/dtype/registry.py                     20      1    95%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     181     11    94%

merlin/schema/schema.py                     211     33    84%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4752   1469    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 401 passed, 1 skipped, 25 warnings in 65.56s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins15455814251405018577.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit f5749a049e38f5dc5319efc80462f56d4fa79d93, no merge conflicts.
Running as SYSTEM
Setting status of f5749a049e38f5dc5319efc80462f56d4fa79d93 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/311/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse f5749a049e38f5dc5319efc80462f56d4fa79d93^{commit} # timeout=10
Checking out Revision f5749a049e38f5dc5319efc80462f56d4fa79d93 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f5749a049e38f5dc5319efc80462f56d4fa79d93 # timeout=10
Commit message: "Minor fixes"
 > git rev-list --no-walk 8b1d3b93e4bc137211630266da30ae7a44310705 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins18328646729320720429.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.8.0+15.gf5749a0.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.14,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.14,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.8.0+15.gf5749a0,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3211937915'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 401 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ..............                          [ 18%]

tests/unit/io/test_io.py ............................................... [ 30%]

................................................................         [ 46%]

tests/unit/schema/test_column_schemas.py .......................         [ 52%]

tests/unit/schema/test_schema.py .............                           [ 55%]

tests/unit/schema/test_schema_io.py .................................... [ 64%]

........................................................................ [ 82%]

........................................................                 [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 45837 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38345 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 35817 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 37339 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 36657 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43217 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    212    40%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     145     71    51%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     17      1    94%

merlin/dtype/base.py                         29      3    90%

merlin/dtype/dtypes.py                       17      0   100%

merlin/dtype/mappings.py                      7      0   100%

merlin/dtype/registry.py                     20      1    95%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     181     11    94%

merlin/schema/schema.py                     211     33    84%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4752   1469    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 401 passed, 1 skipped, 25 warnings in 65.88s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins5265509445592630837.sh

@karlhigley
Prototype: merlin.dtypes module
982ed40 
This proof-of-concept code demonstrates how Merlin could create its own dtypes that would allow us to standardize the types used in Merlin schemas, while maintaining compatibility with the variety of libraries we use and integrate with that have their own dtypes.
@karlhigley
Incorporate Merlin dtypes into ColumnSchema
eb4f3b0
@karlhigley
Extract dtypes and mappings from merlin.dtype module's init
39f0b9a
@karlhigley
Minor fixes
7dcdc56
@karlhigley
Add ElementUnit enum and Dtype.elemunit field
ac0d0f7
@karlhigley
Add better error messages to DType.to() when mappings not found
998f1f3
@karlhigley
Add convenience methods to_python/to_numpy to DType` class
56f35af
@karlhigley
Extend the available datetime64 types to include units
abe2036
@karlhigley
Add the datetime types to the Numpy dtype mapping
7f84f70
@karlhigley
Prefer np.dtype(...) objects over dtype classes in Numpy mapping
5fbf5c6
@karlhigley karlhigley force-pushed the feature/merlin-dtypes branch from f5749a0 to 5fbf5c6 Compare November 28, 2022 17:12
@karlhigley karlhigley marked this pull request as ready for review November 30, 2022 18:53
@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit c334a45a2d9071bc2e1ce2ea56657428e3888a77, no merge conflicts.
Running as SYSTEM
Setting status of c334a45a2d9071bc2e1ce2ea56657428e3888a77 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/325/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse c334a45a2d9071bc2e1ce2ea56657428e3888a77^{commit} # timeout=10
Checking out Revision c334a45a2d9071bc2e1ce2ea56657428e3888a77 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c334a45a2d9071bc2e1ce2ea56657428e3888a77 # timeout=10
Commit message: "Placate pylint about modules not being callable"
 > git rev-list --no-walk 2e830b574a6c8d62fb763ac68ed8a6a0aabe59a1 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins2012920387679441559.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.9.0+51.gc334a45.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.19,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.19,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.9.0+51.gc334a45,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1994970959'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 403 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ....                                    [ 16%]

tests/unit/io/test_io.py ............................................... [ 27%]

................................................................         [ 43%]

tests/unit/schema/test_column_schemas.py ............................... [ 51%]

.                                                                        [ 51%]

tests/unit/schema/test_schema.py .............                           [ 54%]

tests/unit/schema/test_schema_io.py .................................... [ 63%]

........................................................................ [ 81%]

...........................................................              [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "


../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 40971 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 46561 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42673 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 35411 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 39655 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 40281 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    210    41%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     143     71    50%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     27      3    89%

merlin/dtype/base.py                         50     12    76%

merlin/dtype/dtypes.py                       25      0   100%

merlin/dtype/mapping.py                      28      2    93%

merlin/dtype/mappings/init.py             6      0   100%

merlin/dtype/mappings/numpy.py               21      3    86%

merlin/dtype/mappings/pandas.py               9      3    67%

merlin/dtype/mappings/python.py               4      0   100%

merlin/dtype/mappings/tf.py                  10      3    70%

merlin/dtype/mappings/torch.py               11      3    73%

merlin/dtype/mappings/triton.py               9      3    67%

merlin/dtype/registry.py                     25      5    80%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     183     11    94%

merlin/schema/schema.py                     216     29    87%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4892   1495    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 403 passed, 1 skipped, 33 warnings in 65.14s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins11218378626961101458.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit 5eebc794fae039059532daefdaf6f949f9d0cbe9, no merge conflicts.
Running as SYSTEM
Setting status of 5eebc794fae039059532daefdaf6f949f9d0cbe9 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/326/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse 5eebc794fae039059532daefdaf6f949f9d0cbe9^{commit} # timeout=10
Checking out Revision 5eebc794fae039059532daefdaf6f949f9d0cbe9 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5eebc794fae039059532daefdaf6f949f9d0cbe9 # timeout=10
Commit message: "Fix import formatting"
 > git rev-list --no-walk c334a45a2d9071bc2e1ce2ea56657428e3888a77 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins7022155620132443478.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.9.0+52.g5eebc79.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.19,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.19,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.9.0+52.g5eebc79,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='530624342'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 403 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ....                                    [ 16%]

tests/unit/io/test_io.py ............................................... [ 27%]

................................................................         [ 43%]

tests/unit/schema/test_column_schemas.py ............................... [ 51%]

.                                                                        [ 51%]

tests/unit/schema/test_schema.py .............                           [ 54%]

tests/unit/schema/test_schema_io.py .................................... [ 63%]

........................................................................ [ 81%]

...........................................................              [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "


../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38031 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 39275 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44737 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 37091 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 41341 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33519 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    210    41%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     143     71    50%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     27      3    89%

merlin/dtype/base.py                         50     12    76%

merlin/dtype/dtypes.py                       25      0   100%

merlin/dtype/mapping.py                      28      2    93%

merlin/dtype/mappings/init.py             1      0   100%

merlin/dtype/mappings/numpy.py               21      3    86%

merlin/dtype/mappings/pandas.py               9      3    67%

merlin/dtype/mappings/python.py               4      0   100%

merlin/dtype/mappings/tf.py                  10      3    70%

merlin/dtype/mappings/torch.py               11      3    73%

merlin/dtype/mappings/triton.py               9      3    67%

merlin/dtype/registry.py                     25      5    80%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     183     11    94%

merlin/schema/schema.py                     216     29    87%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4887   1495    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 403 passed, 1 skipped, 33 warnings in 65.44s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins15799818355770895674.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit ff0d88ff1ea34fe604e2d44d56956080af15102b, no merge conflicts.
Running as SYSTEM
Setting status of ff0d88ff1ea34fe604e2d44d56956080af15102b to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/327/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse ff0d88ff1ea34fe604e2d44d56956080af15102b^{commit} # timeout=10
Checking out Revision ff0d88ff1ea34fe604e2d44d56956080af15102b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ff0d88ff1ea34fe604e2d44d56956080af15102b # timeout=10
Commit message: "Merge branch 'main' into feature/merlin-dtypes"
 > git rev-list --no-walk 5eebc794fae039059532daefdaf6f949f9d0cbe9 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins15753114492566715908.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.9.0+54.gff0d88f.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.19,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.19,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.9.0+54.gff0d88f,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3932957584'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 403 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ....                                    [ 16%]

tests/unit/io/test_io.py ............................................... [ 27%]

................................................................         [ 43%]

tests/unit/schema/test_column_schemas.py ............................... [ 51%]

.                                                                        [ 51%]

tests/unit/schema/test_schema.py .............                           [ 54%]

tests/unit/schema/test_schema_io.py .................................... [ 63%]

........................................................................ [ 81%]

...........................................................              [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "


../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33303 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43259 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42611 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43089 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 39213 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38377 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    210    41%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     143     71    50%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtype/init.py                     27      3    89%

merlin/dtype/base.py                         50     12    76%

merlin/dtype/dtypes.py                       25      0   100%

merlin/dtype/mapping.py                      28      2    93%

merlin/dtype/mappings/init.py             1      0   100%

merlin/dtype/mappings/numpy.py               21      3    86%

merlin/dtype/mappings/pandas.py               9      3    67%

merlin/dtype/mappings/python.py               4      0   100%

merlin/dtype/mappings/tf.py                  10      3    70%

merlin/dtype/mappings/torch.py               11      3    73%

merlin/dtype/mappings/triton.py               9      3    67%

merlin/dtype/registry.py                     25      5    80%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     183     11    94%

merlin/schema/schema.py                     216     29    87%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4887   1495    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 403 passed, 1 skipped, 33 warnings in 65.35s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins1036540167726043697.sh

@karlhigley karlhigley added this to the Merlin 22.12 milestone Nov 30, 2022
@karlhigley
Adjust the syntax for using dtypes
c2c1f96 
Importing `merlin.dtypes as mn` gives us the familiar `library_abbrev.dtype()` syntax without the complexity of a callable module, which was tripping up pylint in downstream libraries. This is a bit clearer anyway.
@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit c2c1f96762abc011a33827123445bb0ca4118d0d, no merge conflicts.
Running as SYSTEM
Setting status of c2c1f96762abc011a33827123445bb0ca4118d0d to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/328/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_core/workspace
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse c2c1f96762abc011a33827123445bb0ca4118d0d^{commit} # timeout=10
Checking out Revision c2c1f96762abc011a33827123445bb0ca4118d0d (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c2c1f96762abc011a33827123445bb0ca4118d0d # timeout=10
Commit message: "Adjust the syntax for using dtypes"
 > git rev-list --no-walk ff0d88ff1ea34fe604e2d44d56956080af15102b # timeout=10
[workspace] $ /bin/bash /tmp/jenkins3445148113283854823.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.9.0+55.gc2c1f96.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.9.0+55.gc2c1f96,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='466422395'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 403 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ....                                    [ 16%]

tests/unit/io/test_io.py ............................................... [ 27%]

................................................................         [ 43%]

tests/unit/schema/test_column_schemas.py ............................... [ 51%]

.                                                                        [ 51%]

tests/unit/schema/test_schema.py .............                           [ 54%]

tests/unit/schema/test_schema_io.py .................................... [ 63%]

........................................................................ [ 81%]

...........................................................              [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "


../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:869: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 36191 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 46429 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 36929 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 37377 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 32783 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43691 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    210    41%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     143     71    50%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtypes/init.py                    24      3    88%

merlin/dtypes/aliases.py                     25      0   100%

merlin/dtypes/base.py                        50     12    76%

merlin/dtypes/mapping.py                     28      2    93%

merlin/dtypes/mappings/init.py            1      0   100%

merlin/dtypes/mappings/numpy.py              21      3    86%

merlin/dtypes/mappings/pandas.py              9      3    67%

merlin/dtypes/mappings/python.py              4      0   100%

merlin/dtypes/mappings/tf.py                 10      3    70%

merlin/dtypes/mappings/torch.py              11      3    73%

merlin/dtypes/mappings/triton.py              9      3    67%

merlin/dtypes/registry.py                    25      5    80%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        603     69    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     183     11    94%

merlin/schema/schema.py                     216     29    87%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4884   1495    69%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 403 passed, 1 skipped, 33 warnings in 66.20s (0:01:06) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins8797092822222021126.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit d961c5c9d82d6b5fb269eb23b9b1d4e7e3dcbcd5, no merge conflicts.
Running as SYSTEM
Setting status of d961c5c9d82d6b5fb269eb23b9b1d4e7e3dcbcd5 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/329/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_core/workspace
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse d961c5c9d82d6b5fb269eb23b9b1d4e7e3dcbcd5^{commit} # timeout=10
Checking out Revision d961c5c9d82d6b5fb269eb23b9b1d4e7e3dcbcd5 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d961c5c9d82d6b5fb269eb23b9b1d4e7e3dcbcd5 # timeout=10
Commit message: "Merge branch 'main' into feature/merlin-dtypes"
 > git rev-list --no-walk c2c1f96762abc011a33827123445bb0ca4118d0d # timeout=10
[workspace] $ /bin/bash /tmp/jenkins15745005988310180386.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.9.0+57.gd961c5c.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.9.0+57.gd961c5c,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3013946943'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 403 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ....                                    [ 16%]

tests/unit/io/test_io.py ............................................... [ 27%]

................................................................         [ 43%]

tests/unit/schema/test_column_schemas.py ............................... [ 51%]

.                                                                        [ 51%]

tests/unit/schema/test_schema.py .............                           [ 54%]

tests/unit/schema/test_schema_io.py .................................... [ 63%]

........................................................................ [ 81%]

...........................................................              [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "


../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:579: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/io/test_io.py::test_parquet_aggregate_files[True]

tests/unit/io/test_io.py::test_parquet_aggregate_files[True]

tests/unit/io/test_io.py::test_parquet_aggregate_files[False]

tests/unit/io/test_io.py::test_parquet_aggregate_files[False]

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:245: FutureWarning: gather_statistics is now deprecated and will be ignored.

warnings.warn(


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 40323 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42263 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 37649 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 35125 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 46043 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33365 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    210    41%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      55     15    73%

merlin/dag/executors.py                     143     71    50%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtypes/init.py                    24      3    88%

merlin/dtypes/aliases.py                     25      0   100%

merlin/dtypes/base.py                        50     12    76%

merlin/dtypes/mapping.py                     28      2    93%

merlin/dtypes/mappings/init.py            1      0   100%

merlin/dtypes/mappings/numpy.py              21      3    86%

merlin/dtypes/mappings/pandas.py              9      3    67%

merlin/dtypes/mappings/python.py              4      0   100%

merlin/dtypes/mappings/tf.py                 10      3    70%

merlin/dtypes/mappings/torch.py              11      3    73%

merlin/dtypes/mappings/triton.py              9      3    67%

merlin/dtypes/registry.py                    25      5    80%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        624     70    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     183     11    94%

merlin/schema/schema.py                     216     29    87%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4905   1496    70%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 403 passed, 1 skipped, 37 warnings in 66.30s (0:01:06) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins18442453578320897003.sh

@karlhigley karlhigley changed the title Create standard Merlin dtypes in the merlin.dtype module Create standard Merlin dtypes in the merlin.dtypes module Dec 2, 2022
@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #170 of commit 4049ab0f1c682e1f33799b578ffe362fea619fe1, no merge conflicts.
Running as SYSTEM
Setting status of 4049ab0f1c682e1f33799b578ffe362fea619fe1 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_core/330/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_core/workspace
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/170/*:refs/remotes/origin/pr/170/* # timeout=10
 > git rev-parse 4049ab0f1c682e1f33799b578ffe362fea619fe1^{commit} # timeout=10
Checking out Revision 4049ab0f1c682e1f33799b578ffe362fea619fe1 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4049ab0f1c682e1f33799b578ffe362fea619fe1 # timeout=10
Commit message: "Adjust syntax to `md.dtype()`, add unknown dtype, clean up preprocessing"
 > git rev-list --no-walk d961c5c9d82d6b5fb269eb23b9b1d4e7e3dcbcd5 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins15349145291400317719.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_core/core/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_core/core/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_core/core/.tox/.tmp/package/1/merlin-core-0.9.0+58.g4049ab0.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.22,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.22,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.9.0+58.g4049ab0,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.1.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1224598970'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, cov-4.0.0, xdist-3.1.0
collected 403 items / 1 skipped

tests/unit/core/test_dispatch.py ....                                    [  0%]

tests/unit/core/test_protocols.py .........                              [  3%]

tests/unit/core/test_version.py .                                        [  3%]

tests/unit/dag/test_base_operator.py ....                                [  4%]

tests/unit/dag/test_column_selector.py ..............................    [ 11%]

tests/unit/dag/test_dictarray.py ...                                     [ 12%]

tests/unit/dag/test_executors.py ..                                      [ 13%]

tests/unit/dag/test_graph.py ....                                        [ 14%]

tests/unit/dag/ops/test_selection.py ....                                [ 15%]

tests/unit/dtypes/test_module.py ....                                    [ 16%]

tests/unit/io/test_io.py ............................................... [ 27%]

................................................................         [ 43%]

tests/unit/schema/test_column_schemas.py ............................... [ 51%]

.                                                                        [ 51%]

tests/unit/schema/test_schema.py .............                           [ 54%]

tests/unit/schema/test_schema_io.py .................................... [ 63%]

........................................................................ [ 81%]

...........................................................              [ 96%]

tests/unit/schema/test_tags.py .......                                   [ 98%]

tests/unit/utils/test_utils.py ........                                  [100%]


=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "


../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,


../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 6 files. Did not have enough partitions to create 7 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 8 files. Did not have enough partitions to create 9 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 8 files. Did not have enough partitions to create 10 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 8 files. Did not have enough partitions to create 11 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 13 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 14 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 15 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 16 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 17 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 18 files.

warnings.warn(


tests/unit/io/test_io.py::test_io_partitions_push

/var/jenkins_home/workspace/merlin_core/core/merlin/io/dataset.py:866: UserWarning: Only creating 12 files. Did not have enough partitions to create 19 files.

warnings.warn(


tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:579: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]


tests/unit/io/test_io.py::test_parquet_aggregate_files[True]

tests/unit/io/test_io.py::test_parquet_aggregate_files[True]

tests/unit/io/test_io.py::test_parquet_aggregate_files[False]

tests/unit/io/test_io.py::test_parquet_aggregate_files[False]

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:245: FutureWarning: gather_statistics is now deprecated and will be ignored.

warnings.warn(


tests/unit/schema/test_column_schemas.py::test_column_schema_tags_normalize

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_schema_io.py::test_json_serialization_with_embedded_dicts

tests/unit/schema/test_schema_io.py::test_merlin_to_proto_to_json_to_merlin

tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/schema/test_tags.py::test_tagset_atomizes_compound_tags

/var/jenkins_home/workspace/merlin_core/core/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.TEXT_TOKENIZED have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.TEXT: 'text'>, <Tags.TOKENIZED: 'tokenized'>].

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 41641 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 35417 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44805 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 43363 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44417 instead

warnings.warn(


tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38663 instead

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                      Stmts   Miss  Cover


merlin/core/init.py                       2      0   100%

merlin/core/_version.py                     354    205    42%

merlin/core/compat.py                        10      4    60%

merlin/core/dispatch.py                     356    210    41%

merlin/core/protocols.py                     99     45    55%

merlin/core/utils.py                        195     56    71%

merlin/dag/init.py                        5      0   100%

merlin/dag/base_operator.py                 122     20    84%

merlin/dag/dictarray.py                      56     15    73%

merlin/dag/executors.py                     144     71    51%

merlin/dag/graph.py                          99     35    65%

merlin/dag/node.py                          344    161    53%

merlin/dag/ops/init.py                    4      0   100%

merlin/dag/ops/concat_columns.py             17      4    76%

merlin/dag/ops/selection.py                  22      0   100%

merlin/dag/ops/subset_columns.py             12      4    67%

merlin/dag/ops/subtraction.py                21     11    48%

merlin/dag/selector.py                      101      6    94%

merlin/dtypes/init.py                    17      1    94%

merlin/dtypes/aliases.py                     26      0   100%

merlin/dtypes/base.py                        51     12    76%

merlin/dtypes/mapping.py                     47      3    94%

merlin/dtypes/mappings/init.py            1      0   100%

merlin/dtypes/mappings/cudf.py               16      3    81%

merlin/dtypes/mappings/numpy.py               5      0   100%

merlin/dtypes/mappings/pandas.py             11      3    73%

merlin/dtypes/mappings/python.py              6      0   100%

merlin/dtypes/mappings/tf.py                 11      3    73%

merlin/dtypes/mappings/torch.py              11      3    73%

merlin/dtypes/mappings/triton.py              9      3    67%

merlin/dtypes/registry.py                    29      5    83%

merlin/io/init.py                         4      0   100%

merlin/io/avro.py                            88     88     0%

merlin/io/csv.py                             57      6    89%

merlin/io/dask.py                           181     53    71%

merlin/io/dataframe_engine.py                61      5    92%

merlin/io/dataframe_iter.py                  21      1    95%

merlin/io/dataset.py                        347     54    84%

merlin/io/dataset_engine.py                  37      8    78%

merlin/io/fsspec_utils.py                   127    108    15%

merlin/io/hugectr.py                         45     35    22%

merlin/io/parquet.py                        624     70    89%

merlin/io/shuffle.py                         38     12    68%

merlin/io/worker.py                          80     66    18%

merlin/io/writer.py                         190     52    73%

merlin/io/writer_factory.py                  18      4    78%

merlin/schema/init.py                     2      0   100%

merlin/schema/io/init.py                  0      0   100%

merlin/schema/io/proto_utils.py              20      4    80%

merlin/schema/io/schema_bp.py               306      5    98%

merlin/schema/io/tensorflow_metadata.py     185     12    94%

merlin/schema/schema.py                     217     29    87%

merlin/schema/tags.py                        82      1    99%


TOTAL                                      4933   1496    70%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/io/test_avro.py:34: could not import 'uavro': No module named 'uavro'

============ 403 passed, 1 skipped, 37 warnings in 65.45s (0:01:05) ============

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins3766378790914040585.sh

@karlhigley karlhigley mentioned this pull request Dec 5, 2022
Merged
@karlhigley karlhigley requested review from jperez999 and angmc December 5, 2022 03:36
@karlhigley karlhigley mentioned this pull request Dec 5, 2022
Merged
karlhigley
karlhigley commented Dec 5, 2022
View reviewed changes
merlin/dag/executors.py

# TODO: Add a utility that condenses the known methods of fetching dtypes
# from series/arrays into a single function, so that Tensorflow specific
# code doesn't leak into the executors
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The need for this new utility was already latent in the existing code, but it becomes a lot clearer once you have a cross-framework way to translate dtypes. Chose not to tackle it here since that would expand the scope of this already somewhat large PR.

@karlhigley
Copy link
Contributor Author

The tests on the downstream repos that run on this PR don't run against the downstream repo branches that adjust for these changes, so they'll continue failing here even if the PR is fine. The tests on the downstream repo PRs do run against this branch, so we should be looking at them to determine when we're ready to merge this.

jperez999
jperez999 reviewed Dec 5, 2022
View reviewed changes
merlin/dtypes/aliases.py
string = DType("str", ElementType.String)
boolean = DType("bool", ElementType.Bool)
object_ = DType("object", ElementType.Object)
unknown = DType("unknown", ElementType.Unknown)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dtype is essential. replacing none with our own logic, so we can catch.

@jperez999
Copy link
Collaborator

Also didn't have a place to comment, but we may want to refactor the executor functionality to use the merlin dtypes to detect whether your a list or not. I dont know if that should happen by probing the dtype directly (for is_list and is_ragged) or by refactoring is_list_dtype.

oliverholworthy
oliverholworthy reviewed Dec 6, 2022
View reviewed changes
merlin/dtypes/base.py Outdated
"""

name: str
elemtype: ElementType
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one small question about naming. If users of this DType class are expected to use the element type, might be clearer here to use the full name for this element_type and same for the below two attrs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's a reasonable suggestion. These started out following the itemtype convention from numpy, which is how this ended up as elemtype. There's something to be said for brevity, but I agree it's clearer to use the full words for the attribute names.

@karlhigley karlhigley merged commit a93c332 into NVIDIA-Merlin:main Jan 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oliverholworthy oliverholworthy oliverholworthy approved these changes

@jperez999 jperez999 jperez999 approved these changes

@rjzamora rjzamora Awaiting requested review from rjzamora

@nv-alaiacano nv-alaiacano Awaiting requested review from nv-alaiacano

@angmc angmc Awaiting requested review from angmc

Assignees

@karlhigley karlhigley

Labels
enhancement New feature or request
Projects
None yet
Milestone
Merlin 22.12
Development

Successfully merging this pull request may close these issues.
5 participants
@karlhigley @nvidia-merlin-bot @jperez999 @oliverholworthy @angmc