Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: Segfault with incorrect ports passing to MultiPortModulesStage #953

Closed
2 tasks done
bsuryadevara opened this issue May 19, 2023 · 1 comment · Fixed by #1829
Closed
2 tasks done

[BUG]: Segfault with incorrect ports passing to MultiPortModulesStage #953

bsuryadevara opened this issue May 19, 2023 · 1 comment · Fixed by #1829
Assignees
@dagardner-nv
Labels
bug Something isn't working
Milestone
24.03 - MNMG

Comments

@bsuryadevara
Copy link
Contributor

Version

23.07

Which installation method(s) does this occur on?

No response

Describe the bug.

MulitPortModuleStage used in the pipeline with incorrect ports is hitting segmentation fault.

here are the scenarios that raises ValueError as the provided ports are not matched with module ports, followed by segfault:

module_input_ports = ["input_0", "input_1"]
provided_input_ports_stage = ["input_0", "input_2"]

module_input_ports = ["input_0", "input_1", "input_2"]
provided_input_ports_stage = ["input_0", "input_2"]

Here in this case it just throws assertion error, but not segfault

module_input_ports = ["input_0"]
provided_input_ports_stage = ["input_0", "input_1"]

====================================================================================== 1 passed, 1 warning in 4.82s =======================================================================================
F20230516 18:19:25.776926 3928398 edge_holder.hpp:61] A node was destructed which still had dependent connections. Nodes must be kept alive while dependent connections are still active
*** Check failure stack trace: ***
    @     0x7fd483c7ef8d  google::LogMessage::Fail()
    @     0x7fd483c82e67  google::LogMessage::SendToLog()
    @     0x7fd483c7ea55  google::LogMessage::Flush()
    @     0x7fd483c7feaa  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fd4847bf57e  mrc::edge::EdgeHolder<>::~EdgeHolder()
    @     0x7fd4847c1fb9  mrc::pymrc::PythonNode<>::~PythonNode()
    @     0x7fd4847a30e7  mrc::segment::Runnable<>::~Runnable()
    @     0x7fd48479a622  std::_Rb_tree<>::_M_erase()
    @     0x7fd48479af6b  mrc::pymrc::PythonSegmentModule::~PythonSegmentModule()
    @     0x7fd480746eda  (unknown)
    @     0x7fd480791f27  (unknown)
    @     0x7fd4848c0700  (unknown)
    @     0x5623d7f4a8d8  frame_dealloc
    @     0x5623d7f5abca  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f5ac2e  tb_dealloc
    @     0x5623d7f604ea  BaseException_dealloc
    @     0x7fd4869f7eaa  FutureObj_clear
    @     0x7fd4869f7f30  TaskObj_clear
    @     0x5623d7f34795  gc_collect_main
    @     0x5623d7fe898e  gc_collect_with_callback
    @     0x5623d801f2a3  PyGC_Collect
    @     0x5623d801ea52  Py_FinalizeEx.localalias
    @     0x5623d801f578  Py_Exit
    @     0x5623d801c98b  handle_system_exit
Aborted (core dumped)

Minimum reproducible example

import cudf
from morpheus.pipeline.pipeline import Pipeline
from morpheus.stages.general.multi_port_modules_stage import MultiPortModulesStage
from morpheus.stages.input.in_memory_source_stage import InMemorySourceStage
from morpheus.stages.output.in_memory_sink_stage import InMemorySinkStage
from morpheus.config import Config
from morpheus.config import CppConfig

def test_multi_port_pipeline():
    CppConfig.set_should_use_cpp(True)
    config = Config()

    df = cudf.DataFrame({
            'col1': [1, 2, 3, 4, 5],
            'col2': [1.1, 2.2, 3.3, 4.4, 5.5],
            'col3': ['a', 'b', 'c', 'd', 'e'],
            'col4': [True, False, True, False, True]
        })

    pipe = Pipeline(config)

    expected_input_ports = ["input_0", "input_1"]
    provided_input_ports = ["input_0", "input_1", "input_2"]

    multiplexer_module_conf = {
        "module_id": "multiplexer",
        "namespace": "morpheus_test",
        "module_name": "multiplexer",
        "input_ports": expected_input_ports,
        "output_port": "output"
    }

    source_stage_1 = pipe.add_stage(InMemorySourceStage(config, [df]))
    source_stage_2 = pipe.add_stage(InMemorySourceStage(config, [df]))

    mux_stage = pipe.add_stage(
        MultiPortModulesStage(config,
                              multiplexer_module_conf,
                              input_ports=provided_input_ports,
                              output_ports=["output"]))

    pipe.add_edge(source_stage_1, mux_stage.input_ports[0])
    pipe.add_edge(source_stage_2, mux_stage.input_ports[1])
    pipe.add_edge(source_stage_2, mux_stage.input_ports[2])
    sink_stage = pipe.add_stage(InMemorySinkStage(config))

    pipe.add_edge(mux_stage, sink_stage)

    pipe.run()


if __name__ == "__main__":
    test_multi_port_pipeline()


### Relevant log output

```shell
/home/bsuryadevara/mambaforge/envs/morpheus/lib/python3.10/site-packages/numba/core/errors.py:175: UserWarning: Insufficiently recent colorama version found. Numba requires colorama >= 0.3.9
  warnings.warn(msg)
WARNING: Logging before InitGoogleLogging() is written to STDERR
E20230519 09:51:24.374199 2220292 builder.cpp:75] Exception during segment initializer. Segment name: main, Segment Rank: 0. Exception message:
ValueError: Provided input ports do not match module input ports. Module: ['input_0', 'input_1'], Provided: ['input_0', 'input_1', 'input_2'].

At:
  /home/bsuryadevara/Github/morpheus/morpheus/stages/general/multi_port_modules_stage.py(105): _validate_ports
  /home/bsuryadevara/Github/morpheus/morpheus/stages/general/multi_port_modules_stage.py(117): _build
  /home/bsuryadevara/Github/morpheus/morpheus/pipeline/stream_wrapper.py(327): build
  /home/bsuryadevara/Github/morpheus/morpheus/pipeline/stream_wrapper.py(349): build
  /home/bsuryadevara/Github/morpheus/morpheus/pipeline/pipeline.py(257): inner_build
E20230519 09:51:24.374406 2220292 controller.cpp:64] exception caught while performing update - this is fatal - issuing kill
E20230519 09:51:24.374444 2220292 context.cpp:124] rank: 0; size: 1; tid: 140668200547904; fid: 0x7fefa8041000: set_exception issued; issuing kill to current runnable. Exception msg: ValueError: Provided input ports do not match module input ports. Module: ['input_0', 'input_1'], Provided: ['input_0', 'input_1', 'input_2'].

At:
  /home/bsuryadevara/Github/morpheus/morpheus/stages/general/multi_port_modules_stage.py(105): _validate_ports
  /home/bsuryadevara/Github/morpheus/morpheus/stages/general/multi_port_modules_stage.py(117): _build
  /home/bsuryadevara/Github/morpheus/morpheus/pipeline/stream_wrapper.py(327): build
  /home/bsuryadevara/Github/morpheus/morpheus/pipeline/stream_wrapper.py(349): build
  /home/bsuryadevara/Github/morpheus/morpheus/pipeline/pipeline.py(257): inner_build
E20230519 09:51:24.374497 2220292 manager.cpp:89] error detected on controller
E20230519 09:51:24.374667 2220237 runner.cpp:189] Runner::await_join - an exception was caught while awaiting on one or more contexts/instances - rethrowing
Traceback (most recent call last):
  File "/home/bsuryadevara/Github/morpheus/tests/test.py", line 54, in <module>
    test_multi_port_pipeline()
  File "/home/bsuryadevara/Github/morpheus/tests/test.py", line 50, in test_multi_port_pipeline
    pipe.run()
  File "/home/bsuryadevara/Github/morpheus/morpheus/pipeline/pipeline.py", line 598, in run
    asyncio.run(self.run_async())
  File "/home/bsuryadevara/mambaforge/envs/morpheus/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/bsuryadevara/mambaforge/envs/morpheus/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/bsuryadevara/Github/morpheus/morpheus/pipeline/pipeline.py", line 576, in run_async
    await self.join()
  File "/home/bsuryadevara/Github/morpheus/morpheus/pipeline/pipeline.py", line 327, in join
    await self._mrc_executor.join_async()
  File "/home/bsuryadevara/Github/morpheus/morpheus/pipeline/pipeline.py", line 257, in inner_build
    stage.build(builder)
  File "/home/bsuryadevara/Github/morpheus/morpheus/pipeline/stream_wrapper.py", line 349, in build
    dep.build(builder, do_propagate=do_propagate)
  File "/home/bsuryadevara/Github/morpheus/morpheus/pipeline/stream_wrapper.py", line 327, in build
    out_ports_pair = self._build(builder, in_ports_pairs)
  File "/home/bsuryadevara/Github/morpheus/morpheus/stages/general/multi_port_modules_stage.py", line 117, in _build
    self._validate_ports(module)
  File "/home/bsuryadevara/Github/morpheus/morpheus/stages/general/multi_port_modules_stage.py", line 105, in _validate_ports
    raise ValueError(f"Provided input ports do not match module input ports. Module: {input_ids}, "
ValueError: Provided input ports do not match module input ports. Module: ['input_0', 'input_1'], Provided: ['input_0', 'input_1', 'input_2'].
F20230519 09:51:24.591825 2220237 edge_holder.hpp:61] A node was destructed which still had dependent connections. Nodes must be kept alive while dependent connections are still active
*** Check failure stack trace: ***
Aborted (core dumped)

Full env printout

<details><summary>Click here to see environment details</summary><pre>
     
     **git***
     commit 0628b74ee20083a8bf0dcb0936a6417549b5220e (HEAD -> 937-fea-migrate-generic-components-implemented-for-azure-ad-ai-workflow, upstream/pull-request/939, origin/937-fea-migrate-generic-components-implemented-for-azure-ad-ai-workflow)
     Author: Bhargav Suryadevara <bsuryadevara@nvidia.com>
     Date:   Wed May 17 19:15:46 2023 -0500
     
     updated imports
     **git submodules***
     a006d1e7ca2856cb290225632e05daa7030ea7aa external/morpheus-visualizations (v23.07.00a-5-ga006d1e)
     c80e5de07299a0b2b4911633b9c3eda9548bfed0 external/utilities (v23.07.00a-12-gc80e5de)
     
     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=22.04
     DISTRIB_CODENAME=jammy
     DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"
     PRETTY_NAME="Ubuntu 22.04.2 LTS"
     NAME="Ubuntu"
     VERSION_ID="22.04"
     VERSION="22.04.2 LTS (Jammy Jellyfish)"
     VERSION_CODENAME=jammy
     ID=ubuntu
     ID_LIKE=debian
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     UBUNTU_CODENAME=jammy
     VARIANT="NVIDIA IT Managed"
     Linux 0673044-lcedt 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
     
     ***GPU Information***
     Fri May 19 09:53:13 2023
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  Quadro RTX 8000     On   | 00000000:15:00.0 Off |                  Off |
     | 33%   31C    P8     7W / 260W |      6MiB / 49152MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     |   1  Quadro RTX 8000     On   | 00000000:2D:00.0  On |                  Off |
     | 34%   42C    P8    24W / 260W |    952MiB / 49152MiB |     40%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+
     
     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |    0   N/A  N/A      8260      G   /usr/lib/xorg/Xorg                  4MiB |
     |    1   N/A  N/A      8260      G   /usr/lib/xorg/Xorg                346MiB |
     |    1   N/A  N/A      8397      G   /usr/bin/gnome-shell              130MiB |
     |    1   N/A  N/A    356592      G   ...464593094905334240,131072       26MiB |
     |    1   N/A  N/A    356626      G   ...veSuggestionsOnlyOnDemand       64MiB |
     |    1   N/A  N/A    356628      G   ...879205004037129708,131072      240MiB |
     |    1   N/A  N/A    356822      G   ...476594505442620371,131072       48MiB |
     |    1   N/A  N/A    791418      G   ...RendererForSitePerProcess       39MiB |
     |    1   N/A  N/A   1375536      G   gnome-control-center               27MiB |
     +-----------------------------------------------------------------------------+
     
     ***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Address sizes:                   46 bits physical, 48 bits virtual
     Byte Order:                      Little Endian
     CPU(s):                          12
     On-line CPU(s) list:             0-11
     Vendor ID:                       GenuineIntel
     Model name:                      Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
     CPU family:                      6
     Model:                           85
     Thread(s) per core:              2
     Core(s) per socket:              6
     Socket(s):                       1
     Stepping:                        4
     CPU max MHz:                     3700.0000
     CPU min MHz:                     1200.0000
     BogoMIPS:                        6800.00
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities
     Virtualization:                  VT-x
     L1d cache:                       192 KiB (6 instances)
     L1i cache:                       192 KiB (6 instances)
     L2 cache:                        6 MiB (6 instances)
     L3 cache:                        19.3 MiB (1 instance)
     NUMA node(s):                    1
     NUMA node0 CPU(s):               0-11
     Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
     Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
     Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Meltdown:          Mitigation; PTI
     Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
     Vulnerability Retbleed:          Mitigation; IBRS
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable
     
     ***CMake***
     /home/bsuryadevara/mambaforge/envs/morpheus/bin/cmake
     cmake version 3.24.3
     
     CMake suite maintained and supported by Kitware (kitware.com/cmake).
     
     ***g++***
     /home/bsuryadevara/mambaforge/envs/morpheus/bin/g++
     g++ (conda-forge gcc 11.2.0-16) 11.2.0
     Copyright (C) 2021 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
     
     
     ***nvcc***
     /home/bsuryadevara/mambaforge/envs/morpheus/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2022 NVIDIA Corporation
     Built on Wed_Sep_21_10:33:58_PDT_2022
     Cuda compilation tools, release 11.8, V11.8.89
     Build cuda_11.8.r11.8/compiler.31833905_0
     
     ***Python***
     /home/bsuryadevara/mambaforge/envs/morpheus/bin/python
     Python 3.10.11
     
     ***Environment Variables***
     PATH                            : /home/bsuryadevara/mambaforge/envs/morpheus/bin:/home/bsuryadevara/mambaforge/condabin:/home/bsuryadevara/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/opt/p4v/bin:/opt/pycharm-community-2020.3.3/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    : /home/bsuryadevara/mambaforge/envs/morpheus
     PYTHON_PATH                     :
     
     ***conda packages***
     /home/bsuryadevara/mambaforge/condabin/conda
     # packages in environment at /home/bsuryadevara/mambaforge/envs/morpheus:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                 conda_forge    conda-forge
     _openmp_mutex             4.5                       2_gnu    conda-forge
     _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
     aiobotocore               2.5.0                    pypi_0    pypi
     aiofiles                  22.1.0                   pypi_0    pypi
     aioitertools              0.11.0                   pypi_0    pypi
     aiosqlite                 0.19.0                   pypi_0    pypi
     alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
     alembic                   1.10.4             pyhd8ed1ab_0    conda-forge
     anyio                     3.6.2                    pypi_0    pypi
     appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
     argon2-cffi               21.3.0                   pypi_0    pypi
     argon2-cffi-bindings      21.2.0                   pypi_0    pypi
     arrow                     1.2.3                    pypi_0    pypi
     arrow-cpp                 10.0.1          ha770c72_14_cuda    conda-forge
     asn1crypto                1.5.1              pyhd8ed1ab_0    conda-forge
     asttokens                 2.2.1                    pypi_0    pypi
     async-lru                 2.0.2                    pypi_0    pypi
     atk-1.0                   2.38.0               hd4edc92_1    conda-forge
     attrs                     22.2.0             pyh71513ae_0    conda-forge
     autoconf                  2.71            pl5321h2b4cb7a_1    conda-forge
     automake                  1.16.5          pl5321ha770c72_0    conda-forge
     aws-c-auth                0.6.26               h987a71b_2    conda-forge
     aws-c-cal                 0.5.21               h48707d8_2    conda-forge
     aws-c-common              0.8.14               h0b41bf4_0    conda-forge
     aws-c-compression         0.2.16               h03acc5a_5    conda-forge
     aws-c-event-stream        0.2.20               h00877a2_4    conda-forge
     aws-c-http                0.7.6                hf342b9f_0    conda-forge
     aws-c-io                  0.13.19              h5b20300_3    conda-forge
     aws-c-mqtt                0.8.6               hc4349f7_12    conda-forge
     aws-c-s3                  0.2.7                h909e904_1    conda-forge
     aws-c-sdkutils            0.1.9                h03acc5a_0    conda-forge
     aws-checksums             0.1.14               h03acc5a_5    conda-forge
     aws-crt-cpp               0.19.8              hf7fbfca_12    conda-forge
     aws-sdk-cpp               1.10.57              h17c43bd_8    conda-forge
     awscli                    1.27.130                 pypi_0    pypi
     babel                     2.12.1             pyhd8ed1ab_1    conda-forge
     backcall                  0.2.0                    pypi_0    pypi
     benchmark                 1.6.1                h9c3ff4c_0    conda-forge
     binutils                  2.36.1               hdd6e379_2    conda-forge
     binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
     binutils_linux-64         2.36                hf3e587d_10    conda-forge
     bleach                    6.0.0                    pypi_0    pypi
     blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
     bokeh                     3.1.0              pyhd8ed1ab_0    conda-forge
     boost-cpp                 1.74.0               h75c5d50_8    conda-forge
     boto3                     1.26.124                 pypi_0    pypi
     botocore                  1.29.130                 pypi_0    pypi
     breathe                   4.34.0                   pypi_0    pypi
     brotli                    1.0.9                    pypi_0    pypi
     brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
     bzip2                     1.0.8                h7f98852_4    conda-forge
     c-ares                    1.18.1               h7f98852_0    conda-forge
     c-compiler                1.5.2                h0b41bf4_0    conda-forge
     ca-certificates           2022.12.7            ha878542_0    conda-forge
     cachetools                5.0.0              pyhd8ed1ab_0    conda-forge
     cairo                     1.16.0            ha61ee94_1014    conda-forge
     ccache                    4.8                  hfcdc2af_0    conda-forge
     certifi                   2023.5.7                 pypi_0    pypi
     cffi                      1.15.1          py310h255011f_3    conda-forge
     charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
     clang                     14.0.6               ha770c72_0    conda-forge
     clang-14                  14.0.6          default_h2e3cab8_0    conda-forge
     clang-format              14.0.6          default_h2e3cab8_0    conda-forge
     clang-format-14           14.0.6          default_h2e3cab8_0    conda-forge
     clang-tools               14.0.6          default_h2e3cab8_0    conda-forge
     clangdev                  14.0.6          default_h2e3cab8_0    conda-forge
     clangxx                   14.0.6          default_ha074a59_0    conda-forge
     click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
     cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
     cmake                     3.24.3               h816a3e0_0    conda-forge
     colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
     comm                      0.1.3                    pypi_0    pypi
     configargparse            1.5.3              pyhd8ed1ab_0    conda-forge
     configparser              5.3.0              pyhd8ed1ab_0    conda-forge
     contourpy                 1.0.7           py310hdf3cbec_0    conda-forge
     coverage                  7.2.5           py310h2372a71_0    conda-forge
     cryptography              40.0.2          py310h34c0648_0    conda-forge
     cubinlinker               0.2.2           py310hf09951c_0    rapidsai
     cuda-cccl                 11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-command-line-tools   11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-compiler             11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-cudart               11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-cudart-dev           11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-cuobjdump            11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-cupti                11.8.87                       0    nvidia/label/cuda-11.8.0
     cuda-cuxxfilt             11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-documentation        11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-driver-dev           11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-gdb                  11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-libraries            11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-libraries-dev        11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-memcheck             11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-nsight               11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-nsight-compute       11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-nvcc                 11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-nvdisasm             11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-nvml-dev             11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-nvprof               11.8.87                       0    nvidia/label/cuda-11.8.0
     cuda-nvprune              11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-nvrtc                11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-nvrtc-dev            11.8.89                       0    nvidia/label/cuda-11.8.0
     cuda-nvtx                 11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-nvvp                 11.8.87                       0    nvidia/label/cuda-11.8.0
     cuda-profiler-api         11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-python               11.8.1          py310h3fd9d12_0    nvidia
     cuda-sanitizer-api        11.8.86                       0    nvidia/label/cuda-11.8.0
     cuda-toolkit              11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-tools                11.8.0                        0    nvidia/label/cuda-11.8.0
     cuda-visual-tools         11.8.0                        0    nvidia/label/cuda-11.8.0
     cudatoolkit               11.8.0              h37601d7_11    conda-forge
     cudf                      23.02.00        cuda_11_py310_g5ad4a85b9d_0    rapidsai
     cupy                      11.6.0          py310h9216885_0    conda-forge
     curl                      8.0.1                h588be90_0    conda-forge
     cxx-compiler              1.5.2                hf52228f_0    conda-forge
     cyrus-sasl                2.1.27               h9033bb2_6    conda-forge
     cython                    0.29.24         py310h122e73d_1    conda-forge
     cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
     dask                      2023.4.1                 pypi_0    pypi
     dask-core                 2022.7.0           pyhd8ed1ab_0    conda-forge
     databricks-cli            0.17.6             pyhd8ed1ab_0    conda-forge
     datacompy                 0.8.4              pyhd8ed1ab_0    conda-forge
     debugpy                   1.6.7                    pypi_0    pypi
     defusedxml                0.7.1                    pypi_0    pypi
     dill                      0.3.6              pyhd8ed1ab_1    conda-forge
     distributed               2023.4.1                 pypi_0    pypi
     distro                    1.8.0              pyhd8ed1ab_0    conda-forge
     dlpack                    0.5                  h9c3ff4c_0    conda-forge
     docker-compose            1.29.2          py310hff52083_2    conda-forge
     docker-py                 5.0.3              pyhd8ed1ab_4    conda-forge
     docker-pycreds            0.4.0                      py_0    conda-forge
     dockerpty                 0.4.1                      py_0    conda-forge
     docopt                    0.6.2                      py_1    conda-forge
     docutils                  0.17.1                   pypi_0    pypi
     doxygen                   1.9.2                hb166930_0    conda-forge
     entrypoints               0.4                pyhd8ed1ab_0    conda-forge
     exceptiongroup            1.1.1              pyhd8ed1ab_0    conda-forge
     executing                 1.2.0                    pypi_0    pypi
     exhale                    0.3.6                    pypi_0    pypi
     expat                     2.5.0                hcb278e6_1    conda-forge
     faker                     12.3.0             pyhd8ed1ab_0    conda-forge
     fastavro                  1.7.3           py310h1fa729e_0    conda-forge
     fastjsonschema            2.16.3                   pypi_0    pypi
     fastrlock                 0.8             py310hd8f1fbe_3    conda-forge
     flake8                    6.0.0              pyhd8ed1ab_0    conda-forge
     flake8-docstrings         1.7.0                    pypi_0    pypi
     flake8-tuple              0.4.1                    pypi_0    pypi
     flask                     2.3.2              pyhd8ed1ab_0    conda-forge
     flatbuffers               2.0.8                hcb278e6_1    conda-forge
     font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
     font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
     font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
     font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
     fontconfig                2.14.2               h14ed4e7_0    conda-forge
     fonts-conda-ecosystem     1                             0    conda-forge
     fonts-conda-forge         1                             0    conda-forge
     fqdn                      1.5.1                    pypi_0    pypi
     freetype                  2.12.1               hca18f0e_1    conda-forge
     fribidi                   1.0.10               h36c2ea0_0    conda-forge
     fsspec                    2023.4.0           pyh1a96a4e_0    conda-forge
     gcc                       11.2.0              h702ea55_10    conda-forge
     gcc_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
     gcc_linux-64              11.2.0              h39a9532_10    conda-forge
     gdk-pixbuf                2.42.10              h05c8ddd_0    conda-forge
     gds-tools                 1.4.0.31                      0    nvidia/label/cuda-11.8.0
     gettext                   0.21.1               h27087fc_0    conda-forge
     gevent                    22.10.2                  pypi_0    pypi
     geventhttpclient          2.0.9                    pypi_0    pypi
     gflags                    2.2.2             he1b5a44_1004    conda-forge
     giflib                    5.2.1                h0b41bf4_3    conda-forge
     git                       2.40.1          pl5321h86e50cf_0    conda-forge
     git-lfs                   3.2.0                ha770c72_0    conda-forge
     gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
     gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
     glog                      0.6.0                h6f12383_0    conda-forge
     gmock                     1.10.0               h4bd325d_7    conda-forge
     gputil                    1.4.0              pyh9f0ad1d_0    conda-forge
     graphite2                 1.3.13            h58526e2_1001    conda-forge
     graphviz                  7.1.0                h2e5815a_0    conda-forge
     greenlet                  2.0.2           py310heca2aa9_0    conda-forge
     grpc-cpp                  1.51.1               h27aab58_3    conda-forge
     grpcio                    1.51.1          py310heca2aa9_3    conda-forge
     gtest                     1.10.0               h4bd325d_7    conda-forge
     gtk2                      2.24.33              h90689f9_2    conda-forge
     gts                       0.7.6                h64030ff_2    conda-forge
     gunicorn                  20.1.0          py310hff52083_3    conda-forge
     gxx                       11.2.0              h702ea55_10    conda-forge
     gxx_impl_linux-64         11.2.0              h82a94d6_16    conda-forge
     gxx_linux-64              11.2.0              hacbe6df_10    conda-forge
     harfbuzz                  6.0.0                h8e241bc_0    conda-forge
     icu                       70.1                 h27087fc_0    conda-forge
     idna                      3.4                pyhd8ed1ab_0    conda-forge
     imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
     importlib-metadata        5.2.0              pyha770c72_0    conda-forge
     importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
     include-what-you-use      0.18                 h27087fc_0    conda-forge
     iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
     ipykernel                 6.22.0                   pypi_0    pypi
     ipython                   8.13.1                   pypi_0    pypi
     ipython-genutils          0.2.0                    pypi_0    pypi
     ipywidgets                8.0.6                    pypi_0    pypi
     isoduration               20.11.0                  pypi_0    pypi
     isort                     5.12.0             pyhd8ed1ab_1    conda-forge
     itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
     jedi                      0.18.2                   pypi_0    pypi
     jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
     jmespath                  1.0.1                    pypi_0    pypi
     joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
     jpeg                      9e                   h0b41bf4_3    conda-forge
     json5                     0.9.11                   pypi_0    pypi
     jsonpointer               2.3                      pypi_0    pypi
     jsonschema                4.17.3                   pypi_0    pypi
     jupyter-client            8.2.0                    pypi_0    pypi
     jupyter-core              4.12.0                   pypi_0    pypi
     jupyter-events            0.6.3                    pypi_0    pypi
     jupyter-lsp               2.1.0                    pypi_0    pypi
     jupyter-server            2.5.0                    pypi_0    pypi
     jupyter-server-fileid     0.9.0                    pypi_0    pypi
     jupyter-server-terminals  0.4.4                    pypi_0    pypi
     jupyter-server-ydoc       0.8.0                    pypi_0    pypi
     jupyter-ydoc              0.2.4                    pypi_0    pypi
     jupyterlab                4.0.0                    pypi_0    pypi
     jupyterlab-pygments       0.2.2                    pypi_0    pypi
     jupyterlab-server         2.22.1                   pypi_0    pypi
     jupyterlab-widgets        3.0.7                    pypi_0    pypi
     kafka-python              2.0.2                    pypi_0    pypi
     kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
     keyutils                  1.6.1                h166bdaf_0    conda-forge
     krb5                      1.20.1               h81ceb04_0    conda-forge
     lcms2                     2.15                 hfd0df8a_0    conda-forge
     ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
     lerc                      4.0.0                h27087fc_0    conda-forge
     libabseil                 20230125.0      cxx17_hcb278e6_1    conda-forge
     libarrow                  10.0.1          h255618e_14_cuda    conda-forge
     libblas                   3.9.0           16_linux64_openblas    conda-forge
     libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
     libbrotlidec              1.0.9                h166bdaf_8    conda-forge
     libbrotlienc              1.0.9                h166bdaf_8    conda-forge
     libcblas                  3.9.0           16_linux64_openblas    conda-forge
     libclang                  14.0.6          default_h2e3cab8_0    conda-forge
     libclang-cpp              14.0.6          default_h2e3cab8_0    conda-forge
     libclang-cpp14            14.0.6          default_h2e3cab8_0    conda-forge
     libclang13                14.0.6          default_h3a83d3e_0    conda-forge
     libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
     libcublas                 11.11.3.6                     0    nvidia/label/cuda-11.8.0
     libcublas-dev             11.11.3.6                     0    nvidia/label/cuda-11.8.0
     libcudf                   23.02.00        cuda11_g5ad4a85b9d_0    rapidsai
     libcufft                  10.9.0.58                     0    nvidia/label/cuda-11.8.0
     libcufft-dev              10.9.0.58                     0    nvidia/label/cuda-11.8.0
     libcufile                 1.4.0.31                      0    nvidia/label/cuda-11.8.0
     libcufile-dev             1.4.0.31                      0    nvidia/label/cuda-11.8.0
     libcurand                 10.3.0.86                     0    nvidia/label/cuda-11.8.0
     libcurand-dev             10.3.0.86                     0    nvidia/label/cuda-11.8.0
     libcurl                   8.0.1                h588be90_0    conda-forge
     libcusolver               11.4.1.48                     0    nvidia/label/cuda-11.8.0
     libcusolver-dev           11.4.1.48                     0    nvidia/label/cuda-11.8.0
     libcusparse               11.7.5.86                     0    nvidia/label/cuda-11.8.0
     libcusparse-dev           11.7.5.86                     0    nvidia/label/cuda-11.8.0
     libdeflate                1.17                 h0b41bf4_0    conda-forge
     libedit                   3.1.20191231         he28a2e2_2    conda-forge
     libev                     4.33                 h516909a_1    conda-forge
     libevent                  2.1.10               h28343ad_4    conda-forge
     libexpat                  2.5.0                hcb278e6_1    conda-forge
     libffi                    3.4.2                h7f98852_5    conda-forge
     libgcc-devel_linux-64     11.2.0              h0952999_16    conda-forge
     libgcc-ng                 12.2.0              h65d4601_19    conda-forge
     libgcrypt                 1.10.1               h166bdaf_0    conda-forge
     libgd                     2.3.3                h5aea950_4    conda-forge
     libgfortran-ng            12.2.0              h69a702a_19    conda-forge
     libgfortran5              12.2.0              h337968e_19    conda-forge
     libglib                   2.76.2               hebfc3b9_0    conda-forge
     libgomp                   12.2.0              h65d4601_19    conda-forge
     libgoogle-cloud           2.8.0                h3c06191_0    conda-forge
     libgpg-error              1.46                 h620e276_0    conda-forge
     libgrpc                   1.51.1               hcf146ea_3    conda-forge
     libgsasl                  1.8.0                         2    conda-forge
     libhiredis                1.0.2                h2cc385e_0    conda-forge
     libhwloc                  2.5.0                h6746aa3_0    conda-forge
     libiconv                  1.17                 h166bdaf_0    conda-forge
     liblapack                 3.9.0           16_linux64_openblas    conda-forge
     libllvm11                 11.1.0               he0ac6c6_5    conda-forge
     libllvm14                 14.0.6               he0ac6c6_1    conda-forge
     libmrc                    23.07.00a       cuda_11.8_haf4b24b_27    nvidia/label/dev
     libnghttp2                1.52.0               h61bc06f_0    conda-forge
     libnpp                    11.8.0.86                     0    nvidia/label/cuda-11.8.0
     libnpp-dev                11.8.0.86                     0    nvidia/label/cuda-11.8.0
     libnsl                    2.0.0                h7f98852_0    conda-forge
     libntlm                   1.4               h7f98852_1002    conda-forge
     libnvjpeg                 11.9.0.86                     0    nvidia/label/cuda-11.8.0
     libnvjpeg-dev             11.9.0.86                     0    nvidia/label/cuda-11.8.0
     libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
     libpng                    1.6.39               h753d276_0    conda-forge
     libprotobuf               3.21.12              h3eb15da_0    conda-forge
     librdkafka                1.7.0                hb1989a6_1    conda-forge
     librmm                    23.02.00        cuda11_g48e8f2a8_0    rapidsai
     librsvg                   2.54.4               h7abd40a_0    conda-forge
     libsanitizer              11.2.0              he4da1e4_16    conda-forge
     libsqlite                 3.42.0               h2797004_0    conda-forge
     libssh2                   1.10.0               hf14f497_3    conda-forge
     libstdcxx-devel_linux-64  11.2.0              h0952999_16    conda-forge
     libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
     libthrift                 0.18.0               h5e4af38_0    conda-forge
     libtiff                   4.5.0                h6adf6a1_2    conda-forge
     libtool                   2.4.7                h27087fc_0    conda-forge
     libutf8proc               2.8.0                h166bdaf_0    conda-forge
     libuuid                   2.38.1               h0b41bf4_0    conda-forge
     libuv                     1.44.2               h166bdaf_0    conda-forge
     libwebp                   1.2.4                h1daa5a0_1    conda-forge
     libwebp-base              1.2.4                h166bdaf_0    conda-forge
     libxcb                    1.13              h7f98852_1004    conda-forge
     libxml2                   2.10.3               hca2bb57_4    conda-forge
     libzlib                   1.2.13               h166bdaf_4    conda-forge
     llvm-tools                14.0.6               he0ac6c6_1    conda-forge
     llvmdev                   14.0.6               he0ac6c6_1    conda-forge
     llvmlite                  0.39.1          py310h58363a5_1    conda-forge
     locket                    1.0.0              pyhd8ed1ab_0    conda-forge
     lxml                      4.9.2                    pypi_0    pypi
     lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
     lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
     m4                        1.4.18            h516909a_1001    conda-forge
     mako                      1.2.4              pyhd8ed1ab_0    conda-forge
     markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
     markupsafe                2.1.2           py310h1fa729e_0    conda-forge
     matplotlib-inline         0.1.6                    pypi_0    pypi
     mccabe                    0.7.0              pyhd8ed1ab_0    conda-forge
     mdit-py-plugins           0.3.5              pyhd8ed1ab_0    conda-forge
     mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
     mistune                   2.0.5                    pypi_0    pypi
     mlflow                    1.30.0          py310ha13cd29_0    conda-forge
     morpheus                  23.7.0a0+75.g04489473.dirty          pypi_0    pypi
     mrc                       23.07.00a       cuda_11.8_py310_hfcec58f_27    nvidia/label/dev
     msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
     mypy                      1.2.0                    pypi_0    pypi
     mypy-extensions           1.0.0                    pypi_0    pypi
     myst-parser               0.17.2                   pypi_0    pypi
     nbclassic                 0.5.6                    pypi_0    pypi
     nbclient                  0.7.4                    pypi_0    pypi
     nbconvert                 7.3.1                    pypi_0    pypi
     nbformat                  5.8.0                    pypi_0    pypi
     nbsphinx                  0.9.1                    pypi_0    pypi
     ncurses                   6.3                  h27087fc_1    conda-forge
     nest-asyncio              1.5.6                    pypi_0    pypi
     networkx                  3.1                pyhd8ed1ab_0    conda-forge
     ninja                     1.10.2               h4bd325d_1    conda-forge
     nlohmann_json             3.9.1                h9c3ff4c_1    conda-forge
     nodejs                    18.15.0              h8d033a5_0    conda-forge
     notebook                  6.5.4                    pypi_0    pypi
     notebook-shim             0.2.3                    pypi_0    pypi
     nsight-compute            2022.3.0.22                   0    nvidia/label/cuda-11.8.0
     numba                     0.56.4          py310h0e39c9b_1    conda-forge
     numpy                     1.23.5          py310h53a5b5f_0    conda-forge
     numpydoc                  1.4.0              pyhd8ed1ab_1    conda-forge
     nvidia-pyindex            1.0.9                    pypi_0    pypi
     nvtx                      0.2.5           py310h1fa729e_0    conda-forge
     oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
     openjpeg                  2.5.0                hfec8fc6_2    conda-forge
     openssl                   3.1.0                hd590300_3    conda-forge
     orc                       1.8.2                hfdbbad2_2    conda-forge
     ordered-set               4.1.0              pyhd8ed1ab_0    conda-forge
     packaging                 21.3               pyhd8ed1ab_0    conda-forge
     pandas                    1.3.5           py310hb5077e9_0    conda-forge
     pandocfilters             1.5.0                    pypi_0    pypi
     pango                     1.50.14              hd33c08f_0    conda-forge
     parquet-cpp               1.5.1                         2    conda-forge
     parso                     0.8.3                    pypi_0    pypi
     partd                     1.4.0              pyhd8ed1ab_0    conda-forge
     pcre2                     10.40                hc3806b6_0    conda-forge
     perl                      5.32.1          2_h7f98852_perl5    conda-forge
     pexpect                   4.8.0                    pypi_0    pypi
     pickleshare               0.7.5                    pypi_0    pypi
     pillow                    9.4.0           py310h023d228_1    conda-forge
     pip                       23.1.2             pyhd8ed1ab_0    conda-forge
     pixman                    0.40.0               h36c2ea0_0    conda-forge
     pkg-config                0.29.2            h36c2ea0_1008    conda-forge
     platformdirs              3.5.0              pyhd8ed1ab_0    conda-forge
     pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
     pooch                     1.7.0              pyha770c72_3    conda-forge
     port-for                  0.6.3                    pypi_0    pypi
     prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
     prometheus_flask_exporter 0.22.4             pyhd8ed1ab_0    conda-forge
     protobuf                  4.21.12         py310heca2aa9_0    conda-forge
     psutil                    5.9.5           py310h1fa729e_0    conda-forge
     pthread-stubs             0.4               h36c2ea0_1001    conda-forge
     ptxcompiler               0.7.0           py310h01a121a_3    conda-forge
     ptyprocess                0.7.0                    pypi_0    pypi
     pure-eval                 0.2.2                    pypi_0    pypi
     py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
     pyarrow                   10.0.1          py310hc81d9b2_14_cuda    conda-forge
     pybind11-stubgen          0.10.5             pyhd8ed1ab_0    conda-forge
     pycodestyle               2.10.0             pyhd8ed1ab_0    conda-forge
     pycparser                 2.21               pyhd8ed1ab_0    conda-forge
     pydocstyle                6.3.0                    pypi_0    pypi
     pydot                     1.4.2           py310hff52083_3    conda-forge
     pyflakes                  3.0.1              pyhd8ed1ab_0    conda-forge
     pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
     pyjwt                     2.6.0              pyhd8ed1ab_0    conda-forge
     pyopenssl                 23.1.1             pyhd8ed1ab_0    conda-forge
     pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
     pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
     pysocks                   1.7.1              pyha2e5f31_6    conda-forge
     pytest                    7.3.1                    pypi_0    pypi
     pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-cov                4.0.0              pyhd8ed1ab_0    conda-forge
     pytest-kafka              0.5.1                     dev_0    <develop>
     python                    3.10.11         he550d4f_0_cpython    conda-forge
     python-confluent-kafka    1.7.0           py310h6acc77f_2    conda-forge
     python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
     python-dotenv             0.21.1             pyhd8ed1ab_0    conda-forge
     python-graphviz           0.20.1             pyh22cad53_0    conda-forge
     python-json-logger        2.0.7                    pypi_0    pypi
     python-rapidjson          1.10                     pypi_0    pypi
     python_abi                3.10                    3_cp310    conda-forge
     pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
     pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
     pyyaml                    5.4.1           py310h5764c6d_4    conda-forge
     pyzmq                     25.0.2                   pypi_0    pypi
     querystring_parser        1.2.4                      py_0    conda-forge
     rapidjson                 1.1.0             he1b5a44_1002    conda-forge
     re2                       2023.02.02           hcb278e6_0    conda-forge
     readline                  8.2                  h8228510_1    conda-forge
     requests                  2.29.0             pyhd8ed1ab_0    conda-forge
     rfc3339-validator         0.1.4                    pypi_0    pypi
     rfc3986-validator         0.1.1                    pypi_0    pypi
     rhash                     1.4.3                h166bdaf_0    conda-forge
     rmm                       23.02.00        cuda11_py310_g48e8f2a8_0    rapidsai
     s2n                       1.3.41               h3358134_0    conda-forge
     s3fs                      2023.4.0                 pypi_0    pypi
     s3transfer                0.6.0                    pypi_0    pypi
     scikit-build              0.17.1             pyh56297ac_0    conda-forge
     scikit-learn              1.2.2           py310h41b6a48_1    conda-forge
     scipy                     1.10.1          py310h8deb116_0    conda-forge
     send2trash                1.8.2                    pypi_0    pypi
     setuptools                59.8.0          py310hff52083_1    conda-forge
     six                       1.16.0             pyh6c4a22f_0    conda-forge
     smmap                     3.0.5              pyh44b312d_0    conda-forge
     snappy                    1.1.10               h9fff704_0    conda-forge
     sniffio                   1.3.0                    pypi_0    pypi
     snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
     sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
     spdlog                    1.8.5                h4bd325d_1    conda-forge
     sphinx                    4.5.0                    pypi_0    pypi
     sphinx_rtd_theme          1.2.0              pyha770c72_0    conda-forge
     sphinxcontrib-applehelp   1.0.4              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge
     sphinxcontrib-htmlhelp    2.0.1              pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jquery      4.1                pyhd8ed1ab_0    conda-forge
     sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge
     sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
     sphinxcontrib-serializinghtml 1.1.5              pyhd8ed1ab_2    conda-forge
     sqlalchemy                1.4.46          py310h1fa729e_0    conda-forge
     sqlparse                  0.4.4              pyhd8ed1ab_0    conda-forge
     stack-data                0.6.2                    pypi_0    pypi
     sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
     tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
     tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
     terminado                 0.17.1                   pypi_0    pypi
     texttable                 1.6.7              pyhd8ed1ab_0    conda-forge
     threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
     tinycss2                  1.2.1                    pypi_0    pypi
     tk                        8.6.12               h27826a3_0    conda-forge
     toml                      0.10.2             pyhd8ed1ab_0    conda-forge
     tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
     toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
     torch                     1.13.1+cu116             pypi_0    pypi
     tornado                   6.3.1                    pypi_0    pypi
     tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
     traitlets                 5.9.0                    pypi_0    pypi
     tritonclient              2.17.0                   pypi_0    pypi
     typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
     typing_extensions         4.5.0              pyha770c72_0    conda-forge
     typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
     tzdata                    2023c                h71feb2d_0    conda-forge
     ucx                       1.13.1               h538f049_1    conda-forge
     uri-template              1.2.0                    pypi_0    pypi
     urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
     watchdog                  2.1.9           py310hff52083_1    conda-forge
     webcolors                 1.13                     pypi_0    pypi
     webencodings              0.5.1                    pypi_0    pypi
     websocket-client          0.57.0          py310hff52083_6    conda-forge
     websockets                11.0.3                   pypi_0    pypi
     werkzeug                  2.3.3              pyhd8ed1ab_0    conda-forge
     wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
     widgetsnbextension        4.0.7                    pypi_0    pypi
     xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
     xorg-libice               1.0.10               h7f98852_0    conda-forge
     xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
     xorg-libx11               1.8.4                h0b41bf4_0    conda-forge
     xorg-libxau               1.0.9                h7f98852_0    conda-forge
     xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
     xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
     xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
     xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
     xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
     xorg-xproto               7.0.31            h7f98852_1007    conda-forge
     xyzservices               2023.2.0           pyhd8ed1ab_0    conda-forge
     xz                        5.2.6                h166bdaf_0    conda-forge
     y-py                      0.5.9                    pypi_0    pypi
     yaml                      0.2.5                h7f98852_2    conda-forge
     yapf                      0.32.0             pyhd8ed1ab_0    conda-forge
     ypy-websocket             0.8.2                    pypi_0    pypi
     zict                      3.0.0              pyhd8ed1ab_0    conda-forge
     zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
     zlib                      1.2.13               h166bdaf_4    conda-forge
     zope-event                4.6                      pypi_0    pypi
     zope-interface            6.0                      pypi_0    pypi
     zstd                      1.5.2                h3eb15da_6    conda-forge
     
</pre></details>

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@bsuryadevara bsuryadevara added the bug Something isn't working label May 19, 2023
@mdemoret-nv mdemoret-nv added this to the 24.03 - MNMG milestone Dec 13, 2023
@dagardner-nv dagardner-nv self-assigned this Jan 17, 2024
@dagardner-nv dagardner-nv moved this from Todo to In Progress in Morpheus Boards Jan 17, 2024
@dagardner-nv
Copy link
Contributor

This is caused by nv-morpheus/MRC#360

The stage requires a MRC builder instance to load the module, which requires the validation step to be performed during MRC build.

@dagardner-nv dagardner-nv moved this from In Progress to Todo in Morpheus Boards Jan 17, 2024
@dagardner-nv dagardner-nv moved this from Todo to In Progress in Morpheus Boards Jan 19, 2024
dagardner-nv added a commit to dagardner-nv/Morpheus that referenced this issue Jan 19, 2024
@dagardner-nv
Add test for issue nv-morpheus#953
98cada9
@dagardner-nv dagardner-nv mentioned this issue Aug 1, 2024
Merged
@morpheus-bot-test morpheus-bot-test bot moved this from In Progress to Review - Ready for Review in Morpheus Boards Aug 1, 2024
@rapids-bot rapids-bot bot closed this as completed in #1829 Aug 2, 2024
@rapids-bot rapids-bot bot closed this as completed in ad9249c Aug 2, 2024
@github-project-automation github-project-automation bot moved this from Review - Ready for Review to Done in Morpheus Boards Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dagardner-nv dagardner-nv

Labels
bug Something isn't working
Projects
Morpheus Boards
Status: Done
Milestone
24.03 - MNMG
3 participants
@bsuryadevara @mdemoret-nv @dagardner-nv