OSError: /opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail12infer_schema20make_function_schemaENS_8ArrayRefINS1_11ArgumentDefEEES4_

[Shreyas-NR]

Hi @qianglin-xlnx @lishixlnx

I'm working on the training yolox model in the vitis ai pytorch conda env using the deployable scripts.
facing the OS ERROR issue

(vitis-ai-pytorch) Vitis-AI /workspace/code > python3 tools/train.py -f exps/example/yolox_voc/yolox_voc_s.py -d 8 -b 64 --fp16 -o -c weights/yolox_s.pth
2022-11-04 08:56:22.142 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 5 initialization finished.
2022-11-04 08:56:22.163 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 6 initialization finished.
2022-11-04 08:56:22.175 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 1 initialization finished.
2022-11-04 08:56:22.200 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 0 initialization finished.
2022-11-04 08:56:22.211 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 7 initialization finished.
2022-11-04 08:56:22.221 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 3 initialization finished.
2022-11-04 08:56:22.229 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 4 initialization finished.
2022-11-04 08:56:22.236 | INFO     | yolox.core.launch:_distributed_worker:130 - Rank 2 initialization finished.
2022-11-04 08:56:33.300 | INFO     | yolox.utils.setup_env:configure_omp:60 -
***************************************************************
We set `OMP_NUM_THREADS` for each process to 1 to speed up.
please further tune the variable for optimal performance.
***************************************************************
2022-11-04 08:56:33 | INFO     | yolox.core.trainer:140 - args: Namespace(batch_size=64, cache=False, ckpt='weights/yolox_s.pth', devices=8, dist_backend='nccl', dist_url=None, exp_file='exps/example/yolox_voc/yolox_voc_s.py', experiment_name='yolox_voc_s', fp16=True, machine_rank=0, name=None, num_machines=1, occupy=True, opts=[], resume=False, start_epoch=None)
2022-11-04 08:56:33 | INFO     | yolox.core.trainer:141 - exp value:
╒══════════════════╤════════════════════════════╕
│ keys             │ values                     │
╞══════════════════╪════════════════════════════╡
│ seed             │ None                       │
├──────────────────┼────────────────────────────┤
│ output_dir       │ './YOLOX_outputs'          │
├──────────────────┼────────────────────────────┤
│ print_interval   │ 10                         │
├──────────────────┼────────────────────────────┤
│ eval_interval    │ 10                         │
├──────────────────┼────────────────────────────┤
│ num_classes      │ 1                          │
├──────────────────┼────────────────────────────┤
│ depth            │ 0.33                       │
├──────────────────┼────────────────────────────┤
│ width            │ 0.5                        │
├──────────────────┼────────────────────────────┤
│ act              │ 'lrelu'                    │
├──────────────────┼────────────────────────────┤
│ data_num_workers │ 4                          │
├──────────────────┼────────────────────────────┤
│ input_size       │ (640, 640)                 │
├──────────────────┼────────────────────────────┤
│ multiscale_range │ 5                          │
├──────────────────┼────────────────────────────┤
│ data_dir         │ None                       │
├──────────────────┼────────────────────────────┤
│ train_ann        │ 'instances_train2017.json' │
├──────────────────┼────────────────────────────┤
│ val_ann          │ 'instances_val2017.json'   │
├──────────────────┼────────────────────────────┤
│ mosaic_prob      │ 1.0                        │
├──────────────────┼────────────────────────────┤
│ mixup_prob       │ 1.0                        │
├──────────────────┼────────────────────────────┤
│ hsv_prob         │ 1.0                        │
├──────────────────┼────────────────────────────┤
│ flip_prob        │ 0.5                        │
├──────────────────┼────────────────────────────┤
│ degrees          │ 10.0                       │
├──────────────────┼────────────────────────────┤
│ translate        │ 0.1                        │
├──────────────────┼────────────────────────────┤
│ mosaic_scale     │ (0.1, 2)                   │
├──────────────────┼────────────────────────────┤
│ mixup_scale      │ (0.5, 1.5)                 │
├──────────────────┼────────────────────────────┤
│ shear            │ 2.0                        │
├──────────────────┼────────────────────────────┤
│ enable_mixup     │ True                       │
├──────────────────┼────────────────────────────┤
│ warmup_epochs    │ 1                          │
├──────────────────┼────────────────────────────┤
│ max_epoch        │ 3                          │
├──────────────────┼────────────────────────────┤
│ warmup_lr        │ 0                          │
├──────────────────┼────────────────────────────┤
│ basic_lr_per_img │ 0.00015625                 │
├──────────────────┼────────────────────────────┤
│ scheduler        │ 'yoloxwarmcos'             │
├──────────────────┼────────────────────────────┤
│ no_aug_epochs    │ 15                         │
├──────────────────┼────────────────────────────┤
│ min_lr_ratio     │ 0.05                       │
├──────────────────┼────────────────────────────┤
│ ema              │ True                       │
├──────────────────┼────────────────────────────┤
│ weight_decay     │ 0.0005                     │
├──────────────────┼────────────────────────────┤
│ momentum         │ 0.9                        │
├──────────────────┼────────────────────────────┤
│ exp_name         │ 'yolox_voc_s'              │
├──────────────────┼────────────────────────────┤
│ test_size        │ (640, 640)                 │
├──────────────────┼────────────────────────────┤
│ test_conf        │ 0.01                       │
├──────────────────┼────────────────────────────┤
│ nmsthre          │ 0.65                       │
╘══════════════════╧════════════════════════════╛
2022-11-04 08:56:35 | ERROR    | yolox.core.launch:161 - An error has been caught in function '_distributed_worker', process 'SpawnProcess-1' (444), thread 'MainThread' (139887121491776):
Traceback (most recent call last):

  File "<string>", line 1, in <module>
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
               │     └ 9
               └ <function _main at 0x7f3a00dbb050>
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/multiprocessing/spawn.py", line 118, in _main
    return self._bootstrap()
           │    └ <function BaseProcess._bootstrap at 0x7f3a00e6f950>
           └ <SpawnProcess(SpawnProcess-1, started)>
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    │    └ <function BaseProcess.run at 0x7f3a00e68f80>
    └ <SpawnProcess(SpawnProcess-1, started)>
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <SpawnProcess(SpawnProcess-1, started)>
    │    │        │    └ (<function _distributed_worker at 0x7f3895d43ef0>, 0, (<function main at 0x7f3a00cca050>, 8, 8, 0, 'nccl', 'tcp://127.0.0.1:3...
    │    │        └ <SpawnProcess(SpawnProcess-1, started)>
    │    └ <function _wrap at 0x7f38a31c27a0>
    └ <SpawnProcess(SpawnProcess-1, started)>
  File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
    │  │   └ (<function main at 0x7f3a00cca050>, 8, 8, 0, 'nccl', 'tcp://127.0.0.1:37325', (╒══════════════════╤══════════════════════════...
    │  └ 0
    └ <function _distributed_worker at 0x7f3895d43ef0>

> File "/workspace/code/yolox/core/launch.py", line 161, in _distributed_worker
    main_func(*args)
    │          └ (╒══════════════════╤════════════════════════════╕
    │            │ keys             │ values                     │
    │            ╞══════════════════╪════...
    └ <function main at 0x7f3a00cca050>

  File "/workspace/code/tools/train.py", line 124, in main
    trainer.train()
    │       └ <function Trainer.train at 0x7f3895cde050>
    └ <yolox.core.trainer.Trainer object at 0x7f38949c5e10>

  File "/workspace/code/yolox/core/trainer.py", line 84, in train
    self.before_train()
    │    └ <function Trainer.before_train at 0x7f38945c8680>
    └ <yolox.core.trainer.Trainer object at 0x7f38949c5e10>

  File "/workspace/code/yolox/core/trainer.py", line 145, in before_train
    model = self.exp.get_model()
            │    │   └ <function ExpDeploy.get_model at 0x7f387adc4710>
            │    └ ╒══════════════════╤════════════════════════════╕
            │      │ keys             │ values                     │
            │      ╞══════════════════╪═════...
            └ <yolox.core.trainer.Trainer object at 0x7f38949c5e10>

  File "/workspace/code/yolox/exp/yolox_base_deploy.py", line 362, in get_model
    from yolox.models.yolox_deploy import YOLOX

  File "/workspace/code/yolox/models/__init__.py", line 19, in <module>
    from .darknet import CSPDarknet, Darknet

  File "/workspace/code/yolox/models/darknet.py", line 21, in <module>
    from .network_blocks import BaseConv, CSPLayer, DWConv, Focus, ResLayer, SPPBottleneck

  File "/workspace/code/yolox/models/network_blocks.py", line 19, in <module>
    import pytorch_nndct.nn.modules.functional as QF

  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/__init__.py", line 14, in <module>
    from .apis import *
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/apis.py", line 25, in <module>
    from .qproc import TorchQuantProcessor
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/qproc/__init__.py", line 1, in <module>
    from .base import TorchQuantProcessor, dump_xmodel
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/qproc/base.py", line 30, in <module>
    from pytorch_nndct.quantization import TORCHQuantizer, FakeQuantizer
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/quantization/__init__.py", line 2, in <module>
    from .torch_qalgo import *
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/quantization/torch_qalgo.py", line 28, in <module>
    from pytorch_nndct.nn import fake_quantize_per_tensor, fake_quantize_per_channel
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/__init__.py", line 1, in <module>
    from pytorch_nndct.nn.modules import functional
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/modules/__init__.py", line 16, in <module>
    from .sigmoid import *
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/modules/sigmoid.py", line 26, in <module>
    from .fix_ops import NndctSigmoidTableLookup, NndctSigmoidSimulation
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/modules/fix_ops.py", line 26, in <module>
    from ..load_kernels import *
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/load_kernels.py", line 31, in <module>
    torch.ops.load_library(lib_abspath)
    │     │   │            └ '/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-linux-gnu...
    │     │   └ <function _Ops.load_library at 0x7f38a325da70>
    │     └ <module 'torch.ops' from '/home/vitis-ai-user/.local/lib/python3.7/site-packages/torch/_ops.py'>
    └ <module 'torch' from '/home/vitis-ai-user/.local/lib/python3.7/site-packages/torch/__init__.py'>
  File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/torch/_ops.py", line 105, in load_library
    ctypes.CDLL(path)
    │      │    └ '/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-linux-gnu...
    │      └ <class 'ctypes.CDLL'>
    └ <module 'ctypes' from '/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/ctypes/__init__.py'>
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
    │    │         │       │    │      └ 0
    │    │         │       │    └ '/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-linux-gnu...
    │    │         │       └ <CDLL '/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-lin...
    │    │         └ <built-in function dlopen>
    │    └ 0
    └ <CDLL '/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-lin...

OSError: /opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.7/site-packages/pytorch_nndct/nn/_kernels.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail12infer_schema20make_function_schemaENS_8ArrayRefINS1_11ArgumentDefEEES4_
(vitis-ai-pytorch) Vitis-AI /workspace/code >

thanks

Originally posted by @Shreyas-NR in #990 (comment)

[Shreyas-NR]

Identified what's missing in the PyTorch conda env

#990 (comment)

[wangxd-xlnx]

Hi @Shreyas-NR

It seems that the pytorch version is specified in the requirements.txt of model, which is inconsistent with the default pytorch version installed in docker. You could have a check whether it's specified the version of pytorch and torchvision in requirements.txt.

If it's different from the version comes with docker, you could use this script '/opt/vitis_ai/scripts/replace_pytorch.sh' to switch the version of pytorch and torchvision. Then a new conda environment for PyTorch can be created.

For more details, you could refer to this link.
https://github.com/Xilinx/Vitis-AI/tree/master/src/Vitis-AI-Quantizer/vai_q_pytorch#quick-start-in-docker-environment

[qianglin-xlnx]

Closing since no activity for more than 1 month, please reopen if you still have any questions, thanks

[MohamedRaizz]

When I'm installing docker vitis 3.0 i faced like this error can u please give solution for this

#0 444.5 channels:
#0 444.5 - conda-forge
#0 444.5 - file:///scratch/conda-channel-wego/wegotf1
#0 444.5 + mamba create -n vitis-ai-tensorflow vai_q_tensorflow tensorflow-onnx -c file:///scratch/conda-channel -c conda-forge -c anaconda
#0 444.6 Traceback (most recent call last):
#0 444.6 File "/opt/vitis_ai/conda/condabin/mamba", line 7, in
#0 444.6 from mamba.mamba import main
#0 444.6 File "/opt/vitis_ai/conda/lib/python3.9/site-packages/mamba/mamba.py", line 51, in
#0 444.6 from mamba import repoquery as repoquery_api
#0 444.6 File "/opt/vitis_ai/conda/lib/python3.9/site-packages/mamba/repoquery.py", line 9, in
#0 444.6 from mamba.utils import init_api_context, load_channels
#0 444.6 File "/opt/vitis_ai/conda/lib/python3.9/site-packages/mamba/utils.py", line 17, in
#0 444.6 from conda.core.index import _supplement_index_with_system, check_allowlist
#0 444.6 ImportError: cannot import name 'check_allowlist' from 'conda.core.index' (/opt/vitis_ai/conda/lib/python3.9/site-packages/conda/core/index.py)

vitis-ai-cpu.Dockerfile:32

30 | ADD conda/banner.sh /etc/
31 | ADD conda/${DOCKER_TYPE}conda/bashrc /etc/bash.bashrc
32 | >>> RUN if [[ -n "${TARGET_FRAMEWORK}" ]]; then bash ./install${TARGET_FRAMEWORK}.sh; fi
33 | USER root
34 | RUN mkdir -p ${VAI_ROOT}/conda/pkgs && chmod 777 ${VAI_ROOT}/conda/pkgs && ./install_vairuntime.sh && rm -fr ./*

ERROR: failed to solve: process "/bin/bash -c if [[ -n "${TARGET_FRAMEWORK}" ]]; then bash ./install_${TARGET_FRAMEWORK}.sh; fi" did not complete successfully: exit code: 1
Error response from daemon: No such image: xilinx/vitis-ai-tensorflow-cpu:3.0.0.001

[janifer112x]

Hi @MohamedRaizz，
are you using the latest commit in master branch? we have fixed this issue last week.

[MohamedRaizz]

now i commit with master branch also same issue we got ?

0 734.5 + export PATH
#0 734.5 + /opt/vitis_ai/conda/bin/conda config --remove channels defaults
#0 734.7 + cat /home/vitis-ai-user/.condarc
#0 734.7 channels:
#0 734.7 - file:///scratch/conda-channel
#0 734.7 + mamba env create -f /scratch/cpu_conda/vitis-ai-wego-torch.yml
#0 734.9 Traceback (most recent call last):
#0 734.9 File "/opt/vitis_ai/conda/condabin/mamba", line 7, in
#0 734.9 from mamba.mamba import main
#0 734.9 File "/opt/vitis_ai/conda/lib/python3.9/site-packages/mamba/mamba.py", line 51, in
#0 734.9 from mamba import repoquery as repoquery_api
#0 734.9 File "/opt/vitis_ai/conda/lib/python3.9/site-packages/mamba/repoquery.py", line 9, in
#0 734.9 from mamba.utils import init_api_context, load_channels
#0 734.9 File "/opt/vitis_ai/conda/lib/python3.9/site-packages/mamba/utils.py", line 17, in
#0 734.9 from conda.core.index import _supplement_index_with_system, check_allowlist
#0 734.9 ImportError: cannot import name 'check_allowlist' from 'conda.core.index' (/opt/vitis_ai/conda/lib/python3.9/site-packages/conda/core/index.py)

ERROR: failed to solve: executor failed running [/bin/bash -c if [[ -n "${TARGET_FRAMEWORK}" ]]; then bash ./install_${TARGET_FRAMEWORK}.sh; fi]: exit code: 1
Error response from daemon: No such image: xilinx/vitis-ai-pytorch-cpu:3.0.0.001

[MohamedRaizz]

@janifer112x i'm doing model compilation in vitis-ai (master) i got those error in particular model can u please let me know ?

(vitis-ai-tensorflow) vitis-ai-user@user:/workspace/CNN_compiler$ vai_c_tensorflow -f ../CNN_src/tf_bert-base_SQuAD_128_22.34G_3.0/quantized/pre_norm_bert_base_int8.pb -a arch.json -o ../CNN_bin/tf_bert-base_SQuAD_128_22.34G_3.0/ -n tf_bert-base_SQuAD_128_22.34G_3.0

---

• VITIS_AI Compilation - Xilinx Inc.

---

[INFO] Namespace(batchsize=1, inputs_shape=None, layout='NHWC', model_files=['../CNN_src/tf_bert-base_SQuAD_128_22.34G_3.0/quantized/pre_norm_bert_base_int8.pb'], model_type='tensorflow', named_inputs_shape=None, out_filename='/tmp/tf_bert-base_SQuAD_128_22.34G_3.0_0x101000056010407_org.xmodel', proto=None)
[INFO] tensorflow model: /workspace/CNN_src/tf_bert-base_SQuAD_128_22.34G_3.0/quantized/pre_norm_bert_base_int8.pb
[INFO] parse raw model : 0%|▎ | 12/4416 [00:00<00:00, 13789.49it/s]
Traceback (most recent call last):
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/bin/xnnc-run", line 33, in
sys.exit(load_entry_point('xnnc==3.0.0', 'console_scripts', 'xnnc-run')())
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/main.py", line 49, in main
runner.normal_run(args)
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/runner.py", line 123, in normal_run
target=target,
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/xconverter.py", line 145, in run
model_files, model_type, _layout, in_shapes, batchsize
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/core.py", line 123, in make_xmodel
model_type=model_t,
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/translator/tensorflow_translator.py", line 110, in to_xmodel
model_type,
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/translator/tensorflow_translator.py", line 176, in create_xmodel
name, layers, layout, in_shapes, batchsize
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/translator/tensorflow_translator.py", line 292, in __create_xmodel_from_tf1
batchsize,
File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/xnnc/translator/tensorflow_translator.py", line 3647, in __generate_xmodel
assert isinstance(val, (list, bool, int, float, str, bytes, map)), f"Unsupported data type!"
AssertionError: Unsupported data type!