-
Notifications
You must be signed in to change notification settings - Fork 567
Closed
Description
flashinfer-python 0.2.6.post1 miscompiles trtllm_utils.so. I can reproduce the issue with both AOT and JIT compilation of the shared library.
>>> import flashinfer.utils
>>> flashinfer.utils.get_trtllm_utils_module()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/tmp/flashinfer_python-0.2.6.post1/flashinfer/utils.py", line 496, in get_trtllm_utils_module
).build_and_load()
^^^^^^^^^^^^^^^^
File "/tmp/flashinfer_python-0.2.6.post1/flashinfer/jit/core.py", line 120, in build_and_load
loader.load_library(so_path)
File "/tmp/venv/lib64/python3.12/site-packages/torch/_ops.py", line 1392, in load_library
ctypes.CDLL(path)
File "/usr/lib64/python3.12/ctypes/__init__.py", line 379, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/user/.cache/flashinfer/100_120_75_80_86_87_89_90/cached_ops/trtllm_utils/trtllm_utils.so: undefined symbol: _ZTIN12tensorrt_llm6common13TllmExceptionEI think that
flashinfer/flashinfer/utils.py
Lines 475 to 481 in 2706062
| def get_trtllm_utils_module(): | |
| return gen_jit_spec( | |
| "trtllm_utils", | |
| [ | |
| jit_env.FLASHINFER_CSRC_DIR | |
| / "nv_internal/tensorrt_llm/kernels/delayStream.cu", | |
| ], |
jit_env.FLASHINFER_CSRC_DIR / "nv_internal/cpp/common/envUtils.cpp",
jit_env.FLASHINFER_CSRC_DIR / "nv_internal/cpp/common/logger.cpp",
jit_env.FLASHINFER_CSRC_DIR / "nv_internal/cpp/common/stringUtils.cpp",
jit_env.FLASHINFER_CSRC_DIR / "nv_internal/cpp/common/tllmException.cpp",
Metadata
Metadata
Assignees
Labels
No labels