Skip to content

🐛 [Bug] Weight streaming test fail #3460

Closed
@keehyuna

Description

@keehyuna

Bug Description

Segmentation fault error from test_004_weight_streaming test. cudagraph.reset() is in call stack

File "/usr/local/lib/python3.12/dist-packages/torch/cuda/graphs.py", line 92 in reset
File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/runtime/_CudaGraphsTorchTensorRTModule.py", line 83 in forward
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1762 in _call_impl
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1751 in _wrapped_call_impl
File "/opt/pytorch/torch_tensorrt/tests/py/dynamo/runtime/test_004_weight_streaming.py", line 364 in test_trt_model
File "/opt/pytorch/torch_tensorrt/tests/py/dynamo/runtime/test_004_weight_streaming.py", line 385 in test_runtime_state_change

To Reproduce

Steps to reproduce the behavior:

  1. pytest test_004_weight_streaming.py

Expected behavior

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • Torch-TensorRT Version (e.g. 1.0.0):
  • PyTorch Version (e.g. 1.0):
  • CPU Architecture:
  • OS (e.g., Linux):
  • How you installed PyTorch (conda, pip, libtorch, source):
  • Build command you used (if compiling from source):
  • Are you using local sources or building from archives:
  • Python version:
  • CUDA version:
  • GPU models and configuration:
  • Any other relevant information:

Additional context

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions