Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Lambda and numpy() cannot coexist in a script #432

Closed
zhiwei-fang opened this issue Feb 23, 2024 · 2 comments
Closed

[Bug] Lambda and numpy() cannot coexist in a script #432

zhiwei-fang opened this issue Feb 23, 2024 · 2 comments
Labels
bug Something isn't working Task Or "Story" in JIRA's term.
Milestone

Comments

@zhiwei-fang
Copy link
Contributor

zhiwei-fang commented Feb 23, 2024

Describe the bug
I make a test code at hidet/python/hidet/graph/ops. The code snapshot:

import numpy as np
import hidet as hi
from hidet import ops, Tensor

from .transform import transpose
from hidet.graph.tensor import asarray

shape = [3, 2]
dtype=np.float32
data = np.random.randn(*shape).astype(dtype)
f = lambda x: x
hidet_result = transpose(hi.asarray(data).cuda()).cpu().numpy()
hi.cuda.synchronize()
print(hidet_result)

This will report a Segmentation fault (core dumped).
However, if we remove the last numpy() in the definition of hidet_result, the segmentation fault will gone.
If we remove the defintion of f = lambda x: x, the the segmentation fault will gone too. Note the lambda function f is unrelated to the hidet code at all.

To Reproduce
Install the hidet (public version), build from source. Run the code by python -m hidet.graph.ops.test_transpose.

Expected behavior
Should remove the segmentation fault.

Enviroment

  • OS: Ubuntu 22.04.3 LTS
  • GPU: RTX 3090
  • Others: CUDA 12.1, GPU Driver Version: 545.23.08
@zhiwei-fang zhiwei-fang added the bug Something isn't working label Feb 23, 2024
@wangshangsam wangshangsam added the Task Or "Story" in JIRA's term. label Feb 27, 2024
@arcturusannamalai
Copy link

can you attach a stack trace? I wonder if it can be a case where hidet code is accessing an object garbage collected by Python interpreter. To check this you can do,

import gc
gc.disable()

at the beginning of your code and run with the numpy and f objects present.

@wangshangsam wangshangsam added this to the 0.4.0 milestone Apr 16, 2024
vadiklyutiy pushed a commit that referenced this issue Dec 19, 2024
- Added more llms to Regression test and complete test run is here:

https://github.com/CentML/hidet/actions/runs/10528073493/job/29172874637

- ~~Accuracy check for masked llms with token ids instead of raw logits
tensor~~

---------

Co-authored-by: Zhumakhan <nazirzhumakhan@gmail,.com>
vadiklyutiy pushed a commit that referenced this issue Dec 20, 2024
- Added more llms to Regression test and complete test run is here:

https://github.com/CentML/hidet/actions/runs/10528073493/job/29172874637

- ~~Accuracy check for masked llms with token ids instead of raw logits
tensor~~

---------

Co-authored-by: Zhumakhan <nazirzhumakhan@gmail,.com>
@vadiklyutiy
Copy link
Collaborator

no actual

vadiklyutiy pushed a commit that referenced this issue Dec 26, 2024
- Added more llms to Regression test and complete test run is here:

https://github.com/CentML/hidet/actions/runs/10528073493/job/29172874637

- ~~Accuracy check for masked llms with token ids instead of raw logits
tensor~~

---------

Co-authored-by: Zhumakhan <nazirzhumakhan@gmail,.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Task Or "Story" in JIRA's term.
Projects
None yet
Development

No branches or pull requests

4 participants