Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IR] Creation-time constant fold for constant expressions #209

Merged
merged 5 commits into from
May 5, 2023
Merged

[IR] Creation-time constant fold for constant expressions #209

merged 5 commits into from
May 5, 2023

Conversation

yaoyaoding
Copy link
Member

Previously, when we do some arhtimatic operations for constant, we will keep them in our IR:

from hidet.ir.expr import convert
a = convert(1)
b = convert(2)
c = a + b
print(c)  # would be Add(Constant(1), Constant(2))

With this PR, we will do the computation when we create the Add expression, and replace it with hidet.ir.Constant:

from hidet.ir.expr import convert
a = convert(1)
b = convert(2)
c = a + b
print(c)  # would be Constant(3)

This PR also clean the hidet.ir.dialects.pattern.py. We do not many fancy pattern matching any more.

@yaoyaoding yaoyaoding merged commit daae22e into hidet-org:main May 5, 2023
@yaoyaoding yaoyaoding deleted the const-fold branch May 5, 2023 04:40
vadiklyutiy pushed a commit that referenced this pull request Jul 22, 2024
@zhumakhan @vadiklyutiy
steal_weight option fixes && fixes for mistral model (#209)
156268b 
steal_weight option works as expected after removing all references to
torch tensors. `HidetModule` has `torch_params` and `hidet_params`
attributes. In modules such as `HidetLinear` and
`HidetMultiheadAttention` we don't need to keep original weight tensors
in `torch_params` and `hidet_params`. Instead, their transposed copies
are stored in those modules. Removing original weight tensors allows us
to have some additional free space on GPU.

Now we are able to compile LLama2-7b model with 12 Gib of weights on
24Gib RTX 3090 GPU. This is the state of GPU memory after compiling with
hidet:

- Allocated by hidet: 21179 MiB
- Allocated by torch: 314 MiB

For a reference, torch.compile with `backend=inductor` and
`mode=max-autotune`:
- Allocated by hidet: 0 Mib
- Allocated by torch: 12925 Mib


This script is used to test llama2 model:
https://drive.google.com/file/d/1Baz5MrC9wWg9ceirmKZtkMPbv4u3CAuc/view?usp=sharing

---------

Co-authored-by: Zhumakhan <nazirzhumakhan@gmail,.com>
vadiklyutiy pushed a commit that referenced this pull request Jul 23, 2024
@zhumakhan @vadiklyutiy
steal_weight option fixes && fixes for mistral model (#209)
9728c21 
steal_weight option works as expected after removing all references to
torch tensors. `HidetModule` has `torch_params` and `hidet_params`
attributes. In modules such as `HidetLinear` and
`HidetMultiheadAttention` we don't need to keep original weight tensors
in `torch_params` and `hidet_params`. Instead, their transposed copies
are stored in those modules. Removing original weight tensors allows us
to have some additional free space on GPU.

Now we are able to compile LLama2-7b model with 12 Gib of weights on
24Gib RTX 3090 GPU. This is the state of GPU memory after compiling with
hidet:

- Allocated by hidet: 21179 MiB
- Allocated by torch: 314 MiB

For a reference, torch.compile with `backend=inductor` and
`mode=max-autotune`:
- Allocated by hidet: 0 Mib
- Allocated by torch: 12925 Mib


This script is used to test llama2 model:
https://drive.google.com/file/d/1Baz5MrC9wWg9ceirmKZtkMPbv4u3CAuc/view?usp=sharing

---------

Co-authored-by: Zhumakhan <nazirzhumakhan@gmail,.com>
vadiklyutiy pushed a commit that referenced this pull request Dec 26, 2024
@zhumakhan
steal_weight option fixes && fixes for mistral model (#209)
edbbd93 
steal_weight option works as expected after removing all references to
torch tensors. `HidetModule` has `torch_params` and `hidet_params`
attributes. In modules such as `HidetLinear` and
`HidetMultiheadAttention` we don't need to keep original weight tensors
in `torch_params` and `hidet_params`. Instead, their transposed copies
are stored in those modules. Removing original weight tensors allows us
to have some additional free space on GPU.

Now we are able to compile LLama2-7b model with 12 Gib of weights on
24Gib RTX 3090 GPU. This is the state of GPU memory after compiling with
hidet:

- Allocated by hidet: 21179 MiB
- Allocated by torch: 314 MiB

For a reference, torch.compile with `backend=inductor` and
`mode=max-autotune`:
- Allocated by hidet: 0 Mib
- Allocated by torch: 12925 Mib


This script is used to test llama2 model:
https://drive.google.com/file/d/1Baz5MrC9wWg9ceirmKZtkMPbv4u3CAuc/view?usp=sharing

---------

Co-authored-by: Zhumakhan <nazirzhumakhan@gmail,.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@yaoyaoding