Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Convert the output of TransformerBlock to ragged tensor #1025

Closed
sararb opened this issue Mar 21, 2023 · 0 comments · Fixed by #1022
Closed

[FEA] Convert the output of TransformerBlock to ragged tensor #1025

sararb opened this issue Mar 21, 2023 · 0 comments · Fixed by #1022
Labels
area/session-based enhancement New feature or request
Milestone

Comments

@sararb
Copy link
Contributor

sararb commented Mar 21, 2023

To align with the HuggingFace transformer layer that requires dense inputs, we convert ragged inputs to dense before calling the TransformerBlock. As a result of this conversion, the outputs are also dense.

This approach can be costly because it means computing logit scores for all positions, even the padded ones. For example, this can impact performance when applying weight-tying multiplication between the hidden representation and all items' embeddings.

It would be helpful to convert the output of the transformer block to a ragged format, which would eliminate the need for padding and avoid unnecessary computation.

@sararb sararb added enhancement New feature or request area/session-based labels Mar 21, 2023
@sararb sararb added this to the Merlin 23.03 milestone Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/session-based enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant