Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache: standardize cache interface #29005

Closed
wants to merge 30 commits into from

Conversation

gante
Copy link
Member

@gante gante commented Feb 13, 2024

What does this PR do?

In #27931, where the static cache was introduced, we noticed it had the following hard requirements:

  1. The model instance holds the cache, as opposed to being a tensor passed around;
  2. Each layer has its own cache, as opposed to a single cache for all layers.

This contrasts with previous implementations (e.g. DynamicCache). Given the hard requirements of the static cache, and the resulting benefits, this PR migrates the interface for all cache classes so as to match the static cache. As a result, the modeling code becomes slightly simpler 🤗

@gante
Copy link
Member Author

gante commented Feb 13, 2024

Model correctness depends on #28937, rebasing with its contents

@gante gante force-pushed the standardize_cache_api branch from 0ab4af6 to 1f88b6d Compare February 14, 2024 12:20
@gante gante force-pushed the standardize_cache_api branch from 1f88b6d to e69eec2 Compare February 15, 2024 15:15
@gante
Copy link
Member Author

gante commented Feb 21, 2024

Closed in place of #29180 (merge conflicts 🤷 )

@gante gante closed this Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants