Skip to content

Conversation

coyotte508
Copy link
Member

@coyotte508 coyotte508 commented Mar 18, 2025

Fix #1284 : Store chunks in cache when they're reused by later terms, and free cache when a chunk is read for the last time

Follow up #1257

@coyotte508 coyotte508 added the hub @huggingface/hub related label Mar 18, 2025
@coyotte508 coyotte508 marked this pull request as ready for review March 19, 2025 00:10
@coyotte508
Copy link
Member Author

cc @bpronan @assafvayner for viz / review maybe

cc @hanouticelina too

@coyotte508 coyotte508 merged commit 204cd30 into main Mar 21, 2025
5 checks passed
@coyotte508 coyotte508 deleted the xet-download-optim branch March 21, 2025 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hub @huggingface/hub related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Keep in cache chunks that may be needed in later parts of the download stream
1 participant