-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: FileCachedDiskArray #195
Comments
This is a nice idea, in particular when dealing with remote arrays. Thinking about an appropriate abstraction, would it be possible to handle this through the LRU Cache itself? DiskArrays only implements the chunk logic for cached arrays but the caching of data itself is completely handled by LRUCache.jl. Maybe these mmap and disk ideas would be good extensions to that package? |
It looks like LRUCache does not really care what to cache, which is nice so I added an option to |
That sounds great! Excited to see it. This would probably also interact well with RechunkedDiskArray ( |
For this use case, would you need a cache that is a bit more permanent? The current implementation stores the cached chunks in /tmp and cleans automatically when the process exits. Feel free to propose an interface for a more permanent cache if you think of use cases where you need the cache to survive sessions. |
It would be nice to have a larger memory limit on CachedDiskArrays when doing computation, and to not occupy RAM that could be used for data or computation.
In this case, might it make sense to have a
FileCachedDiskArray
, that does the same thing as CachedDiskArray but using files instead of a RAM cache? Or maybe even unpack andmmap
a whole disk array if it's sufficiently small. Not sure what the best abstraction is here, but more flexibility would always be good.The text was updated successfully, but these errors were encountered: