Commits on Mar 11, 2023

1. Fix race with eviction when reading from FileCache (#6592) …

   The previous implementation had an inherent race condition where a
   zero-reference count IndexInput read from the cache could be evicted
   before the IndexInput was cloned (and therefore had its reference count
   incremented). Since the IndexInputs are stateful this is very bad. The
   least-recently-used semantics meant that in a properly-configured system
   this would be unlikely since accessing a zero-reference count item would
   move it to be most-recently used and therefore least likely to be
   evicted. However, there was still a latent bug that was possible to
   encounter (see issue #6295).
   
   The only way to fix this, as far as I can see, is to change the cache
   behavior so that fetching an item from the cache atomically
   increments its reference count. This also led to a change to
   TransferManager to ensure that all requests for an item ultimately read
   through the cache to eliminate any possibility of a race. I have
   implement some concurrent unit tests that put the cache into a
   worst-case thrashing scenario to ensure that concurrent access never
   closes an IndexInput while it is still being used.
   
   Signed-off-by: Andrew Ross <andrross@amazon.com>
   (cherry picked from commit d139ebc)
   Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

   

   github-actions[bot] committed Mar 11, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 1797548
   • Browse repository at this point

   Copy the full SHA

   1797548 View commit details

   Browse the repository at this point in the history