GCSFuse Errors in Google Cloud Console: CANCELLED ReadObject & NOT_FOUND GetObjectMetadata

[ashmeenkaur]

I'm seeing client errors in my Google Cloud Console Metrics Explorer dashboard related to GCSFuse. Specifically, I'm getting CANCELLED ReadObject and NOT_FOUND GetObjectMetadata errors. How can I troubleshoot and fix these?

[ashmeenkaur]

Both CANCELLED ReadObject and NOT_FOUND GetObjectMetadata errors are generally expected behaviors within GCSFuse and don't necessarily indicate a problem.

CANCELLED ReadObject:

• These errors typically occur when you perform range reads on a file handle.
• GCSFuse uses an adaptive reader to optimize performance. It starts with a larger buffer (200MB) and adjusts down to a smaller buffer (1MB) if it detects random access patterns.
• When a read request can't be served by the existing reader (e.g., it's outside the current buffer range), or if the next read is significantly far from the current reader position, GCSFuse closes the existing reader and opens a new one at the appropriate offset.
  This reader closure is what triggers the CANCELLED ReadObject error you're seeing.

NOT_FOUND GetObjectMetadata:

• These are also expected due to how GCSFuse handles file and directory lookups. When the kernel asks GCSFuse to check for a file/directory, GCSFuse needs to determine if it exists on GCS.

  • With --implicit-dir flag set to false: GCSFuse makes two stat calls to GCS - one for the object name itself and one for the object name with a trailing slash (to check if it's a directory). One of these is bound to fail, as only one (either the file or the directory) should exist.
  • With --implicit-dir flag set to true: GCSFuse makes one list call (to determine if an implicitly defined directory exists on GCS) and one stat call, and again one of these calls is expected to return no results (stat will fail with NOT_FOUND or List will return 0 results).
  • More details here.

Key Points:

• These errors are part of GCSFuse's internal workings and optimization strategies.
• They shouldn't impact your overall GCSFuse functionality.
• If you're experiencing performance issues or other unexpected behavior alongside these errors, further investigation might be needed, but in isolation, they are not a cause for concern.