BlockCache::OpenFile : Failed to get block list

[Niko-CZ]

Which version of blobfuse was used?

blobfuse2-2.3.2-1.x86_64
fuse-2.9.2-11.el7.x86_64
fuse3-3.6.1-4.el7.x86_64
fuse3-libs-3.6.1-4.el7.x86_64

Which OS distribution and version are you using?

Red Hat Enterprise Linux release 8.9 (Ootpa) x86_64
4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

If relevant, please share your mount command.

/usr/bin/blobfuse2 mount /oracle/Blob_BAK --config-file=/etc/Blobfuse2-BlockCacheConfig.yaml;
cat /etc/Blobfuse2-BlockCacheConfig.yaml;
allow-other: true
logging:
type: base
level: log_debug
file-path: /var/log/blobfuse2/blobfuse2.log
max-file-size-mb: 64
file-count: 2
track-time: true
components:
-libfuse
-block_cache
-attr_cache
-azstorage
health_monitor:
enable-monitoring: true
stats-poll-interval-sec: 10
process-monitor-interval-sec: 30
output-path: /mnt/resource/blobfuse2_healthmon
monitor-disable-list:
-blobfuse_stats
libfuse:
fuse-trace: true
default-permission: 0777
attribute-expiration-sec: 240
entry-expiration-sec: 240
negative-entry-expiration-sec: 120
ignore-open-flags: true
block_cache:
block-size-mb: 256
mem-size-mb: 20480
prefetch: 20
parallelism: 128
attr_cache:
timeout-sec: 7200
azstorage:
type: block
account-name: xxx
endpoint: xxx
mode: msi
container: xxx
honour-acl: false

What was the issue encountered?

Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_DEBUG [utils.go (156)]: SDK(Retry) : response 200 Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_DEBUG [utils.go (156)]: SDK(Retry) : exit due to non-retriable status code Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_ERR [block_cache.go (349)]: BlockCache::OpenFile : Failed to get block list of X28_3965183084/X28_3965183084_20241008_51_1 [<nil>] Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_ERR [libfuse_handler.go (712)]: Libfuse::libfuse_open : Failed to open X28_3965183084/X28_3965183084_20241008_51_1 [failed to retrieve bloc k list for X28_3965183084/X28_3965183084_20241008_51_1]

In the same time file is being created:
ls -lsa /oracle/Blob_BAK/X28_3965183084/X28_3965183084_20241008_51_1
0 -rwxrwxrwx 1 root root 67117056 Oct 8 13:37 /oracle/Blob_BAK/X28_3965183084/X28_3965183084_20241008_51_1

blobfuse2_log_block_cache.log
We have backups working with the same configuration, but the streaming cache instead of block.
After reading of fresh release notes (https://github.com/Azure/azure-storage-fuse/blob/main/README.md):

streaming mode is being deprecated.

I decided to enable block caching changing configuration part:
stream: block-size-mb: 8 max-buffers: 16 buffer-size-mb: 384
To:
block_cache: block-size-mb: 256 mem-size-mb: 20480 prefetch: 20 parallelism: 128
But facing issue above with any combination of parameters in block_cache section.

Have you found a mitigation/solution?

No.

Please share logs if available.

File blobfuse2_log_block_cache.log is attached.

[ashruti-msft]

In the same time file is being created:

You mean to say that the file was created successfully but you saw that error message in the logs, is that correct?

[Niko-CZ]

In the same time file is being created:

You mean to say that the file was created successfully but you saw that error message in the logs, is that correct?

Yes, file created successfully with the 64M size.

[bharath-krishna]

Hi,

I am also facing exactly similar issue. Unable to list/create any folders in the container.
Here is a debug log,

Mon Nov 11 07:44:50 UTC 2024 : blobfuse2[10940] : [/mnt/XXXXXX] LOG_DEBUG [utils.go (156)]: SDK(Response) : ==> REQUEST/RESPONSE (Try=1/12.735586ms, OpTime=12.753664ms) -- RESPONSE RECEIVED
   HEAD https://XXXXXX.blob.core.windows.net/XXXXXX/data
   Accept: application/xml
   Authorization: REDACTED
   User-Agent: Azure-Storage-Fuse/2.3.2 (Ubuntu 22.04.5 LTS) azsdk-go-azblob/v1.4.0 (go1.22.4; linux)
   X-Ms-Date: Mon, 11 Nov 2024 07:44:50 GMT
   x-ms-version: 2024-05-04
   --------------------------------------------------------------------------------
   RESPONSE Status: 200 OK
   Accept-Ranges: bytes
   Content-Length: 0
   Date: Mon, 11 Nov 2024 07:44:50 GMT
   Etag: "0x8DCE72D5A742E74"
   Last-Modified: Tue, 08 Oct 2024 00:08:35 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Acl: user::rwx,group::r-x,other::---
   X-Ms-Blob-Type: BlockBlob
   X-Ms-Creation-Time: Tue, 08 Oct 2024 00:08:35 GMT
   X-Ms-Group: $superuser
   X-Ms-Lease-State: available
   X-Ms-Lease-Status: REDACTED
   X-Ms-Meta-Hdi_isfolder: REDACTED
   X-Ms-Owner: $superuser
   X-Ms-Permissions: rwxr-x---
   X-Ms-Request-Id: b2393362-b01e-0039-440d-34bc54000000
   X-Ms-Resource-Type: directory
   X-Ms-Server-Encrypted: REDACTED
   X-Ms-Version: 2024-05-04

Mon Nov 11 07:44:50 UTC 2024 : blobfuse2[10940] : [/mnt/XXXXXX] LOG_DEBUG [utils.go (156)]: SDK(Retry) : response 200
Mon Nov 11 07:44:50 UTC 2024 : blobfuse2[10940] : [/mnt/XXXXXX] LOG_DEBUG [utils.go (156)]: SDK(Retry) : exit due to non-retriable status code

Is there any fix for this?

[vibhansa-msft]

Can you share the complete log file with debug logs enabled. Above log shows success reply from the backend and does not point to any error condition. Also, can you share list does not work only for some time after mount or it does not work ever ?