Using azure blob fails to upload splits

[Barre]

After ingesting around 100M docs, I started to get such errors:

WARN merge{merge_split_id=01JCZ8FMKQ06XP94SYK95TB9E8 split_ids=["01JCY8PPA9CQ6QGCK29J4RCSG5", "01JCY998YBNJKP9PF9HMG6487S", "01JCY4SFWYDDYBCVRD2ZPVJ026"] typ=Merge}:uploader: quickwit_indexing::actors::uploader: Failed to upload split. Killing! cause=failed uploading key 01JCZ8FMKQ06XP94SYK95TB9E8.split in bucket azure://quickwit/indexes/

Of which the root cause seems to be:

2024-11-18T10:27:37.793408+01:00 merklemap quickwit[876138]: 0: storage error(kind=Service, source=Azure error wrapper(inner=server returned error status which will not be retried: 413))

My first guess is that quickwit is trying to upload parts that are too large? https://learn.microsoft.com/en-us/troubleshoot/azure/azure-storage/blobs/connectivity/request-body-large

Expected behavior
Using azure blob storage should not fail.

Configuration:

1. Quickwit 0.8.2 (x86_64-unknown-linux-gnu 2024-06-17T16:36:47Z 42766b8)

version: 0.8

index_id: data

doc_mapping:
  field_mappings:
    - name: id
      type: u64
      fast: true
      stored: true
    - name: raw_data
      type: bytes
      stored: true
      indexed: false

indexing_settings:
  commit_timeout_secs: 60

[Barre]

At first glance, it seems that my assumption is correct as quickwit is not doing any specific handling for azure very low max part size

https://azure.microsoft.com/en-us/blog/general-availability-larger-block-blobs-in-azure-storage/

https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-storage/src/object_storage/policy.rs#L73

[fulmicoton]

Thank you for your report. We won't have time to fix and experiment with Azure unfortunately. If someone wants to help with this?

[Barre]

FYI: After further testing, it seems like my PR fixes uploads but something is still wrong (corrupted splits?)

quickwit_search::error: search internal error: search failed for the following splits: 01JD7MEQT70400M5PNQFYVZ955, 01JD7MCX0GKKJ457TD63Q5EC51, 01JD7MB235CPVSDMTYN18DFHR7, 01JD7M96VCPBRZ9A7749EWHDYQ. For instance, split 01JD7MEQT70400M5PNQFYVZ955 failed with the following error message: internal error: `spawned task in root join failed: task 35394 panicked with message "assertion failed: end <= orig_range.end"`

I'd be happy to fix this issue myself but from what I understand, you won't have time to perform a review as of now?

[fulmicoton]

No I will be able to review, and even help. I just do not want to go through playing with Azure :)

[fulmicoton]

Oh I have nothing against Azure! It's just I don't have it configured right now.