Add block support to AzureBlobStoreRepositoryTests #46664
Conversation
This commit adds support for Put Block API to the internal HTT server used in Azure repository integration tests.
tlrx
added
>test
|
Pinging @elastic/es-distributed |
| @Override | ||
| BlobRequestOptions getBlobRequestOptionsForWriteBlob() { | ||
| BlobRequestOptions options = new BlobRequestOptions(); | ||
| options.setSingleBlobPutThresholdInBytes(Math.toIntExact(ByteSizeUnit.MB.toBytes(1))); |
There was a problem hiding this comment.
This allows to lower the threshold between single Put Blob request and Put Blocks + Put Blocks List requests.
|
|
||
| } else if (Regex.simpleMatch("HEAD /container/*", request)) { | ||
| BytesReference blob = blobs.get(exchange.getRequestURI().toString()); | ||
| final BytesReference blob = blobs.get(exchange.getRequestURI().getPath()); |
There was a problem hiding this comment.
This was wrong (ie, query parameters were part of the blob name)
| protected String requestUniqueId(final HttpExchange exchange) { | ||
| // Azure SDK client provides a unique ID per request | ||
| return exchange.getRequestHeaders().getFirst(Constants.HeaderConstants.CLIENT_REQUEST_ID_HEADER); | ||
| final String requestId = exchange.getRequestHeaders().getFirst(Constants.HeaderConstants.CLIENT_REQUEST_ID_HEADER); |
There was a problem hiding this comment.
I did not notice it but HEAD and GET blob share the same request id. When reading a blob from Azure using input stream, the SDK first execute a HEAD request to know the size of the object to read and then retrieve the content using 1 or more requests, all with the same request id.
original-brownbear
left a comment
original-brownbear
left a comment
There was a problem hiding this comment.
LGTM :) Thanks again for this one!
| * AzureRepositoryPlugin that allows to set very low values for the Azure's client retry policy | ||
| * Test the snapshot and restore of an index which has large segments files. | ||
| */ | ||
| public void testSnapshotWithLargeSegmentFiles() throws Exception { |
There was a problem hiding this comment.
Are we gonna build S3 multi-part uploads as well and then move this to the abstract parent? :)
...-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobStoreRepositoryTests.java
Outdated
Show resolved
Hide resolved
|
Jenkins test this |
|
@elasticmachine update branch |
|
Thanks @original-brownbear ! |
This commit adds support for Put Block API to the internal HTTP server used in Azure repository integration tests. This allows to test the behavior of the Azure SDK client when the Azure Storage service returns errors when uploading Blob in multiple blocks or when downloading a blob using ranged downloads.
4 participants
This commit adds support for Put Block API to the internal HTT server
used in Azure repository integration tests. This allows to test the
behavior of the Azure SDK client when the Azure Storage service
returns errors when uploading Blob in multiple blocks or when
downloading a blob using ranged downloads.