Azure storage returns empty <Blobs>

[petr-tichy]

We have hit (a possibly undocumented) behavior of Azure List Blobs operation returning empty node:

<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://myaccount.blob.core.windows.net/" ContainerName="mycontainer"> <Marker>2!212!MDAwMTE1IWRhdGEvZmRiYmFja3VwL2t2cmFuZ2VzL3NuYXBzaG90LjAwMDAwNjYyMjQxNDYyNTg5Mi8xNS9yYW5nZSw2NjIzMDE4MTQ4MjUzLGVmNTIyNmU3NDZmZTQ5MWExOTZhNjE5YTA0YmM2MmJjLDEwNDg1NzYhMDAwMDI4ITk5OTktMTItMzFUMjM6NTk6NTkuOTk5OTk5OVoh</Marker>
 <MaxResults>1000</MaxResults>
 <Delimiter>/</Delimiter>
 <Blobs />
<NextMarker>2!212!MDAwMTE0IWRhdGEvZmRiYmFja3VwL2t2cmFuZ2VzL3NuYXBzaG90LjAwMDAwNjYyNDk3Mjg5MTI4Ny80L3JhbmdlLDY2MjUzNjM5MjY4MzMsMjgxNzgxOTI4ZGJkMjRhN2FmZTNiNTJkZTFhZjFkOGUsMTA0ODU3NiEwMDAwMjghOTk5OS0xMi0zMVQyMzo1OTo1OS45OTk5OTk5WiE-</NextMarker>
</EnumerationResults>" 

This causes

java.util.NoSuchElementException: null
	at java.base/java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:760)
	at java.base/java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:780)
	at com.google.common.collect.Iterators.getLast(Iterators.java:876)
	at com.google.common.collect.Iterables.getLast(Iterables.java:794)
	at org.gaul.s3proxy.S3ProxyHandler.handleBlobList(S3ProxyHandler.java:1464)
	at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:672)
	at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:76)
	at org.gaul.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

Comparing with debug output of az storage blob list, it seems this is OK and azure-cli simply issues a next request.

Thanks

Petr

[petr-tichy]

After some testing it looks like simply sending a response to s3 client without either one of CommonPrefixes or Contents elements is not expected, at least by s3cmd and our backup tool. I suspect that s3proxy would have to handle this internally, issuing the next request(s) and returning it to the client only when it finally contains either CommonPrefixes or Contents

s3cmd fails with

DEBUG: s3cmd version 2.1.0
...

DEBUG: Canonical Request:
GET
/backup/
delimiter=%2F&prefix=data%2Fbackup%2Fsnapshots%2F
host:localhost:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200820T191251Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-date': '20200820T191251Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=none/20200820/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=53e4e909916546be4527f62475fcbb2ace0778ded959864eb05f1a7a512aecf4', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(backup): localhost:8080
DEBUG: ConnMan.get(): re-using connection: http://localhost:8080#1
DEBUG: format_uri(): /backup/?delimiter=%2F&prefix=data%2Fbackup%2Fsnapshots%2F
DEBUG: Sending request method_string='GET', uri='/backup/?delimiter=%2F&prefix=data%2Fbackup%2Fsnapshots%2F', headers={'x-amz-date': '20200820T191251Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=none/20200820/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=53e4e909916546be4527f62475fcbb2ace0778ded959864eb05f1a7a512aecf4', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://localhost:8080#2)
DEBUG: Response:
{'data': b'<?xml version=\'1.0\' encoding=\'UTF-8\'?><ListBucketResult xmlns="h'
         b'ttp://s3.amazonaws.com/doc/2006-03-01/"><Name>backup</Name><Prefi'
         b'x>data/backup/snapshots/</Prefix><MaxKeys>1000</MaxKeys><Marker/>'
         b'<Delimiter>/</Delimiter><IsTruncated>true</IsTruncated><NextMarker>2'
         b'!160!MDAwMDc1IWRhdGEvZmRiYmFja3VwL3NuYXBzaG90cy9zbmFwc2hvdCw2NjM2ODY'
         b'xNDQzNzAzLDY3MjMyOTExNzcxNjksNDYzMTM0MTYzNDYyOCEwMDAwMjghMTYwMS0wMS0'
         b'wMVQwMDowMDowMC4wMDAwMDAwWiE-</NextMarker></ListBucketResult>',
 'headers': {'content-type': 'application/xml; charset=UTF-8',
             'date': 'Thu, 20 Aug 2020 19:12:51 GMT',
             'server': 'Jetty(9.2.z-SNAPSHOT)',
             'transfer-encoding': 'chunked',
             'x-amz-request-id': '4442587FB7D0A2F9'},
 'reason': 'OK',
 'status': 200}

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd -c s3cfg --no-ssl --debug ls s3://backup/data/backup/snapshots/
Problem: <class 'IndexError: list index out of range
S3cmd:   2.1.0
python:   3.8.5 (default, Jul 21 2020, 10:48:26)
[Clang 11.0.3 (clang-1103.0.32.62)]
environment LANG=cs_CZ.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3121, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 3030, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 152, in cmd_ls
    subcmd_bucket_list(s3, uri, cfg.limit)
  File "/usr/local/bin/s3cmd", line 188, in subcmd_bucket_list
    response = s3.bucket_list(bucket, prefix = prefix, limit = limit)
  File "/usr/local/Cellar/s3cmd/2.1.0/libexec/lib/python3.8/site-packages/S3/S3.py", line 321, in bucket_list
    for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
  File "/usr/local/Cellar/s3cmd/2.1.0/libexec/lib/python3.8/site-packages/S3/S3.py", line 368, in bucket_list_streaming
    uri_params['marker'] = current_prefixes[-1]["Prefix"]
IndexError: list index out of range