deeplake does not support aliyun oss storage to make an empty dataset?

[wuhuaiwei]

my code is as followed :

ds = deeplake.empty(f's3://$bucket/{dataset_name}',
                            creds={
                                'aws_access_key_id': s3_ak,
                                'aws_secret_access_key': s3_sk,
                                'endpoint_url': 'https://oss-cn-shenzhen-internal.aliyuncs.com'
                            }
                           )

However, it raise errors that: An error occurred (SecondLevelDomainForbidden) when calling the GetObject operation: Please use virtual hosted style to access.

I read the oss docs, it says that aliyun oss only support virtual hosted type.

"Virtual hosted style是指将Bucket置于Host Header的访问方式。基于安全考虑，OSS仅支持virtual hosted访问方式。所以在S3迁移至OSS后，客户端应用需要进行相应设置。部分S3工具默认使用Path style，也需要进行相应配置，否则可能导致OSS报错，并禁止访问。"

So, I change my code as followed to the virtal hosted type:

ds = deeplake.empty(f's3://{dataset_name}',
                            creds={
                                'aws_access_key_id': s3_ak,
                                'aws_secret_access_key': s3_sk,
                                'endpoint_url': 'https://$bucket.oss-cn-shenzhen-internal.aliyuncs.com'
                            }
                           )

However, it raise error again: botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: The specified key does not exist.

I read the oss docs again, and find the doc:
https://developer.alibaba.com/docs/doc.htm?treeId=620&articleId=116348&docType=1

it says that aliyun oss doesn't support the ListObjectsV2 API, so it will raise error if we use aliyun oss as the deeplake storage.

Can anyone solve this problem or have a solution ? Thank you very much!

[nvoxland]

There are a couple places in https://github.com/activeloopai/deeplake/blob/f9884a6d1259af032c58c1aea4e728d3292f09e1/deeplake/core/storage/s3.py which use the list_objects_v2 and that is what's apparently not compatible with aliyun.

I don't have access to an aliyun system to be able to see what alternatives work and what other problems get hit beyond that ListObjectesV2. @wuhuaiwei, would you be able to find what it takes to get the s3.py calls to be compatible and send a pull requeset?

[wuhuaiwei]

It seems that when deeplake finds the dataset does not exist , it will check the storage path is empty or not. In this case, deeplake will use the list_objects_v2 API.

the execption happend in this code in the file 'dataset.py':

dataset.py:

  def _populate_meta(self, address: Optional[str] = None, verbose=True):
        if address is None:
            commit_id = self._get_commit_id_for_address("main", self.version_state)
        else:
            commit_id = self._get_commit_id_for_address(address, self.version_state)
        if dataset_exists(self.storage, commit_id):
            load_meta(self)
        **elif not self.storage.empty():  ### the execption happens** 
            # dataset does not exist, but the path was not empty
            raise PathNotEmptyException  

the detail execeptions are as followed:

Traceback (most recent call last):
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\dataset\dataset.py", line 253, in init
self._set_derived_attributes(address=address)
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\dataset\dataset.py", line 2367, in _set_derived_attributes
self._populate_meta(
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\dataset\dataset.py", line 1888, in _populate_meta
elif not self.storage.empty():
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\storage\provider.py", line 193, in empty
return len(self) - int(lock_key in self) <= 0
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\storage\lru_cache.py", line 360, in len
return len(self._all_keys())
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\storage\lru_cache.py", line 433, in _all_keys
key_set = set().union(key_set, self.cache_storage._all_keys())
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\storage\s3.py", line 355, in
return ("/".join(name.split("/")[len_path:]) for name in self._keys_iterator())
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\deeplake\core\storage\s3.py", line 339, in _keys_iterator
for page in paginator.paginate(
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\paginate.py", line 269, in iter
response = self._make_request(current_kwargs)
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\paginate.py", line 357, in _make_request
return self._method(**current_kwargs)
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\huaiwei.wu\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: The specified key does not exist.

[nvoxland]

It looks like it's in deeplake\core\storage\s3.py, line 339. It also gets used in rename on 407.

Are you able to create a custom version of s3.py that removes the list_objects_v2 in those places? If so, does everything work for you? Or do you hit errors with another difference in aliyun?