Support Alibaba OSS Cloud

[AlexanderChiuluvB]

I want to support the tool upload data to OSS Cloud. Can I be assigned with this task?

[Slach]

great , any contribution is welcome
please look to ./test/ folder and try to figure out how to write tests

Alibaba OSS have something like Azurite or minio for test locally?

[AlexanderChiuluvB]

grea , any contribution is welcome please look to ./test/ folder and try to figure out how to write tests

Alibaba OSS have something like Azurite or minio for test locally?

OSS is S3 compatiable, minio can serve as a gateway for OSS

[Slach]

@AlexanderChiuluvB any news? Could I help with anything?

[AlexanderChiuluvB]

sorry I have been a little bit busy lately I will start push forward it may be 1 wk later

[john8628]

@AlexanderChiuluvB @Slach any update, we can test it recently

[Slach]

@john8628 is Alibaba OSS is S3 compatible, all should works with S3 protocol

[john8628]

@Slach thanks for your reply ,i will try it ;

[pigzhuzhu55]

Can you provide an example of Alibaba OSS by S3 configuration, thank you.

[john8628]

#all the config should be following the s3 config; it works for me

remote_storage: s3 
s3:
  access_key: "oss key"
  secret_key: "oss secret_key"
  bucket: "oss bucket"
  endpoint: "oss endpoint"
  region: "oss region"
  assume_role_arn: ""
  force_path_style: false
  path: dd/backup/pro      
  disable_ssl: false
  compression_level: 1
  compression_format: tar
  sse: ""
  disable_cert_verification: false
  storage_class: STANDARD
  concurrency: 1
  part_size: 0
  debug: false

[and1990]

@john8628 Hi, I use Oss as my remote storage, the backup data can be uploaded to Oss successfully, but clickhouse-backup list remote shows "The specified key does not exist". Do you have encountered this problem? Thanks.

[john8628]

Yes, it often happens when your path or your endpoint is not right;

[and1990]

@john8628 How did you solve this problem?

[john8628]

check your path ; ak/sk and endpoint is inner network?

[and1990]

check your path ; ak/sk and endpoint is inner network?

Could you give me some hints? This is my configuration.

s3:
    access_key: "xxxxx"
    secret_key: "xxxx"
    bucket: "test-clickhouse-backup"
    endpoint: "https://test-clickhouse-backup.oss-cn-beijing-internal.aliyuncs.com/"
    region: "oss-cn-beijing"
    acl: private
    assume_role_arn: ""
    force_path_style: false
    path: "core3"
    disable_ssl: false
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: false
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 3
    part_size: 0
    max_parts_count: 5000
    allow_multipart_download: false
    object_labels: {}
    debug: false
gcs:

Now, the data can be uploaded to oss successfully, but the command clickhouse-backup list remote shows "The specified key does not exist". The oss Web UI is like this.

[john8628]

oh , endpoint is not right , following the https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints ; and if your abosolute path is oss://{bucket}{specific path} then your path should be specific path;your path should be test-clickhouse-backup/core3/

[and1990]

oh , endpoint is not right , following the https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints ; and if your abosolute path is oss://{bucket}{specific path} then your path should be specific path;your path should be test-clickhouse-backup/core3/

Actually, I have tested before with the endpoint oss-cn-beijing-internal.aliyuncs.com, and it shows the error "HostID: oss-cn-beijing-internal.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access."

It means that the bucket name must be added to the head of endpoints.

[Slach]

@and1990
try

s3:
 bucket: test-clickhouse-backup
 force_path_style: true
 path: core3
 endpoint: https://oss-cn-beijing-internal.aliyuncs.com

[easayliu]

use old version ,it can help you

                - name: S3_ENDPOINT
                  value: oss-cn-hangzhou-internal.aliyuncs.com
                - name: S3_FORCE_PATH_STYLE
                  value: "false"