custom TLS self-signed cert for S3 return error even when S3_DISABLE_CERT_VALIDATION=true

[sanjeev3d]

An error occurred while attempting to upload a backup to S3 [MINIO]. The error message indicates that the maximum number of attempts was exceeded and there was a failure to verify the TLS certificate.

Error Message

SELECT status, error FROM system.backup_actions WHERE command='upload chi-cliff-cliffcluster-replica0-shard0.ns-rm-yb-clicktt.svc.cluster.local-full-2024-07-25-07-50-12'

error: one of upload table go-routine return error: 
one of uploadTableData go-routine return error: can't upload: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error 
StatusCode: 0, RequestID: , HostID: , 
request send failed, 
Put "https://[xxxxxxxxxx]:30000/yb-backup/backup/shard-shard0/chi-cliff-cliffcluster-replica0-shard0.ns-rm-yb-clicktt.svc.cluster.local-full-2024-07-25-07-50-12/shadow/test_db/mydata/default_4_0_0_0.tar?x-id=PutObject": 
tls: failed to verify certificate: x509: certificate signed by unknown authority

The configmap for clickhouse-backup container is as follows:

data:
  ALLOW_EMPTY_BACKUPS: "true"
  API_CREATE_INTEGRATION_TABLES: "true"
  API_LISTEN: 0.0.0.0:7171
  BACKUPS_TO_KEEP_REMOTE: "3"
  LOG_LEVEL: debug
  REMOTE_STORAGE: s3
  S3_ACCESS_KEY: xxxxx
  S3_ACL: private
  S3_BUCKET: yb-backup
  S3_DEBUG: "true"
  S3_DISABLE_SSL: "true"
  S3_ENDPOINT: xxxxxxxx
  S3_FORCE_PATH_STYLE: "true"
  S3_PATH: backup/shard-{shard}
  S3_DISABLE_CERT_VALIDATION: "true"
  S3_SECRET_KEY: xxxxx

[Slach]

S3_ENDPOINT: xxxxxxxx

what exactly endpoint do you use?

http://yourhost:port?
or
https://yourhost:port?

[Slach]

Which s3 provider do you trying to use? minio?

[sanjeev3d]

@Slach

I have tried with both but getting these error in two different case

case 1: http://yourhost:port/?

SELECT status,error FROM system.backup_actions WHERE command='upload  chi-cliff-cliffcluster-replica0-shard0.ns-rm-yb-clicktt.svc.click.local-full-2024-07-25-16-01-13'
error	one of upload table go-routine return error: one of uploadTableData go-routine return error: can\'t upload: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: , HostID: , deserialization failed, failed to copy error response body, read tcp [xxxxxx]:43800->[xxxxxxxx]:30000: read: connection reset by peer

case2: https://yourhost:port/?

error one of upload table go-routine return error: one of uploadTableData go-routine return error: can\'t upload: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "https://[xxxxxx]:30000/yb-backup/backup/shard-shard0/chi-cliff-cliffcluster-replica0-shard0.ns-rm-yb-clicktt.svc.click.local-full-2024-07-25-07-50-12/shadow/test_db/mydata/default_4_0_0_0.tar?x-id=PutObject": tls: failed to verify certificate: x509: certificate signed by unknown authority

Yes I'm using MINIO

[Slach]

ok. how exactly do you run minio?
this is helm chart or something else?
by default minio docker image use http
provide more context and details about your minio installation

remove

S3_DISABLE_SSL: "true"

and continue to use https://xxx:30000/

[sanjeev3d]

Minio is deployed using Operator in k8s and this secured one [SSL integrated]

This is our centralized MINIO.

In our another env I have tested backup with unsecured MINIO [without SSL integration] and that is working fine

[sanjeev3d]

After removing S3_DISABLE_SSL: "true" still getting auth issue

error one of upload table go-routine return error: one of uploadTableData go-routine return error: can\'t upload: operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "https://[240b:c020:104:5479:b442:2::]:30000/yb-backup/backup/shard-shard0/chi-cliff-cliffcluster-replica0-shard0.ns-rm-yb-clicktt.svc.uhn7tt7r5.local-full-2024-07-25-16-42-48/shadow/test_db/mydata/default_4_0_0_0.tar?x-id=PutObject": tls: failed to verify certificate: x509: certificate signed by unknown authority

[Slach]

@sanjeev3d could you check 2.5.22 version with
S3_DEBUG=1 clickhouse-backup create_remote test_backup
and share logs without sensitive information, if failed?

[2peter3]

@sanjeev3d try to setup the bucket with Current Status: Versioned in the minio GUI.

[Slach]

@2peter3 is issue still relevant for you with minio on 2.6.2 clickhouse-backup version?