Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

An error occurred while uploading backup to Cos #1053

Open
liubo-it opened this issue Dec 3, 2024 · 4 comments
Open

An error occurred while uploading backup to Cos #1053

liubo-it opened this issue Dec 3, 2024 · 4 comments

Comments

@liubo-it
Copy link

liubo-it commented Dec 3, 2024

command：

/data/tingyun/clickhouse-backup/clickhouse-backup   upload 2024-12-02 --config=/data/tingyun/clickhouse-backup/apm.yml

config

general:
  remote_storage: cos
  max_file_size: 107374182400
  disable_progress_bar: false
  backups_to_keep_local: 1
  backups_to_keep_remote: 32
  log_level: info
  allow_empty_backups: false
clickhouse:
  username: default
  password: xxx
  host: wukong-release-apmck-iossd-505.txbj.tingyun.com
  port: 9000
  disk_mapping: {}
  skip_tables:
  - system.*
  - information_schema.*
  - INFORMATION_SCHEMA.*
  timeout: 5m
  freeze_by_part: false
  secure: false
  skip_verify: false
  sync_replicated_tables: true
  skip_sync_replica_timeouts: true
  log_sql_queries: false
  check_replicas_before_attach: false
  check_parts_columns: false
cos:
  url: "https://wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com"
  timeout: 3m
  secret_id: "xxx"
  secret_key: "xxx"
  path: "apm/03/wukong-release-apmck-iossd-505.txbj.tingyun.com"
  compression_format: tar
  compression_level: 1

description
An error occurred when I used upload to upload backup data to Cos.

error

2024-12-02 20:34:53.508 INF pkg/backup/upload.go:168 > done duration=65ms operation=upload_data progress=141/158 size=860B table=apm.svr_metric_data_baseline_application_instance_dist version=2.6.3
2024-12-02 20:34:53.555 INF pkg/backup/upload.go:168 > done duration=75ms operation=upload_data progress=142/158 size=650B table=aspm.aspm_risk_trace_local_request_count_dist version=2.6.3
2024-12-02 20:34:53.585 INF pkg/backup/upload.go:168 > done duration=77ms operation=upload_data progress=143/158 size=1.67KiB table=apm.svr_component_op_trace_dist version=2.6.3
2024-12-02 20:34:53.611 INF pkg/backup/upload.go:168 > done duration=56ms operation=upload_data progress=144/158 size=694B table=apm.svr_log_message_dist version=2.6.3
2024-12-02 20:34:53.631 INF pkg/backup/upload.go:168 > done duration=46ms operation=upload_data progress=145/158 size=634B table=aspm.aspm_risk_trace_local_user_count_dist version=2.6.3
2024-12-02 20:34:53.655 INF pkg/backup/upload.go:168 > done duration=44ms operation=upload_data progress=146/158 size=1.64KiB table=apm.svr_action_data_day_dist version=2.6.3
2024-12-02 20:34:53.672 INF pkg/backup/upload.go:168 > done duration=41ms operation=upload_data progress=147/158 size=844B table=apm.svr_component_trace_all version=2.6.3
2024-12-02 20:34:53.692 INF pkg/backup/upload.go:168 > done duration=37ms operation=upload_data progress=148/158 size=596B table=apm.svr_jar_class_index_dist version=2.6.3
2024-12-02 20:34:53.725 INF pkg/backup/upload.go:168 > done duration=53ms operation=upload_data progress=149/158 size=846B table=apm.svr_component_trace_dist version=2.6.3
2024-12-02 20:34:53.744 INF pkg/backup/upload.go:168 > done duration=52ms operation=upload_data progress=150/158 size=689B table=aspm.aspm_sqlquery_dist version=2.6.3
2024-12-02 20:34:53.783 INF pkg/backup/upload.go:168 > done duration=58ms operation=upload_data progress=151/158 size=1.37KiB table=apm.svr_action_data_baseline_server_service_group_dist version=2.6.3
2024-12-02 20:34:53.830 INF pkg/backup/upload.go:168 > done duration=86ms operation=upload_data progress=152/158 size=1.12KiB table=aspm.aspm_trace_risk_dist version=2.6.3
2024-12-02 20:34:53.856 INF pkg/backup/upload.go:168 > done duration=72ms operation=upload_data progress=153/158 size=541B table=apm.svr_health_targets_dist version=2.6.3
2024-12-02 20:34:53.882 INF pkg/backup/upload.go:168 > done duration=52ms operation=upload_data progress=154/158 size=1.83KiB table=apm.svr_error_trace_dist version=2.6.3
2024-12-02 20:34:53.902 INF pkg/backup/upload.go:168 > done duration=46ms operation=upload_data progress=155/158 size=1.36KiB table=apm.svr_action_data_baseline_server_biz_system_dist version=2.6.3
2024-12-02 20:34:53.917 INF pkg/backup/upload.go:168 > done duration=35ms operation=upload_data progress=156/158 size=782B table=apm.svr_health_events_dist version=2.6.3
2024-12-02 20:34:53.951 INF pkg/backup/upload.go:168 > done duration=48ms operation=upload_data progress=157/158 size=553B table=aspm.aspm_trace_total_dist version=2.6.3
2024-12-02 20:34:53.982 INF pkg/backup/upload.go:168 > done duration=64ms operation=upload_data progress=158/158 size=1.00KiB table=apm.svr_global_topo_edge_test_dist version=2.6.3
2024-12-02 20:34:59.230 INF pkg/backup/upload.go:168 > done duration=1m8.479s operation=upload_data progress=57/158 size=1.50GiB table=apm.svr_component_op_trace_local version=2.6.3
2024-12-02 20:35:54.099 INF pkg/backup/upload.go:168 > done duration=2m5.841s operation=upload_data progress=53/158 size=3.08GiB table=apm.svr_metric_data_day_local version=2.6.3
2024-12-02 20:36:06.058 INF pkg/backup/upload.go:168 > done duration=2m17.196s operation=upload_data progress=56/158 size=2.00GiB table=apm.svr_log_message_local version=2.6.3
2024-12-02 20:36:13.941 INF pkg/backup/upload.go:168 > done duration=2m27.133s operation=upload_data progress=2/158 size=10.23GiB table=apm.svr_action_data_baseline_server_application_action_local version=2.6.3
2024-12-02 20:39:36.830 INF pkg/backup/upload.go:168 > done duration=5m37.499s operation=upload_data progress=59/158 size=3.52GiB table=apm.svr_health_targets_local version=2.6.3
2024-12-02 20:41:13.074 INF pkg/backup/upload.go:168 > done duration=7m25.816s operation=upload_data progress=36/158 size=15.22GiB table=apm.sys_gauge_data_hour_local version=2.6.3
2024-12-02 20:43:14.404 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /1-20240825_0_18845_31/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.406 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
2024-12-02 20:43:14.405 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled
@Slach
Copy link
Collaborator

Slach commented Dec 3, 2024

could you remove max_file_size from general section

general:
  max_file_size: 107374182400

and increase timeout to 10m

cos:
 timeout: 10m
 debug: true

and try again and share logs?

@liubo-it
Copy link
Author

liubo-it commented Dec 4, 2024

could you remove max_file_size from general section

general:
  max_file_size: 107374182400

and increase timeout to 10m

cos:
 timeout: 10m
 debug: true

and try again and share logs?

PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Faspm%2Faspm_trace_risk_log_local%2Fdefault_9%252D20241103_0_266_123.tar HTTP/1.1^M
Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M
Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733314541;1733318141&q-key-time=1733314541;1733318141&q-header-list=host&q-url-param-list=&q-signature=7b1f41e622448e25d5a4e74be37fd8187a944a99^M
User-Agent: cos-go-sdk-v5/0.7.55^M
^M


PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Faspm%2Faspm_trace_risk_log_local%2Fdefault_9%252D20241201_116_116_0.tar HTTP/1.1^M
Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M
Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733314541;1733318141&q-key-time=1733314541;1733318141&q-header-list=host&q-url-param-list=&q-signature=1664c28baaa5fd22f4eada1388a26dc9b8ad91f6^M
User-Agent: cos-go-sdk-v5/0.7.55^M
^M


PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Faspm%2Faspm_instance_event_log%2Fdefault_12%252D20241110_0_0_0.tar HTTP/1.1^M
Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M
Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733314541;1733318141&q-key-time=1733314541;1733318141&q-header-list=host&q-url-param-list=&q-signature=8a97d26562b12052e844e01cfeaf60cad84eadc0^M
User-Agent: cos-go-sdk-v5/0.7.55^M
^M


/UploadCompressedStream
Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733316961;1733320561&q-key-time=1733316961;1733320561&q-header-list=host&q-url-param-list=&q-signature=3577b2c3c1d14aebfc21fbbc2d166509e2b0fd4e^M
User-Agent: cos-go-sdk-v5/0.7.55^M
^M


HTTP/1.1 200 OK^M
Content-Length: 20992^M
Accept-Ranges: bytes^M
Connection: keep-alive^M
Content-Type: application/x-tar^M
Date: Wed, 04 Dec 2024 12:56:01 GMT^M
Etag: "972a07539bb4f13912f38d87f451bd05"^M
Last-Modified: Wed, 04 Dec 2024 12:56:01 GMT^M
Server: tencent-cos^M
X-Cos-Hash-Crc64ecma: 5261643989400821985^M
X-Cos-Request-Id: Njc1MDUxNjFfNzEwN2YxZV9iZDM4Xzc3ZWRkZDg=^M
^M

PUT /apm%2F03%2Fwukong-release-apmck-iossd-505.txbj.tingyun.com%2F2024-12-04%2Fshadow%2Fapm%2Fsvr_component_data_min_local%2Fdefault_14%252D20241124_0_24124_39.tar HTTP/1.1^M
Host: wukong-apm-clickhouse-1251882848.cos.ap-beijing.myqcloud.com^M
Authorization: q-sign-algorithm=sha1&q-ak=xxx&q-sign-time=1733316961;1733320561&q-key-time=1733316961;1733320561&q-header-list=host&q-url-param-list=&q-signature=eabbed16fbe847ab9a2a4a4c1ac688c906d507e0^M
User-Agent: cos-go-sdk-v5/0.7.55^M
^M


2024-12-04 20:56:01.917 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /10-20241124_0_31339_15/component_exclusive.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241201_20622_23749_5/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241201_0_16817_7/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_0_15275_8/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_15276_28242_6/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /12-20241124_0_46582_22/component_count.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /4-20241124_0_53903_17/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241201_16818_20621_5/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_54293_56925_5/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_28243_40264_6/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241124_40265_54292_6/action_guid.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240929_0_23827_16/success_count.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241103_0_55050_15/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241006_0_48477_19/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241013_0_50243_17/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241027_0_45805_14/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241020_0_48849_14/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240825_0_57638_16/resp_time.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240915_0_55707_15/count.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240922_28971_57292_14/resp_time_micro.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240901_0_54628_12/count.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240908_21782_53096_11/resp_time_micro.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20241110_20970_60208_19/resp_time_micro.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240908_0_21781_12/success_count.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: file /6-20240922_0_28970_14/resp_time_micro.bin: writing data: io: read/write on closed pipe
2024-12-04 20:56:01.918 ERR pkg/backup/upload.go:561 > UploadCompressedStream return error: context canceled

It seems that there are still issues

@liubo-it
Copy link
Author

hi, @Slach
I would like to ask a question on this issue。

For example, if I backup a full amount of data and retrieve it when I restore it, is it due to Clickhouse's own TTL policy expiring if there is data that exceeds the retention time? Or will it not recover?

@Slach
Copy link
Collaborator

Slach commented Dec 10, 2024

question is not related to issue

when you successful, create backup, this is a snapshot of your data parts with tables and other sql object schema definitions

during restore, you will restore backuped data

but if your table contains TTL and TTL is expired then TTL clause will applied during backdroing merges or when TTL thread will spawn ever 4 hour (look https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#merge_with_ttl_timeout)

to avoid it
use https://clickhouse.com/docs/en/sql-reference/statements/system#stop-ttl-merges
SYSTEM STOP TTL MERGES

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Slach @liubo-it