/backup/download, /backup/upload API return context canceled in /backup/actions #814

mglotov · 2024-01-18T06:32:04Z

Downloading backups from GCS stopped working after upgrading clickhouse-backup to 2.4.14
Error message:

2024/01/18 06:23:07.646574  info done                      backup=chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 duration=393ms logger=backuper operation=download size=13.71KiB
2024/01/18 06:23:07.646825  info clickhouse connection closed logger=clickhouse
2024/01/18 06:23:07.646865  info Update backup metrics start (onlyLocal=true) logger=server
2024/01/18 06:23:07.646908  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/01/18 06:23:07.648085  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/01/18 06:23:07.648108  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/01/18 06:23:07.648141  warn can't get ClickHouse version: context canceled logger=clickhouse
2024/01/18 06:23:07.648195  info clickhouse connection closed logger=clickhouse
2024/01/18 06:23:07.648210 error UpdateBackupMetrics return error: context canceled logger=server
2024/01/18 06:23:07.648231 debug api.status.stop -> status.commands[0] == {ActionRowStatus:{Command:download --schema chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 Status:error Start:2024-01-18 06:23:07 Finish:2024-01-18 06:23:07 Error:context canceled} Ctx:<nil> Cancel:<nil>} logger=status
2024/01/18 06:25:19.409921  info Stopping API server       logger=server.Run
2024/01/18 06:25:19.410198 debug api.status.cancel -> status.commands[0] == {ActionRowStatus:{Command:download --schema chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 Status:cancel Start:2024-01-18 06:23:07 Finish:2024-01-18 06:25:19 Error:canceled during server stop} Ctx:<nil> Cancel:<nil>} logger=status
2024/01/18 06:25:19.410422  warn ListenAndServe get signal: http: Server closed logger=server.Restart

Clickhouse version:

chi-dc1-cluster1-0-0-0.chi-dc1-cluster1-0-0.test.svc.cluster.local :)         SELECT version()

SELECT version()

Query id: 31e1bd4c-f107-456d-a7c4-e8888927440c

┌─version()──┐
│ 22.8.21.38 │
└────────────┘

chi-dc1-cluster1-0-0-0.chi-dc1-cluster1-0-0.test.svc.cluster.local :)         SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'

SELECT value
FROM system.build_options
WHERE name = 'VERSION_INTEGER'

Query id: 500d8882-89fa-476d-86f1-cf8a3423ae31

┌─value────┐
│ 22008021 │
└──────────┘

The text was updated successfully, but these errors were encountered:

Slach · 2024-01-18T08:14:24Z

could you share
clickhouse-backup print-config
without sensitive credentials?

could you provide full log of download command?

It looks like you got SIGSTOP signal to clickhouse-backup process,

2024/01/18 06:25:19.410422  warn ListenAndServe get signal: http: Server closed logger=server.Restart

maybe this is not first time

After that, all commands will stop via cancel context, your command downloaded backup successfully but get context cancel during upgrade backup metrics which executed after each command

How do you run clickhouse? Is this a standalone Linux, docker or kubernetes environment?

mglotov · 2024-01-18T08:20:15Z

could you share clickhouse-backup print-config without sensitive credentials?

could you provide full log of download command?

It looks like you got SIGSTOP signal to clickhouse-backup process,
2024/01/18 06:25:19.410422  warn ListenAndServe get signal: http: Server closed logger=server.Restart
maybe this is not first time

After that, all commands will stop via cancel context, your command downloaded backup successfully but get context cancel during upgrade backup metrics which executed after each command

How do you run clickhouse? Is this a standalone Linux, docker or kubernetes environment?

Clickhouse is run using Altinity clickhouse operator.
The full log:

2024/01/18 06:23:07.241592  info API call POST /backup/download/chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 logger=server
2024/01/18 06:23:07.241869 debug api.status.inProgress -> len(status.commands)=0, inProgress=false logger=status
2024/01/18 06:23:07.243543 debug api.status.Start -> status.commands[0] == {ActionRowStatus:{Command:download --schema chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 Status:in progress Start:2024-01-18 06:23:07 Finish: Error:} Ctx:context.Background.WithCancel Cancel:0x49ff60} logger=status
2024/01/18 06:23:07.244232  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/01/18 06:23:07.246128  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/01/18 06:23:07.246183  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/01/18 06:23:07.248472  info SELECT count() is_disk_type_present FROM system.columns WHERE database='system' AND table='disks' AND name='type' logger=clickhouse
2024/01/18 06:23:07.251208  info SELECT path, any(name) AS name, any(type) AS type FROM system.disks GROUP BY path logger=clickhouse
2024/01/18 06:23:07.253939  info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts logger=clickhouse
2024/01/18 06:23:07.256036  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/01/18 06:23:07.258298  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/01/18 06:23:07.260462 debug /tmp/.clickhouse-backup-metadata.cache.GCS load 1 elements logger=gcs
2024/01/18 06:23:07.429718 debug /tmp/.clickhouse-backup-metadata.cache.GCS save 2 elements logger=gcs
2024/01/18 06:23:07.436746 debug prepare table METADATA concurrent semaphore with concurrency=8 len(tablesForDownload)=10 backup=chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 logger=backuper operation=download
2024/01/18 06:23:07.468575  info done                      backup=chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 duration=32ms logger=backuper operation=download size=424B table_metadata=test
2024/01/18 06:23:07.646574  info done                      backup=chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 duration=393ms logger=backuper operation=download size=13.71KiB
2024/01/18 06:23:07.646825  info clickhouse connection closed logger=clickhouse
2024/01/18 06:23:07.646865  info Update backup metrics start (onlyLocal=true) logger=server
2024/01/18 06:23:07.646908  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/01/18 06:23:07.648085  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/01/18 06:23:07.648108  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/01/18 06:23:07.648141  warn can't get ClickHouse version: context canceled logger=clickhouse
2024/01/18 06:23:07.648195  info clickhouse connection closed logger=clickhouse
2024/01/18 06:23:07.648210 error UpdateBackupMetrics return error: context canceled logger=server
2024/01/18 06:23:07.648231 debug api.status.stop -> status.commands[0] == {ActionRowStatus:{Command:download --schema chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 Status:error Start:2024-01-18 06:23:07 Finish:2024-01-18 06:23:07 Error:context canceled} Ctx:<nil> Cancel:<nil>} logger=status
2024/01/18 06:25:19.409921  info Stopping API server       logger=server.Run
2024/01/18 06:25:19.410198 debug api.status.cancel -> status.commands[0] == {ActionRowStatus:{Command:download --schema chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 Status:cancel Start:2024-01-18 06:23:07 Finish:2024-01-18 06:25:19 Error:canceled during server stop} Ctx:<nil> Cancel:<nil>} logger=status
2024/01/18 06:25:19.410422  warn ListenAndServe get signal: http: Server closed logger=server.Restart

mglotov · 2024-01-18T08:30:56Z

could you share clickhouse-backup print-config without sensitive credentials?

could you provide full log of download command?

It looks like you got SIGSTOP signal to clickhouse-backup process,
2024/01/18 06:25:19.410422  warn ListenAndServe get signal: http: Server closed logger=server.Restart
maybe this is not first time

After that, all commands will stop via cancel context, your command downloaded backup successfully but get context cancel during upgrade backup metrics which executed after each command

How do you run clickhouse? Is this a standalone Linux, docker or kubernetes environment?

chi-dc1-cluster1-0-1-0:/# clickhouse-backup print-config
general:
    remote_storage: gcs
    max_file_size: 0
    disable_progress_bar: true
    backups_to_keep_local: 0
    backups_to_keep_remote: 14
    log_level: debug
    allow_empty_backups: true
    download_concurrency: 4
    upload_concurrency: 1
    use_resumable_state: true
    restore_schema_on_cluster: ""
    upload_by_part: true
    download_by_part: true
    restore_database_mapping: {}
    retries_on_failure: 3
    retries_pause: 30s
    watch_interval: 1h
    full_interval: 24h
    watch_backup_name_template: shard{shard}-{type}-{time:20060102150405}
    sharded_operation_mode: ""
    cpu_nice_priority: 15
    io_nice_priority: idle
    retriesduration: 30s
    watchduration: 1h0m0s
    fullduration: 24h0m0s
clickhouse:
    username: default
    password: ""
    host: localhost
    port: 9000
    disk_mapping: {}
    skip_tables:
        - system.*
        - INFORMATION_SCHEMA.*
        - default.*
    skip_table_engines: []
    timeout: 5m
    freeze_by_part: false
    freeze_by_part_where: ""
    use_embedded_backup_restore: false
    embedded_backup_disk: ""
    backup_mutations: true
    restore_as_attach: false
    check_parts_columns: true
    secure: false
    skip_verify: false
    sync_replicated_tables: false
    log_sql_queries: true
    config_dir: /etc/clickhouse-server/
    restart_command: exec:systemctl restart clickhouse-server
    ignore_not_exists_error_during_freeze: true
    check_replicas_before_attach: true
    tls_key: ""
    tls_cert: ""
    tls_ca: ""
    debug: false
s3:
    access_key: ""
    secret_key: ""
    bucket: ""
    endpoint: ""
    region: us-east-1
    acl: private
    assume_role_arn: ""
    force_path_style: false
    path: ""
    object_disk_path: ""
    disable_ssl: false
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: false
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 5
    part_size: 0
    max_parts_count: 5000
    allow_multipart_download: false
    object_labels: {}
    request_payer: ""
    debug: false
gcs:
    credentials_file: ""
    credentials_json: ""
    credentials_json_encoded: ""
    bucket: BUCKET_NAME
    path: backup/shard-{shard}
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    debug: false
    force_http: false
    endpoint: ""
    storage_class: STANDARD
    object_labels: {}
    custom_storage_class_map: {}
    client_pool_size: 12
cos:
    url: ""
    timeout: 2m
    secret_id: ""
    secret_key: ""
    path: ""
    compression_format: tar
    compression_level: 1
    debug: false
api:
    listen: 0.0.0.0:7171
    enable_metrics: true
    enable_pprof: false
    username: ""
    password: ""
    secure: false
    certificate_file: ""
    private_key_file: ""
    ca_cert_file: ""
    ca_key_file: ""
    create_integration_tables: false
    integration_tables_host: ""
    allow_parallel: false
    complete_resumable_after_restart: true
ftp:
    address: ""
    timeout: 2m
    username: ""
    password: ""
    tls: false
    skip_tls_verify: false
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 5
    debug: false
sftp:
    address: ""
    port: 22
    username: ""
    password: ""
    key: ""
    path: ""
    object_disk_path: ""
    compression_format: tar
    compression_level: 1
    concurrency: 5
    debug: false
azblob:
    endpoint_schema: https
    endpoint_suffix: core.windows.net
    account_name: ""
    account_key: ""
    sas: ""
    use_managed_identity: false
    container: ""
    path: ""
    object_disk_path: ""
    compression_level: 1
    compression_format: tar
    sse_key: ""
    buffer_size: 0
    buffer_count: 3
    max_parts_count: 5000
    timeout: 15m
custom:
    upload_command: ""
    download_command: ""
    list_command: ""
    delete_command: ""
    command_timeout: 4h
    commandtimeoutduration: 4h0m0s

Slach · 2024-01-18T12:58:16Z

could you share log from container with clickhouse-server (usually clickhouse or clickhouse-pod container name)
from the same time period?

Slach · 2024-01-18T12:59:25Z

for 2024-01-18 06:23:07

mglotov · 2024-01-22T06:21:42Z

could you share log from container with clickhouse-server (usually clickhouse or clickhouse-pod container name) from the same time period?

Explore-logs-2024-01-22 12_19_48.txt

Slach · 2024-01-22T06:36:01Z

could you upgrade to altinity/clickhouse-backup:2.4.18?
and increase timeout

clickhouse:
    timeout: 5m

via backup ConfigMap
or via CLICKHOUSE_TIMEOUT environment variable in podTemplates

mglotov · 2024-01-22T06:38:40Z

could you upgrade to altinity/clickhouse-backup:2.4.18? and increase timeout
clickhouse:
    timeout: 5m
via backup ConfigMap or via CLICKHOUSE_TIMEOUT environment variable in podTemplates

ok. I'll do it a bit later

Slach · 2024-01-22T08:40:34Z

sorry, please setup timeout: 30m
5m is default, but looks like some internals in clickhouse-go works incorrectly

Slach · 2024-01-24T20:15:28Z

@mglotov could you try altinity/clickhouse-backup:2.4.20 we improve connection pooling to ClickHouse, it could help

mglotov · 2024-01-25T05:54:43Z

@mglotov could you try altinity/clickhouse-backup:2.4.20 we improve connection pooling to ClickHouse, it could help

Still getting the same error:

2024/01/25 05:53:26.981257  info chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45/access.tar already processed logger=resumable
2024/01/25 05:53:26.997522 debug chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45/configs.tar not exists on remote storage, skip download logger=downloadBackupRelatedDir
2024/01/25 05:53:26.997816  info done                      backup=chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 duration=73ms logger=backuper operation=download size=13.71KiB
2024/01/25 05:53:26.997918  info clickhouse connection closed logger=clickhouse
2024/01/25 05:53:26.997945  info Update backup metrics start (onlyLocal=true) logger=server
2024/01/25 05:53:26.997974  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/01/25 05:53:27.000781  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/01/25 05:53:27.000829  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/01/25 05:53:27.000846  warn can't get ClickHouse version: context canceled logger=clickhouse
2024/01/25 05:53:27.000898  info clickhouse connection closed logger=clickhouse
2024/01/25 05:53:27.001242 error UpdateBackupMetrics return error: context canceled logger=server
2024/01/25 05:53:27.001337 debug api.status.stop -> status.commands[1] == {ActionRowStatus:{Command:download --schema chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 Status:error Start:2024-01-25 05:53:26 Finish:2024-01-25 05:53:27 Error:context canceled} Ctx:<nil> Cancel:<nil>} logger=status

Tried:

to use 2.4.20 version
set CLICKHOUSE_TIMEOUT to 30m

Slach · 2024-01-25T07:34:39Z

ok something strange,
could you share
how exactly you run your backup restore workflow?
Do you use kind: CronJob?

Could you share the used manifests?

mglotov · 2024-01-25T07:36:54Z

ok something strange, could you share how exactly you run your backup restore workflow? Do you use kind: CronJob?

Could you share the used manifests?

I'm testing it using kind: Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: clickhouse-restore
  namespace: test
spec:
  backoffLimit: 1
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        app: clickhouse-restore
    spec:
      restartPolicy: Never
      containers:
        - name: run-restore
          image: bash
          imagePullPolicy: IfNotPresent
          env:
            - name: BACKUP_NAME
              value: chi-dc1-cluster1-0-0-full-2024-01-17-09-42-45 # Set backup_name that we need to restore
            # use all replicas in each shard to restore schema
            - name: CLICKHOUSE_SCHEMA_RESTORE_SERVICES
              value: chi-dc1-cluster1-0-0,chi-dc1-cluster1-0-1
            # use only first replica in each shard to restore data
            - name: CLICKHOUSE_DATA_RESTORE_SERVICES
              value: chi-dc1-cluster1-0-0
          command:
            - bash
            - -ec
            - |
              declare -A BACKUP_NAMES;
              CLICKHOUSE_SCHEMA_RESTORE_SERVICES=$(echo $CLICKHOUSE_SCHEMA_RESTORE_SERVICES | tr "," " ");
              CLICKHOUSE_DATA_RESTORE_SERVICES=$(echo $CLICKHOUSE_DATA_RESTORE_SERVICES | tr "," " ");
              #Install extra components
              apk add curl jq;

              for SERVER in $CLICKHOUSE_SCHEMA_RESTORE_SERVICES; do
                BACKUP_NAMES[$SERVER]="${BACKUP_NAME}";
                curl -s "$SERVER:7171/backup/download/${BACKUP_NAMES[$SERVER]}?schema" -X POST | jq .;
              done;

              for SERVER in $CLICKHOUSE_SCHEMA_RESTORE_SERVICES; do
                while [[ "in progress" == $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"download --schema ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; do
                  echo "Download is still in progress ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  sleep 1;
                done;
                if [[ "success" != $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"download --schema ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; then
                  echo "error download --schema ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"download --schema ${BACKUP_NAMES[$SERVER]}\") | .error";
                  exit 1;
                fi;
              done;

              for SERVER in $CLICKHOUSE_SCHEMA_RESTORE_SERVICES; do
                curl -s "$SERVER:7171/backup/restore/${BACKUP_NAMES[$SERVER]}?schema&rm&rbac" -X POST | jq .;
              done;

              for SERVER in $CLICKHOUSE_SCHEMA_RESTORE_SERVICES; do
                while [[ "in progress" == $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"restore --schema --rm --rbac ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; do
                  echo "Restore is still in progress ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  sleep 1;
                done;
                if [[ "success" != $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"restore --schema --rm --rbac ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; then
                  echo "error restore --schema --rm --rbac ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"restore --schema --rm --rbac ${BACKUP_NAMES[$SERVER]}\") | .error";
                  exit 1;
                fi;
                curl -s "$SERVER:7171/backup/delete/local/${BACKUP_NAMES[$SERVER]}" -X POST | jq .;
              done;

              for SERVER in $CLICKHOUSE_DATA_RESTORE_SERVICES; do
                BACKUP_NAMES[$SERVER]="${BACKUP_NAME}";
                curl -s "$SERVER:7171/backup/download/${BACKUP_NAMES[$SERVER]}" -X POST | jq .;
              done;

              for SERVER in $CLICKHOUSE_DATA_RESTORE_SERVICES; do
                while [[ "in progress" == $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"download ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; do
                  echo "Download is still in progress ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  sleep 1;
                done;
                if [[ "success" != $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"download ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; then
                  echo "error download ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"download ${BACKUP_NAMES[$SERVER]}\") | .error";
                  exit 1;
                fi;
              done;

              for SERVER in $CLICKHOUSE_DATA_RESTORE_SERVICES; do
                curl -s "$SERVER:7171/backup/restore/${BACKUP_NAMES[$SERVER]}?data" -X POST | jq .;
              done;

              for SERVER in $CLICKHOUSE_DATA_RESTORE_SERVICES; do
                while [[ "in progress" == $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"restore --data ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; do
                  echo "Restore is still in progress ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  sleep 1;
                done;
                if [[ "success" != $(curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"restore --data ${BACKUP_NAMES[$SERVER]}\") | .status") ]]; then
                  echo "error restore --data ${BACKUP_NAMES[$SERVER]} on $SERVER";
                  curl -s "$SERVER:7171/backup/actions" | jq -r ". | select(.command == \"restore --data ${BACKUP_NAMES[$SERVER]}\") | .error";
                  exit 1;
                fi;
                curl -s "$SERVER:7171/backup/delete/local/${BACKUP_NAMES[$SERVER]}" -X POST | jq .;
              done;

              echo "RESTORE FINISHED"

it works fine if I use 2.4.13 version.

Slach · 2024-01-25T09:14:59Z

ok. reproduced locally, will fix ASAP

…error, fix #814

Slach added this to the 2.4.21 milestone Jan 25, 2024

Slach changed the title ~~Downloading backups via API stopped working~~ /backup/download, /backup/upload API return context canceled in /backup/actions Jan 25, 2024

Slach closed this as completed in 5e296ff Jan 25, 2024

Slach added a commit that referenced this issue Jan 25, 2024

refactoring execution UpdateBackupMetrics, to avoid context canceled …

f2edb85

…error, fix #814

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

/backup/download, /backup/upload API return context canceled in /backup/actions #814

/backup/download, /backup/upload API return context canceled in /backup/actions #814

mglotov commented Jan 18, 2024

Slach commented Jan 18, 2024 •

edited

Loading

mglotov commented Jan 18, 2024

mglotov commented Jan 18, 2024

Slach commented Jan 18, 2024

Slach commented Jan 18, 2024

mglotov commented Jan 22, 2024

Slach commented Jan 22, 2024

mglotov commented Jan 22, 2024 •

edited

Loading

Slach commented Jan 22, 2024 •

edited

Loading

Slach commented Jan 24, 2024

mglotov commented Jan 25, 2024 •

edited

Loading

Slach commented Jan 25, 2024

mglotov commented Jan 25, 2024

Slach commented Jan 25, 2024

/backup/download, /backup/upload API return context canceled in /backup/actions #814

/backup/download, /backup/upload API return context canceled in /backup/actions #814

Comments

mglotov commented Jan 18, 2024

Slach commented Jan 18, 2024 • edited Loading

mglotov commented Jan 18, 2024

mglotov commented Jan 18, 2024

Slach commented Jan 18, 2024

Slach commented Jan 18, 2024

mglotov commented Jan 22, 2024

Slach commented Jan 22, 2024

mglotov commented Jan 22, 2024 • edited Loading

Slach commented Jan 22, 2024 • edited Loading

Slach commented Jan 24, 2024

mglotov commented Jan 25, 2024 • edited Loading

Slach commented Jan 25, 2024

mglotov commented Jan 25, 2024

Slach commented Jan 25, 2024

Slach commented Jan 18, 2024 •

edited

Loading

mglotov commented Jan 22, 2024 •

edited

Loading

Slach commented Jan 22, 2024 •

edited

Loading

mglotov commented Jan 25, 2024 •

edited

Loading