The specified blob does not exist when downloading an embedded backup from azure blob storage

[oliverwoe-avl]

Downloading with CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE enabled yields this error:

Action download mybackup-135946 failed: one of Download go-routine return error: one of downloadTableData go-routine return error: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/runner/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.15.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=BlobNotFound) =====
Description=The specified blob does not exist.
RequestId:a620bb5a-b01e-0024-755c-2f0c0a000000
Time:..., Details: 
   Code: BlobNotFound
   GET https://container.blob.core.windows.net/backups/mybackup-135946/shadow/mybackup/TINYLOG_TABLE/backups_azure/author.bin/mybackup-135946/shadow/mybackup/TINYLOG_TABLE/backups_azure/author.bin?timeout=901
   Authorization: REDACTED

According to this error message the path is not built correctly. Instead of using https://container.blob.core.windows.net/backups/mybackup-135946/shadow/mybackup/TINYLOG_TABLE/backups_azure/author.bin (which exists), another part (/mybackup-135946/shadow/mybackup/TINYLOG_TABLE/backups_azure/author.bin?timeout=901) is appended.

The result is this weird url https://container.blob.core.windows.net/backups/mybackup-135946/shadow/mybackup/TINYLOG_TABLE/backups_azure/author.bin/mybackup-135946/shadow/mybackup/TINYLOG_TABLE/backups_azure/author.bin?timeout=901.

This is the config I am using currently:
clickhouse-backup:

- name: ALLOW_EMPTY_BACKUPS
  value: "true"
- name: API_LISTEN
  value: "0.0.0.0:7171"
- name: API_CREATE_INTEGRATION_TABLES
  value: "true"
- name: AZBLOB_COMPRESSION_FORMAT
  value: "none"
- name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
  value: "true"
- name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
  value: "backups_azure"
- name: CLICKHOUSE_TIMEOUT
  value: "24h"

clickhouse:

<yandex>
    <backups>
        <allowed_disk>backups_local</allowed_disk>
        <allowed_disk>backups_azure</allowed_disk>
    </backups>
    <storage_configuration>
        <disks>
            <backups_azure>
                <account_key>...</account_key>
                <account_name>container</account_name>
                <cache_enabled>false</cache_enabled>
                <container_name>backups</container_name>
                <storage_account_url>https://container.blob.core.windows.net</storage_account_url>
                <type>azure_blob_storage</type>
            </backups_azure>
            <backups_local>
                <path>/var/lib/clickhouse/backup/</path>
                <type>local</type>
            </backups_local>
        </disks>
    </storage_configuration>
</yandex>

PS: thanks for helping me out with the upload part! And I am really sorry for bothering you twice today

[Slach]

it looks like bug

could you share

SELECT * FROM system.disks;
SELECT * FROM system.storage_policies;

on destination clickhouse server

[Slach]

which clickhouse-backup version do you use?
try to
clickhouse-backup:2.4.14

[oliverwoe-avl]

which clickhouse-backup version do you use? try to clickhouse-backup:2.4.14

Yes I updated now to version 2.4.14 but the issue still persists. Clickhouse version is 23.8.2.7

it looks like bug

could you share

SELECT * FROM system.disks;
SELECT * FROM system.storage_policies;

on destination clickhouse server

SOURCE
name         |path                                    |free_space          |total_space         |unreserved_space    |keep_free_space|type              |is_encrypted|is_read_only|is_write_once|is_remote|is_broken|cache_path|
-------------+----------------------------------------+--------------------+--------------------+--------------------+---------------+------------------+------------+------------+-------------+---------+---------+----------+
backups_azure|/var/lib/clickhouse/disks/backups_azure/|18446744073709551615|18446744073709551615|18446744073709551615|              0|azure_blob_storage|           0|           0|            0|        1|        0|          |
backups_local|/var/lib/clickhouse/backup/             |        521720692736|        536870912000|        521720692736|              0|local             |           0|           0|            0|        0|        0|          |
default      |/var/lib/clickhouse/                    |        521720692736|        536870912000|        521720692736|              0|local             |           0|           0|            0|        0|        0|          |

policy_name|volume_name|volume_priority|disks      |volume_type|max_data_part_size|move_factor|prefer_not_to_merge|perform_ttl_move_on_insert|load_balancing|
-----------+-----------+---------------+-----------+-----------+------------------+-----------+-------------------+--------------------------+--------------+
default    |default    |              1|['default']|JBOD       |                 0|        0.0|                  0|                         1|ROUND_ROBIN   |


DESTINATION
name         |path                                    |free_space          |total_space         |unreserved_space    |keep_free_space|type              |is_encrypted|is_read_only|is_write_once|is_remote|is_broken|cache_path|
-------------+----------------------------------------+--------------------+--------------------+--------------------+---------------+------------------+------------+------------+-------------+---------+---------+----------+
backups_azure|/var/lib/clickhouse/disks/backups_azure/|18446744073709551615|18446744073709551615|18446744073709551615|              0|azure_blob_storage|           0|           0|            0|        1|        0|          |
backups_local|/var/lib/clickhouse/backup/             |       2026805198848|       2199023255552|       2026805198848|              0|local             |           0|           0|            0|        0|        0|          |
default      |/var/lib/clickhouse/                    |       2026805198848|       2199023255552|       2026805198848|              0|local             |           0|           0|            0|        0|        0|          |

policy_name|volume_name|volume_priority|disks      |volume_type|max_data_part_size|move_factor|prefer_not_to_merge|perform_ttl_move_on_insert|load_balancing|
-----------+-----------+---------------+-----------+-----------+------------------+-----------+-------------------+--------------------------+--------------+
default    |default    |              1|['default']|JBOD       |                 0|        0.0|                  0|                         1|ROUND_ROBIN   |

Disks and policies on both servers are the same (I can also re-produce this issue when source = destination host)

And just for clarification I use the API endpoint /backups/download/{backup_name} endpoint (also to create and upload), if this helps...

[Slach]

Same recommendations with #799 (comment)

you need to change /etc/clickhouse-backup/config.yml

azblob:
  path: "/clickhouse_backup_data/"
  object_disk_path: "/clickhouse_backup_object_disk_data/"