Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare v2.5 #843

Merged
merged 89 commits into from
Apr 8, 2024
Merged

prepare v2.5 #843

merged 89 commits into from
Apr 8, 2024

Conversation

Slach
Copy link
Collaborator

@Slach Slach commented Feb 21, 2024

IMPROVEMENTS

  • complete removed support for legacy backups, created with version prior v1.0
  • removed disable_progress_bar config option and related progress bar code
  • added --delete-source parameter for upload and create_remote commands to explicitly delete local backup during upload, fix 777
  • added support for --env ENV_NAME=value cli parameter for allow dynamically override any config parameter, fix 821
  • added support for use_embedded_backup_restore: true with empty embedded_backup_disk value, tested on S3/GCS over S3/AzureBlobStorage, fix 695
  • --rbac, --rbac-only, --configs, --configs-only now works with use_embedded_backup_restore: true
    -- --data for restore with use_embedded_backup_restore: true will use allow_non_empty_tables=true to allow fix 756
  • added --diff-from-remote parameter for create command, will copy only new data parts object disk data, also allows to download properly object disk data from required backup during restore, fix 865
  • added support of native Clickhouse incremental backup for use_embedded_backup_restore: true fix 735
  • added GCS_CHUNK_SIZE config parameter, try to speedup GCS upload fix 874, thanks @dermasmid
  • added --remote-backup cli parameter to tables command and GET /backup/table, fix 778
  • added rbac_always_backup: true option to default config, will create backup for RBAC objects automatically, restore still require --rbac to avoid destructive actions, fix 793
  • added rbac_conflict_resolution: recreate option for RBAC object name conflicts during restore, fix 851
  • added upload_max_bytes_per_seconds and download_max_bytes_per_seconds config options to allow throttling without CAP_SYS_NICE, fix 817
  • added clickhouse_backup_in_progress_commands metric, fix 836
  • switched to golang 1.22
  • updated all third-party SDK to latest versions
  • added clickhouse/clickhouse-server:24.3 to CI/CD

BUG FIXES

  • continue S3_MAX_PARTS_COUNT default value from 2000 to 4000 to continue decrease memory usage for S3
  • changed minimal part size for multipart upload in CopyObject from 5Mb to 10Mb
  • restore SQL UDF functions after restore tables
  • execute ALTER TABLE ... DROP PARTITION instead of DROP TABLE for restore and restore_remote with parameters --data --partitions=..., fix 756
  • fix wrong behavior for freeze_by_part + freeze_by_part_where, fix 855
  • apply CLICKHOUSE_SKIP_TABLES_ENGINES during create command
  • fixed behavior for upload / download when .inner. table missing for MATERIALIZED VIEW by table pattern, fix 765
  • fixed ObjectDisks + CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE: true - shall skip upload object disk content, fix 799
  • fixed connection to clickhouse-server behavior when long clickhouse-server startup time and docker-entrypoint.d processing, will infinite reconnect each 5 seconds, until success, fix 857
  • fixed USE_EMBEDDED_BACKUP_RESTORE=true behavior to allow use backup disk with type local, fix 882
  • fixed wrong list command behavior, it shall scann all system.disks path not only default disk to find pratially created backups, fix 873
  • fixed create --rbac behavior, don't create access folder if no RBAC objects is present
  • fixed behavior when system.disks contains disk which not present in any storage_policies, fix 845

@coveralls
Copy link

coveralls commented Feb 28, 2024

Pull Request Test Coverage Report for Build 8590641064

Details

  • 1166 of 1654 (70.5%) changed or added relevant lines in 25 files are covered.
  • 55 unchanged lines in 9 files lost coverage.
  • Overall coverage increased (+0.5%) to 67.339%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/backup/watch.go 1 2 50.0%
pkg/filesystemhelper/filesystemhelper.go 38 40 95.0%
pkg/storage/sftp.go 9 11 81.82%
pkg/storage/s3.go 8 11 72.73%
pkg/clickhouse/clickhouse.go 31 35 88.57%
pkg/config/config.go 15 21 71.43%
pkg/server/server.go 22 28 78.57%
pkg/storage/general.go 41 47 87.23%
pkg/storage/cos.go 0 8 0.0%
pkg/backup/delete.go 39 49 79.59%
Files with Coverage Reduction New Missed Lines %
pkg/storage/sftp.go 1 73.23%
pkg/backup/upload.go 1 68.28%
pkg/storage/gcs.go 2 54.61%
pkg/backup/download.go 4 73.94%
pkg/storage/ftp.go 4 76.89%
pkg/backup/create.go 7 74.47%
pkg/backup/restore.go 10 66.96%
pkg/backup/delete.go 13 63.1%
pkg/storage/general.go 13 63.64%
Totals Coverage Status
Change from base Build 8455012384: 0.5%
Covered Lines: 8641
Relevant Lines: 12832

💛 - Coveralls

…edded_backup_disk` value, tested on S3/GCS over S3/AzureBlobStorage, fix #695
…`restore` and `restore_remote` with parameters `--data --partitions=...`, fix #756
…`restore` and `restore_remote` with parameters `--data --partitions=...`, fix #756
…`restore` and `restore_remote` with parameters `--data --partitions=...`, fix #756
…use `allow_non_empty_tables=true` to allow fix #756
… parameter for allow dynamically override any config parameter, fix #821
Slach added 23 commits April 3, 2024 20:47
…e` - will never upload object disk content, fix #799
…name conflicts during restore, fix #851, change approach to SQL based
…erver startup time and `docker-entrypoint.d` processing, will infinite reconnect each 5 seconds, until success, fix #857
… disk with type `local`, fix #882, asd complete removed support for legacy backups, created with version prior v1.0
…th not only default disk to find pratially created backups, fix #873

fixed create `--rbac` behavior, don't create access folder if no RBAC objects is present
@Slach Slach merged commit 99f6873 into master Apr 8, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment