chunk size relation to s3 cold / hot storage performance

[benjaminpreiss]

Heya everyone!

I am wondering:

1. What is the relation of chunk size and backup / restore performance on hot / cold repos. I reckon that a bigger chunk size will make uploads / downloads faster?
2. Why would I want small chunks?
3. If so, what is the sweet spot regarding the chunk size, for data > 2TB?
4. Are up- and downloads (backup / restore) parallelized?

Anyway, can the chunk size be influenced in rustic?

[aawsome]

Hi @benjaminpreiss

First note that there are 2 different things:

• the chunk size which determines at which point big files are split into chunks - those are then used for the dedupliction. As we use a CDC this is not a fixed size, but rather a size range. Also, for small files we always get "chunks" which are the full file, i.e. the size can be much smaller than the size range. This is general the case for the "last" chunk which is just the remaining of the file even if it is not big enough.
• the pack size which is the size of a pack file - which packs multiple chunks into a file to be saved on the repository storage backend. This again is also just a targeted pack size as the chunks itself are variable in size. And again, packs can be much smaller if there are no more chunks left to pack.

As you are talking about the backend performance, the pack size is IMO the one you are talking about. And yes, that can be influenced, see https://rustic.cli.rs/docs/commands/init/intro.html or use rustic config -h to see the various option.

Chunk sizes can not be changed currently they are set to be in the range of 512kb up to 8MiB.

[benjaminpreiss]

Thank you so much for your detailed answer and the links!

• Would you say that the current default for pack size is the optimum for repos of any size?
• Or should I tweak it for smaller / bigger repos?
• If I expect all the files that I want to backup to be of a specific size, how would that influence my ideal pack size
• What happens if I want to change the pack size, after I have made some backups already (e.g. to cold storage)?

I am just trying to understand what the rules are that govern a good pack size choice 🙏