[feature] Allow Hauler to chunk the haul (aka tarball)

[zackbradys]

Is your feature request related to a problem? Please describe.

• Yes, please see below

Describe the solution you'd like

• For hauler to have ability to spilt the hauls (aka tarball) into separate and usable chunks, with the max size set by the user. Each haul can be used with hauler store load independently and there is no spilt assets/content/etc...
• Possible implementation below:

zackbradys@Zacks-MBP ~ % hauler store save -h
Save a content store to a store archive

Usage:
  hauler store save [flags]

Flags:
  -f, --filename string   Name of archive (default "pkg.tar.zst")
  -s, --size string       Maximum size of the haul (in GB, example "8GB")
  -h, --help              help for save

Global Flags:
      --cache string       Location of where to store cache data (defaults to $XDG_CACHE_DIR/hauler)
  -l, --log-level string    (default "info")
  -s, --store string       Location to create store at (default "store")

Describe alternatives you've considered

• Adding limited amounts of content to the store before hauler store save to ensure it is below the size maximum.
• Manually splitting the tarball and combining back into one tarball, once on the air gapped network.

Additional context

• Many customers have specific requirements when moving assets onto an air gapped network, including amount of files or maximum size of files, or even grouped files. If customers have to manually chunk the haul and/or determined a limited amount of content to have in the store at one time, it will be difficult to manage so having the ability for hauler to intelligent spilt the haul into usable chunks will be very useful.

[zackbradys]

CC: @aleiner @mddamato

[mitchty]

Couldn't this just be left to ye olde split(1) command with the end archive like so? Outside of split -b being a bit of a pain to use to split by size not sure this would really gain much versus just using the old school unix commands.

$ du -hs rancher-2.8.2.tar.zst*    
24G     rancher-2.8.2.tar.zst
1.0K    rancher-2.8.2.tar.zst.sha256sum
$ split -b $((1024**2*(10*1024))) rancher-2.8.2.tar.zst split
$ du -ks rancher-2.8.2.tar.zst* split*
24568189        rancher-2.8.2.tar.zst
1       rancher-2.8.2.tar.zst.sha256sum
10491313        splitaa
10491889        splitab
3585005 splitac
$ du -hs rancher-2.8.2.tar.zst* split*
24G     rancher-2.8.2.tar.zst
1.0K    rancher-2.8.2.tar.zst.sha256sum
11G     splitaa
11G     splitab
3.5G    splitac
$ rm rancher-2.8.2.tar.zst 
$ echo this would be on the destination after syncing all the split\* files
$ cat split* > !$   
cat split* > rancher-2.8.2.tar.zst
$ rm split*
$ sha256sum --check rancher-2.8.2.tar.zst.sha256sum

[zackbradys]

hey @mitchty, yes you are able to do this outside of hauler, but the value would be to have it natively done within hauler so we would be able to spilt the haul into usable chunks.

if you spilt the haul, outside of hauler, then you need all portions of the haul before loading into the airgap.

[wcrum]

@zackbradys would you want to upgrade to github.com/mholt/archiver/v4 before introducing chunking? This could also be done in conjunction with supporting multiple file formats?