Enable automatic multipart upload for replication

[bwplotka]

Currently, we spam with warning logs about guessed file size fail. For a good reason, as some object storage providers indeed has to know file size upfront. See: thanos-io/thanos#678 Current hack allows guessing if the reader is an os.File.

However current replicate implementation streams from Get method straight to Upload avoiding large local memory buffering. This, however, means that object storage client has to be smarter to deduce if we want multipart upload or not without knowing overall upload size. GCS client has this implemented, but e.g both S3 and Aliyun client does not have automatic multipart upload deduction, so replicate will never use multipart functionality for those storages.

This is blocked on: thanos-io/thanos#678

Multipart might be essential for large files. It allows improved retry logic and concurrency. For our current use cases, NOT having this might be fine so let's jump on this only if required.

AC:

• Replicate can use multipart upload feature for large files replication.

Help wanted! (:

Log from run:

level=info ts=2019-10-29T13:45:29.641192675Z caller=main.go:145 msg="Tracing will be disabled"
level=info ts=2019-10-29T13:45:29.641627665Z caller=factory.go:43 component=replicate msg="loading bucket configuration"
level=info ts=2019-10-29T13:45:29.645146195Z caller=factory.go:43 component=replicate msg="loading bucket configuration"
level=info ts=2019-10-29T13:45:29.672897436Z caller=replicate.go:171 component=replicate msg="starting replication"
level=info ts=2019-10-29T13:45:29.67334713Z caller=replicate.go:137 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="running replication attempt"
level=info ts=2019-10-29T13:45:29.673091945Z caller=main.go:236 component=replicate msg="Listening for metrics" address=0.0.0.0:10902
level=warn ts=2019-10-29T13:45:29.773949245Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP4VMSK4VEVJ8D97MNK6HX/chunks/000001
level=info ts=2019-10-29T13:45:31.072717458Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP4VMSK4VEVJ8D97MNK6HX/chunks/000001
level=warn ts=2019-10-29T13:45:31.085079443Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP4VMSK4VEVJ8D97MNK6HX/index
level=info ts=2019-10-29T13:45:31.636109684Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP4VMSK4VEVJ8D97MNK6HX/index
level=warn ts=2019-10-29T13:45:31.636260541Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP4VMSK4VEVJ8D97MNK6HX/meta.json
level=warn ts=2019-10-29T13:45:31.875072133Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5YP832QR70D6TY6E54NV/chunks/000001
level=info ts=2019-10-29T13:45:32.788662576Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN5YP832QR70D6TY6E54NV/chunks/000001
level=warn ts=2019-10-29T13:45:32.801842697Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5YP832QR70D6TY6E54NV/index
level=info ts=2019-10-29T13:45:33.454589811Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN5YP832QR70D6TY6E54NV/index
level=warn ts=2019-10-29T13:45:33.454774874Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5YP832QR70D6TY6E54NV/meta.json
level=warn ts=2019-10-29T13:45:33.660587338Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5VBAZST7N8YZBH62RG6C/chunks/000001
level=info ts=2019-10-29T13:45:48.602867872Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN5VBAZST7N8YZBH62RG6C/chunks/000001
level=warn ts=2019-10-29T13:45:48.614270696Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5VBAZST7N8YZBH62RG6C/index
level=info ts=2019-10-29T13:45:55.683780245Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN5VBAZST7N8YZBH62RG6C/index
level=warn ts=2019-10-29T13:45:55.683937575Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5VBAZST7N8YZBH62RG6C/meta.json
level=warn ts=2019-10-29T13:45:55.920142369Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP4RNVZ94135ZA6B10EMRR/chunks/000001
level=info ts=2019-10-29T13:46:20.861287214Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP4RNVZ94135ZA6B10EMRR/chunks/000001
level=warn ts=2019-10-29T13:46:20.875305564Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP4RNVZ94135ZA6B10EMRR/index
level=info ts=2019-10-29T13:46:23.904545905Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP4RNVZ94135ZA6B10EMRR/index
level=warn ts=2019-10-29T13:46:23.90486319Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP4RNVZ94135ZA6B10EMRR/meta.json
level=warn ts=2019-10-29T13:46:24.123511025Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5CDRY6W5870M23TN87WM/chunks/000001
level=info ts=2019-10-29T13:46:39.008805279Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN5CDRY6W5870M23TN87WM/chunks/000001
level=warn ts=2019-10-29T13:46:39.022871112Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5CDRY6W5870M23TN87WM/index
level=info ts=2019-10-29T13:46:46.069727033Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN5CDRY6W5870M23TN87WM/index
level=warn ts=2019-10-29T13:46:46.069834422Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN5CDRY6W5870M23TN87WM/meta.json
level=warn ts=2019-10-29T13:46:46.306926875Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP41SEFNF95Z3XKC8WDCDJ/chunks/000001
level=info ts=2019-10-29T13:47:11.135822277Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP41SEFNF95Z3XKC8WDCDJ/chunks/000001
level=warn ts=2019-10-29T13:47:11.15034015Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP41SEFNF95Z3XKC8WDCDJ/index
level=info ts=2019-10-29T13:47:14.158928487Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP41SEFNF95Z3XKC8WDCDJ/index
level=warn ts=2019-10-29T13:47:14.159020458Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP41SEFNF95Z3XKC8WDCDJ/meta.json
level=warn ts=2019-10-29T13:47:14.381442834Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4XZWTXR4NRM9P5JJ5TSR/chunks/000001
level=info ts=2019-10-29T13:47:29.30825647Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4XZWTXR4NRM9P5JJ5TSR/chunks/000001
level=warn ts=2019-10-29T13:47:29.321866577Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4XZWTXR4NRM9P5JJ5TSR/index
level=info ts=2019-10-29T13:47:36.358419703Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4XZWTXR4NRM9P5JJ5TSR/index
level=warn ts=2019-10-29T13:47:36.358573525Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4XZWTXR4NRM9P5JJ5TSR/meta.json
level=warn ts=2019-10-29T13:47:36.623591373Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP3AQMC4D6QK4C7B6DW3T3/chunks/000001
level=info ts=2019-10-29T13:48:01.423700604Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP3AQMC4D6QK4C7B6DW3T3/chunks/000001
level=warn ts=2019-10-29T13:48:01.437768117Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP3AQMC4D6QK4C7B6DW3T3/index
level=info ts=2019-10-29T13:48:04.452045073Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP3AQMC4D6QK4C7B6DW3T3/index
level=warn ts=2019-10-29T13:48:04.452216777Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP3AQMC4D6QK4C7B6DW3T3/meta.json
level=warn ts=2019-10-29T13:48:04.683854138Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4EQH4EM248F0DAVAZ3EN/chunks/000001
level=info ts=2019-10-29T13:48:07.426047312Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4EQH4EM248F0DAVAZ3EN/chunks/000001
level=warn ts=2019-10-29T13:48:07.437082409Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4EQH4EM248F0DAVAZ3EN/index
level=info ts=2019-10-29T13:48:08.965177068Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4EQH4EM248F0DAVAZ3EN/index
level=warn ts=2019-10-29T13:48:08.965224341Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4EQH4EM248F0DAVAZ3EN/meta.json
level=warn ts=2019-10-29T13:48:09.179431493Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2MNV4TYJGKEJ8RW12NTQ/chunks/000001
level=info ts=2019-10-29T13:48:13.498317515Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2MNV4TYJGKEJ8RW12NTQ/chunks/000001
level=warn ts=2019-10-29T13:48:13.51245926Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2MNV4TYJGKEJ8RW12NTQ/index
level=info ts=2019-10-29T13:48:14.352109074Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2MNV4TYJGKEJ8RW12NTQ/index
level=warn ts=2019-10-29T13:48:14.352170427Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2MNV4TYJGKEJ8RW12NTQ/meta.json
level=warn ts=2019-10-29T13:48:14.584715283Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2GM87TM3ZHSRVKTHJ8KK/chunks/000001
level=info ts=2019-10-29T13:48:18.98957326Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2GM87TM3ZHSRVKTHJ8KK/chunks/000001
level=warn ts=2019-10-29T13:48:19.005584946Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2GM87TM3ZHSRVKTHJ8KK/index
level=info ts=2019-10-29T13:48:19.852417173Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2GM87TM3ZHSRVKTHJ8KK/index
level=warn ts=2019-10-29T13:48:19.852580491Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2GM87TM3ZHSRVKTHJ8KK/meta.json
level=warn ts=2019-10-29T13:48:20.075538249Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4CMPPEF01P8HA5CPCHFR/chunks/000001
level=info ts=2019-10-29T13:48:22.871991663Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4CMPPEF01P8HA5CPCHFR/chunks/000001
level=warn ts=2019-10-29T13:48:22.883769964Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4CMPPEF01P8HA5CPCHFR/index
level=info ts=2019-10-29T13:48:24.425504485Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4CMPPEF01P8HA5CPCHFR/index
level=warn ts=2019-10-29T13:48:24.425745753Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4CMPPEF01P8HA5CPCHFR/meta.json
level=warn ts=2019-10-29T13:48:24.627092844Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2C3PBMJG6QSDM7MRG23T/chunks/000001
level=info ts=2019-10-29T13:48:25.960042414Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2C3PBMJG6QSDM7MRG23T/chunks/000001
level=warn ts=2019-10-29T13:48:25.974457685Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2C3PBMJG6QSDM7MRG23T/index
level=info ts=2019-10-29T13:48:26.539956456Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2C3PBMJG6QSDM7MRG23T/index
level=warn ts=2019-10-29T13:48:26.540016748Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2C3PBMJG6QSDM7MRG23T/meta.json
level=warn ts=2019-10-29T13:48:26.750901534Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4ANHP7XYRD4XR6SCAHCQ/chunks/000001
level=info ts=2019-10-29T13:48:27.669487424Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4ANHP7XYRD4XR6SCAHCQ/chunks/000001
level=warn ts=2019-10-29T13:48:27.685069544Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4ANHP7XYRD4XR6SCAHCQ/index
level=info ts=2019-10-29T13:48:28.300732707Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4ANHP7XYRD4XR6SCAHCQ/index
level=warn ts=2019-10-29T13:48:28.300951363Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4ANHP7XYRD4XR6SCAHCQ/meta.json
level=warn ts=2019-10-29T13:48:28.548658796Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2A9VZF1Q81ZF7X95RZ5F/chunks/000001
level=info ts=2019-10-29T13:48:29.869166834Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2A9VZF1Q81ZF7X95RZ5F/chunks/000001
level=warn ts=2019-10-29T13:48:29.88423152Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2A9VZF1Q81ZF7X95RZ5F/index
level=info ts=2019-10-29T13:48:30.395158926Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP2A9VZF1Q81ZF7X95RZ5F/index
level=warn ts=2019-10-29T13:48:30.395327127Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP2A9VZF1Q81ZF7X95RZ5F/meta.json
level=warn ts=2019-10-29T13:48:30.593405073Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN49XY34ST8H45B3XP3AD8/chunks/000001
level=info ts=2019-10-29T13:48:31.437256609Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN49XY34ST8H45B3XP3AD8/chunks/000001
level=warn ts=2019-10-29T13:48:31.453081079Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN49XY34ST8H45B3XP3AD8/index
level=info ts=2019-10-29T13:48:32.06716499Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN49XY34ST8H45B3XP3AD8/index
level=warn ts=2019-10-29T13:48:32.067250656Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN49XY34ST8H45B3XP3AD8/meta.json
level=warn ts=2019-10-29T13:48:32.288886846Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP289X5NG9Q8JYPETQXC4P/chunks/000001
level=info ts=2019-10-29T13:48:33.832122178Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP289X5NG9Q8JYPETQXC4P/chunks/000001
level=warn ts=2019-10-29T13:48:33.845121989Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP289X5NG9Q8JYPETQXC4P/index
level=info ts=2019-10-29T13:48:34.407296034Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBP289X5NG9Q8JYPETQXC4P/index
level=warn ts=2019-10-29T13:48:34.407433038Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBP289X5NG9Q8JYPETQXC4P/meta.json
level=warn ts=2019-10-29T13:48:34.61403667Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4965FREYEN5AD3SZ4XVM/chunks/000001
level=info ts=2019-10-29T13:48:35.502596372Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4965FREYEN5AD3SZ4XVM/chunks/000001
level=warn ts=2019-10-29T13:48:35.516222774Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4965FREYEN5AD3SZ4XVM/index
level=info ts=2019-10-29T13:48:36.135179229Z caller=scheme.go:310 component=replicate replication-run-id=01DRBW7AT9JC4GP2ZYW4PCJ1AR msg="object replicated" object=01DRBN4965FREYEN5AD3SZ4XVM/index
level=warn ts=2019-10-29T13:48:36.13537203Z caller=s3.go:309 component=replicate msg="could not guess file size for multipart upload" name=01DRBN4965FREYEN5AD3SZ4XVM/meta.json
level=info ts=2019-10-29T13:48:36.302723329Z caller=main.go:192 msg=exiting