Consider merging this with Thanos shipper package.

[bwplotka]

Something we missed in the design phase is that this repo is really functionally similar to shipper package used by Thanos. In fact, the shipper would use some fresh improvements we did here (e.g to fix thanos-io/thanos#1670), On top of this, on blockbench repo we agreed to have separate sync CLI command that will do block shipping from the filesystem (see this discussion). Again really similar functionality as Thanos replicate.

Looks like all of this would be nice to be a generalized package for shipping/replicating blocks between two places, am I missing something? 🤔 I am not sure if we should attack this now if we agree on this, but I would propose:

• Adding replicate improvements (replication ID, upload ordering, tests) to the shipper. Happy to rename it replicator
• Adding sync CLI command that will use shipper/replicator
• Allow --daemon and --single-run option.

Sorry for spotting this so late. cc @brancz WDYT?

[brancz]

Leaving the component named “sidecar” seems weird but other than that this sounds good to me!

[cdickmann]

I also have a use case where I want to instruct Prometheus to create a snapshot and then take that snapshot (i.e. its blocks) and ship them to S3. I have abused thanos sidecar, such that I start the daemon, watch the thanos.shipper.json to see when all blocks are referenced in the uploaded array, and then kill the daemon again.

So having a mode for sidecar, or a thanos replicate command which does a single run (without running store API) would be awesome. Personally I would love it if I could say --replicate-snapshot where Thanos calls Prometheus to create a snapshot, and then upload that snapshot. But obviously that can be scripted easily once thanos replicate is available.

[bwplotka]

We have now sync from local disk: #21

Leftover work (help wanted):

• Move to Thanos repo (if agree so)
• Remove shipper and use replicate instead. Blockers:
  • Fix multipart support: Enable automatic multipart upload for replication #22
  • Alllow injecting custom Thanos meta: ext labels

[bwplotka]

Kind of happening thanos-io/thanos#2113