Handle Metadata from xtz-shots

[nicolasochem]

This introduces under the hood "magic" to download the right snapshot for you, when using default values.

There was a previous attempt #394, however since then we have made some progress in formalizing the metadata at xtz-shots.io, to the point where it makes sense to introduce these changes instead.

The metadata file https://xtz-shots.io/tezos-snapshots.json is intended to become some sort of standard across ecosystem snapshot providers.

It provides a flat list of snapshots across tezos networks, containing size in bytes and sha256sum.

We introduce a new value snapshot_source which is defaulting to the xtz-shots metadata URL. I hope marigold will provide one such file soon as well.

When this file is set, config-generator will download it, and find the right snapshot, based on the network name and the octez version. It will then pass it to the node pods for download.

We also introduce prefer_tarballs which will make the nodes download a tarball instead of a snapshot. This defaults to False to preserve today's behavior.

The variables full_snapshot_url archive_tarball_url etc... still exist. They used to have default set to mainnet URLs from xtz-shots, but they default to null now.

If you set these variables, they will override the metadata behavior that we are introducing. As before, you can't set a value for snapshot and tarball of same history mode at the same time, or helm will error.

When using the new method, two new behaviors are introduced:

1. snapshot-downloader verifies the free space on the filesystem, and does not attempt to download if it is insufficient
2. after downloading, it will check the sha256sum against the one in metadata.

Now, our "quick start" guide will become more compelling: with just one line of helm, it will be possible to start a mainnet node, a ghostnet node, or any network on xtz-shots.

Test:

These 4 values yaml are provided as reference.

I tested them all on limanet (because artifacst are small and fast).

When it's merged and released, the tezos_k8s_images value can go and only node_config_network:chain_name needs to be set.

/# starts a limanet rolling node, downloads a snapshot, verifies checksum and unpacks it
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet

/# starts a limanet rolling node, downloads a tarball, verifies checksum and unpacks it
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
prefer_tarballs: true

/# verify that old method of downloading tarball still works
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
rolling_tarball_url: https://limanet.xtz-shots.io/rolling-tarball
rolling_snapshot_url: null

/# verify that old method of downloading snapshot still works
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
rolling_tarball_url: null
rolling_snapshot_url: https://limanet.xtz-shots.io/rolling

I also tried a private chain with mkchain with no parameters. it works

I am updating the python version to 3.10 in utils because I wanted to use python's new case statement. I tried 3.11 but pytezos is broken, I opened a ticket and referenced it in the source code.

[nicolasochem]

I retested all scenarios listed in PR description with bf43020, it still works. I also tried with latest master branch build, but it fails for a separate reason. Please re-review @harryttd

[nicolasochem]

Is there anything we can do to stop the process from continuously trying to download the same file when the hash doesn't match?

no, but you could argue it's the fault of the snapshot website so it's fine.

[nicolasochem]

I retested all scenarios listed in PR description with 0a98d1b, it still works. Please re-review @harryttd

[harryttd]

I haven't tested only reviewed the code but i'm assuming it all works now.

[nicolasochem]

Yes, I just retested everything on the very latest commit, it's fine. I also tried spinning up a rolling node while only setting the archive tarball url, to verify that the file is not being created.