From 48a1be3e0123c5a18a0efbf62300166c55222c23 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Thu, 15 Dec 2022 16:08:46 -0800
Subject: [PATCH 01/30] Handle Metadata from xtz-shots

This introduces under the hood "magic" to download the right snapshot
for you, when using default values.

There was a previous attempt #394, however since then we have made some
progress in formalizing the metadata at xtz-shots.io, to the point where
it makes sense to introduce these changes instead.

The metadata file https://xtz-shots.io/tezos-snapshots.json is intended
to become some sort of standard across ecosystem snapshot providers.

It provides a flat list of snapshots across tezos networks, containing
size in bytes and sha256sum.

We introduce a new value `snapshot_source` which is defaulting to the
xtz-shots metadata URL. I hope marigold will provide one such file soon
as well.

When this file is set, config-generator will download it, and find the
right snapshot, based on the network name and the octez version. It will
then pass it to the node pods for download.

We also introduce `prefer_tarballs` which will make the nodes download a
tarball instead of a snapshot. This defaults to False to preserve
today's behavior.

The variables `full_snapshot_url` `archive_tarball_url` etc... still
exist. They used to have default set to mainnet URLs from xtz-shots, but
they default to null now.

If you set these variables, they will override the metadata behavior
that we are introducing. As before, you can't set a value for snapshot
and tarball of same history mode at the same time, or helm will error.

When using the new method, three new behaviors are introduced:

1. snapshot-downloader verifies the free space on the filesystem, and
   does not attempt to download if it is insufficient
1. after downloading, it will check the sha256sum against the one in
   metadata.
1. when importing a snapshot (not a tarball), it passes the block hash
   to the import command. The octez official documentation recommends to
   do this. If there is a mismatch, it will error out.

Now, our "quick start" guide will become more compelling: with just one
line of helm, it will be possible to start a mainnet node, a ghostnet
node, or any network on xtz-shots.

Test:

These 4 values yaml are provided as reference.

I tested them all on limanet (because artifacst are small and fast).

When it's merged and released, the `tezos_k8s_images` value can go and
only `node_config_network:chain_name` needs to be set.

```
/# starts a limanet rolling node, downloads a snapshot, verifies checksum and unpacks it
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
```

```
/# starts a limanet rolling node, downloads a tarball, verifies checksum and unpacks it
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
prefer_tarballs: true
```

```
/# verify that old method of downloading tarball still works
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
rolling_tarball_url: https://limanet.xtz-shots.io/rolling-tarball
rolling_snapshot_url: null
```

```
/# verify that old method of downloading snapshot still works
tezos_k8s_images:
  utils: ghcr.io/oxheadalpha/tezos-k8s-utils:snapshot_metadata
node_config_network:
  chain_name: limanet
rolling_tarball_url: null
rolling_snapshot_url: https://limanet.xtz-shots.io/rolling
```

I also tried a private chain with mkchain with no parameters. it works

I am updating the python version to 3.10 in utils because I wanted to
use python's new `case` statement. I tried 3.11 but pytezos is broken, I
opened a ticket and referenced it in the source code.
---
 charts/tezos/scripts/snapshot-importer.sh |  16 ++-
 charts/tezos/templates/_helpers.tpl       |   2 +-
 charts/tezos/templates/configs.yaml       |   3 +
 charts/tezos/values.yaml                  |  34 ++++--
 mkchain/tqchain/mkchain.py                |   5 +-
 test/charts/mainnet.expect.yaml           |  25 +++-
 test/charts/mainnet2.expect.yaml          |  41 +++++--
 test/charts/private-chain.expect.yaml     |   3 +
 test/charts/private-chain.in.yaml         |   4 +-
 utils/Dockerfile                          |   6 +-
 utils/config-generator.py                 | 137 +++++++++++++++++++---
 utils/snapshot-downloader.sh              |  79 +++++++------
 12 files changed, 270 insertions(+), 85 deletions(-)

diff --git a/charts/tezos/scripts/snapshot-importer.sh b/charts/tezos/scripts/snapshot-importer.sh
index ec3befdfb..dcaf634c9 100644
--- a/charts/tezos/scripts/snapshot-importer.sh
+++ b/charts/tezos/scripts/snapshot-importer.sh
@@ -1,4 +1,4 @@
-set -ex
+set -e
 
 bin_dir="/usr/local/bin"
 data_dir="/var/tezos"
@@ -6,6 +6,14 @@ node_dir="$data_dir/node"
 node_data_dir="$node_dir/data"
 node="$bin_dir/tezos-node"
 snapshot_file=${node_dir}/chain.snapshot
+if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+    block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
+fi
+
+if [ ! -f ${snapshot_file} ]; then
+    echo "No snapshot to import."
+    exit 0
+fi
 
 if [ -d ${node_data_dir}/context ]; then
     echo "Blockchain has already been imported. If a tarball"
@@ -16,8 +24,12 @@ fi
 
 cp -v /etc/tezos/config.json ${node_data_dir}
 
+if [ "${block_hash}" != "" ]; then
+  block_hash_arg="--block ${block_hash}"
+fi
+
 ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
-    --network $CHAIN_NAME
+    --network $CHAIN_NAME ${block_hash_arg}
 find ${node_dir}
 
 rm -rvf ${snapshot_file}
diff --git a/charts/tezos/templates/_helpers.tpl b/charts/tezos/templates/_helpers.tpl
index 33405d529..ca02d505b 100644
--- a/charts/tezos/templates/_helpers.tpl
+++ b/charts/tezos/templates/_helpers.tpl
@@ -56,7 +56,7 @@
 {{- define "tezos.shouldDownloadSnapshot" -}}
   {{- if or (.Values.full_snapshot_url) (.Values.full_tarball_url)
             (.Values.rolling_snapshot_url) (.Values.rolling_tarball_url)
-            (.Values.archive_tarball_url) }}
+            (.Values.archive_tarball_url) (.Values.snapshot_source) }}
     {{- if or (and (.Values.rolling_tarball_url) (.Values.rolling_snapshot_url))
         (and (.Values.full_tarball_url) (.Values.full_snapshot_url))
     }}
diff --git a/charts/tezos/templates/configs.yaml b/charts/tezos/templates/configs.yaml
index ad0adf1d8..88b2872f7 100644
--- a/charts/tezos/templates/configs.yaml
+++ b/charts/tezos/templates/configs.yaml
@@ -16,6 +16,9 @@ data:
   ROLLING_SNAPSHOT_URL: "{{ .Values.rolling_snapshot_url }}"
   ROLLING_TARBALL_URL: "{{ .Values.rolling_tarball_url }}"
   ARCHIVE_TARBALL_URL: "{{ .Values.archive_tarball_url }}"
+  PREFER_TARBALLS: "{{ .Values.prefer_tarballs }}"
+  SNAPSHOT_SOURCE: "{{ .Values.snapshot_source }}"
+  OCTEZ_VERSION: "{{ .Values.images.octez }}"
   NODE_GLOBALS: |
 {{ .Values.node_globals | mustToPrettyJson | indent 4 }}
 
diff --git a/charts/tezos/values.yaml b/charts/tezos/values.yaml
index d2aedda8a..38aec2463 100644
--- a/charts/tezos/values.yaml
+++ b/charts/tezos/values.yaml
@@ -265,17 +265,31 @@ signers: {}
 # ```
 # End Signers
 
-## Where full and rolling history mode nodes will get their Tezos snapshots from.
-full_snapshot_url: https://mainnet.xtz-shots.io/full
-rolling_snapshot_url: https://mainnet.xtz-shots.io/rolling
+# When spinning up nodes, tezos-k8s will attempt to download a snapshot
+# from a known source.
+# If you want to sync from scratch, set to 'null'
+# For a private chain, set to 'null'
+snapshot_source: https://xtz-shots.io/tezos-snapshots.json
 
-## Alternatively to a Tezos snapshot, you can download an LZ4-compressed
-## filesystem tar of a node's data directory by setting `archive_tarball_url`
-## and `rolling_tarball_url` fields to the URL of the file. NOTE:
-## `rolling_tarball_url` and `rolling_snapshot_url` are mutually exclusive and
-## cannot both be specified at the same time.
-archive_tarball_url: https://mainnet.xtz-shots.io/archive-tarball
-# rolling_tarball_url: "https://mainnet.xtz-shots.io/rolling-tarball
+# By default, tezos-k8s will download and unpack snapshots.
+# A tarball is a LZ4-compressed filesystem tar of a node's data directory.
+# You must trust the tarball provider to provide good data, as no check is
+# performed by the node.
+# If you prefer tarballs, set to "true" below.
+prefer_tarballs: false
+
+# By default, tezos-k8s will attempt to download the right artifact from
+# `snapshot_source` set above.
+# You can override and hard-code a snapshot URL source below.
+# When any of the below variables is set, `snapshot_source` above will be
+# ignored for all artifact types.
+## NOTE: `*_tarball_url` and `*_snapshot_url` are mutually exclusive
+## and cannot both be specified at the same time.
+full_snapshot_url: null
+full_tarball_url: null
+rolling_snapshot_url: null
+archive_tarball_url: null
+rolling_tarball_url: null
 
 # List of peers for nodes to connect to. Gets set under config.json `p2p` field
 bootstrap_peers: []
diff --git a/mkchain/tqchain/mkchain.py b/mkchain/tqchain/mkchain.py
index 83d317f1d..7bcb63d7f 100644
--- a/mkchain/tqchain/mkchain.py
+++ b/mkchain/tqchain/mkchain.py
@@ -178,10 +178,7 @@ def main():
             "zerotier_token": args.zerotier_token,
         },
         # Custom chains should not pull snapshots or tarballs
-        "full_snapshot_url": None,
-        "rolling_snapshot_url": None,
-        "archive_tarball_url": None,
-        "rolling_tarball_url": None,
+        "snapshot_source": None,
         "node_globals": {
             # Needs a quotedstring otherwise helm interprets "Y" as true and it does not work
             "env": {
diff --git a/test/charts/mainnet.expect.yaml b/test/charts/mainnet.expect.yaml
index 4d9ab2c59..ed5b877c5 100644
--- a/test/charts/mainnet.expect.yaml
+++ b/test/charts/mainnet.expect.yaml
@@ -26,11 +26,14 @@ data:
         },
       "protocol_activation": null
     }
-  FULL_SNAPSHOT_URL: "https://mainnet.xtz-shots.io/full"
+  FULL_SNAPSHOT_URL: ""
   FULL_TARBALL_URL: ""
-  ROLLING_SNAPSHOT_URL: "https://mainnet.xtz-shots.io/rolling"
+  ROLLING_SNAPSHOT_URL: ""
   ROLLING_TARBALL_URL: ""
-  ARCHIVE_TARBALL_URL: "https://mainnet.xtz-shots.io/archive-tarball"
+  ARCHIVE_TARBALL_URL: ""
+  PREFER_TARBALLS: "false"
+  SNAPSHOT_SOURCE: "https://xtz-shots.io/tezos-snapshots.json"
+  OCTEZ_VERSION: "tezos/tezos:v14-release"
   NODE_GLOBALS: |
     {
       "env": {}
@@ -310,7 +313,7 @@ spec:
           args:
             - "-c"
             - |
-              set -ex
+              set -e
               
               bin_dir="/usr/local/bin"
               data_dir="/var/tezos"
@@ -318,6 +321,14 @@ spec:
               node_data_dir="$node_dir/data"
               node="$bin_dir/tezos-node"
               snapshot_file=${node_dir}/chain.snapshot
+              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+                  block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
+              fi
+              
+              if [ ! -f ${snapshot_file} ]; then
+                  echo "No snapshot to import."
+                  exit 0
+              fi
               
               if [ -d ${node_data_dir}/context ]; then
                   echo "Blockchain has already been imported. If a tarball"
@@ -328,8 +339,12 @@ spec:
               
               cp -v /etc/tezos/config.json ${node_data_dir}
               
+              if [ "${block_hash}" != "" ]; then
+                block_hash_arg="--block ${block_hash}"
+              fi
+              
               ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
-                  --network $CHAIN_NAME
+                  --network $CHAIN_NAME ${block_hash_arg}
               find ${node_dir}
               
               rm -rvf ${snapshot_file}
diff --git a/test/charts/mainnet2.expect.yaml b/test/charts/mainnet2.expect.yaml
index 997114f31..f7f41321c 100644
--- a/test/charts/mainnet2.expect.yaml
+++ b/test/charts/mainnet2.expect.yaml
@@ -26,11 +26,14 @@ data:
         },
       "protocol_activation": null
     }
-  FULL_SNAPSHOT_URL: "https://mainnet.xtz-shots.io/full"
+  FULL_SNAPSHOT_URL: ""
   FULL_TARBALL_URL: ""
-  ROLLING_SNAPSHOT_URL: "https://mainnet.xtz-shots.io/rolling"
+  ROLLING_SNAPSHOT_URL: ""
   ROLLING_TARBALL_URL: ""
-  ARCHIVE_TARBALL_URL: "https://mainnet.xtz-shots.io/archive-tarball"
+  ARCHIVE_TARBALL_URL: ""
+  PREFER_TARBALLS: "false"
+  SNAPSHOT_SOURCE: "https://xtz-shots.io/tezos-snapshots.json"
+  OCTEZ_VERSION: "tezos/tezos:v14-release"
   NODE_GLOBALS: |
     {
       "env": {
@@ -409,7 +412,7 @@ spec:
           args:
             - "-c"
             - |
-              set -ex
+              set -e
               
               bin_dir="/usr/local/bin"
               data_dir="/var/tezos"
@@ -417,6 +420,14 @@ spec:
               node_data_dir="$node_dir/data"
               node="$bin_dir/tezos-node"
               snapshot_file=${node_dir}/chain.snapshot
+              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+                  block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
+              fi
+              
+              if [ ! -f ${snapshot_file} ]; then
+                  echo "No snapshot to import."
+                  exit 0
+              fi
               
               if [ -d ${node_data_dir}/context ]; then
                   echo "Blockchain has already been imported. If a tarball"
@@ -427,8 +438,12 @@ spec:
               
               cp -v /etc/tezos/config.json ${node_data_dir}
               
+              if [ "${block_hash}" != "" ]; then
+                block_hash_arg="--block ${block_hash}"
+              fi
+              
               ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
-                  --network $CHAIN_NAME
+                  --network $CHAIN_NAME ${block_hash_arg}
               find ${node_dir}
               
               rm -rvf ${snapshot_file}
@@ -762,7 +777,7 @@ spec:
           args:
             - "-c"
             - |
-              set -ex
+              set -e
               
               bin_dir="/usr/local/bin"
               data_dir="/var/tezos"
@@ -770,6 +785,14 @@ spec:
               node_data_dir="$node_dir/data"
               node="$bin_dir/tezos-node"
               snapshot_file=${node_dir}/chain.snapshot
+              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+                  block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
+              fi
+              
+              if [ ! -f ${snapshot_file} ]; then
+                  echo "No snapshot to import."
+                  exit 0
+              fi
               
               if [ -d ${node_data_dir}/context ]; then
                   echo "Blockchain has already been imported. If a tarball"
@@ -780,8 +803,12 @@ spec:
               
               cp -v /etc/tezos/config.json ${node_data_dir}
               
+              if [ "${block_hash}" != "" ]; then
+                block_hash_arg="--block ${block_hash}"
+              fi
+              
               ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
-                  --network $CHAIN_NAME
+                  --network $CHAIN_NAME ${block_hash_arg}
               find ${node_dir}
               
               rm -rvf ${snapshot_file}
diff --git a/test/charts/private-chain.expect.yaml b/test/charts/private-chain.expect.yaml
index 8762bf39f..2e72671c1 100644
--- a/test/charts/private-chain.expect.yaml
+++ b/test/charts/private-chain.expect.yaml
@@ -103,6 +103,9 @@ data:
   ROLLING_SNAPSHOT_URL: ""
   ROLLING_TARBALL_URL: ""
   ARCHIVE_TARBALL_URL: ""
+  PREFER_TARBALLS: "false"
+  SNAPSHOT_SOURCE: ""
+  OCTEZ_VERSION: "tezos/tezos:v14-release"
   NODE_GLOBALS: |
     {
       "env": {}
diff --git a/test/charts/private-chain.in.yaml b/test/charts/private-chain.in.yaml
index 2432002a0..6469aee92 100644
--- a/test/charts/private-chain.in.yaml
+++ b/test/charts/private-chain.in.yaml
@@ -72,9 +72,7 @@ activation:
     sc_rollup_max_available_messages: 1000000
 bootstrap_peers: []
 expected_proof_of_work: 0
-full_snapshot_url: null
-rolling_snapshot_url: null
-archive_tarball_url: null
+snapshot_source: null
 images:
   octez: 'tezos/tezos:v14-release'
 is_invitation: false
diff --git a/utils/Dockerfile b/utils/Dockerfile
index 7817a1d24..84e5c125d 100644
--- a/utils/Dockerfile
+++ b/utils/Dockerfile
@@ -1,4 +1,6 @@
-FROM python:3.9-alpine
+FROM python:3.10-alpine
+# TODO: update to 3.11 once the bug is fixed:
+# https://github.com/baking-bad/pytezos/issues/336
 ENV PYTHONUNBUFFERED=1
 
 #
@@ -14,7 +16,7 @@ RUN     PIP="pip --no-cache install"					\
   $APK_ADD --virtual .build-deps gcc python3-dev			\
   libffi-dev musl-dev make		\
   && $APK_ADD libsodium-dev libsecp256k1-dev gmp-dev			\
-  && $APK_ADD zeromq-dev						\
+  && $APK_ADD zeromq-dev findmnt						\
   && $PIP install base58 pynacl					\
   && $PIP install mnemonic pytezos requests				\
   && $PIP install pyblake2 pysodium flask \
diff --git a/utils/config-generator.py b/utils/config-generator.py
index 6b93a7f6e..37b3c3937 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -2,6 +2,7 @@
 import collections
 import json
 import os
+import re
 import requests
 import socket
 from grp import getgrnam
@@ -14,7 +15,7 @@
 from pytezos import pytezos
 from base58 import b58encode_check
 
-with open('/etc/secret-volume/ACCOUNTS', 'r') as secret_file:
+with open("/etc/secret-volume/ACCOUNTS", "r") as secret_file:
     ACCOUNTS = json.loads(secret_file.read())
 CHAIN_PARAMS = json.loads(os.environ["CHAIN_PARAMS"])
 DATA_DIR = "/var/tezos/node/data"
@@ -141,17 +142,29 @@ def main():
                 "ERROR: No bootstrap peers found for this non-bootstrap node"
             )
 
-        config_json = json.dumps(
-            create_node_config_json(
-                bootstrap_peers,
-                my_zerotier_ip,
-            ),
+        node_config = create_node_config_json(
+            bootstrap_peers,
+            my_zerotier_ip,
+        )
+        node_config_json = json.dumps(
+            node_config,
+            indent=2,
+        )
+        node_snapshot_config = create_node_snapshot_config_json(
+            NETWORK_CONFIG.get("chain_name"), node_config["shell"]["history_mode"]
+        )
+        node_snapshot_config_json = json.dumps(
+            node_snapshot_config,
             indent=2,
         )
         print("Generated config.json :")
-        print(config_json)
+        print(node_config_json)
+        print("Generated snapshot_config.json :")
+        print(node_snapshot_config_json)
         with open("/etc/tezos/config.json", "w") as json_file:
-            print(config_json, file=json_file)
+            print(node_config_json, file=json_file)
+        with open("/var/tezos/snapshot_config.json", "w") as json_file:
+            print(node_snapshot_config_json, file=json_file)
 
 
 # If NETWORK_CONFIG["genesis"]["block"] hasn't been specified, we generate a
@@ -249,7 +262,9 @@ def verify_this_bakers_account(accounts):
         # We can count on accounts[acct]["type"] because import_keys will
         # fill it in when it is missing.
         if not (accounts[acct]["type"] == "secret" or signer):
-            raise Exception(f"ERROR: Either a secret key or a signer_url should be provided for {acct}")
+            raise Exception(
+                f"ERROR: Either a secret key or a signer_url should be provided for {acct}"
+            )
 
 
 #
@@ -282,7 +297,9 @@ def fill_in_missing_keys(all_accounts):
 
     for account_name, account_values in all_accounts.items():
         if "type" in account_values:
-            raise Exception("Deprecated field 'type' passed by helm, but helm should have pruned it.")
+            raise Exception(
+                "Deprecated field 'type' passed by helm, but helm should have pruned it."
+            )
         account_key = account_values.get("key")
 
         if account_key == None:
@@ -324,7 +341,9 @@ def expose_secret_key(account_name):
 
 
 def pod_requires_secret_key(account_values):
-    return MY_POD_TYPE in ["activating", "signing"] and "signer_url" not in account_values
+    return (
+        MY_POD_TYPE in ["activating", "signing"] and "signer_url" not in account_values
+    )
 
 
 #
@@ -521,9 +540,11 @@ def get_genesis_pubkey():
                 genesis_pubkey = pubkey["value"]["key"]
                 break
         if not genesis_pubkey:
-            raise Exception("ERROR: Couldn't find the genesis_pubkey. " +
-                            "This generally happens if you forgot to " +
-                            "define an account for the activation account")
+            raise Exception(
+                "ERROR: Couldn't find the genesis_pubkey. "
+                + "This generally happens if you forgot to "
+                + "define an account for the activation account"
+            )
         return genesis_pubkey
 
 
@@ -552,7 +573,7 @@ def create_node_config_json(
         "data-dir": DATA_DIR,
         "rpc": {
             "listen-addrs": [f"{os.getenv('MY_POD_IP')}:8732", "127.0.0.1:8732"],
-            "acl": [ { "address": os.getenv('MY_POD_IP'), "blacklist": [] } ]
+            "acl": [{"address": os.getenv("MY_POD_IP"), "blacklist": []}],
         },
         "p2p": {
             "bootstrap-peers": bootstrap_peers,
@@ -600,5 +621,91 @@ def create_node_config_json(
     return node_config
 
 
+def create_node_snapshot_config_json(network_name, history_mode):
+    """Create this node's snapshot config"""
+
+    snapshot_source = os.environ["SNAPSHOT_SOURCE"]
+    prefer_tarballs = os.environ["PREFER_TARBALLS"] == "true"
+    artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
+    rolling_tarball_url = os.environ["ROLLING_TARBALL_URL"]
+    full_tarball_url = os.environ["FULL_TARBALL_URL"]
+    archive_tarball_url = os.environ["ARCHIVE_TARBALL_URL"]
+    rolling_snapshot_url = os.environ["ROLLING_SNAPSHOT_URL"]
+    full_snapshot_url = os.environ["FULL_SNAPSHOT_URL"]
+    if (
+        rolling_tarball_url
+        or full_tarball_url
+        or rolling_snapshot_url
+        or full_snapshot_url
+        or archive_tarball_url
+    ):
+        print("Snapshot or tarball URL found, will ignore snapshot_source")
+        match history_mode:
+            case "rolling":
+                if rolling_tarball_url:
+                    return {"url": rolling_tarball_url, "artifact_type": "tarball"}
+                elif rolling_snapshot_url:
+                    return {
+                        "url": rolling_snapshot_url,
+                        "artifact_type": "tezos-snapshot",
+                    }
+                return {}
+            case "full":
+                if full_tarball_url:
+                    return {"url": full_tarball_url, "artifact_type": "tarball"}
+                elif full_snapshot_url:
+                    return {"url": full_snapshot_url, "artifact_type": "tezos-snapshot"}
+                return {}
+            case "archive":
+                if archive_tarball_url:
+                    return {"url": archive_tarball_url, "artifact_type": "tarball"}
+                return {}
+            case _:
+                return {}
+
+    octez_container_version = os.environ["OCTEZ_VERSION"]
+    if snapshot_source:
+        try:
+            all_snapshots = requests.get(snapshot_source).json()
+        except Exception:
+            return {}
+    else:
+        return {}
+    try:
+        octez_long_version = octez_container_version.split(":")[1]
+        octez_version_re = re.search(r"v\d\d", octez_long_version)
+        octez_version = None
+        if octez_version_re:
+            octez_version = octez_version_re.group().replace("v", "")
+    except Exception:
+        octez_version = None
+
+    print(
+        f"""
+Searching for snapshots from {snapshot_source}
+with history mode {history_mode}
+and artifact type {artifact_type}
+and chain name {network_name}
+and octez version {octez_version}.
+    """
+    )
+    # find snapshot matching all the requested fields
+    try:
+        matching_snapshots = [
+            s
+            for s in all_snapshots
+            if s["history_mode"] == history_mode
+            and s["artifact_type"] == artifact_type
+            and s["chain_name"] == network_name
+            and octez_version in s["tezos_version"]
+        ]
+        matching_snapshots = sorted(matching_snapshots, key=lambda d: d["block_height"])
+
+        matching_snapshot = matching_snapshots[-1]
+        return matching_snapshot
+    except Exception:
+        return None
+
+
 if __name__ == "__main__":
     main()
diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 31452311b..15540227d 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -19,48 +19,55 @@ fi
 
 echo "Did not find a pre-existing blockchain."
 
-my_nodes_history_mode=$(< /etc/tezos/config.json jq -r "
-				.shell.history_mode
-				|if type == \"object\" then
-					(keys|.[0])
-				 else .
-				 end")
-
-echo "My nodes history mode: '$my_nodes_history_mode'"
-
-snapshot_url=""
-tarball_url=""
-case "$my_nodes_history_mode" in
-  full)     snapshot_url="$FULL_SNAPSHOT_URL"
-            tarball_url="$FULL_TARBALL_URL";;
-
-  rolling)  snapshot_url="$ROLLING_SNAPSHOT_URL"
-            tarball_url="$ROLLING_TARBALL_URL";;
-
-  archive)  tarball_url="$ARCHIVE_TARBALL_URL";;
-
-  *)        echo "Invalid node history mode: '$my_nodes_history_mode'"
-            exit 1;;
-esac
-
-if [ -z "$snapshot_url" ] && [ -z "$tarball_url" ]; then
-  echo "ERROR: No snapshot or tarball url specified."
-  exit 1
+if [ "$(cat ${data_dir}/snapshot_config.json)" == "null" ]; then
+  echo "No snapshot config found, nothing to do."
+  exit 0
 fi
 
-if [ -n "$snapshot_url" ] && [ -n "$tarball_url" ]; then
-  echo "ERROR: Either only a snapshot or tarball url may be specified per Tezos node history mode."
-fi
+echo "Tezos snapshot config is:"
+cat ${data_dir}/snapshot_config.json
 
+artifact_url=$(cat ${data_dir}/snapshot_config.json | jq -r '.url')
+artifact_type=$(cat ${data_dir}/snapshot_config.json | jq -r '.artifact_type')
 mkdir -p "$node_data_dir"
 
-if [ -n "$snapshot_url" ]; then
-  echo "Downloading $snapshot_url"
+download() {
+  # Smart Downloading function. When relevant metadata is accessible, it:
+  # * checks that there is enough space to download the file
+  # * verifies the sha256sum
+  filesize_bytes=$(cat ${data_dir}/snapshot_config.json | jq -r '.filesize_bytes // empty')
+  sha256=$(cat ${data_dir}/snapshot_config.json | jq -r '.sha256 // empty')
+  if [ ! -z "${filesize_bytes}" ]; then
+    free_space=$(findmnt -bno size -T ${data_dir})
+    echo "Free space available in filesystem: ${free_space}" >&2
+    if [ "${filesize_bytes}" -gt "${free_space}" ]; then
+      echo "Error: not enough disk space available to download artifact." >&2
+      exit 1
+    else
+      echo "There is sufficient free space to download the artifact of size ${filesize_bytes}." >&2
+    fi
+  fi
+  curl -LfsS $1 | tee >(sha256sum > ${snapshot_file}.sha256sum)
+  if [ ! -z "${sha256}" ]; then
+    if [ "${sha256}" != "$(cat ${snapshot_file}.sha256sum | head -c 64)" ]; then
+      echo "Error: sha256 checksum of the downloaded file did not match checksum from metadata file." >&2
+      exit 1
+    else
+      echo "Snapshot sha256sum check successful." >&2
+    fi
+  fi
+}
+if [ "${artifact_type}" == "tezos-snapshot" ]; then
+  echo "Downloading $artifact_url"
   echo '{ "version": "0.0.4" }' > "$node_dir/version.json"
-  curl -LfsS -o "$snapshot_file" "$snapshot_url"
-elif [ -n "$tarball_url" ]; then
-  echo "Downloading and extracting tarball from $tarball_url"
-  curl -LfsS "$tarball_url" | lz4 -d | tar -x -C "$data_dir"
+  block_hash=$(cat ${data_dir}/snapshot_config.json | jq -r '.block_hash // empty')
+  download "$artifact_url" > "$snapshot_file"
+  if [ ! -z "${block_hash}" ]; then
+    echo ${block_hash} > ${snapshot_file}.block_hash
+  fi
+elif [ "${artifact_type}" == "tarball" ]; then
+  echo "Downloading and extracting tarball from $artifact_url"
+  download "$artifact_url" | lz4 -d | tar -x -C "$data_dir"
 fi
 
 chown -R 1000 "$data_dir"

From c98794af3a71c7ab0707b81fec87aff998d03ada Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:17:08 -0800
Subject: [PATCH 02/30] eval PREFER_TARBALLS as a python bool

Co-authored-by: Aryeh Harris <harryttd@users.noreply.github.com>
---
 utils/config-generator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 37b3c3937..c2e04cfd0 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -625,7 +625,7 @@ def create_node_snapshot_config_json(network_name, history_mode):
     """Create this node's snapshot config"""
 
     snapshot_source = os.environ["SNAPSHOT_SOURCE"]
-    prefer_tarballs = os.environ["PREFER_TARBALLS"] == "true"
+    prefer_tarballs = os.environ["PREFER_TARBALLS"] == True
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
     rolling_tarball_url = os.environ["ROLLING_TARBALL_URL"]
     full_tarball_url = os.environ["FULL_TARBALL_URL"]

From 6bc71d1fc0c876783db3948b8646ddb436198e18 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:21:03 -0800
Subject: [PATCH 03/30] look up CHAIN_NAME inside function per review

---
 utils/config-generator.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 69ecccb12..925543698 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -151,7 +151,7 @@ def main():
             indent=2,
         )
         node_snapshot_config = create_node_snapshot_config_json(
-            NETWORK_CONFIG.get("chain_name"), node_config["shell"]["history_mode"]
+            node_config["shell"]["history_mode"]
         )
         node_snapshot_config_json = json.dumps(
             node_snapshot_config,
@@ -621,9 +621,10 @@ def create_node_config_json(
     return node_config
 
 
-def create_node_snapshot_config_json(network_name, history_mode):
+def create_node_snapshot_config_json(history_mode):
     """Create this node's snapshot config"""
 
+    network_name = NETWORK_CONFIG.get("chain_name")
     snapshot_source = os.environ["SNAPSHOT_SOURCE"]
     prefer_tarballs = os.environ["PREFER_TARBALLS"] == True
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"

From 3f968f08ed4a9f18c4c68a08bbce0b79e56ba538 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:26:46 -0800
Subject: [PATCH 04/30] simplify logic to grab the octez version with regexp

Co-authored-by: Aryeh Harris <harryttd@users.noreply.github.com>
---
 utils/config-generator.py | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 925543698..03352897f 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -674,10 +674,8 @@ def create_node_snapshot_config_json(history_mode):
         return {}
     try:
         octez_long_version = octez_container_version.split(":")[1]
-        octez_version_re = re.search(r"v\d\d", octez_long_version)
-        octez_version = None
-        if octez_version_re:
-            octez_version = octez_version_re.group().replace("v", "")
+        octez_version_re = re.search(r"v(\d+)", octez_long_version)
+        octez_version = octez_version_re and octez_version_re.group(1)
     except Exception:
         octez_version = None
 

From e6ce5d1b37e4d7190e72b399458b14ce597f751a Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:27:55 -0800
Subject: [PATCH 05/30] better explanation of snapshot_source in default
 values.yaml

Co-authored-by: Aryeh Harris <harryttd@users.noreply.github.com>
---
 charts/tezos/values.yaml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/charts/tezos/values.yaml b/charts/tezos/values.yaml
index 21fa62cdd..d46237894 100644
--- a/charts/tezos/values.yaml
+++ b/charts/tezos/values.yaml
@@ -273,10 +273,10 @@ signers: {}
 # ```
 # End Signers
 
-# When spinning up nodes, tezos-k8s will attempt to download a snapshot
-# from a known source.
-# If you want to sync from scratch, set to 'null'
-# For a private chain, set to 'null'
+# When spinning up nodes, tezos-k8s will attempt to download a snapshot from a
+# known source. This should be a url to a json metadata file in the format
+# xtz-shots uses. If you want to sync from scratch or for a private chain, set
+# to `null`.
 snapshot_source: https://xtz-shots.io/tezos-snapshots.json
 
 # By default, tezos-k8s will download and unpack snapshots.

From 84d44e4a03055cc609498cffc71024ba806da770 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:31:05 -0800
Subject: [PATCH 06/30] grab env var `SNAPSHOT_SOURCE` where it's actually
 needed

---
 utils/config-generator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 03352897f..041902fd4 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -625,7 +625,6 @@ def create_node_snapshot_config_json(history_mode):
     """Create this node's snapshot config"""
 
     network_name = NETWORK_CONFIG.get("chain_name")
-    snapshot_source = os.environ["SNAPSHOT_SOURCE"]
     prefer_tarballs = os.environ["PREFER_TARBALLS"] == True
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
     rolling_tarball_url = os.environ["ROLLING_TARBALL_URL"]
@@ -665,6 +664,7 @@ def create_node_snapshot_config_json(history_mode):
                 return {}
 
     octez_container_version = os.environ["OCTEZ_VERSION"]
+    snapshot_source = os.environ["SNAPSHOT_SOURCE"]
     if snapshot_source:
         try:
             all_snapshots = requests.get(snapshot_source).json()

From 8737fd3d9784fa74b56f4326add068a7e2064a9f Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:33:36 -0800
Subject: [PATCH 07/30] lint values.yaml explanation of hardcoded snapshot URLs

Co-authored-by: Aryeh Harris <harryttd@users.noreply.github.com>
---
 charts/tezos/values.yaml | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/charts/tezos/values.yaml b/charts/tezos/values.yaml
index d46237894..4b155730b 100644
--- a/charts/tezos/values.yaml
+++ b/charts/tezos/values.yaml
@@ -287,16 +287,15 @@ snapshot_source: https://xtz-shots.io/tezos-snapshots.json
 prefer_tarballs: false
 
 # By default, tezos-k8s will attempt to download the right artifact from
-# `snapshot_source` set above.
-# You can override and hard-code a snapshot URL source below.
-# When any of the below variables is set, `snapshot_source` above will be
-# ignored for all artifact types.
+# `snapshot_source` set above. You can override and hard-code a snapshot URL
+# source below. When any of the below variables are set, `snapshot_source` above
+# will be ignored for all artifact types.
 ## NOTE: `*_tarball_url` and `*_snapshot_url` are mutually exclusive
 ## and cannot both be specified at the same time.
+archive_tarball_url: null # e.g. https://mainnet.xtz-shots.io/archive-tarball
 full_snapshot_url: null
 full_tarball_url: null
 rolling_snapshot_url: null
-archive_tarball_url: null
 rolling_tarball_url: null
 
 # List of peers for nodes to connect to. Gets set under config.json `p2p` field

From 33fae452d887ce8240d496f9bfc9ac877b26dc1e Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:57:42 -0800
Subject: [PATCH 08/30] return None instead of {} when snapshot not found

---
 utils/config-generator.py    | 15 ++++++++-------
 utils/snapshot-downloader.sh |  1 +
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 041902fd4..356f7e0a3 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -649,29 +649,30 @@ def create_node_snapshot_config_json(history_mode):
                         "url": rolling_snapshot_url,
                         "artifact_type": "tezos-snapshot",
                     }
-                return {}
+                return
             case "full":
                 if full_tarball_url:
                     return {"url": full_tarball_url, "artifact_type": "tarball"}
                 elif full_snapshot_url:
                     return {"url": full_snapshot_url, "artifact_type": "tezos-snapshot"}
-                return {}
+                return
             case "archive":
                 if archive_tarball_url:
                     return {"url": archive_tarball_url, "artifact_type": "tarball"}
-                return {}
+                return
             case _:
-                return {}
+                return
 
     octez_container_version = os.environ["OCTEZ_VERSION"]
     snapshot_source = os.environ["SNAPSHOT_SOURCE"]
     if snapshot_source:
         try:
             all_snapshots = requests.get(snapshot_source).json()
-        except Exception:
-            return {}
+        except Exception as e:
+            print(f"Error while fetching {snapshot_source}: {e}")
+            return
     else:
-        return {}
+        return
     try:
         octez_long_version = octez_container_version.split(":")[1]
         octez_version_re = re.search(r"v(\d+)", octez_long_version)
diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 15540227d..390325c83 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -57,6 +57,7 @@ download() {
     fi
   fi
 }
+
 if [ "${artifact_type}" == "tezos-snapshot" ]; then
   echo "Downloading $artifact_url"
   echo '{ "version": "0.0.4" }' > "$node_dir/version.json"

From b65e0d2add76388ab8fa818adcdd3d55894aa325 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 13:58:01 -0800
Subject: [PATCH 09/30] when octez_version is not known, pick snapshot of any
 version

---
 utils/config-generator.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 356f7e0a3..9a4a36500 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -697,8 +697,9 @@ def create_node_snapshot_config_json(history_mode):
             if s["history_mode"] == history_mode
             and s["artifact_type"] == artifact_type
             and s["chain_name"] == network_name
-            and octez_version in s["tezos_version"]
         ]
+        if octez_version:
+            matching_snapshots = [ s for s in matching_snapshots if  octez_version in s["tezos_version"] ]
         matching_snapshots = sorted(matching_snapshots, key=lambda d: d["block_height"])
 
         matching_snapshot = matching_snapshots[-1]

From 973930aca6e5346f069f8f2dd401eb88fcb7dc07 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 14:01:07 -0800
Subject: [PATCH 10/30] throw an error when history mode is unknown

---
 utils/config-generator.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 9a4a36500..939399429 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -5,6 +5,7 @@
 import re
 import requests
 import socket
+import sys
 from grp import getgrnam
 from hashlib import blake2b
 from pathlib import Path
@@ -661,7 +662,8 @@ def create_node_snapshot_config_json(history_mode):
                     return {"url": archive_tarball_url, "artifact_type": "tarball"}
                 return
             case _:
-                return
+                print(f"Error: history mode {history_mode} is not known.")
+                sys.exit(1)
 
     octez_container_version = os.environ["OCTEZ_VERSION"]
     snapshot_source = os.environ["SNAPSHOT_SOURCE"]

From aa35ff334a6783ec60c9a79eddb2caa1daab9a27 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 14:02:02 -0800
Subject: [PATCH 11/30] snapshot importer: verify there is a snapshot to import
 first

---
 charts/tezos/scripts/snapshot-importer.sh | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/charts/tezos/scripts/snapshot-importer.sh b/charts/tezos/scripts/snapshot-importer.sh
index 22e2d366a..4db23bfd6 100644
--- a/charts/tezos/scripts/snapshot-importer.sh
+++ b/charts/tezos/scripts/snapshot-importer.sh
@@ -6,15 +6,16 @@ node_dir="$data_dir/node"
 node_data_dir="$node_dir/data"
 node="$bin_dir/octez-node"
 snapshot_file=${node_dir}/chain.snapshot
-if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
-    block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
-fi
 
 if [ ! -f ${snapshot_file} ]; then
     echo "No snapshot to import."
     exit 0
 fi
 
+if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+    block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
+fi
+
 if [ -d ${node_data_dir}/context ]; then
     echo "Blockchain has already been imported. If a tarball"
     echo "instead of a regular tezos snapshot was used, it was"

From a17117c0c39576308fe3e7bd7b0ea839107c80be Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 15:04:32 -0800
Subject: [PATCH 12/30] fix true/false handling for `prefer_tarballs`

---
 utils/config-generator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 939399429..6910bebf8 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -626,7 +626,7 @@ def create_node_snapshot_config_json(history_mode):
     """Create this node's snapshot config"""
 
     network_name = NETWORK_CONFIG.get("chain_name")
-    prefer_tarballs = os.environ["PREFER_TARBALLS"] == True
+    prefer_tarballs = os.environ["PREFER_TARBALLS"].lower() in ('true', '1', 't')
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
     rolling_tarball_url = os.environ["ROLLING_TARBALL_URL"]
     full_tarball_url = os.environ["FULL_TARBALL_URL"]

From e168f84e6e038f6476ae77c421e47e76256ac911 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 16:06:35 -0800
Subject: [PATCH 13/30] handle sha256 check failure for tarball; delete data

---
 utils/snapshot-downloader.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 390325c83..4093b2b8a 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -51,7 +51,7 @@ download() {
   if [ ! -z "${sha256}" ]; then
     if [ "${sha256}" != "$(cat ${snapshot_file}.sha256sum | head -c 64)" ]; then
       echo "Error: sha256 checksum of the downloaded file did not match checksum from metadata file." >&2
-      exit 1
+      touch ${data_dir}/sha256sum_failed
     else
       echo "Snapshot sha256sum check successful." >&2
     fi
@@ -63,12 +63,23 @@ if [ "${artifact_type}" == "tezos-snapshot" ]; then
   echo '{ "version": "0.0.4" }' > "$node_dir/version.json"
   block_hash=$(cat ${data_dir}/snapshot_config.json | jq -r '.block_hash // empty')
   download "$artifact_url" > "$snapshot_file"
+  if [ -f "${data_dir}/sha256sum_failed" ]; then
+    # sha256 failure
+    rm -rvf "${data_dir}/sha256sum_failed"
+    exit 1
+  fi
   if [ ! -z "${block_hash}" ]; then
     echo ${block_hash} > ${snapshot_file}.block_hash
   fi
 elif [ "${artifact_type}" == "tarball" ]; then
   echo "Downloading and extracting tarball from $artifact_url"
   download "$artifact_url" | lz4 -d | tar -x -C "$data_dir"
+  if [ -f "${data_dir}/sha256sum_failed" ]; then
+    echo "sha256 check failed, deleting data"
+    rm -rvf "${node_data_dir}"
+    rm -rvf "${data_dir}/sha256sum_failed"
+    exit 1
+  fi
 fi
 
 chown -R 1000 "$data_dir"

From c5b553b4190a955418c58da962c598598f9f4de3 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 16:09:21 -0800
Subject: [PATCH 14/30] also handle disk check failure

---
 utils/snapshot-downloader.sh | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 4093b2b8a..054dfbe24 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -42,7 +42,7 @@ download() {
     echo "Free space available in filesystem: ${free_space}" >&2
     if [ "${filesize_bytes}" -gt "${free_space}" ]; then
       echo "Error: not enough disk space available to download artifact." >&2
-      exit 1
+      touch ${data_dir}/disk_space_failed
     else
       echo "There is sufficient free space to download the artifact of size ${filesize_bytes}." >&2
     fi
@@ -81,6 +81,10 @@ elif [ "${artifact_type}" == "tarball" ]; then
     exit 1
   fi
 fi
+if [ -f "${data_dir}/disk_space_failed" ]; then
+  rm -rvf "${data_dir}/disk_space_failed"
+  exit 1
+fi
 
 chown -R 1000 "$data_dir"
 ls -lR "$data_dir"

From 24d73f632e5433754bfc508f3f8704a4e1c1984e Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 16:27:50 -0800
Subject: [PATCH 15/30] handle case where octez image is defined in statefulset

---
 utils/config-generator.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 6910bebf8..aed10acf9 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -665,7 +665,10 @@ def create_node_snapshot_config_json(history_mode):
                 print(f"Error: history mode {history_mode} is not known.")
                 sys.exit(1)
 
-    octez_container_version = os.environ["OCTEZ_VERSION"]
+    if "images" in MY_POD_CLASS and "octez" in MY_POD_CLASS["images"]:
+        octez_container_version = MY_POD_CLASS["images"]["octez"]
+    else:
+        octez_container_version = os.environ["OCTEZ_VERSION"]
     snapshot_source = os.environ["SNAPSHOT_SOURCE"]
     if snapshot_source:
         try:

From 73bce9fa73c2394d157dcc68b5e5fdc1427985fe Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 16:50:53 -0800
Subject: [PATCH 16/30] query env vars with get()

---
 utils/config-generator.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index aed10acf9..0005ebbba 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -668,8 +668,8 @@ def create_node_snapshot_config_json(history_mode):
     if "images" in MY_POD_CLASS and "octez" in MY_POD_CLASS["images"]:
         octez_container_version = MY_POD_CLASS["images"]["octez"]
     else:
-        octez_container_version = os.environ["OCTEZ_VERSION"]
-    snapshot_source = os.environ["SNAPSHOT_SOURCE"]
+        octez_container_version = os.environ.get("OCTEZ_VERSION")
+    snapshot_source = os.environ.get("SNAPSHOT_SOURCE")
     if snapshot_source:
         try:
             all_snapshots = requests.get(snapshot_source).json()
@@ -705,7 +705,7 @@ def create_node_snapshot_config_json(history_mode):
         ]
         if octez_version:
             matching_snapshots = [ s for s in matching_snapshots if  octez_version in s["tezos_version"] ]
-        matching_snapshots = sorted(matching_snapshots, key=lambda d: d["block_height"])
+        matching_snapshots = sorted(matching_snapshots, key=lambda d: d.get("block_height"))
 
         matching_snapshot = matching_snapshots[-1]
         return matching_snapshot

From 7dec366626aa829a02dd624cb0f7be838144fc3d Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 16:52:48 -0800
Subject: [PATCH 17/30] query manual snapshot env vars with get()

---
 utils/config-generator.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 0005ebbba..a2b6547b8 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -628,11 +628,11 @@ def create_node_snapshot_config_json(history_mode):
     network_name = NETWORK_CONFIG.get("chain_name")
     prefer_tarballs = os.environ["PREFER_TARBALLS"].lower() in ('true', '1', 't')
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
-    rolling_tarball_url = os.environ["ROLLING_TARBALL_URL"]
-    full_tarball_url = os.environ["FULL_TARBALL_URL"]
-    archive_tarball_url = os.environ["ARCHIVE_TARBALL_URL"]
-    rolling_snapshot_url = os.environ["ROLLING_SNAPSHOT_URL"]
-    full_snapshot_url = os.environ["FULL_SNAPSHOT_URL"]
+    rolling_tarball_url = os.environ.get("ROLLING_TARBALL_URL")
+    full_tarball_url = os.environ.get("FULL_TARBALL_URL")
+    archive_tarball_url = os.environ.get("ARCHIVE_TARBALL_URL")
+    rolling_snapshot_url = os.environ.get("ROLLING_SNAPSHOT_URL")
+    full_snapshot_url = os.environ.get("FULL_SNAPSHOT_URL")
     if (
         rolling_tarball_url
         or full_tarball_url

From ff32e1b18dc3894852f63f02ecfd921826148a04 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 16:54:36 -0800
Subject: [PATCH 18/30] simplify snapshot-importer

---
 charts/tezos/scripts/snapshot-importer.sh | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/charts/tezos/scripts/snapshot-importer.sh b/charts/tezos/scripts/snapshot-importer.sh
index 4db23bfd6..6977fca4b 100644
--- a/charts/tezos/scripts/snapshot-importer.sh
+++ b/charts/tezos/scripts/snapshot-importer.sh
@@ -12,10 +12,6 @@ if [ ! -f ${snapshot_file} ]; then
     exit 0
 fi
 
-if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
-    block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
-fi
-
 if [ -d ${node_data_dir}/context ]; then
     echo "Blockchain has already been imported. If a tarball"
     echo "instead of a regular tezos snapshot was used, it was"
@@ -25,8 +21,8 @@ fi
 
 cp -v /etc/tezos/config.json ${node_data_dir}
 
-if [ "${block_hash}" != "" ]; then
-  block_hash_arg="--block ${block_hash}"
+if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+    block_hash_arg="--block $(cat ${node_dir}/chain.snapshot.block_hash)"
 fi
 
 ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \

From 866e3118b37e71f7dccb73e36546d98f58f33235 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 17:02:11 -0800
Subject: [PATCH 19/30] do not wrap sort function in try/except, return last if
 exists

---
 utils/config-generator.py | 28 ++++++++++++----------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index a2b6547b8..c57b169b5 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -695,22 +695,18 @@ def create_node_snapshot_config_json(history_mode):
     """
     )
     # find snapshot matching all the requested fields
-    try:
-        matching_snapshots = [
-            s
-            for s in all_snapshots
-            if s["history_mode"] == history_mode
-            and s["artifact_type"] == artifact_type
-            and s["chain_name"] == network_name
-        ]
-        if octez_version:
-            matching_snapshots = [ s for s in matching_snapshots if  octez_version in s["tezos_version"] ]
-        matching_snapshots = sorted(matching_snapshots, key=lambda d: d.get("block_height"))
-
-        matching_snapshot = matching_snapshots[-1]
-        return matching_snapshot
-    except Exception:
-        return None
+    matching_snapshots = [
+        s
+        for s in all_snapshots
+        if s["history_mode"] == history_mode
+        and s["artifact_type"] == artifact_type
+        and s["chain_name"] == network_name
+    ]
+    if octez_version:
+        matching_snapshots = [ s for s in matching_snapshots if  octez_version in s["tezos_version"] ]
+    matching_snapshots = sorted(matching_snapshots, key=lambda d: d.get("block_height"))
+
+    return matching_snapshots[-1] if len(matching_snapshots) else None
 
 
 if __name__ == "__main__":

From db481da4abfda735ee69cbc6d2fd96ecd87c70c6 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 17:05:46 -0800
Subject: [PATCH 20/30] fix tests

---
 test/charts/mainnet.expect.yaml       |  9 +++------
 test/charts/mainnet2.expect.yaml      | 16 +++++-----------
 test/charts/private-chain.expect.yaml |  2 +-
 3 files changed, 9 insertions(+), 18 deletions(-)

diff --git a/test/charts/mainnet.expect.yaml b/test/charts/mainnet.expect.yaml
index 2eb2c2c43..4c6b923f6 100644
--- a/test/charts/mainnet.expect.yaml
+++ b/test/charts/mainnet.expect.yaml
@@ -33,7 +33,7 @@ data:
   ARCHIVE_TARBALL_URL: ""
   PREFER_TARBALLS: "false"
   SNAPSHOT_SOURCE: "https://xtz-shots.io/tezos-snapshots.json"
-  OCTEZ_VERSION: "tezos/tezos:v14-release"
+  OCTEZ_VERSION: "tezos/tezos:v15-release"
   NODE_GLOBALS: |
     {
       "env": {}
@@ -328,9 +328,6 @@ spec:
               node_data_dir="$node_dir/data"
               node="$bin_dir/octez-node"
               snapshot_file=${node_dir}/chain.snapshot
-              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
-                  block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
-              fi
               
               if [ ! -f ${snapshot_file} ]; then
                   echo "No snapshot to import."
@@ -346,8 +343,8 @@ spec:
               
               cp -v /etc/tezos/config.json ${node_data_dir}
               
-              if [ "${block_hash}" != "" ]; then
-                block_hash_arg="--block ${block_hash}"
+              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+                  block_hash_arg="--block $(cat ${node_dir}/chain.snapshot.block_hash)"
               fi
               
               ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
diff --git a/test/charts/mainnet2.expect.yaml b/test/charts/mainnet2.expect.yaml
index 4b217c94c..07b71ee3e 100644
--- a/test/charts/mainnet2.expect.yaml
+++ b/test/charts/mainnet2.expect.yaml
@@ -33,7 +33,7 @@ data:
   ARCHIVE_TARBALL_URL: ""
   PREFER_TARBALLS: "false"
   SNAPSHOT_SOURCE: "https://xtz-shots.io/tezos-snapshots.json"
-  OCTEZ_VERSION: "tezos/tezos:v14-release"
+  OCTEZ_VERSION: "tezos/tezos:v15-release"
   NODE_GLOBALS: |
     {
       "env": {
@@ -437,9 +437,6 @@ spec:
               node_data_dir="$node_dir/data"
               node="$bin_dir/octez-node"
               snapshot_file=${node_dir}/chain.snapshot
-              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
-                  block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
-              fi
               
               if [ ! -f ${snapshot_file} ]; then
                   echo "No snapshot to import."
@@ -455,8 +452,8 @@ spec:
               
               cp -v /etc/tezos/config.json ${node_data_dir}
               
-              if [ "${block_hash}" != "" ]; then
-                block_hash_arg="--block ${block_hash}"
+              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+                  block_hash_arg="--block $(cat ${node_dir}/chain.snapshot.block_hash)"
               fi
               
               ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
@@ -812,9 +809,6 @@ spec:
               node_data_dir="$node_dir/data"
               node="$bin_dir/octez-node"
               snapshot_file=${node_dir}/chain.snapshot
-              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
-                  block_hash=$(cat ${node_dir}/chain.snapshot.block_hash)
-              fi
               
               if [ ! -f ${snapshot_file} ]; then
                   echo "No snapshot to import."
@@ -830,8 +824,8 @@ spec:
               
               cp -v /etc/tezos/config.json ${node_data_dir}
               
-              if [ "${block_hash}" != "" ]; then
-                block_hash_arg="--block ${block_hash}"
+              if [ -f ${node_dir}/chain.snapshot.block_hash ]; then
+                  block_hash_arg="--block $(cat ${node_dir}/chain.snapshot.block_hash)"
               fi
               
               ${node} snapshot import ${snapshot_file} --data-dir ${node_data_dir} \
diff --git a/test/charts/private-chain.expect.yaml b/test/charts/private-chain.expect.yaml
index e0ea7c2ae..51be255cc 100644
--- a/test/charts/private-chain.expect.yaml
+++ b/test/charts/private-chain.expect.yaml
@@ -105,7 +105,7 @@ data:
   ARCHIVE_TARBALL_URL: ""
   PREFER_TARBALLS: "false"
   SNAPSHOT_SOURCE: ""
-  OCTEZ_VERSION: "tezos/tezos:v14-release"
+  OCTEZ_VERSION: "tezos/tezos:v15-release"
   NODE_GLOBALS: |
     {
       "env": {}

From bf43020b4bfcee31133208edf337a1335c55de0d Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 17:12:32 -0800
Subject: [PATCH 21/30] black

---
 utils/config-generator.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index c57b169b5..8cfcf566a 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -626,7 +626,7 @@ def create_node_snapshot_config_json(history_mode):
     """Create this node's snapshot config"""
 
     network_name = NETWORK_CONFIG.get("chain_name")
-    prefer_tarballs = os.environ["PREFER_TARBALLS"].lower() in ('true', '1', 't')
+    prefer_tarballs = os.environ["PREFER_TARBALLS"].lower() in ("true", "1", "t")
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
     rolling_tarball_url = os.environ.get("ROLLING_TARBALL_URL")
     full_tarball_url = os.environ.get("FULL_TARBALL_URL")
@@ -703,7 +703,9 @@ def create_node_snapshot_config_json(history_mode):
         and s["chain_name"] == network_name
     ]
     if octez_version:
-        matching_snapshots = [ s for s in matching_snapshots if  octez_version in s["tezos_version"] ]
+        matching_snapshots = [
+            s for s in matching_snapshots if octez_version in s["tezos_version"]
+        ]
     matching_snapshots = sorted(matching_snapshots, key=lambda d: d.get("block_height"))
 
     return matching_snapshots[-1] if len(matching_snapshots) else None

From 109b916c41ea3dfc23ed3b8154bf04096eda65db Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 17:51:34 -0800
Subject: [PATCH 22/30] get PREFER_TARBALLS env var

---
 utils/config-generator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 8cfcf566a..9738b0697 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -626,7 +626,7 @@ def create_node_snapshot_config_json(history_mode):
     """Create this node's snapshot config"""
 
     network_name = NETWORK_CONFIG.get("chain_name")
-    prefer_tarballs = os.environ["PREFER_TARBALLS"].lower() in ("true", "1", "t")
+    prefer_tarballs = os.environ.get("PREFER_TARBALLS").lower() in ("true", "1", "t")
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
     rolling_tarball_url = os.environ.get("ROLLING_TARBALL_URL")
     full_tarball_url = os.environ.get("FULL_TARBALL_URL")

From 01b79031273cc4ec2e2812adcb6b7a8123e8800e Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 17:56:22 -0800
Subject: [PATCH 23/30] do not download file if there is no space

---
 utils/snapshot-downloader.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 054dfbe24..1b724a8db 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -43,6 +43,7 @@ download() {
     if [ "${filesize_bytes}" -gt "${free_space}" ]; then
       echo "Error: not enough disk space available to download artifact." >&2
       touch ${data_dir}/disk_space_failed
+      return 1
     else
       echo "There is sufficient free space to download the artifact of size ${filesize_bytes}." >&2
     fi

From 2e58d5a8653c537c09046f49908063ffe2b056b8 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 17:58:23 -0800
Subject: [PATCH 24/30] default to "" so it doesn't do lower() on none type

---
 utils/config-generator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 9738b0697..bfb6f99ff 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -626,7 +626,7 @@ def create_node_snapshot_config_json(history_mode):
     """Create this node's snapshot config"""
 
     network_name = NETWORK_CONFIG.get("chain_name")
-    prefer_tarballs = os.environ.get("PREFER_TARBALLS").lower() in ("true", "1", "t")
+    prefer_tarballs = os.environ.get("PREFER_TARBALLS", "").lower() in ("true", "1", "t")
     artifact_type = "tarball" if prefer_tarballs else "tezos-snapshot"
     rolling_tarball_url = os.environ.get("ROLLING_TARBALL_URL")
     full_tarball_url = os.environ.get("FULL_TARBALL_URL")

From 88a0b2b10fd705ab9c09efd2e14217633962cf6d Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 18:01:16 -0800
Subject: [PATCH 25/30] use get() for every attribute lookup from snapshot
 metadata file

---
 utils/config-generator.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index bfb6f99ff..7a497de60 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -698,15 +698,15 @@ def create_node_snapshot_config_json(history_mode):
     matching_snapshots = [
         s
         for s in all_snapshots
-        if s["history_mode"] == history_mode
-        and s["artifact_type"] == artifact_type
-        and s["chain_name"] == network_name
+        if s.get("history_mode") == history_mode
+        and s.get("artifact_type") == artifact_type
+        and s.get("chain_name") == network_name
     ]
     if octez_version:
         matching_snapshots = [
-            s for s in matching_snapshots if octez_version in s["tezos_version"]
+            s for s in matching_snapshots if octez_version in s.get("tezos_version")
         ]
-    matching_snapshots = sorted(matching_snapshots, key=lambda d: d.get("block_height"))
+    matching_snapshots = sorted(matching_snapshots, key=lambda s: s.get("block_height"))
 
     return matching_snapshots[-1] if len(matching_snapshots) else None
 

From 9e9a6705e44a9731c746e673d2ef4031b128fe34 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 18:02:47 -0800
Subject: [PATCH 26/30] rm snapshot when sha256 fails

---
 utils/snapshot-downloader.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 1b724a8db..e2ff160c3 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -66,6 +66,7 @@ if [ "${artifact_type}" == "tezos-snapshot" ]; then
   download "$artifact_url" > "$snapshot_file"
   if [ -f "${data_dir}/sha256sum_failed" ]; then
     # sha256 failure
+    rm -rvf ${snapshot_file}
     rm -rvf "${data_dir}/sha256sum_failed"
     exit 1
   fi

From e06fd562b75d67d4593a3ef3facd70384c0a7ebd Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 18:10:49 -0800
Subject: [PATCH 27/30] log too small size

---
 utils/snapshot-downloader.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index e2ff160c3..bd5046e56 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -41,7 +41,7 @@ download() {
     free_space=$(findmnt -bno size -T ${data_dir})
     echo "Free space available in filesystem: ${free_space}" >&2
     if [ "${filesize_bytes}" -gt "${free_space}" ]; then
-      echo "Error: not enough disk space available to download artifact." >&2
+      echo "Error: not enough disk space available (${free_space} bytes) to download artifact of size ${filesize_bytes} bytes." >&2
       touch ${data_dir}/disk_space_failed
       return 1
     else

From 1d5094be69a7900efc91141c8728c3edca0bad1b Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 18:13:04 -0800
Subject: [PATCH 28/30] function return when sha256 check fail (not needed now
 but mb later)

---
 utils/snapshot-downloader.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index bd5046e56..6e7797a99 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -53,6 +53,7 @@ download() {
     if [ "${sha256}" != "$(cat ${snapshot_file}.sha256sum | head -c 64)" ]; then
       echo "Error: sha256 checksum of the downloaded file did not match checksum from metadata file." >&2
       touch ${data_dir}/sha256sum_failed
+      return 1
     else
       echo "Snapshot sha256sum check successful." >&2
     fi

From 0a98d1bd970e38a98ebd653f934198bc8e3d7097 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 18:14:12 -0800
Subject: [PATCH 29/30] default tezos_version when not found in snapshot
 matching func

Co-authored-by: Aryeh Harris <harryttd@users.noreply.github.com>
---
 utils/config-generator.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index 7a497de60..b1d588768 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -704,7 +704,7 @@ def create_node_snapshot_config_json(history_mode):
     ]
     if octez_version:
         matching_snapshots = [
-            s for s in matching_snapshots if octez_version in s.get("tezos_version")
+            s for s in matching_snapshots if octez_version in s.get("tezos_version", "")
         ]
     matching_snapshots = sorted(matching_snapshots, key=lambda s: s.get("block_height"))
 

From bf50654868985216601384f7f53dfb20036f66d8 Mon Sep 17 00:00:00 2001
From: Nicolas Ochem <nicolas.ochem@gmail.com>
Date: Tue, 27 Dec 2022 18:34:40 -0800
Subject: [PATCH 30/30] do not create snapshot_config.json when there's no
 artifact

---
 utils/config-generator.py    | 9 +++++----
 utils/snapshot-downloader.sh | 2 +-
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/utils/config-generator.py b/utils/config-generator.py
index b1d588768..7b0dcf62b 100755
--- a/utils/config-generator.py
+++ b/utils/config-generator.py
@@ -160,12 +160,13 @@ def main():
         )
         print("Generated config.json :")
         print(node_config_json)
-        print("Generated snapshot_config.json :")
-        print(node_snapshot_config_json)
         with open("/etc/tezos/config.json", "w") as json_file:
             print(node_config_json, file=json_file)
-        with open("/var/tezos/snapshot_config.json", "w") as json_file:
-            print(node_snapshot_config_json, file=json_file)
+        if node_snapshot_config:
+            print("Generated snapshot_config.json :")
+            print(node_snapshot_config_json)
+            with open("/var/tezos/snapshot_config.json", "w") as json_file:
+                print(node_snapshot_config_json, file=json_file)
 
 
 # If NETWORK_CONFIG["genesis"]["block"] hasn't been specified, we generate a
diff --git a/utils/snapshot-downloader.sh b/utils/snapshot-downloader.sh
index 6e7797a99..9a2a10bb6 100755
--- a/utils/snapshot-downloader.sh
+++ b/utils/snapshot-downloader.sh
@@ -19,7 +19,7 @@ fi
 
 echo "Did not find a pre-existing blockchain."
 
-if [ "$(cat ${data_dir}/snapshot_config.json)" == "null" ]; then
+if [ ! -f ${data_dir}/snapshot_config.json ]; then
   echo "No snapshot config found, nothing to do."
   exit 0
 fi