SnapshotEngine: New metadata format and schema validation

[orcutt989]

This PR is a drastic change to the metadata created by the Snapshot Engine. It also provides a mechanism to validate this data via a JSON schema URL.

Changes

tezos-snapshots.json

list of all snapshots currently available at xtz-shots.io across all network names

old

[
    {
        "block_hash": "BMXiuyjbgbKuDxSJmyWcbE1WmCQ13U4jkjndNzReq8MGYFSeFaa",
        "block_height": 3132408,
        "block_timestamp": "2023-02-10T01:28:29Z",
        "filename": "tezos-mainnet-rolling-tarball-3132408.lz4",
        "url": "https://mainnet-v15.xtz-shots.io/tezos-mainnet-rolling-tarball-3132408.lz4",
        "sha256": "fc425fa78c349775ee8645c037eaf52509d5008404dba589596967a38757b9bb",
        "filesize_bytes": 4615157177,
        "filesize": "4G",
        "chain_name": "mainnet",
        "history_mode": "rolling",
        "artifact_type": "tarball",
        ...

new

{
    "date_generated": "2023-02-17T12:46:40Z",
    "org": "Oxhead Alpha",
    "$schema": "https://raw.githubusercontent.com/oxheadalpha/tezos-snapshot-metadata-schema/main/tezos-snapshot-metadata.schema.json",
    "data": [
        {
            "block_hash": "BMXiuyjbgbKuDxSJmyWcbE1WmCQ13U4jkjndNzReq8MGYFSeFaa",
            "block_height": 3132408,
            "block_timestamp": "2023-02-10T01:28:29Z",
            "filename": "tezos-mainnet-rolling-tarball-3132408.lz4",
            "url": "https://mainnet-v15.xtz-shots.io/tezos-mainnet-rolling-tarball-3132408.lz4",
            "sha256": "fc425fa78c349775ee8645c037eaf52509d5008404dba589596967a38757b9bb",
            "filesize_bytes": 4615157177,
            "filesize": "4G",
            "chain_name": "mainnet",
            "history_mode": "rolling",
            "artifact_type": "tarball",
        ...

Artifact metadata files

Each Tezos file system artifact has had its own invididual metadata file. This file is named <FILENAME>.json. We have added new informatin and fixed typing of certain fields.

old

{
  "block_hash": "BMHknnrtX9i4k7gSD7MF769vfbggimHhDEax5pkCuo5coNtPLvo",
  "block_height": 697318,
  "block_timestamp": "2023-02-17T09:55:40Z",
  "filename": "limanet-697318.rolling",
  "url": "https://limanet-v15.xtz-shots.io/limanet-697318.rolling",
  "sha256": "e59db7bb352fd005be661e17a8d9b291c0096ef4260dc4427520e30b194cbecc",
  "filesize_bytes": 116346880,
  "filesize": "110M",
  "chain_name": "limanet",
  "history_mode": "rolling",
  "artifact_type": "tezos-snapshot",
  "tezos_version": "763259c5 (2022-12-01 10:20:58 +0000) (15.1)"
}

new

{
  "block_hash": "BMHknnrtX9i4k7gSD7MF769vfbggimHhDEax5pkCuo5coNtPLvo",
  "block_height": 697318,
  "block_timestamp": "2023-02-17T09:55:40Z",
  "filename": "limanet-697318.rolling",
  "url": "https://limanet-v15.xtz-shots.io/limanet-697318.rolling",
  "sha256": "e59db7bb352fd005be661e17a8d9b291c0096ef4260dc4427520e30b194cbecc",
  "filesize_bytes": 116346880,
  "filesize": "110M",
  "chain_name": "limanet",
  "history_mode": "rolling",
  "artifact_type": "tezos-snapshot",
  "tezos_version": {
    "implementation": "octez",
    "version": {
      "major": 15,
      "minor": 1,
      "additional_info": "release"
    },
    "commit_info": {
      "commit_hash": "763259c5",
      "commit_date": "2022-12-01 10:20:58 +0000"
    }
  },
  "snapshot_version": 4
}

[itkach]

lets have the json schema here as well, have the generated artifacts validated against it and have the schema published (or have schema published and fetch it here to perform validation)

[nicolasochem]

Thanks. looks good and it's already deployed and works.