Skip to content

Commit

Permalink
Merge pull request #2 from stacchain/write-disk
Browse files Browse the repository at this point in the history
Write merkle tree json file for data verification
  • Loading branch information
jonhealy1 authored Nov 19, 2024
2 parents 0f2fb5b + 4335458 commit 14b1d7c
Show file tree
Hide file tree
Showing 39 changed files with 1,225 additions and 388 deletions.
17 changes: 9 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

## [v0.2.0] - 2024-11-16
## [v0.3.0] - 2024-11-20

### Added

- Enhanced collection processing to support nested subdirectories for items [#1](https://github.com/stacchain/stac-merkle-tree-cli/pull/1)
- Test to ensure `merkle:root` values remain the same whether items are nested or not [#1](https://github.com/stacchain/stac-merkle-tree-cli/pull/1)
- Merkle tree verification JSON to help Users check the produced merkle values and hierarchical structure of their STAC Catalog post-processing [#2](https://github.com/stacchain/stac-merkle-tree-cli/pull/2)

## [v0.1.1] - 2024-11-16
## [v0.2.0] - 2024-11-16

### Fixed
### Added

- fixed incorrect license identifier
- Enhanced collection processing to support nested subdirectories for items [#1](https://github.com/stacchain/stac-merkle-tree-cli/pull/1)
- Test to ensure `merkle:root` values remain the same whether items are nested or not [#1](https://github.com/stacchain/stac-merkle-tree-cli/pull/1)

## [v0.1.0] - 2024-11-16

- first release

[Unreleased]: https://github.com/stacchain/stac-merkle-tree-cli/tree/v0.1.1...main
[v0.1.1]: https://github.com/stacchain/stac-merkle-tree-cli/tree/v0.1.0...v0.1.1
[Unreleased]: https://github.com/stacchain/stac-merkle-tree-cli/tree/v0.3.0...main
[v0.3.0]: https://github.com/stacchain/stac-merkle-tree-cli/tree/v0.2.0...v0.3.0
[v0.2.0]: https://github.com/stacchain/stac-merkle-tree-cli/tree/v0.1.0...v0.2.0
[v0.1.0]: https://github.com/stacchain/stac-merkle-tree-cli/tree/v0.1.0
131 changes: 124 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,12 +125,18 @@ catalog/

After installing the package, you can use the `stac-merkle-tree-cli` command to compute and add Merkle information to your STAC catalog.

Navigate to the directory containing your catalog.json file and run the command as follows:

```bash
stac-merkle-tree-cli path/to/catalog.json
stac-merkle-tree-cli path/to/catalog_directory [OPTIONS]
```

#### Parameters:

- path/to/catalog_directory: (Required) Path to the root directory containing catalog.json.

#### Options:

- --merkle-tree-file TEXT: (Optional) Path to the output Merkle tree structure file. Defaults to merkle_tree.json within the provided catalog_directory.

### Example

Assuming your directory structure is as follows:
Expand All @@ -154,20 +160,20 @@ my_stac_catalog/
Run the tool:

```bash
stac-merkle-tree-cli my_stac_catalog/catalog.json
stac-merkle-tree-cli my_stac_catalog/
```

Expected Output:

```
Processed Item: /path/to/my_stac_catalog/collections/collection1/item1.json
Processed Item: /path/to/my_stac_catalog/collections/collection1/item2.json
Processed Item: /path/to/my_stac_catalog/collections/collection1/item2/item2.json
Processed Collection: /path/to/my_stac_catalog/collections/collection1/collection.json
Processed Item: /path/to/my_stac_catalog/collections/collection2/item1.json
Processed Item: /path/to/my_stac_catalog/collections/collection2/item1/item1.json
Processed Item: /path/to/my_stac_catalog/collections/collection2/item2.json
Processed Collection: /path/to/my_stac_catalog/collections/collection2/collection.json
Processed Catalog: /path/to/my_stac_catalog/catalog.json
Merkle info computation and addition completed.
Merkle tree structure saved to /path/to/my_stac_catalog/merkle_tree.json
```

## Merkle Tree Extension Specification
Expand Down Expand Up @@ -203,6 +209,76 @@ All STAC objects processed by this tool will include the Merkle extension URL in

After running the tool, each STAC object will be updated with the appropriate Merkle fields.

### Merkle Tree Structure (merkle_tree.json)

The tool generates a `merkle_tree.json` file that represents the hierarchical Merkle tree of your STAC catalog. Below is an example of the `merkle_tree.json` structure:

```json
{
"node_id": "Catalogue",
"type": "Catalog",
"merkle:object_hash": "b14fd102417c1d673f481bc053d19946aefdc27d84c584989b23c676c897bd5a",
"merkle:root": "2c637f0bae066e89de80839f3468f73e396e9d1498faefc469f0fd1039e19e0c",
"children": [
{
"node_id": "COP-DEM",
"type": "Collection",
"merkle:object_hash": "17789b31f8ae304de8dbe2350a15263dbf5e31adfc0d17a997e7e55f4cfc2f53",
"merkle:root": "2f4aa32184fbe70bd385d5b6b6e6d4ec5eb8b2e43611b441febcdf407c4e0030",
"children": [
{
"node_id": "DEM1_SAR_DGE_30_20101212T230244_20140325T230302_ADS_000000_1jTi",
"type": "Item",
"merkle:object_hash": "ce9f56e695ab1751b8f0c8d9ef1f1ecedaf04574ec3077e70e7426ec9fc61ea4"
}
]
},
{
"node_id": "TERRAAQUA",
"type": "Collection",
"merkle:object_hash": "6ae6f97edd2994b632b415ff810af38639faa84544aa8a33a88bdf867a649374",
"merkle:root": "6ae6f97edd2994b632b415ff810af38639faa84544aa8a33a88bdf867a649374",
"children": []
},
{
"node_id": "S2GLC",
"type": "Collection",
"merkle:object_hash": "84ab0e102924c012d4cf2a3b3e10ed4f768f695001174cfd5d9c75d4335b7a48",
"merkle:root": "33631c1a3d9339ffc66b3f3a3eb3de8f558bcabe4900494b55ca17aff851e661",
"children": [
{
"node_id": "S2GLC_T30TWT_2017",
"type": "Item",
"merkle:object_hash": "3a3803a0dae5dbaf9561aeb4cce2770bf38b5da4b71ca67398fb24d48c43a68f"
}
]
}
]
}
```

#### Structure Explanation:

- **Root Node** (Catalogue):
- node_id: Identifier of the Catalog.
- type: Specifies that this node is a Catalog.
- merkle:object_hash: Hash of the Catalog's metadata.
- merkle:root: The Merkle root representing the entire Catalog.
- children: Array containing child nodes, which can be Collections or Items.
- **Child Nodes** (e.g., COP-DEM, TERRAAQUA, S2GLC):

- node_id: Identifier of the Collection.
- type: Specifies that this node is a Collection.
- merkle:object_hash: Hash of the Collection's metadata.
- merkle:root: The Merkle root representing the Collection, calculated from its children.
- children: Array containing child nodes, which can be Items or further sub-Collections.

- **Leaf Nodes** (e.g., DEM1_SAR_DGE_30_20101212T230244_20140325T230302_ADS_000000_1jTi, S2GLC_T30TWT_2017):
- node_id: Identifier of the Item.
- type: Specifies that this node is an Item.
- merkle:object_hash: Hash of the Item's metadata.
- No merkle:root or children: As Items are leaf nodes, they do not contain these fields.

### Catalog (catalog.json)

```json
Expand Down Expand Up @@ -272,3 +348,44 @@ After running the tool, each STAC object will be updated with the appropriate Me
## Contributing

Contributions are welcome! If you encounter issues or have suggestions for improvements, please open an issue or submit a pull request on the [GitHub repository](https://github.com/stacchain/stac-merkle-tree-cli).

## Verification Steps

### 1. Run the CLI Tool:

```bash
stac-merkle-tree-cli path/to/catalog_directory
```

### 2. Check the Output:

- **Console Output**: You should see logs indicating the processing of Items, Collections, and the Catalog.

```ruby
Processed Item: /path/to/catalog_directory/collections/collection1/item1.json
Processed Item: /path/to/catalog_directory/collections/collection1/item2/item2.json
Processed Collection: /path/to/catalog_directory/collections/collection1/collection.json
Processed Item: /path/to/catalog_directory/collections/collection2/item1/item1.json
Processed Item: /path/to/catalog_directory/collections/collection2/item2.json
Processed Collection: /path/to/catalog_directory/collections/collection2/collection.json
Processed Catalog: /path/to/catalog_directory/catalog.json
Merkle tree structure saved to /path/to/catalog_directory/merkle_tree.json
```

- **Merkle Tree JSON**: Verify that the `merkle_tree.json` (or your specified output file) accurately represents the hierarchical structure of your STAC catalog with correct `merkle:object_hash` and `merkle:root` values.

### 3. Verify Integrity:

- **Catalog**: Ensure that the `catalog.json` now includes `merkle:object_hash`, `merkle:root`, and `merkle:hash_method`.

- **Collections**: Each `collection.json` should include `merkle:object_hash`, `merkle:root`, and `merkle:hash_method`.

- **Items**: Each Item's JSON should have `merkle:object_hash` within the properties field.

### 4. Run Tests:

Ensure that all tests pass by executing:

```bash
pytest -v
```
6 changes: 3 additions & 3 deletions example_catalog/catalog.json
Original file line number Diff line number Diff line change
Expand Up @@ -55,14 +55,14 @@
"stac_extensions": [
"https://stacchain.github.io/merkle-tree/v1.0.0/schema.json"
],
"merkle:object_hash": "4adf81a9aba90fe1f483ec3bbe7c628309e598239cee45d4aeafa1bc8c767ae6",
"merkle:root": "6dcb155aa44f0384cf23af6843fb0505d752b11aaaa0778829889b9d598c176f",
"merkle:object_hash": "b14fd102417c1d673f481bc053d19946aefdc27d84c584989b23c676c897bd5a",
"merkle:root": "2c637f0bae066e89de80839f3468f73e396e9d1498faefc469f0fd1039e19e0c",
"merkle:hash_method": {
"function": "sha256",
"fields": [
"*"
],
"ordering": "ascending",
"description": "Computed by including the merkle:root of collections and the catalogs own merkle:object_hash."
"description": "Computed by including the merkle:root of collections and the catalog's own merkle:object_hash."
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T23:02:44.000000Z",
"end_datetime": "2014-03-25T23:03:02.000000Z",
"start_datetime": "2010-12-12T23:02:44.000000Z",
"merkle:object_hash": "32c896520ed1fc38a93e59877cb6a88a5e795721eeae3bf6e7bd29206efb87af"
"merkle:object_hash": "ce9f56e695ab1751b8f0c8d9ef1f1ecedaf04574ec3077e70e7426ec9fc61ea4"
},
"bbox": [
99,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T23:02:44.000000Z",
"end_datetime": "2014-03-29T11:37:10.000000Z",
"start_datetime": "2010-12-12T23:02:44.000000Z",
"merkle:object_hash": "9c6b0b011ec783f4bcdec7c712c2443c3c6eca6d1a0d043842a7c8d587d767c4"
"merkle:object_hash": "ac66e07717b56e8421c8fec00b2b300afd49d30a8ec9c6d505df3b0568de9c77"
},
"bbox": [
100,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T10:36:47.000000Z",
"end_datetime": "2013-04-05T10:30:47.000000Z",
"start_datetime": "2010-12-15T10:36:47.000000Z",
"merkle:object_hash": "762a6152760f8d4868e4036b2ad009a948485e2626c51c6db2a506fc2c7e491c"
"merkle:object_hash": "39969fd5f4a3a170ff39df8f2c13ebca66aab40890275a94ce798e281b85d54d"
},
"bbox": [
98,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T10:36:47.000000Z",
"end_datetime": "2013-04-05T10:30:47.000000Z",
"start_datetime": "2010-12-15T10:36:47.000000Z",
"merkle:object_hash": "b56b4a839810159bfbcd77fe458bcf12173e796a6750b31399166c67cfd29c33"
"merkle:object_hash": "22a31ab988181f280a0bfc6872556ad1d670373bd997f711389d057e1b1f531a"
},
"bbox": [
100,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T20:36:49.000000Z",
"end_datetime": "2014-09-13T09:09:54.000000Z",
"start_datetime": "2010-12-15T20:36:49.000000Z",
"merkle:object_hash": "df89c14ec6017be8ad68b8a51f1152671520a8959c65a7a377cfc2b789f30708"
"merkle:object_hash": "3c22648957e7c76f75a9fbfdf7c164e7d745d9e4ebe809db15d2474e01be4764"
},
"bbox": [
135,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T20:36:49.000000Z",
"end_datetime": "2014-09-13T09:09:54.000000Z",
"start_datetime": "2010-12-15T20:36:49.000000Z",
"merkle:object_hash": "7365039727dfa4456a355cd02585e713af706692716bf0b6050c1b8e82e1308f"
"merkle:object_hash": "6d16f23e2fe2849ce0316c012e9284b413f7c086a11d6de421c0582b397f513e"
},
"bbox": [
135,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T20:39:14.000000Z",
"end_datetime": "2012-12-26T20:48:52.000000Z",
"start_datetime": "2010-12-15T20:39:14.000000Z",
"merkle:object_hash": "b2bb980dc3b1d92106f009b6c732b7140533e622f3e06801dedd643eb9a8614c"
"merkle:object_hash": "ed99b4348ede45959a9cf471490f0f2c6a106bb4337a46d07d4fc4b4988f099f"
},
"bbox": [
134,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-16T00:50:27.000000Z",
"end_datetime": "2013-03-31T00:52:08.000000Z",
"start_datetime": "2010-12-16T00:50:27.000000Z",
"merkle:object_hash": "1d3e8b68bfa0c90b75cbbddec7cccc674ee732d90e83dcb2afa7b97958096626"
"merkle:object_hash": "90a0e5265d02fd58ab81437d14ef1c318ec1b86167ac935d6abb598602798690"
},
"bbox": [
-111,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-16T00:50:27.000000Z",
"end_datetime": "2013-04-11T00:52:16.000000Z",
"start_datetime": "2010-12-16T00:50:27.000000Z",
"merkle:object_hash": "4543f10e43c01172bfa78b0a822c5619e4d7cd6b711475355930ade977c92d23"
"merkle:object_hash": "176a6f0026a763a85909d64d3eeac37873cd7c970fae685a6f37bd16d806b3fe"
},
"bbox": [
-112,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T10:24:16.000000Z",
"end_datetime": "2010-12-12T10:24:17.000000Z",
"start_datetime": "2010-12-12T10:24:16.000000Z",
"merkle:object_hash": "c3c057f15200da00d392d8608a77190019a06c092bd7814670c3426653e1ca78"
"merkle:object_hash": "ffaee98a244aaad0f970100f0f3b11bf3ffd3f5de74fa473ed86cc851c73dbc9"
},
"bbox": [
-72,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T10:24:18.000000Z",
"end_datetime": "2010-12-12T10:24:19.000000Z",
"start_datetime": "2010-12-12T10:24:18.000000Z",
"merkle:object_hash": "8c1277ae875d680ba189a884e63cd66ed9ac0c94b2ba5eb55d3ae11847814f6d"
"merkle:object_hash": "6d7dca89eec5da5e31023b7e24b9527e33a021099c8e74791ecda860f96091ab"
},
"bbox": [
-72,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T20:36:49.000000Z",
"end_datetime": "2014-09-13T09:09:54.000000Z",
"start_datetime": "2010-12-15T20:36:49.000000Z",
"merkle:object_hash": "ded2686d7956fde26b4941fd1a4294d7822f54bd7bf21f6275eff176484757db"
"merkle:object_hash": "88f4d9bb5b1512f214f1ca60e6ff2bac28640fd0a00339d01c453b6db9fd7f88"
},
"bbox": [
135,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-15T20:39:14.000000Z",
"end_datetime": "2013-04-04T20:49:08.000000Z",
"start_datetime": "2010-12-15T20:39:14.000000Z",
"merkle:object_hash": "e6f9dfc50f4bda3d7d07448d15e6528cb34b1d64f5a95ac77058004356afe2d8"
"merkle:object_hash": "457aa1fbc4005627077b55cd673a4457f2afc8a15ef3b776af8df60e60e8e1ab"
},
"bbox": [
133,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-16T10:04:43.000000Z",
"end_datetime": "2014-04-15T21:42:54.000000Z",
"start_datetime": "2010-12-16T10:04:43.000000Z",
"merkle:object_hash": "da40bec7fb828698829f9b69bcf2c926a8f7967044bc15cf049335bfa2bded3a"
"merkle:object_hash": "e7b65b76ed20f947d8a2b4ae126b832d9f07fb18822863556c9fa0ac40dae873"
},
"bbox": [
122,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-17T20:02:31.000000Z",
"end_datetime": "2014-08-02T08:34:58.000000Z",
"start_datetime": "2010-12-17T20:02:31.000000Z",
"merkle:object_hash": "09eae78bd6d1a6385d330d96832329c30c27868321a886220ec8d4eb7f729b3f"
"merkle:object_hash": "d0114075968ff1a3860eaf90d85317304f96757d6313f396eb556546a9c95006"
},
"bbox": [
144,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T08:49:36.000000Z",
"end_datetime": "2010-12-12T08:49:37.000000Z",
"start_datetime": "2010-12-12T08:49:36.000000Z",
"merkle:object_hash": "718637ce64200c90a8981916961cbaa994684aa362208ae4c4588f99dafa3834"
"merkle:object_hash": "7611aa1e37ee256bb346d0405c210c9e2256c43e7ed9854b4a91ef75caf59d09"
},
"bbox": [
-49,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T08:49:42.000000Z",
"end_datetime": "2010-12-12T08:49:43.000000Z",
"start_datetime": "2010-12-12T08:49:42.000000Z",
"merkle:object_hash": "81a60b7a33c7c17b486a4b3e77b2a74211aa2a4c868d1db8a39f8442800cdb7c"
"merkle:object_hash": "c4a241c5917f7fcb82bedca10127ad22504b3462c0d3704d76a5f549e2c24010"
},
"bbox": [
-48,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-12T10:24:00.000000Z",
"end_datetime": "2010-12-12T10:24:01.000000Z",
"start_datetime": "2010-12-12T10:24:00.000000Z",
"merkle:object_hash": "0372d094f1c112b8df328070bcabdc34d3bbe043803f89fb259c0d1510a5bee5"
"merkle:object_hash": "ae063aa10078d5316f36718b92f966c9f96f63f621839bd031730eb068c1c265"
},
"bbox": [
-72,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-13T03:47:46.000000Z",
"end_datetime": "2010-12-13T03:47:47.000000Z",
"start_datetime": "2010-12-13T03:47:46.000000Z",
"merkle:object_hash": "550175301f559ef22b03c179a2ece08cde642c5502bbeccc40dd55b6526fb6e2"
"merkle:object_hash": "c1511bd4889e8078010287a820d3fcc216cad1ab48374e324fb7098fd1c3f882"
},
"bbox": [
26,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"datetime": "2010-12-17T22:42:03.000000Z",
"end_datetime": "2010-12-17T22:42:04.000000Z",
"start_datetime": "2010-12-17T22:42:03.000000Z",
"merkle:object_hash": "176aa74d27edd9327ada0801fa79ccba1348e69d2f82d0bed44eda5b59908a31"
"merkle:object_hash": "9c33c4c49913483588fb7e2aa8084e675649ed253083ab62ac1ebebcfc16a944"
},
"bbox": [
-84,
Expand Down
Loading

0 comments on commit 14b1d7c

Please sign in to comment.