From 286ef6445ffe3dedcb7d29bcb27ed49759e2bb01 Mon Sep 17 00:00:00 2001 From: Henry Jewell Date: Thu, 22 Aug 2024 18:28:53 +0100 Subject: [PATCH 1/5] draft docs for veracity online inclusion proof re AB#9653 --- .../developer-patterns/veracity/index.md | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 content/developers/developer-patterns/veracity/index.md diff --git a/content/developers/developer-patterns/veracity/index.md b/content/developers/developer-patterns/veracity/index.md new file mode 100644 index 000000000..420acd351 --- /dev/null +++ b/content/developers/developer-patterns/veracity/index.md @@ -0,0 +1,152 @@ +--- +title: "Veracity" +description: "Supporting independent verification of your events" +lead: "Exploring the merkle log with the DataTrails Veracity" +date: 2021-06-09T13:49:35+01:00 +lastmod: 2021-06-09T13:49:35+01:00 +draft: false +images: [] +menu: + developers: + parent: "developer-patterns" +weight: 110 +toc: true +aliases: + - /docs/developer-patterns/veracity/ + +--- + +## Overview +Veracity is an open-source command line tool developed by DataTrails. It lets you explore the merkle +log and prove that your critical data is present and correct. By default it connects to the DataTrails +service to obtain the relevant merkle log for verification. The offline mode verifies event data +against local copies of merkle logs demonstrating the decentralizability of the proof layer. + +In this tutorial we'll explore how you can use Veracity to: +1. Prove the inclusion of events that matter in the DataTrails merkle log with `verify-inclusion` +2. Explore the DataTrails merkle log using the `node` command + +{{< note >}} +**Note:** Veracity has a range of power-user commands that are beyond the scope of this tutorial. If +you'd like to know more, please see the [Veracity Repo](https://github.com/datatrails/veracity?tab=readme-ov-file) +{{< /note >}} + +## Prerequisites +- [Downloaded and installed Veracity](https://github.com/datatrails/veracity/releases) + +## Verifying Critical Event Data +DataTrails captures events that matter to your business and lets you prove what happened at a later +date. Here are two walkthroughs showing how to perform the proof with Veracity for online and +offline data scenarios. + +{{< tabs name="event-verification" >}} +{{{< tab name="Online" >}} +#### Online Verification +We're going to walk through an example of proving that a publicly attested event exists on the +live DataTrails merkle log. + +#### Setup +```sh +EVENT_ID=publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818 +DATATRAILS_URL=https://app.datatrails.ai +TENANT_ID=tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +``` + +We just set some variables that reference the public tenant in DataTrails and a public event that +we'd like to verify the inclusion of. + +#### Loading the Event +```sh +curl -sL $DATATRAILS_URL/archivist/v2/$EVENT_ID > event.json +``` + +If you inspect the contents of `event.json` you will see something like this (some fields omitted +for salience.) +```json +{ + "identity": "publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818", + "asset_identity": "publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed", + "tenant_identity": "tenant/ee52ae46-d4fb-4030-9888-4696ef4b27da", + "event_attributes": { + "arc_display_type": "Business Critical Action", + "quality_system_ref": "1112345", + "approval_status": "2", + "arc_description": "An important event to your business" + }, + "timestamp_declared": "2024-08-22T16:28:54Z", + "timestamp_accepted": "2024-08-22T16:28:54Z", + "timestamp_committed": "2024-08-22T16:29:04.130Z", + "confirmation_status": "CONFIRMED", + "merklelog_entry": { + "commit": { + "index": "5772", + "idtimestamp": "01917aeb9103048500" + }, + "confirm": { + "mmr_size": "5774", + "root": "Z5S0ewjARI26IP04vJOC5pnH2V/M/BETAB4pojIZFkQ=", + "timestamp": "1724344145109", + "idtimestamp": "", + "signed_tree_head": "" + }, + } +} +``` + +#### Prove Event Inclusion +```sh +cat event.json | veracity \ + --data-url $DATATRAILS_URL/verifiabledata \ + --tenant=$PUBLIC_TENANT_ID \ + --loglevel=INFO \ + verify-included +``` + +Its that simple. Note that by default Veracity produces no output on success to enable simple build +system integration. By supplying `--loglevel=INFO` we get some insight into what the tool is doing: + +```sh +... +verifying for tenant: tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +verifying: 5772 2889 01917aeb9103048500 publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818 +OK|5772 2889|[c46a47677b043602dba8a9d1db3215207d1e2f4bdbb19bc07592602fa745b3b7, 18b5d6be487dc0b87d14cb7a389a6cf936aab2427dd26c1b230653f692964f06, a68a7678739a2e00431c25bf3d810b4f417830c3a95cfc692e771d6d54e37fa6, 907c561fd157a5a022aa4e42807bfca082c54d98531831847ad5414a1ad2b492, 9dfeaef9e86d6b857170245ec4cfc5d98fea11bba3937e211d134ab548eb743e, 04602adc424529275ce3415d55f31413743b67bf7e7fae03c90b08f1f5422264] +``` +{{< note >}} +##### Detecting Tampering +Try tampering with `event.json` and re-running `verify-inclusion` to observe the failure: + +```sh +sed "s/Business Critical Action/Malicious Action/g" event.json | \ + ./veracity \ + --data-url $DATATRAILS_URL/verifiabledata \ + --tenant=$PUBLIC_TENANT_ID \ + --loglevel=INFO \ + verify-included +``` + +```sh +... +verifying for tenant: tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +verifying: 5772 2889 01917aeb9103048500 publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818 +XX|5772 2889 + +error: the entry is not in the log. for tenant tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +the entry is not in the log. for tenant tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +``` +{{< /note >}} + +{{< /tab >}}} +{{{< tab name="Offline" >}} + +Coming soon ... + +{{< /tab >}}} +{{< /tabs >}} + +## Exploring the Merkle Log + +Coming soon ... + +## Further Reading + +Coming soon ... From b128591c1abde45f419252bd4f61f439daa14c1b Mon Sep 17 00:00:00 2001 From: Henry Jewell Date: Thu, 22 Aug 2024 19:36:32 +0100 Subject: [PATCH 2/5] reformatted and added offline verification re AB#9653 --- .../developer-patterns/veracity/index.md | 55 ++++++++++++++----- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/content/developers/developer-patterns/veracity/index.md b/content/developers/developer-patterns/veracity/index.md index 420acd351..9135b5180 100644 --- a/content/developers/developer-patterns/veracity/index.md +++ b/content/developers/developer-patterns/veracity/index.md @@ -1,9 +1,9 @@ --- title: "Veracity" description: "Supporting independent verification of your events" -lead: "Exploring the merkle log with the DataTrails Veracity" -date: 2021-06-09T13:49:35+01:00 -lastmod: 2021-06-09T13:49:35+01:00 +lead: "Exploring DataTrails' merkle log with Veracity" +date: 2024-08-22T19:35:35+01:00 +lastmod: 2024-08-22T19:35:35+01:00 draft: false images: [] menu: @@ -37,13 +37,7 @@ you'd like to know more, please see the [Veracity Repo](https://github.com/datat ## Verifying Critical Event Data DataTrails captures events that matter to your business and lets you prove what happened at a later date. Here are two walkthroughs showing how to perform the proof with Veracity for online and -offline data scenarios. - -{{< tabs name="event-verification" >}} -{{{< tab name="Online" >}} -#### Online Verification -We're going to walk through an example of proving that a publicly attested event exists on the -live DataTrails merkle log. +offline data scenarios. We're going to walk through an example of proving that a publicly attested event exists on the live DataTrails merkle log. #### Setup ```sh @@ -135,13 +129,44 @@ the entry is not in the log. for tenant tenant/6ea5cd00-c711-3649-6914-7b125928b ``` {{< /note >}} -{{< /tab >}}} -{{{< tab name="Offline" >}} +#### Offline Verification +Veracity can be used to verify the inclusion of an event in an offline backup of a DataTrails +merkle log. We can do this by supplying a `--data-local` argument instead of `--data-url`. First, +we'll need to download a copy of that log. -Coming soon ... +{{< note >}} +**Note:** Massifs are how DataTrails break the merkle log down into manageable chunks that record +up to a fixed number of events. Once that limit is reached, a new massif is started. `--data-local` +accepts a single massif file or a directory. The event we're verifying is contained by the first +massif. +{{< /note >}} + +```sh +curl -H "x-ms-blob-type: BlockBlob" -H "x-ms-version: 2019-12-12" https://app.datatrails.ai/verifiabledata/merklelogs/v1/mmrs/tenant/6ea5cd00-c711-3649-6914-7b125928bbb4/0/massifs/0000000000000000.log -o mmr.log +``` + +We can now run the `verify-included` command again using our local log data and the output will be +the same. -{{< /tab >}}} -{{< /tabs >}} +```sh +cat event.json | \ + ./veracity \ + --data-local mmr.log \ + --tenant=$PUBLIC_TENANT_ID \ + --loglevel=INFO \ + verify-included +``` + +{{< note >}} +**Note:** Proof paths shown in the output were complete at time of writing. As the log grows the +proof path increases in length. See [this article](https://docs.datatrails.ai/developers/developer-patterns/navigating-merklelogs/) for a deep-dive into our merkle log. +{{< /note >}} + +```sh +verifying for tenant: tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +verifying: 5772 2889 01917aeb9103048500 publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818 +OK|5772 2889|[c46a47677b043602dba8a9d1db3215207d1e2f4bdbb19bc07592602fa745b3b7, 18b5d6be487dc0b87d14cb7a389a6cf936aab2427dd26c1b230653f692964f06, a68a7678739a2e00431c25bf3d810b4f417830c3a95cfc692e771d6d54e37fa6, 907c561fd157a5a022aa4e42807bfca082c54d98531831847ad5414a1ad2b492, 9dfeaef9e86d6b857170245ec4cfc5d98fea11bba3937e211d134ab548eb743e, 04602adc424529275ce3415d55f31413743b67bf7e7fae03c90b08f1f5422264] +``` ## Exploring the Merkle Log From c93f54120305818edc8dd70daca990aadaca4bf4 Mon Sep 17 00:00:00 2001 From: Henry Jewell Date: Thu, 22 Aug 2024 20:14:24 +0100 Subject: [PATCH 3/5] add node command guide and wording updates re AB#9653 --- .../developer-patterns/veracity/index.md | 79 +++++++++++-------- 1 file changed, 47 insertions(+), 32 deletions(-) diff --git a/content/developers/developer-patterns/veracity/index.md b/content/developers/developer-patterns/veracity/index.md index 9135b5180..b1ab0c92e 100644 --- a/content/developers/developer-patterns/veracity/index.md +++ b/content/developers/developer-patterns/veracity/index.md @@ -17,45 +17,45 @@ aliases: --- ## Overview -Veracity is an open-source command line tool developed by DataTrails. It lets you explore the merkle -log and prove that your critical data is present and correct. By default it connects to the DataTrails -service to obtain the relevant merkle log for verification. The offline mode verifies event data -against local copies of merkle logs demonstrating the decentralizability of the proof layer. +Veracity is an open-source command line tool developed by DataTrails. With it, you can explore the +merkle log and prove the inclusion of your event data. By default it connects to the DataTrails +service to obtain a copy of the merkle log. Veracity can also work from copies of the merkle +log on disk. -In this tutorial we'll explore how you can use Veracity to: +In this guide we'll explore how you can use Veracity to: 1. Prove the inclusion of events that matter in the DataTrails merkle log with `verify-inclusion` 2. Explore the DataTrails merkle log using the `node` command -{{< note >}} -**Note:** Veracity has a range of power-user commands that are beyond the scope of this tutorial. If -you'd like to know more, please see the [Veracity Repo](https://github.com/datatrails/veracity?tab=readme-ov-file) -{{< /note >}} - ## Prerequisites -- [Downloaded and installed Veracity](https://github.com/datatrails/veracity/releases) +- Have downloaded and installed [Veracity](https://github.com/datatrails/veracity/releases) using the +instructions found [here](https://github.com/datatrails/veracity?tab=readme-ov-file#installation) + +## Verifying Event Data +DataTrails records the events that matter to your business and lets you prove them at a later date. +This guide will show how to do this for both online and offline data scenarios. -## Verifying Critical Event Data -DataTrails captures events that matter to your business and lets you prove what happened at a later -date. Here are two walkthroughs showing how to perform the proof with Veracity for online and -offline data scenarios. We're going to walk through an example of proving that a publicly attested event exists on the live DataTrails merkle log. +For simplicity we'll walk through an example of proving that a publicly attested event exists on the +merkle log for the public tenant on DataTrails. If you want to try this with your own data, simply +download a copy of your event from the DataTrails API and supply your tenant ID instead of the public +one. #### Setup +Lets set some variables that reference the public tenant in DataTrails and a public event that +we'd like to verify the inclusion of. + ```sh EVENT_ID=publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818 DATATRAILS_URL=https://app.datatrails.ai TENANT_ID=tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 ``` -We just set some variables that reference the public tenant in DataTrails and a public event that -we'd like to verify the inclusion of. - #### Loading the Event ```sh curl -sL $DATATRAILS_URL/archivist/v2/$EVENT_ID > event.json ``` -If you inspect the contents of `event.json` you will see something like this (some fields omitted -for salience.) +If you inspect the contents of `event.json` you will see something like this (with some fields omitted +for brevity.) ```json { "identity": "publicassets/046ad7b4-dc99-4f90-9511-d2fad2e72bed/events/fef3c753-52e5-406b-8e41-8a36a2cc4818", @@ -107,7 +107,8 @@ OK|5772 2889|[c46a47677b043602dba8a9d1db3215207d1e2f4bdbb19bc07592602fa745b3b7, ``` {{< note >}} ##### Detecting Tampering -Try tampering with `event.json` and re-running `verify-inclusion` to observe the failure: +Adversaries tampering with critical data is a serious risk, but DataTrails makes this straightforward +to detect. Try tampering with `event.json` and re-running `verify-inclusion` to observe the failure: ```sh sed "s/Business Critical Action/Malicious Action/g" event.json | \ @@ -132,21 +133,22 @@ the entry is not in the log. for tenant tenant/6ea5cd00-c711-3649-6914-7b125928b #### Offline Verification Veracity can be used to verify the inclusion of an event in an offline backup of a DataTrails merkle log. We can do this by supplying a `--data-local` argument instead of `--data-url`. First, -we'll need to download a copy of that log. +we'll need to get a copy of the massif. {{< note >}} -**Note:** Massifs are how DataTrails break the merkle log down into manageable chunks that record -up to a fixed number of events. Once that limit is reached, a new massif is started. `--data-local` -accepts a single massif file or a directory. The event we're verifying is contained by the first -massif. +**Note:** DataTrails break the merkle log down into manageable chunks called massifs. Once each massif +is full, a new one is started. The filenames are numbered (e.g. 0000000000000000.log, 0000000000000001.log) to indicate order. + +The argument `--data-local` accepts a single massif file or a directory containing multiple. The event +we're verifying in this example is contained within the first massif. {{< /note >}} ```sh curl -H "x-ms-blob-type: BlockBlob" -H "x-ms-version: 2019-12-12" https://app.datatrails.ai/verifiabledata/merklelogs/v1/mmrs/tenant/6ea5cd00-c711-3649-6914-7b125928bbb4/0/massifs/0000000000000000.log -o mmr.log ``` -We can now run the `verify-included` command again using our local log data and the output will be -the same. +When we run the `verify-included` command using our local copy of the massif, it will also verify +successfully with the outputs matching. ```sh cat event.json | \ @@ -159,7 +161,7 @@ cat event.json | \ {{< note >}} **Note:** Proof paths shown in the output were complete at time of writing. As the log grows the -proof path increases in length. See [this article](https://docs.datatrails.ai/developers/developer-patterns/navigating-merklelogs/) for a deep-dive into our merkle log. +proof path increases in length. See [this article](developers/developer-patterns/navigating-merklelogs/) for a deep-dive into our merkle log. {{< /note >}} ```sh @@ -170,8 +172,21 @@ OK|5772 2889|[c46a47677b043602dba8a9d1db3215207d1e2f4bdbb19bc07592602fa745b3b7, ## Exploring the Merkle Log -Coming soon ... +The `node` command is a convenience function for retrieving the value of a node in the merkle log +without needing to download the entire massif. Lets use our example event from earlier, which lives +at index 5772 (this works with both `--data-local` and `--data-url`.) + +```sh +veracity --data-url $DATATRAILS_URL/verifiabledata \ + --tenant=$PUBLIC_TENANT_ID \ + node --mmrindex 5772 +``` + +The value returned is the hash stored at that node: -## Further Reading +```sh +26c7061166187363dd156f4f5f1f517a39323af3c70d572de28c5206de160ec2 +``` -Coming soon ... +Leaf nodes in the merkle log contain the hash of the event data (plus some metadata, see [this article](/developers/developer-patterns/navigating-merklelogs/#leaf-nodes-created-by-hashing-event-data)) while +intermediate nodes hash together the content of their left and right children. From 80d09b5d568fdc429fb9b6bed60d6f9aa6022b64 Mon Sep 17 00:00:00 2001 From: Henry Jewell Date: Fri, 23 Aug 2024 09:26:54 +0100 Subject: [PATCH 4/5] fixup! add node command guide and wording updates --- content/developers/developer-patterns/veracity/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/developers/developer-patterns/veracity/index.md b/content/developers/developer-patterns/veracity/index.md index b1ab0c92e..6a03fa61b 100644 --- a/content/developers/developer-patterns/veracity/index.md +++ b/content/developers/developer-patterns/veracity/index.md @@ -40,7 +40,7 @@ download a copy of your event from the DataTrails API and supply your tenant ID one. #### Setup -Lets set some variables that reference the public tenant in DataTrails and a public event that +Let's set some variables that reference the public tenant in DataTrails and a public event that we'd like to verify the inclusion of. ```sh @@ -126,7 +126,7 @@ verifying: 5772 2889 01917aeb9103048500 publicassets/046ad7b4-dc99-4f90-9511-d2f XX|5772 2889 error: the entry is not in the log. for tenant tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 -the entry is not in the log. for tenant tenant/6ea5cd00-c711-3649-6914-7b125928bbb4 +... ``` {{< /note >}} @@ -139,8 +139,8 @@ we'll need to get a copy of the massif. **Note:** DataTrails break the merkle log down into manageable chunks called massifs. Once each massif is full, a new one is started. The filenames are numbered (e.g. 0000000000000000.log, 0000000000000001.log) to indicate order. -The argument `--data-local` accepts a single massif file or a directory containing multiple. The event -we're verifying in this example is contained within the first massif. +The argument `--data-local` accepts either a single massif file or a directory containing multiple +massif files. The event we're verifying in this example is contained within the first massif. {{< /note >}} ```sh From 77c82b2ab4d7ce3aa8f542aaee7cc8e9582d99f8 Mon Sep 17 00:00:00 2001 From: Henry Jewell Date: Fri, 23 Aug 2024 12:03:32 +0100 Subject: [PATCH 5/5] fixup! fixup! add node command guide and wording updates --- content/developers/developer-patterns/veracity/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/developers/developer-patterns/veracity/index.md b/content/developers/developer-patterns/veracity/index.md index 6a03fa61b..543fde602 100644 --- a/content/developers/developer-patterns/veracity/index.md +++ b/content/developers/developer-patterns/veracity/index.md @@ -1,5 +1,5 @@ --- -title: "Veracity" +title: "Independently verifying DataTrails transparent merkle logs" description: "Supporting independent verification of your events" lead: "Exploring DataTrails' merkle log with Veracity" date: 2024-08-22T19:35:35+01:00 @@ -9,7 +9,7 @@ images: [] menu: developers: parent: "developer-patterns" -weight: 110 +weight: 38 toc: true aliases: - /docs/developer-patterns/veracity/