Merge branch 'main' into cwinter_3343

microsoft · Mar 23, 2022 · 6b1c3fa · 6b1c3fa
2 parents 6abbb4d + c341e24
commit 6b1c3fa
Show file tree

Hide file tree

Showing 10 changed files with 209 additions and 70 deletions.
diff --git a/doc/architecture/index.rst b/doc/architecture/index.rst
@@ -57,6 +57,14 @@ Architecture
 
     Indexing system used to speed up historical queries.
 
+    ---
+
+    :fa:`scroll` :doc:`receipts`
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    Receipts can be used with the ledger for audit purposes.
+
+
     ---
 
     :fa:`address-book` :doc:`tls_internals`
@@ -90,6 +98,7 @@ Architecture
     raft_tla
     node_to_node
     indexing
+    receipts
     tls_internals
     tcp_internals
     quic_internals
diff --git a/doc/architecture/receipts.rst b/doc/architecture/receipts.rst
@@ -0,0 +1,9 @@
+Receipts
+========
+
+CCF implements `write` receipts, which are signed proofs associated with a transaction. They serve two main purposes:
+
+1. :ref:`Endorse <use_apps/verify_tx:Application Claims>` claims made by the application logic, ie. a signed statement of fact, verifiable offline and by third parties, equivalent to "this transaction produced this outcome at this position in the ledger".
+2. Together with a copy of the ledger, or other receipts, they can be used to :ref:`audit <audit/receipts:Receipts>` the service and hold the consortium to account.
+
+Internally, receipts are also used to establish the validity of ledger snapshots.
diff --git a/doc/audit/index.rst b/doc/audit/index.rst
@@ -3,6 +3,13 @@ Audit
 
 .. panels::
 
+    :fa:`scroll` :doc:`receipts`
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    Receipts can be used with the ledger for audit purposes.
+
+    ---
+
     :fa:`table` :doc:`builtin_maps`
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -23,17 +30,9 @@ Audit
 
     ---
 
-    :fa:`scroll` :doc:`receipts`
-    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-    Audit individual transactions on the basis of their receipt.
-
-    ---
-
-
 .. toctree::
     :hidden:
 
+    receipts
     builtin_maps
-    python_library
-    receipts
+    python_library
diff --git a/doc/audit/receipts.rst b/doc/audit/receipts.rst
@@ -1,60 +1,21 @@
 Receipts
 ========
 
-Write Receipts
---------------
+In combination with a copy of the ledger, receipts are also useful for audit purposes.
 
-Once a transaction has been committed, it is possible to get a cryptographic receipt over the entry produced in the ledger. That receipt can be verified offline.
+Check for transaction inclusion
+-------------------------------
 
-To obtain a receipt, a user needs to call a :http:GET:`/node/receipt` for a particular transaction ID. Because fetching the information necessary to produce a receipt likely involves a round trip to the ledger, the endpoint is implemented as a historical query.
-This means that the request may return ``202 Accepted`` at first, with a suggested ``Retry-After`` header. A subsequent call will return the actual receipt, for example:
+A user having executed a transaction, fetched a receipt for it, can check for its inclusion in the ledger.
+All they need to do is scan to the corresponding :term:`Transaction ID`, digest the transaction, and compare it with `write_set_digest` in their receipt.
 
-.. code-block:: bash
+Denounce an invalid recovery
+----------------------------
 
-    $ curl -X GET "https://<ccf-node-address>/app/receipt?transaction_id=2.643" --cacert service_cert.pem --key user0_privk.pem --cert user0_cert.pem
+A user having executed a number of transactions, and fetched receipts for them, can denounce a recovery that removes one or more of these transactions.
+This may occur if the consortium approves a catastrophic recovery from a truncated ledger.
 
-    {'cert': '-----BEGIN CERTIFICATE-----\n'
-            'MIIBzjCCAVSgAwIBAgIQGR/ue9CFspRa/g6jSMHFYjAKBggqhkjOPQQDAzAWMRQw\n'
-            'EgYDVQQDDAtDQ0YgTmV0d29yazAeFw0yMjAxMjgxNjAzNDZaFw0yMjAxMjkxNjAz\n'
-            'NDVaMBMxETAPBgNVBAMMCENDRiBOb2RlMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE\n'
-            'wsdpHLNw7xso/g71XzlQjoITiTBOef8gCayOiPJh/W2YfzreOawzD6gVQPSI+iPg\n'
-            'ZPc6smFhtV5bP/WZ2KW0K9Pn+OIjm/jMU5+s3rSgts50cRjlA/k81bUI88dzQzx9\n'
-            'o2owaDAJBgNVHRMEAjAAMB0GA1UdDgQWBBQgtPwYar54AQ4UL0RImVsm6wQQpzAf\n'
-            'BgNVHSMEGDAWgBS2ngksRlVPvwDcLhN57VV+j2WyBTAbBgNVHREEFDAShwR/AAAB\n'
-            'hwR/ZEUlhwR/AAACMAoGCCqGSM49BAMDA2gAMGUCMQDq54yS4Bmfwfcikpy2yL2+\n'
-            'GFemyqNKXheFExRVt2edxVgId+uvIBGjrJEqf6zS/dsCMHVnBCLYRgxpamFkX1BF\n'
-            'BDkVitfTOdYfUDWGV3MIMNdbam9BDNxG4q6XtQr4eb3jqg==\n'
-            '-----END CERTIFICATE-----\n',
-    'leaf_components': {'commit_evidence': 'ce:2.643:55dbbbf04b71c6dcc01dd9d1c0012a6a959aef907398f7e183cc8913c82468d8',
-                        'write_set_digest': 'd0c521504ce2be6b4c22db8e99b14fc475b51bc91224181c75c64aa2cef72b83'},
-    'node_id': '7dfbb9a56ebe8b43c833b34cb227153ef61e4890187fe6164022255dec8f9646',
-    'proof': [{'left': '00a771baf15468ed05d6ef8614b3669fcde6809314650061d64281b5d4faf9ec'},
-              {'left': 'a9c8a36d01aa9dfbfb74c6f6a2cef2efcbd92bd6dfd1f7440302ad5ac7be1577'},
-              {'right': '8e238d95767e6ffe4b20e1a5e93dd7b926cbd86caa83698584a16ad2dd7d60b8'},
-              {'left': 'd4717996ae906cdce0ac47257a4a9445c58474c2f40811e575f804506e5fee9f'},
-              {'left': 'c1c206c4670bd2adee821013695d593f5983ca0994ae74630528da5fb6642205'}],
-    'signature': 'MGQCMHrnwS123oHqUKuQRPsQ+gk6WVutixeOvxcXX79InBgPOxJCoScCOlBnK4UYyLzangIwW9k7IZkMgG076qVv5zcx7OuKb7bKyii1yP1rcakeGVvVMwISeE+Fr3BnFfPD66Df'}
+This user can either:
 
-Note that receipts over signature transactions are a special case, for example:
-
-.. code-block:: bash
-
-    $ curl -X GET "https://<ccf-node-address>/app/receipt?transaction_id=2.35" --cacert service_cert.pem --key user0_privk.pem --cert user0_cert.pem
-
-    {'leaf': 'fdc977c49d3a8bdf986176984e9432a09b5f6fe0c04e0b1c2dd177c03fdca9ec',
-     'node_id': '06fef62c80b6471c7005c1b114166fd1b0e077845f5ad544ad4eea4fb1d31f78',
-     'proof': [],
-     'signature': 'MGQCMACklXqd0ge+gBS8WzewrwtwzRzSKy+bfrLZVx0YHmQvtsqs7dExYESsqrUrB8ZcKwIwS3NPKaGq0w2QlPlCqUC3vQoQvhcZgPHPu2GkFYa7JEOdSKLknNPHaCRv80zx2RGF',
-     'cert': '<PEM string>'}
-
-The proof is empty, and the 'leaf' field is set to the value being signed, which is the root of the Merkle Tree covering all transactions until the signature.
-This allows writing verification code that handles both regular and signature receipts similarly, but it is worth noting that the 'leaf' value for signatures is not
-the digest of the signature transaction itself.
-
-Verifying a receipt involves the following steps:
-
-  - Digest ``commit_evidence`` to produce ``commit_evidence_digest`` and ``claims`` to produce ``claims_digest`` when applicable.
-  - If the receipt contains ``leaf_components``, digest the concatenation ``write_set_digest + commit_evidence_digest + claims_digest`` to produce ``leaf``.
-  - Combine ``leaf`` with the successive elements in ``proof`` to calculate the value of ``root``. See :py:func:`ccf.receipt.root` for a reference implementation.
-  - Verify ``signature`` over the ``root`` using the certificate of the node identified by ``node_id`` and ``cert``. See :py:func:`ccf.receipt.verify` for a reference implementation.
-  - Check that the certificate ``cert`` of ``node_id`` used to sign the receipt is endorsed by the CCF network. See :py:func:`ccf.receipt.check_endorsement` for a reference implementation.
+1. Query the new service for receipts at the same :term:`Transaction ID` values.  If those transactions come back as `INVALID`, because they were truncated, the signature over the old receipts is proof of truncation. If they come back as `COMMITTED` with a different root, the existence of two signatures over different roots at the same TxID is proof that a fork happened.
+2. Scan the ledger, for example using the :doc:`/audit/python_library`, and find the transactions for which they have receipts. The `write_set_digest` in the receipt should match the digest of the transactions on disk. If it doesn't, the signature over the receipt is proof of a fork.
diff --git a/doc/build_apps/api.rst b/doc/build_apps/api.rst
@@ -100,6 +100,13 @@ Supporting Types
 .. doxygenenum:: ccf::ApiResult
    :project: CCF
 
+RPC Context
+-----------
+
+.. doxygenclass:: enclave::RpcContext
+   :project: CCF
+   :members: get_request_body, get_request_query, get_request_path_params, get_request_verb, get_request_path, get_request_headers, get_request_header, get_request_url, set_claims_digest
+
 Historical Queries
 ------------------
 
@@ -114,6 +121,10 @@ Historical Queries
    :project: CCF
    :members:
 
+.. doxygenstruct:: ccf::TxReceipt
+   :project: CCF
+   :members:
+
 JavaScript FFI Plugins
 ----------------------
 

diff --git a/doc/build_apps/logging_cpp.rst b/doc/build_apps/logging_cpp.rst
@@ -214,12 +214,13 @@ A user wanting to tie transaction-specific values to a receipt can do so by atta
     :end-before: SNIPPET_END: set_claims_digest
     :dedent:
 
-CCF will then record the digest of the transaction as the combined digest of the write set, plus this claims digest.
+CCF will then record this transaction as a leaf in the Merkle tree constructed from the combined digest of the write set, this ``claims_digest``, and the commit evidence.
 
-Receipts for transactions that have set a claims digest expose a ``leaf_components``, rather than an opaque ``leaf``,
-which means that a receipt endpoint can choose to reveal the claims and remove their digest from the receipt.
+This ``claims_digest`` will be exposed in receipts under ``leaf_components``. It can then be revealed externally,
+or by the endpoint directly if it has been stored in the ledger. The receipt object deliberately makes the ``claims_digest`` optional,
+to allow the endpoint to remove it when the claims themselves are revealed.
 
-The receipt verification can then only succeed if the revealed claims are digested and their digest combined into a
+Receipt verification can then only succeed if the revealed claims are digested and their digest combined into a
 ``leaf`` that correctly combines with the ``proof`` to form the ``root`` that the signature covers. Receipt verification
 therefore establishes the authenticity of the claims.
 
@@ -229,5 +230,48 @@ therefore establishes the authenticity of the claims.
     :end-before: SNIPPET_END: claims_digest_in_receipt
     :dedent:
 
-A client consuming the output of this endpoint can then digest the claims themselves, combine the digest with the other leaf component
-(``write_set_digest``) to obtain the equivalent ``leaf``.
+A client consuming the output of this endpoint must digest the claims themselves, combine the digest with the other leaf components
+(``write_set_digest`` and ``hash(commit_evidence)``) to obtain the equivalent ``leaf``. See :ref:`use_apps/verify_tx:Receipt Verification` for the full set of steps.
+
+As an example, a logging application may register the contents being logged as a claim:
+
+.. literalinclude:: ../../samples/apps/logging/logging.cpp
+    :language: cpp
+    :start-after: SNIPPET_START: record_public
+    :end-before: SNIPPET_END: record_public
+    :dedent:
+
+And expose an endpoint returning receipts, with that claim expanded:
+
+.. literalinclude:: ../../samples/apps/logging/logging.cpp
+    :language: cpp
+    :start-after: SNIPPET_START: get_historical_with_receipt
+    :end-before: SNIPPET_END: get_historical_with_receipt
+    :dedent:
+
+Receipts from this endpoint will then look like:
+
+.. code-block:: json
+
+    {'msg': 'Public message at idx 5 [0]',
+     'receipt': {'cert': '-----BEGIN CERTIFICATE-----\n'
+                         'MIIBzzCCAVWgAwIBAgIRANKoegKBViucMxSPzftnDB4wCgYIKoZIzj0EAwMwFjEU\n'
+                         'MBIGA1UEAwwLQ0NGIE5ldHdvcmswHhcNMjIwMzE1MjExODIwWhcNMjIwMzE2MjEx\n'
+                         'ODE5WjATMREwDwYDVQQDDAhDQ0YgTm9kZTB2MBAGByqGSM49AgEGBSuBBAAiA2IA\n'
+                         'BG+RJ5qNPOga8shCF3w64yija/ShW46JxrE0n9kDybyRf+L3810GjCvjxSpzTQhX\n'
+                         '5WEF2dou1dG2ppI/KSNQsSfk081lbaB50NADWw+jDCtrq/fKuZ+w9wQSaoSvE5+0\n'
+                         '1qNqMGgwCQYDVR0TBAIwADAdBgNVHQ4EFgQU7tFQR91U1EDhup1XPS3u0w5+R2Yw\n'
+                         'HwYDVR0jBBgwFoAU3aI0vfJMBdWckvv9dKK2UzNCLU0wGwYDVR0RBBQwEocEfwAA\n'
+                         'AYcEfxoNCocEfwAAAjAKBggqhkjOPQQDAwNoADBlAjAiOmvGpatg4Uq8phQkwj/p\n'
+                         'Wj33fih6SUtRHOpdsIKvbV8TDNHRdSo1RKPArDd1w1wCMQDnw9zziS5G8qwvucP3\n'
+                         'gn3htz+2ZPBJRr98AqmRNmgflhgqLQp+jAVPrJaWtD3fDpw=\n'
+                         '-----END CERTIFICATE-----\n',
+                 'leaf_components': {'commit_evidence': 'ce:2.25:54571ec6d0540b364d8343b74dff055932981fd72a24c1399c39ca9c74d2f713',
+                                     'write_set_digest': '08b044fc5b0e9cd03c68d77c949bb815e3d70bd24ad339519df48758430ac0f7'},
+                 'node_id': '95baf92969b4c9e52b4f8fcde830dea9fa0286a8c3a92cda4cffcf8251c06b39',
+                 'proof': [{'left': '50a1a35a50bd2c5a4725907e77f3b1f96f1f9f37482aa18f8e7292e0542d9d23'},
+                           {'left': 'e2184154ac72b304639b923b3c7a0bc04cecbd305de4f103a174a90210cae0dc'},
+                           {'left': 'abc9bcbeff670930c34ebdab0f2d57b56e9d393e4dccdccf2db59b5e34507422'}],
+                 'signature': 'MGUCMHYBgZ3gySdkJ+STUL13EURVBd8354ULC11l/kjx20IwpXrg/aDYLWYf7tsGwqUxPwIxAMH2wJDd9wpwbQrULpaAx5XEifpUfOriKtYo7XiFr05J+BV10U39xa9GBS49OK47QA=='}}
+                 
+Note that the ``claims_digest`` is not present in the ``leaf_components``, and must be re-computed by digesting the ``msg``.
diff --git a/doc/overview/glossary.rst b/doc/overview/glossary.rst
@@ -80,5 +80,8 @@ Glossary
   TLS
     `Transport Layer Security <https://en.wikipedia.org/wiki/Transport_Layer_Security>`_ is an IETF cryptographic protocol standard designed to secure communications between a client and a server over a computer network.
 
+  Transaction ID
+    Unique transaction identifier in CCF, composed of a View and a Sequence Number. Sequence Numbers start from 1, and are contiguous. Views are monotonic.
+
   Users
     Directly interact with the application running in CCF. Their public identity should be voted in by members before they are allowed to issue requests.