From 81835edd3ba1b4a5713529bce92667b899aaa701 Mon Sep 17 00:00:00 2001 From: Shannyn Telander Date: Fri, 29 Jul 2022 13:08:39 -0500 Subject: [PATCH 1/4] Consolidate REST API features and organize This change updates the community planning features list to consolidate REST API designs into one feature document. This document will speak on the designs from the past and the future designs for the Grid REST API. This also alphabetizes the remaining features. Signed-off-by: Shannyn Telander --- community/planning/features.md | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/community/planning/features.md b/community/planning/features.md index b9b4505c..490590f4 100644 --- a/community/planning/features.md +++ b/community/planning/features.md @@ -1,6 +1,6 @@ # Feature Planning @@ -17,18 +17,13 @@ community meeting. | Feature | Description | | ------- | ----------- | -| [Location Sapling]({% link community/planning/location_sapling.md %}) | A front-end for Grid Location functionality | -| [Batch Submission Queuer Strategies]({% link community/planning/batch_queuer_strategies.md %}) | A set of algorithms for determining batch submission order | | [Batch Submitter]({% link community/planning/batch_submitter.md %}) | A daemon for submitting batches | +| [Batch Submission Queuer Strategies]({% link community/planning/batch_queuer_strategies.md %}) | A set of algorithms for determining batch submission order | | [Batch Tracking Store]({% link community/planning/batch_tracking_store.md %}) | A store for providing functionality to interact with the batch tracking database | +| [DLT Event Monitor]({% link community/planning/dlt_event_monitor.md %}) | A proposed future event monitor for syncing batch statuses | +| [DLT Polling Monitor]({% link community/planning/dlt_polling_monitor.md %}) | A proposed future polling monitor for syncing batch statuses | | [High Availability]({% link community/planning/ha_plan.md %}) | A high-availability plan for Grid | -| [High & Low-level Transaction REST API]({% link -community/planning/high_low_level_transaction_rest_api.md %}) | An API for submitting higher-level JSON transactions | -| [Product Sapling]({% link community/planning/product_sapling.md %}) | A front-end for Grid Product functionality | -| [Queuer - One-round serial]({% link community/planning/queuer_strat_one_round_serial.md %}) | A one-round serial queuing strategy for submitting batches to the DLT | +| [Location Sapling]({% link community/planning/location_sapling.md %}) | A front-end for Grid Location functionality | | [Queuer - One-round parallel]({% link community/planning/queuer_strat_one_round_parallel.md %}) | Initial research regarding a one-round parallel queuing strategy for submitting batches to the DLT | -| [Future REST API]({% link community/planning/rest_api/index.md %}) | A proposed future REST API for transacting with Grid | -| [DLT Polling Monitor]({% link community/planning/dlt_polling_monitor.md %}) | A proposed future polling monitor for syncing batch statuses | -| [DLT Event Monitor]({% link community/planning/dlt_event_monitor.md %}) | A proposed future event monitor for syncing batch statuses | -| [Rest API Backwards Compatibility]({% link community/planning/rest_api/backwards_compatibility.md %}) | A proposed backwards compatibility plan for testing the REST API | -| [Griddle Proxy]({% link community/planning/griddle_proxy.md %}) | Proposed proxying functionality for Griddle | +| [Product Sapling]({% link community/planning/product_sapling.md %}) | A front-end for Grid Product functionality | +| [REST API]({% link community/planning/rest_api/index.md %}) | A REST API for transacting with Grid | From bffa142302dd8d5be26282d5df08540dab96c551 Mon Sep 17 00:00:00 2001 From: Shannyn Telander Date: Fri, 29 Jul 2022 13:11:31 -0500 Subject: [PATCH 2/4] Move REST API feature documents into `rest_api` This change moves feature documents for the REST API into a directory, titled `rest_api`. These documents were unlinked from the community features page, as the index page of this directory will link to the historic design documents. Signed-off-by: Shannyn Telander --- community/planning/{ => rest_api}/griddle_proxy.md | 0 .../{ => rest_api}/high_low_level_transaction_rest_api.md | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename community/planning/{ => rest_api}/griddle_proxy.md (100%) rename community/planning/{ => rest_api}/high_low_level_transaction_rest_api.md (100%) diff --git a/community/planning/griddle_proxy.md b/community/planning/rest_api/griddle_proxy.md similarity index 100% rename from community/planning/griddle_proxy.md rename to community/planning/rest_api/griddle_proxy.md diff --git a/community/planning/high_low_level_transaction_rest_api.md b/community/planning/rest_api/high_low_level_transaction_rest_api.md similarity index 100% rename from community/planning/high_low_level_transaction_rest_api.md rename to community/planning/rest_api/high_low_level_transaction_rest_api.md From 9755b43f866cc918bc8bd9938038f1bd6dc4f9d4 Mon Sep 17 00:00:00 2001 From: Shannyn Telander Date: Fri, 29 Jul 2022 16:36:36 -0500 Subject: [PATCH 3/4] Add plan for REST API moving forward This change adds a description of how the REST API will handle batches, for both gridd and griddle. This includes information on the payloads that will be accepted by the new REST API and how specific endpoints will respond. Signed-off-by: Shannyn Telander --- community/planning/rest_api/index.md | 129 ++++++++++++++++++++++++++- 1 file changed, 127 insertions(+), 2 deletions(-) diff --git a/community/planning/rest_api/index.md b/community/planning/rest_api/index.md index 609818b9..dfcdbff4 100644 --- a/community/planning/rest_api/index.md +++ b/community/planning/rest_api/index.md @@ -6,12 +6,137 @@ https://creativecommons.org/licenses/by/4.0/ --> - This document is intended to capture proposed updates to the Grid REST API. This document is a work-in-progress and proposed changes may or may not be implemented as project requirements evolve. -* [Future REST API Reference](/community/planning/rest_api/api/) +## Planned functionality + +Mainly, the REST API will be updated to support more robust batch handling. +The proposed functionality will accept transactions in JSON, the request body, +while arguments for the batch can be submitted in the request headers. Arguments +for a batch may include a service ID, if Grid is running with Splinter, or a +user-created identifier for the batch submitted. More specific headers pertaining +to batch arguments may be added. The array of JSON arguments will be converted +into the appropriate transactions and signed. As Grid will support batch +tracking, the REST API will not directly submit batches. Instead, once a batch +has been signed and built, it will be stored in Grid's database. Other +components will handle the batch for the rest of its lifecycle and the database +entry will reflect updates. + +The REST API is provided by two components, gridd and griddle. Griddle is a +client component that will act as both a proxy and signing utility. Grid may +also perform transaction and batch signing, if configured at run-time. Griddle +will communicate with gridd over REST API and may be run from a separate +security domain. Access to the internet is not required by Griddle, which allows +it to handle sensitive data, such as private key files, without worry this data +may become compromised. As griddle does not have access to gridd's database, it +will also proxy batches it creates to gridd's `/batches` endpoint to be stored +in the database. Database support may be added to griddle in the future. + +All smart contract actions that may be performed in Grid will be represented by +`POST` endpoints. An example of a request for the `POST /organizations` endpoint +is below: + +``` +POST /locations +Content-Type: application/json+grid-simple +Headers: + data_change_id: + // Batch arguments +Body: +[ + { + "namespace": "GS1", + "location_id": "1234", + "properties": [ + "..." + ], + }, +] +``` + +This JSON is converted by the endpoint into it's respective transaction type, +the action of this example would be `CreateLocation`. There may be multiple +transactions in the JSON body. Endpoints receiving these requests must account +for the list of transactions when deserializing the request body. More complex +batches may include transactions with different action types. The `/batches` +endpoint allows for this situation. An example request to this endpoint follows: + +``` +POST /batches +Content-Type: application/json +Headers: + data_change_id: + // Batch arguments +Body: +[ + { + "target": "POST /locations" + "namespace": "GS1", + "location_id": "1234", + "properties": [ + "..." + ], + }, + { + "target": "PUT /locations" + "namespace": "GS1", + "location_id": "5678", + "properties": [ + "..." + ], + }, +] +``` + +This request will create a batch containing two transactions, one for creating +a location and one for updating another location. The endpoint handling this +request will deserialize the request body into transactions, depending on +value of the `target` field. If the endpoint is unable to recognize the targeted +transaction, it will return `400 Bad Request`. A signer object is passed to +each endpoint and is required to build the batch. If the endpoint does not have +access to a signer, it will return `400 Bad Request`. + +Once a batch has been built from the request body and signed, it can be stored +in gridd's database. As endpoints may be provided by both gridd and griddle, +the signed batch should be handled by an implementation-specific object. +Endpoints have access to a trait object, `BatchSubmissionHandler`, which +submits the batch once it is signed. This trait implements a `submit_batches` +method, which takes in pertinent batch information to store. If an endpoint is +provided by griddle, the submission handler is required to send the batch +to gridd to be stored. An endpoint provided by gridd is able to directly store +the batch. + +If a batch is submitted successfully, the endpoint will respond with +status code `202 Accepted` and a JSON body of the identifiers for the batch +stored. An example response body to a request that included a Splinter service +ID and a user-defined ID. + +``` +{ + [ + { + "dlt_batch_id": , + "data_change_id": , + "service_id": , + }, + ] +} +``` + +If Grid is not running with Splinter services, the `service_id` field will not +be returned. Similarly, if no `data_change_id` was submitted in the original +request, it will also not be returned. + +If a batch is not submitted successfully, the endpoint will respond with a +status code to describe the error and a JSON body with any additional data. For +example, an error is returned at the point griddle attempts to send a batch +to gridd, a `SendError` with a message describing what went wrong. The griddle +endpoint may return a `402 Bad Gateway` if it is unable to connect to gridd. + +All previous design documents for the REST API are linked in the following +section. ## Changes From 6452d92ede421ab602b7fe380b4bfa0dd48605e5 Mon Sep 17 00:00:00 2001 From: Shannyn Telander Date: Fri, 29 Jul 2022 16:32:24 -0500 Subject: [PATCH 4/4] Add links and information for historical designs This change adds short descriptions to previously proposed functionality and links to the respective design documents. Signed-off-by: Shannyn Telander --- community/planning/rest_api/index.md | 42 ++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/community/planning/rest_api/index.md b/community/planning/rest_api/index.md index dfcdbff4..0df1b865 100644 --- a/community/planning/rest_api/index.md +++ b/community/planning/rest_api/index.md @@ -138,23 +138,47 @@ endpoint may return a `402 Bad Gateway` if it is unable to connect to gridd. All previous design documents for the REST API are linked in the following section. -## Changes - - - Added `POST` and `PUT` routes for various Grid resources. These endpoints +## Proposed functionality + +- Accept JSON Batches. The Grid REST API will be updated to support + batches encoded in bytes or JSON, providing a more natural + experience for users. An initial design document for the future + REST API, [High-level and Low-level Transaction REST API]({% + link community/planning/rest_api/high_low_level_transaction_rest_api.md %}), + explains that Grid will support both byte-encoded and JSON-encoded + batches. JSON-encoded batches will be signed and encoded by the + Grid daemon, as needed before submission. + +- Proxy. Griddle is a client component that runs separately from the Grid daemon, + communicating with Grid via REST API. Griddle is able to proxy requests to Grid + and will create and sign batches submitted as JSON. The initial design + document, [Griddle Proxy]({% + link community/planning/rest_api/griddle_proxy.md %}), explains this + functionality more thoroughly. + +- Backwards compatibility. Especially as the API changes greatly, setting + forth a plan for handling backwards compatibility will smooth the process + for both developers and users. See [Rest API Backwards + Compatibility]({% link community/planning/rest_api/backwards_compatibility.md %}) + for more details. + +## Proposed API + +* [Future REST API Reference](/community/planning/rest_api/api/) + +- Added `POST` and `PUT` routes for various Grid resources. These endpoints will be used to submit batches to create and update resources. This takes the place of submitting everything through a `POST` to `/batches` and will provide users with a more familiar REST API experience. - - Updated resource schemas. The resource schemas for various Grid features +- Updated resource schemas. The resource schemas for various Grid features have been updated to reflect their protobuf message counterparts and what a user can expect to see when fetching that resource. Some resources do not differ between their create and update messages and have not changed. - - Removed Track and Trace endpoints. This feature will no longer be supported. +- Removed Track and Trace endpoints. This feature will no longer be supported. -## For further consideration +## Proposed for further consideration - - Consider changing `/batch_statuses` to `/batch-statuses`. This would bring + - Change `/batch_statuses` to `/batch-statuses`. This would bring this endpoint in line with other endpoints in this API but consideration must be given to the impact this will have on the corresponding endpoints in Sawtooth and Splinter as well as backwards-compatibility concerns. - - [Rest API Backwards Compatibility]({% link - community/planning/rest_api/backwards_compatibility.md %})