From 96db2edca4022437019bd3e968d4adc8c3bde6f4 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Fri, 1 Apr 2022 14:16:00 +0200 Subject: [PATCH 01/12] Specify the HTTP Gateway protocol --- default.nix | 3 +- spec/http-gateway.did | 34 +++++++++++++ spec/index.adoc | 116 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 151 insertions(+), 2 deletions(-) create mode 100644 spec/http-gateway.did diff --git a/default.nix b/default.nix index e35311795..21ae84914 100644 --- a/default.nix +++ b/default.nix @@ -34,8 +34,7 @@ rec { -R $PWD -D $out/$doc_path/ index.adoc find . -type f -name '*.png' | cpio -pdm $out/$doc_path/ cp *.cddl $out/$doc_path - cp ic.did $out/$doc_path - + cp *.did $out/$doc_path mkdir -p $out/nix-support echo "report spec $out/$doc_path index.html" >> $out/nix-support/hydra-build-products diff --git a/spec/http-gateway.did b/spec/http-gateway.did new file mode 100644 index 000000000..d1c32777f --- /dev/null +++ b/spec/http-gateway.did @@ -0,0 +1,34 @@ +type HeaderField = record { text; text; }; + +type HttpRequest = record { + method: text; + url: text; + headers: vec HeaderField; + body: blob; +}; + +type HttpResponse = record { + status_code: nat16; + headers: vec HeaderField; + body: blob; + upgrade : opt bool; + streaming_strategy: opt StreamingStrategy; +}; + +type StreamingCallbackHttpResponse = record { + body: blob; + token: opt T; +}; + +type StreamingStrategy = variant { + Callback: record { + callback: func (T) -> (opt StreamingCallbackHttpResponse) query; + token: T; + }; +}; + +service : { + http_request: (request: HttpRequest) -> (HttpResponse) query; + http_request_update: (request: HttpRequest) -> (HttpResponse); +} + diff --git a/spec/index.adoc b/spec/index.adoc index 13b3c68c2..f508a019d 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1949,6 +1949,122 @@ lookup_path(["d"], pruned_tree) = Found "morning" lookup_path(["e"], pruned_tree) = Absent .... +[#http-gateway] +== The HTTP Gateway protocol + +This section specifies the _HTTP Gateway protocol_, which allows canisters to handle conventional HTTP requests. + +This feature involves the help of a _HTTP Gateway_ that translates between HTTP requests and the IC protocol. Such a gateway could be a stand-alone proxy, it could be implemented in a web browsers (natively, via plugin or via a service worker) or in other ways. This document describes the interface and semantics of this protocol independent of a concrete Gateway, so that all Gateway implementations can be compatible. + +This protocol sits conceptually on top of the interface specified in the remainder of this document, as the involved parties (HTTP Gateway and canisters) are not part of the “system” described here. It is still included in this file due to its role in the Internet Computer ecosystem and due to the importance of keeping multiple Gateway implementations in sync. + +=== Overview + +A HTTP request by an HTTP client is handled by these steps: + +1. The Gateway resolves the Host of the request to a canister id. +2. The Gateway Candid-encoded the HTTP request data. +3. The Gateway invokes the canister via a query call to `http_request`. +4. The canister handles the request and returns a HTTP response, encoded in Candid, together with additional metadata. +5. If requested by the canister, the Gateway sends the request again via an update call to `http_request_update`. +6. If applicable, the Gateway fetches futher body data via streaming query calls. +7. If applicable, the Gateway validates the certificate of the response. +8. The Gateway sends the response to the HTTP client. + +[#http-gateway-interface] +=== Candid interface + +The following interface description, in https://github.com/dfinity/candid/blob/master/spec/Candid.md[Candid syntax], describes the expecte Canister interface. You can also link:{attachmentsdir}/http-gateway.did[download the file]. +---- +include::{example}http-gateway.did[] +---- + +This type signature is a type signature _scheme_, and the canister can choose any type for the token type `T` used for streaming. The Gateway handles that type generically. + +Only canisters that use the “Upgrade to update calls” feature need to provide the `http_request_update` method. + +NOTE: Canisters not using these upgrade these features can completeley leave out the `streaming_strategy` and/or `upgade` fields in the `HttpResponse` they return, due to how Candid subtyping works. This might simplifiy their code. + +[#http-gateway-name-resolution] +=== Canister resolution + +The Gateway needs to know the canister id of the canister to talk to, and obtains that information from the hostname as follows: + +1. Check that the hostname, taken from the `Host` field of the HTTP request, is of the form `.raw.ic0.app` or `.ic0.app`, or fail. + +2. If the `` is in the following table, use the given canister ids: ++ +.Canister hostname resolution +|=== +|Hostname |Canister id +|identity | rdmx6-jaaaa-aaaaa-aaadq-cai +|nns | qoctq-giaaa-aaaaa-aaaea-cai +|dscvr | h5aet-waaaa-aaaab-qaamq-cai +|personhood | g3wsl-eqaaa-aaaan-aaaaa-cai + +3. Else, if `` a valid textual encoding of a principal, use that principal as the canister ids. + +4. Else fail. + +If the hostname was of the form `.ic0.app`, it is a _safe_ hostname; if it was of the form `.raw.ic0.app` it is a _raw_ hostname. + +=== Request encoding + +The HTTP request is encoded into the `HttpRequest` candid structure. + +The `method` field contains the HTTP methd (e.g. `HTTP`), in upper case. + +The `url` field contains the URL from the HTTP request line, i.e. without protocol or hostname, and including query parameters. + +The `headers` field contains the headers of the HTTP request. + +The `body` field contains the body of the HTTP request (without any content encodings processed by the Gateway). + +=== Upgrade to update calls + +If the canister sets `update = opt true` in the `HttpResponse` reply from `http_request`, then the Gateway ignores all other fields of the reply. The Gateway performs an _update_ call, with the same argument, to `http_request_update`, and uses that response instead. + +The value of the `update` field returned from `http_request_update` is ignored. + +=== Response decoding and streaming + +The Gateway assembles the HTTP response from the given `HttpResponse` record. + +The HTTP response status code is taken from the `status_code` field. + +The HTTP response headers are taken from the `headers` field. + +The HTTP response body is initialized with the value of the `body` field. + +If the `streaming_strategy` field is set, the HTTP Gateway then uses further query calls to obtain further chunks to append to the body: + +1. If the function reference in the `callback` field of the `streaming_strategy` is not a method of the given canister, the Gateway fails the request. + +2. Else, it makes a query call to the give method, passing the `token` value given in the `streaming_strategy` as the argument. + +3. That query method returns a `StreamingCallbackHttpResponse`. The `body` therein is appended to the body of the HTTP response. This is repeated as long as the method returns some token in the `token` field, until that field is `null`. + +=== Response certification + +If the hostname was safe, the HTTP Gateway performs _certificate validation_: + +1. It searches for a response header called `Ic-Certificate` (case-insensitive). + +2. The value of the header must be a structured header according to RFC8941 with fields `certificate` and `tree`, both being byte sequences. + +3. The `certificate` must be a valid certificate as per <>, signed by the root key. If the certificate contains a subnet delegation, the delegation must be valid for the given canister. The timestamp in `/time` must be recent. The subnet state tree in the certificate must reveal the canister’s <>. + +4. The `tree` must be a hash trees as per <>. + +5. The root hash of that `tree` must match the canister’s certified data. + +6. The path `["http_assets",]`, where `url` is the utf8-encoded `url` from the `HttpRequest` must exist and be a leaf. Else, if it does not exist, `["http_assets","/index.html"]` must exist and be a leaf. + +7. That leaf must contain the SHA256 of the _decoded_ body. ++ +The decoded body is the body of the HTTP response (in particular, after assembling streaming chunks), decoded according to the `Content-Encoding` header, if present. Supported encodings for `Content-Encoding` are `gzip` and `deflate.` + + [#abstract-behavior] == Abstract behavior From 6a01a1730f3d6aae4e3190998638f854578d0f00 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Sat, 2 Apr 2022 11:45:14 +0200 Subject: [PATCH 02/12] Review comments --- spec/index.adoc | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index f508a019d..1df7cbe65 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1956,7 +1956,7 @@ This section specifies the _HTTP Gateway protocol_, which allows canisters to ha This feature involves the help of a _HTTP Gateway_ that translates between HTTP requests and the IC protocol. Such a gateway could be a stand-alone proxy, it could be implemented in a web browsers (natively, via plugin or via a service worker) or in other ways. This document describes the interface and semantics of this protocol independent of a concrete Gateway, so that all Gateway implementations can be compatible. -This protocol sits conceptually on top of the interface specified in the remainder of this document, as the involved parties (HTTP Gateway and canisters) are not part of the “system” described here. It is still included in this file due to its role in the Internet Computer ecosystem and due to the importance of keeping multiple Gateway implementations in sync. +Conceptually, this protocol builds on top of the interface specifie d in the remainder of this document, and therefore is an “application-level” interface, not a feature of the core Internet Computer system described in the other sections, and could be a separate document. We nevertheless include this protocol in the Internet Computer Interface Specification because of its important role in the ecosystem and due to the importance of keeping multiple Gateway implementations in sync. === Overview @@ -1967,7 +1967,7 @@ A HTTP request by an HTTP client is handled by these steps: 3. The Gateway invokes the canister via a query call to `http_request`. 4. The canister handles the request and returns a HTTP response, encoded in Candid, together with additional metadata. 5. If requested by the canister, the Gateway sends the request again via an update call to `http_request_update`. -6. If applicable, the Gateway fetches futher body data via streaming query calls. +6. If applicable, the Gateway fetches further body data via streaming query calls. 7. If applicable, the Gateway validates the certificate of the response. 8. The Gateway sends the response to the HTTP client. @@ -1983,7 +1983,7 @@ This type signature is a type signature _scheme_, and the canister can choose an Only canisters that use the “Upgrade to update calls” feature need to provide the `http_request_update` method. -NOTE: Canisters not using these upgrade these features can completeley leave out the `streaming_strategy` and/or `upgade` fields in the `HttpResponse` they return, due to how Candid subtyping works. This might simplifiy their code. +NOTE: Canisters not using these features can completely leave out the `streaming_strategy` and/or `upgade` fields in the `HttpResponse` they return, due to how Candid subtyping works. This might simplify their code. [#http-gateway-name-resolution] === Canister resolution @@ -2022,21 +2022,27 @@ The `body` field contains the body of the HTTP request (without any content enco === Upgrade to update calls -If the canister sets `update = opt true` in the `HttpResponse` reply from `http_request`, then the Gateway ignores all other fields of the reply. The Gateway performs an _update_ call, with the same argument, to `http_request_update`, and uses that response instead. +If the canister sets `update = opt true` in the `HttpResponse` reply from `http_request`, then the Gateway ignores all other fields of the reply. The Gateway performs an _update_ call to `http_request_update`, passing the same `HttpRequest` record as the argument, and uses that response instead. The value of the `update` field returned from `http_request_update` is ignored. +[#http-gateway-streaming] === Response decoding and streaming -The Gateway assembles the HTTP response from the given `HttpResponse` record. +The Gateway assembles the HTTP response from the given `HttpResponse` record: -The HTTP response status code is taken from the `status_code` field. +* The HTTP response status code is taken from the `status_code` field. -The HTTP response headers are taken from the `headers` field. +* The HTTP response headers are taken from the `headers` field. -The HTTP response body is initialized with the value of the `body` field. +* The HTTP response body is initialized with the value of the `body` field, and further assembled as per the <>. -If the `streaming_strategy` field is set, the HTTP Gateway then uses further query calls to obtain further chunks to append to the body: +[#http-gateway-streaming] +=== Response decoding and streaming + +The HTTP Gateway protocol has provisions transfer further chunks of the body data from the canister to the HTTP Gateway, to overcome the message limit of the Internet Computer. This streaming protocol is idependent of any possible streaming of data between the HTTP Gateway and the HTTP client. The gateway may assemble the response in whole before passing it on, or pass the chunks on directly, on the TCP or HTTP level, as it sees fit. When the Gateway is <>, it must not pass on uncertified chunks. + +If the `streaming_strategy` field of the `HttpResponse` is set, the HTTP Gateway then uses further query calls to obtain further chunks to append to the body: 1. If the function reference in the `callback` field of the `streaming_strategy` is not a method of the given canister, the Gateway fails the request. @@ -2044,6 +2050,7 @@ If the `streaming_strategy` field is set, the HTTP Gateway then uses further que 3. That query method returns a `StreamingCallbackHttpResponse`. The `body` therein is appended to the body of the HTTP response. This is repeated as long as the method returns some token in the `token` field, until that field is `null`. +[#http-gateway-certification] === Response certification If the hostname was safe, the HTTP Gateway performs _certificate validation_: @@ -2064,6 +2071,7 @@ If the hostname was safe, the HTTP Gateway performs _certificate validation_: + The decoded body is the body of the HTTP response (in particular, after assembling streaming chunks), decoded according to the `Content-Encoding` header, if present. Supported encodings for `Content-Encoding` are `gzip` and `deflate.` +WARNING: The certification protocol only covers the mapping from request URL to response body. It completely ignores the request method and headers, and does not cover the response headers and status code. [#abstract-behavior] From fbaa36caa6f54d36ec3777baa154c08b62e816ee Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Sat, 2 Apr 2022 11:47:16 +0200 Subject: [PATCH 03/12] Align table --- spec/index.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index 1df7cbe65..db36b624a 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1996,11 +1996,11 @@ The Gateway needs to know the canister id of the canister to talk to, and obtain + .Canister hostname resolution |=== -|Hostname |Canister id -|identity | rdmx6-jaaaa-aaaaa-aaadq-cai -|nns | qoctq-giaaa-aaaaa-aaaea-cai -|dscvr | h5aet-waaaa-aaaab-qaamq-cai -|personhood | g3wsl-eqaaa-aaaan-aaaaa-cai +| Hostname | Canister id +| identity | rdmx6-jaaaa-aaaaa-aaadq-cai +| nns | qoctq-giaaa-aaaaa-aaaea-cai +| dscvr | h5aet-waaaa-aaaab-qaamq-cai +| personhood | g3wsl-eqaaa-aaaan-aaaaa-cai 3. Else, if `` a valid textual encoding of a principal, use that principal as the canister ids. From 0c433a36b69b0e36ad1f35a1558ffc9e4097aa21 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Sat, 2 Apr 2022 11:48:24 +0200 Subject: [PATCH 04/12] Section titles --- spec/index.adoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index db36b624a..33233e33c 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -2026,8 +2026,7 @@ If the canister sets `update = opt true` in the `HttpResponse` reply from `http_ The value of the `update` field returned from `http_request_update` is ignored. -[#http-gateway-streaming] -=== Response decoding and streaming +=== Response decoding The Gateway assembles the HTTP response from the given `HttpResponse` record: @@ -2038,7 +2037,7 @@ The Gateway assembles the HTTP response from the given `HttpResponse` record: * The HTTP response body is initialized with the value of the `body` field, and further assembled as per the <>. [#http-gateway-streaming] -=== Response decoding and streaming +=== Response body streaming The HTTP Gateway protocol has provisions transfer further chunks of the body data from the canister to the HTTP Gateway, to overcome the message limit of the Internet Computer. This streaming protocol is idependent of any possible streaming of data between the HTTP Gateway and the HTTP client. The gateway may assemble the response in whole before passing it on, or pass the chunks on directly, on the TCP or HTTP level, as it sees fit. When the Gateway is <>, it must not pass on uncertified chunks. From f7e612047a644927566afa16a1558201543ddb96 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 5 Apr 2022 11:06:53 +0200 Subject: [PATCH 05/12] Improve description of the `StreamingToken` type, and add stern warning --- spec/http-gateway.did | 13 ++++++++++--- spec/index.adoc | 5 +++-- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/spec/http-gateway.did b/spec/http-gateway.did index d1c32777f..4ec844f65 100644 --- a/spec/http-gateway.did +++ b/spec/http-gateway.did @@ -15,15 +15,22 @@ type HttpResponse = record { streaming_strategy: opt StreamingStrategy; }; +// Each canister that uses the streaming feature gets to choose their concrete +// type; the HTTP Gateway will treat it as an opaque value that is only fed to +// the callback method + +type StreamingToken = /* application-specific type */ + + type StreamingCallbackHttpResponse = record { body: blob; - token: opt T; + token: opt StreamingToken; }; type StreamingStrategy = variant { Callback: record { - callback: func (T) -> (opt StreamingCallbackHttpResponse) query; - token: T; + callback: func (StreamingToken) -> (opt StreamingCallbackHttpResponse) query; + token: StreamingToken; }; }; diff --git a/spec/index.adoc b/spec/index.adoc index 33233e33c..860b25bdf 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1979,8 +1979,6 @@ The following interface description, in https://github.com/dfinity/candid/blob/m include::{example}http-gateway.did[] ---- -This type signature is a type signature _scheme_, and the canister can choose any type for the token type `T` used for streaming. The Gateway handles that type generically. - Only canisters that use the “Upgrade to update calls” feature need to provide the `http_request_update` method. NOTE: Canisters not using these features can completely leave out the `streaming_strategy` and/or `upgade` fields in the `HttpResponse` they return, due to how Candid subtyping works. This might simplify their code. @@ -2049,6 +2047,9 @@ If the `streaming_strategy` field of the `HttpResponse` is set, the HTTP Gateway 3. That query method returns a `StreamingCallbackHttpResponse`. The `body` therein is appended to the body of the HTTP response. This is repeated as long as the method returns some token in the `token` field, until that field is `null`. +WARNING: The type of the `token` value is chosen by the canister; the HTTP Gateway obtains the Candid type from the encoded message from the canister, and use it when passing the token back to the canister. This generic use of Candid is no covered by the Candid specification, and may not be possible in some cases (e.g. when using “future types”). Canister authors may have to use “simple” types. + + [#http-gateway-certification] === Response certification From f7e9bc8efd696303391e588e4962cb6ed3ed71f6 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 5 Apr 2022 11:09:33 +0200 Subject: [PATCH 06/12] Add note about headers --- spec/index.adoc | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index 860b25bdf..3026e5542 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -2010,13 +2010,15 @@ If the hostname was of the form `.ic0.app`, it is a _safe_ hostname; if it The HTTP request is encoded into the `HttpRequest` candid structure. -The `method` field contains the HTTP methd (e.g. `HTTP`), in upper case. +* The `method` field contains the HTTP methd (e.g. `HTTP`), in upper case. -The `url` field contains the URL from the HTTP request line, i.e. without protocol or hostname, and including query parameters. +* The `url` field contains the URL from the HTTP request line, i.e. without protocol or hostname, and including query parameters. -The `headers` field contains the headers of the HTTP request. +* The `headers` field contains the headers of the HTTP request. ++ +NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-cookie` header. -The `body` field contains the body of the HTTP request (without any content encodings processed by the Gateway). +* The `body` field contains the body of the HTTP request (without any content encodings processed by the Gateway). === Upgrade to update calls From 84f308f95f6f4931168a51d329a34c51796c1382 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 5 Apr 2022 11:10:45 +0200 Subject: [PATCH 07/12] Move to the right section --- spec/index.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index 3026e5542..214ad6272 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -2015,8 +2015,6 @@ The HTTP request is encoded into the `HttpRequest` candid structure. * The `url` field contains the URL from the HTTP request line, i.e. without protocol or hostname, and including query parameters. * The `headers` field contains the headers of the HTTP request. -+ -NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-cookie` header. * The `body` field contains the body of the HTTP request (without any content encodings processed by the Gateway). @@ -2033,6 +2031,8 @@ The Gateway assembles the HTTP response from the given `HttpResponse` record: * The HTTP response status code is taken from the `status_code` field. * The HTTP response headers are taken from the `headers` field. ++ +NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-cookie` header. * The HTTP response body is initialized with the value of the `body` field, and further assembled as per the <>. From 153179d17571de9146332a1dab9755f51ae3e498 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 5 Apr 2022 11:12:21 +0200 Subject: [PATCH 08/12] Document additional headers --- spec/index.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/spec/index.adoc b/spec/index.adoc index 214ad6272..1abf472f2 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -2033,6 +2033,16 @@ The Gateway assembles the HTTP response from the given `HttpResponse` record: * The HTTP response headers are taken from the `headers` field. + NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-cookie` header. ++ +NOTE: HTTP Gateway may add addditional headers. In particular, the following headers may be set: ++ +.... +access-control-allow-origin: * +access-control-allow-methods: GET, POST, HEAD, OPTIONS +access-control-allow-headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Cookie +access-control-expose-headers: Content-Length,Content-Range +x-cache-status: MISS +.... * The HTTP response body is initialized with the value of the `body` field, and further assembled as per the <>. From e89a3329af1e5dde11f5d0913ad030e3553f6eb0 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 5 Apr 2022 11:20:41 +0200 Subject: [PATCH 09/12] Fix adoc --- spec/index.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/spec/index.adoc b/spec/index.adoc index 1abf472f2..447da20be 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1999,6 +1999,7 @@ The Gateway needs to know the canister id of the canister to talk to, and obtain | nns | qoctq-giaaa-aaaaa-aaaea-cai | dscvr | h5aet-waaaa-aaaab-qaamq-cai | personhood | g3wsl-eqaaa-aaaan-aaaaa-cai +|=== 3. Else, if `` a valid textual encoding of a principal, use that principal as the canister ids. From 14552682ae7311db8680845f965b1b16b172aefb Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 5 Apr 2022 11:28:02 +0200 Subject: [PATCH 10/12] More adoc fixes --- spec/index.adoc | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index 447da20be..9ba23a070 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1993,13 +1993,13 @@ The Gateway needs to know the canister id of the canister to talk to, and obtain 2. If the `` is in the following table, use the given canister ids: + .Canister hostname resolution -|=== -| Hostname | Canister id -| identity | rdmx6-jaaaa-aaaaa-aaadq-cai -| nns | qoctq-giaaa-aaaaa-aaaea-cai -| dscvr | h5aet-waaaa-aaaab-qaamq-cai -| personhood | g3wsl-eqaaa-aaaan-aaaaa-cai -|=== +|============================================ +| Hostname | Canister id +| `identity` | `rdmx6-jaaaa-aaaaa-aaadq-cai` +| `nns` | `qoctq-giaaa-aaaaa-aaaea-cai` +| `dscvr` | `h5aet-waaaa-aaaab-qaamq-cai` +| `personhood` | `g3wsl-eqaaa-aaaan-aaaaa-cai` +|============================================ 3. Else, if `` a valid textual encoding of a principal, use that principal as the canister ids. @@ -2035,8 +2035,10 @@ The Gateway assembles the HTTP response from the given `HttpResponse` record: + NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-cookie` header. + -NOTE: HTTP Gateway may add addditional headers. In particular, the following headers may be set: -+ +[NOTE] +==== +HTTP Gateway may add addditional headers. In particular, the following headers may be set: + .... access-control-allow-origin: * access-control-allow-methods: GET, POST, HEAD, OPTIONS @@ -2044,6 +2046,7 @@ access-control-allow-headers: DNT,User-Agent,X-Requested-With,If-Modified-Since, access-control-expose-headers: Content-Length,Content-Range x-cache-status: MISS .... +==== * The HTTP response body is initialized with the value of the `body` field, and further assembled as per the <>. From 8ce8f7eef7fc818c74efeaf750c0a445db046f9d Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Thu, 7 Apr 2022 07:39:42 +0200 Subject: [PATCH 11/12] Apply suggestions from Paul Co-authored-by: Paul Young <84700+paulyoung@users.noreply.github.com> --- spec/index.adoc | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/spec/index.adoc b/spec/index.adoc index 9ba23a070..f59e9f001 100644 --- a/spec/index.adoc +++ b/spec/index.adoc @@ -1956,14 +1956,14 @@ This section specifies the _HTTP Gateway protocol_, which allows canisters to ha This feature involves the help of a _HTTP Gateway_ that translates between HTTP requests and the IC protocol. Such a gateway could be a stand-alone proxy, it could be implemented in a web browsers (natively, via plugin or via a service worker) or in other ways. This document describes the interface and semantics of this protocol independent of a concrete Gateway, so that all Gateway implementations can be compatible. -Conceptually, this protocol builds on top of the interface specifie d in the remainder of this document, and therefore is an “application-level” interface, not a feature of the core Internet Computer system described in the other sections, and could be a separate document. We nevertheless include this protocol in the Internet Computer Interface Specification because of its important role in the ecosystem and due to the importance of keeping multiple Gateway implementations in sync. +Conceptually, this protocol builds on top of the interface specified in the remainder of this document, and therefore is an “application-level” interface, not a feature of the core Internet Computer system described in the other sections, and could be a separate document. We nevertheless include this protocol in the Internet Computer Interface Specification because of its important role in the ecosystem and due to the importance of keeping multiple Gateway implementations in sync. === Overview A HTTP request by an HTTP client is handled by these steps: 1. The Gateway resolves the Host of the request to a canister id. -2. The Gateway Candid-encoded the HTTP request data. +2. The Gateway Candid-encodes the HTTP request data. 3. The Gateway invokes the canister via a query call to `http_request`. 4. The canister handles the request and returns a HTTP response, encoded in Candid, together with additional metadata. 5. If requested by the canister, the Gateway sends the request again via an update call to `http_request_update`. @@ -1974,14 +1974,14 @@ A HTTP request by an HTTP client is handled by these steps: [#http-gateway-interface] === Candid interface -The following interface description, in https://github.com/dfinity/candid/blob/master/spec/Candid.md[Candid syntax], describes the expecte Canister interface. You can also link:{attachmentsdir}/http-gateway.did[download the file]. +The following interface description, in https://github.com/dfinity/candid/blob/master/spec/Candid.md[Candid syntax], describes the expected Canister interface. You can also link:{attachmentsdir}/http-gateway.did[download the file]. ---- include::{example}http-gateway.did[] ---- Only canisters that use the “Upgrade to update calls” feature need to provide the `http_request_update` method. -NOTE: Canisters not using these features can completely leave out the `streaming_strategy` and/or `upgade` fields in the `HttpResponse` they return, due to how Candid subtyping works. This might simplify their code. +NOTE: Canisters not using these features can completely leave out the `streaming_strategy` and/or `upgrade` fields in the `HttpResponse` they return, due to how Candid subtyping works. This might simplify their code. [#http-gateway-name-resolution] === Canister resolution @@ -2001,7 +2001,7 @@ The Gateway needs to know the canister id of the canister to talk to, and obtain | `personhood` | `g3wsl-eqaaa-aaaan-aaaaa-cai` |============================================ -3. Else, if `` a valid textual encoding of a principal, use that principal as the canister ids. +3. Else, if `` is a valid textual encoding of a principal, use that principal as the canister id. 4. Else fail. @@ -2009,9 +2009,9 @@ If the hostname was of the form `.ic0.app`, it is a _safe_ hostname; if it === Request encoding -The HTTP request is encoded into the `HttpRequest` candid structure. +The HTTP request is encoded into the `HttpRequest` Candid structure. -* The `method` field contains the HTTP methd (e.g. `HTTP`), in upper case. +* The `method` field contains the HTTP method (e.g. `HTTP`), in upper case. * The `url` field contains the URL from the HTTP request line, i.e. without protocol or hostname, and including query parameters. @@ -2033,11 +2033,11 @@ The Gateway assembles the HTTP response from the given `HttpResponse` record: * The HTTP response headers are taken from the `headers` field. + -NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-cookie` header. +NOTE: Not all Gateway implementations may be able to pass on all forms of headers. In particular, Service Workers are unable to pass on the `Set-Cookie` header. + [NOTE] ==== -HTTP Gateway may add addditional headers. In particular, the following headers may be set: +HTTP Gateways may add additional headers. In particular, the following headers may be set: .... access-control-allow-origin: * @@ -2053,17 +2053,17 @@ x-cache-status: MISS [#http-gateway-streaming] === Response body streaming -The HTTP Gateway protocol has provisions transfer further chunks of the body data from the canister to the HTTP Gateway, to overcome the message limit of the Internet Computer. This streaming protocol is idependent of any possible streaming of data between the HTTP Gateway and the HTTP client. The gateway may assemble the response in whole before passing it on, or pass the chunks on directly, on the TCP or HTTP level, as it sees fit. When the Gateway is <>, it must not pass on uncertified chunks. +The HTTP Gateway protocol has provisions to transfer further chunks of the body data from the canister to the HTTP Gateway, to overcome the message limit of the Internet Computer. This streaming protocol is independent of any possible streaming of data between the HTTP Gateway and the HTTP client. The gateway may assemble the response in whole before passing it on, or pass the chunks on directly, on the TCP or HTTP level, as it sees fit. When the Gateway is <>, it must not pass on uncertified chunks. If the `streaming_strategy` field of the `HttpResponse` is set, the HTTP Gateway then uses further query calls to obtain further chunks to append to the body: 1. If the function reference in the `callback` field of the `streaming_strategy` is not a method of the given canister, the Gateway fails the request. -2. Else, it makes a query call to the give method, passing the `token` value given in the `streaming_strategy` as the argument. +2. Else, it makes a query call to the given method, passing the `token` value given in the `streaming_strategy` as the argument. 3. That query method returns a `StreamingCallbackHttpResponse`. The `body` therein is appended to the body of the HTTP response. This is repeated as long as the method returns some token in the `token` field, until that field is `null`. -WARNING: The type of the `token` value is chosen by the canister; the HTTP Gateway obtains the Candid type from the encoded message from the canister, and use it when passing the token back to the canister. This generic use of Candid is no covered by the Candid specification, and may not be possible in some cases (e.g. when using “future types”). Canister authors may have to use “simple” types. +WARNING: The type of the `token` value is chosen by the canister; the HTTP Gateway obtains the Candid type of the encoded message from the canister, and uses it when passing the token back to the canister. This generic use of Candid is not covered by the Candid specification, and may not be possible in some cases (e.g. when using “future types”). Canister authors may have to use “simple” types. [#http-gateway-certification] @@ -2073,17 +2073,17 @@ If the hostname was safe, the HTTP Gateway performs _certificate validation_: 1. It searches for a response header called `Ic-Certificate` (case-insensitive). -2. The value of the header must be a structured header according to RFC8941 with fields `certificate` and `tree`, both being byte sequences. +2. The value of the header must be a structured header according to RFC 8941 with fields `certificate` and `tree`, both being byte sequences. -3. The `certificate` must be a valid certificate as per <>, signed by the root key. If the certificate contains a subnet delegation, the delegation must be valid for the given canister. The timestamp in `/time` must be recent. The subnet state tree in the certificate must reveal the canister’s <>. +3. The `certificate` must be a valid certificate as per <>, signed by the root key. If the certificate contains a subnet delegation, the delegation must be valid for the given canister. The timestamp in `/time` must be recent. The subnet state tree in the certificate must reveal the canister’s <>. -4. The `tree` must be a hash trees as per <>. +4. The `tree` must be a hash tree as per <>. 5. The root hash of that `tree` must match the canister’s certified data. 6. The path `["http_assets",]`, where `url` is the utf8-encoded `url` from the `HttpRequest` must exist and be a leaf. Else, if it does not exist, `["http_assets","/index.html"]` must exist and be a leaf. -7. That leaf must contain the SHA256 of the _decoded_ body. +7. That leaf must contain the SHA-256 hash of the _decoded_ body. + The decoded body is the body of the HTTP response (in particular, after assembling streaming chunks), decoded according to the `Content-Encoding` header, if present. Supported encodings for `Content-Encoding` are `gzip` and `deflate.` From 861074af81469c9b6eb9fe6511721bc4e2689d04 Mon Sep 17 00:00:00 2001 From: Joachim Breitner Date: Tue, 3 May 2022 10:19:45 +0200 Subject: [PATCH 12/12] Add changelog entry --- spec/changelog.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/spec/changelog.adoc b/spec/changelog.adoc index 00ae26198..5a756ce7b 100644 --- a/spec/changelog.adoc +++ b/spec/changelog.adoc @@ -7,6 +7,7 @@ * Spec: Canister access to performance metrics * Spec: Expose Wasm custom sections in the state tree * Spec: User delegations include a principal scope +* Spec: Include the HTTP Gateway Protocol in this spec [#0_18_3] === 0.18.3 (2022-01-10)