From afeedf862c4fcd742e8e53e69e7f2c74d0d8deeb Mon Sep 17 00:00:00 2001 From: "Henry H. Andrews" Date: Wed, 7 May 2025 13:09:48 -0700 Subject: [PATCH 1/7] Support ordered multipart including streaming This adds support for all `multipart` media types that do not have named parts, including support for streaming such media types. Note that `multipart/mixed` defines the basic processing rules for all `multipart` types, and implementations that encounter unrecognized `multipart` subtypes are required to process them as `multipart/mixed`. Therefore support for `multipart/mixed` addresses all other subtypes to some degree. This builds on the recent support for sequential media types: * `multipart/mixed` and similar meet the definition for a sequential media type, requiring it to be modeled as an array. This does use an expansive definition of "repeating the same structure", where the structure is literally any content with a media type. * As a sequential media type, it also supports `itemSchema` * Adding a parallel `itemEncoding` is the obvious solution to `multipart/mixed` streams requiring an Encoding Object * We have regularly received requests to support truly mixed `multipart/mixed` payloads, and previously claimed such support from 3.0.0 onwards, without actually supporting it. Adding `prefixEncoding` along with `itemEncoding` supports this use case with a clear parallel to `prefixItems`, which is the schema construct needed to support this case. * There is no need for a `prefixSchema` field because the streaming use case requires a repetition of the same schema for each item. Therefore all mixed use cases can use `schema` and `prefixItems` --- src/oas.md | 143 ++++++++++++++++++++++++++--- src/schemas/validation/schema.yaml | 13 ++- 2 files changed, 141 insertions(+), 15 deletions(-) diff --git a/src/oas.md b/src/oas.md index 547c82f42c..68af364bd1 100644 --- a/src/oas.md +++ b/src/oas.md @@ -101,14 +101,17 @@ Some examples of sequential media types (including some that are not IANA-regist application/json-seq application/geo+json-seq text/event-stream + multipart/mixed ``` In the first three above, the repeating structure is any [JSON value](https://tools.ietf.org/html/rfc8259#section-3). -The fourth repeats `application/geo+json`-structured values, while the last repeats a custom text format related to Server-Sent Events. +The fourth repeats `application/geo+json`-structured values, while `text/event-stream` repeats a custom text format related to Server-Sent Events. +The final media type listed above, `multipart/mixed`, provides an ordered list of documents of any media type, and is sometimes streamed. Implementations MUST support mapping sequential media types into the JSON Schema data model by treating them as if the values were in an array in the same order. See [Complete vs Streaming Content](#complete-vs-streaming-content) for more information on handling sequential media type in a streaming context, including special considerations for `text/event-stream` content. +For `multipart` types, see also [Encoding By Position](#encoding-by-position). #### Media Type Registry @@ -1253,7 +1256,9 @@ See [Working With Examples](#working-with-examples) for further guidance regardi | itemSchema | [Schema Object](#schema-object) | A schema describing each item within a [sequential media type](#sequential-media-types). | | example | Any | Example of the media type; see [Working With Examples](#working-with-examples). | | examples | Map[ `string`, [Example Object](#example-object) \| [Reference Object](#reference-object)] | Examples of the media type; see [Working With Examples](#working-with-examples). | -| encoding | Map[`string`, [Encoding Object](#encoding-object)] | A map between a property name and its encoding information, as defined under [Encoding Usage and Restrictions](#encoding-usage-and-restrictions). The `encoding` field SHALL only apply when the media type is `multipart` or `application/x-www-form-urlencoded`. If no Encoding Object is provided for a property, the behavior is determined by the default values documented for the Encoding Object. | +| encoding | Map[`string`, [Encoding Object](#encoding-object)] | A map between a property name and its encoding information, as defined under [Encoding By Name](#encoding-by-name). The `encoding` field SHALL only apply when the media type is `multipart` or `application/x-www-form-urlencoded`. If no Encoding Object is provided for a property, the behavior is determined by the default values documented for the Encoding Object. This field MUST NOT be present if `prefixEncoding` or `itemEncoding` are present. | +| prefixEncoding | [[Encoding Object](#encoding-object)] | An array of positional encoding information, as defined under [Encoding By Position](#encoding-by-position). The `prefixEncoding` field SHALL only apply when the media type is `multipart`. If no Encoding Object is provided for a property, the behavior is determined by the default values documented for the Encoding Object. This field MUST NOT be present if `encoding` is present. | +| itemEncoding | [Encoding Object](#encoding-object) | A single Encoding Object that provides encoding information for multiple array items, as defined under [Encoding By Position](#encoding-by-position). The `itemEncoding` field SHALL only apply when the media type is `multipart`. If no Encoding Object is provided for a property, the behavior is determined by the default values documented for the Encoding Object. This field MUST NOT be present if `encoding` is present. | This object MAY be extended with [Specification Extensions](#specification-extensions). @@ -1273,7 +1278,8 @@ For this use case, `maxLength` MAY be implemented outside of regular JSON Schema ###### Streaming Sequential Media Types -The `itemSchema` field is provided to support streaming use cases for sequential media types. +The `itemSchema` field is provided to support streaming use cases for sequential media types, with `itemEncoding` as a corresponding encoding mechanism for streaming [positional `multipart` media types](#encoding-by-position). + Unlike `schema`, which is applied to the complete content (treated as an array as described in the [sequential media types](#sequential-media-types) section), `itemSchema` MUST be applied to each item in the stream independently, which supports processing each item as it is read from the stream. Both `schema` and `itemSchema` MAY be used in the same Media Type Object. @@ -1309,13 +1315,16 @@ properties: ##### Encoding Usage and Restrictions -The `encoding` field defines how to map each [Encoding Object](#encoding-object) to a specific value in the data. +The three encoding fields define how to map each [Encoding Object](#encoding object) to a specific value in the data. +Each field has its own set of media types with which it can be used; for all other media types all three fields SHALL be ignored. -To use the `encoding` field, a `schema` MUST exist, and the `encoding` field's keys MUST exist in the schema as properties. -Array properties MUST be handled by applying the given Encoding Object to one part per array item, each with the same `name`, as is recommended by [[?RFC7578]] [Section 4.3](https://www.rfc-editor.org/rfc/rfc7578.html#section-4.3) for supplying multiple values per form field. -For all other value types for both top-level non-array properties and for values, including array values, within a top-level array, the Encoding Object MUST be applied to the entire value. +###### Encoding By Name The behavior of the `encoding` field is designed to support web forms, and is therefore only defined for media types structured as name-value pairs that allow repeat values, most notably `application/x-www-form-urlencoded` and `multipart/form-data`. + +To use the `encoding` field, each key under the field MUST exist in the `schema` as a property. +Array properties MUST be handled by applying the given Encoding Object to produce one encoded value per array item, each with the same `name`, as is recommended by [[?RFC7578]] [Section 4.3](https://www.rfc-editor.org/rfc/rfc7578.html#section-4.3) for supplying multiple values per form field. +For all other value types for both top-level non-array properties and for values, including array values, within a top-level array, the Encoding Object MUST be applied to the entire value. The order of these name-value pairs in the target media type is implementation-defined. For `application/x-www-form-urlencoded`, the encoding keys MUST map to parameter names, with the values produced according to the rules of the [Encoding Object](#encoding-object). @@ -1324,15 +1333,29 @@ See [Encoding the `x-www-form-urlencoded` Media Type](#encoding-the-x-www-form-u For `multipart`, the encoding keys MUST map to the [`name` parameter](https://www.rfc-editor.org/rfc/rfc7578#section-4.2) of the `Content-Disposition: form-data` header of each part, as is defined for `multipart/form-data` in [[?RFC7578]]. See [[?RFC7578]] [Section 5](https://www.rfc-editor.org/rfc/rfc7578.html#section-5) for guidance regarding non-ASCII part names. -Other `multipart` media types are not directly supported as they do not define a mechanism for part names. -However, the usage of a `name` [`Content-Disposition` parameter](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-2) is defined for the `form-data` [`Content-Disposition` value](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-1), which is not restricted to `multipart/form-data`. -Implementations MAY choose to support the a `Conent-Disposition` of `form-data` with a `name` parameter in other `multipart` media types in order to use the `encoding` field with them, but this usage is unlikely to be supported by generic `multipart` implementations. - See [Encoding `multipart` Media Types](#encoding-multipart-media-types) for further guidance and examples, both with and without the `encoding` field. +###### Encoding By Position + +Most `multipart` media types, including `multipart/mixed` which defines the underlying rules for parsing all `multipart` types, do not have named parts. +Data for these media types are modeled as an array, with one item per part, in order. + +To use the `prefixEncoding` and/or `itemEncoding` fields, either an array `schema` or `itemSchema` MUST be present. +These fields are analogous to the `prefixItems` and `items` JSON Schema keywords, with `prefixEncoding` (if present) providing an array of Encoding Objects that are each applied to the value at the same position in the data array, and `itemEncoding` applying its single Encoding Object to all remaining items in the array. + +The `itemEncoding` field can also be used with `itemSchema` to support streaming `multipart` content. + +###### Additional Encoding Approaches + +The `prefixEncoding` field can be used with any `multipart` content to require a fixed part order. +This includes `multipart/form-data`, for which the Encoding Object's `headers` field MUST be used to provide the `Content-Disposition` and part name, as no property names exist to provide the names automatically. + +Prior versions of this specifications advised using the `name` [`Content-Disposition` parameter](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-2) of the `form-data` [`Content-Disposition` value](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-1) with `multipart` media types other than `multipart/form-data` in order to work around the limitations of the `encoding` field. +Implementations MAY choose to support this workaround, but as this usage is not common, implementations of non-`form-data` `multipart` media types are unlikely to support it. + ##### Media Type Examples -For form-related media type examples, see the [Encoding Object](#encoding-object). +For form-related and `multipart` media type examples, see the [Encoding Object](#encoding-object). ###### JSON @@ -1645,8 +1668,9 @@ These fields MAY be used either with or without the RFC6570-style serialization This object MAY be extended with [Specification Extensions](#specification-extensions). The default values for `contentType` are as follows, where an _n/a_ in the `contentEncoding` column means that the presence or value of `contentEncoding` is irrelevant. -This table is based on the value to which the Encoding Object is being applied, which as defined under [Encoding Usage and Restrictions](#encoding-usage-and-restrictions) is the array item for properties of type `"array"`, and the entire value for all other types. -Therefore the `array` row in this table applies only to array values inside of a top-level array. +This table is based on the value to which the Encoding Object is being applied as defined under [Encoding Usage and Restrictions](#encoding-usage-and-restrictions). +Note that in the case of [Encoding By Name](#encoding-by-name), this value is the array item for properties of type `"array"`, and the entire value for all other types. +Therefore the `array` row in this table applies only to array values inside of a top-level array when encoding by name. | `type` | `contentEncoding` | Default `contentType` | | ---- | ---- | ---- | @@ -1869,6 +1893,97 @@ requestBody: As seen in the [Encoding Object's `contentType` field documentation](#encoding-content-type), the empty schema for `items` indicates a media type of `application/octet-stream`. +###### Example: Ordered, Unnamed Multipart + +A `multipart/mixed` payload consisting of a JSON metadata document followed by an image which the metadata describes: + +```yaml +multipart/mixed: + schema: + prefixItems: + - # default content type for objects + # is `application/json`type: object + properties: + author: + type: string + created: + type: string + format: datetime + copyright: + type: string + license: + type: string + - # default content type for a schema without `type` + # is `application/octet-stream`, which we need + # to override. + {} + prefixEncoding: + - # Encoding Object defaults are correct for JSON + {} + - contentType: image/* +``` + +###### Example: Ordered Multipart With Required Header + +As described in [[?RFC2557]], a set of HTML pages can be sent in a `multipart/related` payload, preserving links among themselves by defining a `Content-Location` header for each page. + +See [Appendix D](appendix-d-serializing-headers-and-cookies) for an explanation of why `content: {text/plain: {...}}` is used to describe the header value. + +```yaml +multipart/related: + schema: + items: + type: string + itemEncoding: + contentType: text/html + headers: + Content-Location: + required: true + content: + text/plain: + schema: + type: string + format: uri +``` + +While the above example could have used `itemSchema` instead, if the payload is expected to be processed all at once, using `schema` ensures that tools will wait until the complete response is available before processing. + +###### Example: Streaming Multipart + +This example assumes a device that takes large sets of pictures and streams them to the caller. +Unlike the previous example, we use `itemSchema` here because the expectation is that each image is processed as it arrives (or in small batches), since we know that buffering the entire stream will take too much memory. + +```yaml +multipart/mixed: + itemSchema: + $comment: A single data image from the device + itemEncoding: + contentType: image/jpg +``` + +###### Example: Streaming Byte Ranges + +For `multipart/byteranges` [[RFC9110]] [Section 14.6](https://www.rfc-editor.org/rfc/rfc9110.html#section-14.6), a `Content-Range` header is required: + +See [Appendix D](appendix-d-serializing-headers-and-cookies) for an explanation of why `content: {text/plain: {...}}` is used to describe the header value. + +```yaml +multipart/byteranges: + itemSchema: + $comment: A single range of bytes from a video + itemEncoding: + contentType: video/mp4 + headers: + Content-Range: + required: true + content: + text/plain: + schema: + # A suitable "pattern" constraint for this + # header is left as an exercise for the reader + type: string +``` + #### Responses Object A container for the expected responses of an operation. diff --git a/src/schemas/validation/schema.yaml b/src/schemas/validation/schema.yaml index 4b04f9a43f..efe32ee7da 100644 --- a/src/schemas/validation/schema.yaml +++ b/src/schemas/validation/schema.yaml @@ -533,9 +533,20 @@ $defs: type: object additionalProperties: $ref: '#/$defs/encoding' + prefixEncoding: + type: array + items: + $ref: '#/$defs/encoding' + itemEncoding: + $ref: '#/$defs/encoding' allOf: - - $ref: '#/$defs/specification-extensions' - $ref: '#/$defs/examples' + - $ref: '#/$defs/specification-extensions' + - dependentSchema: + encoding: + properties: + prefixEncoding: false + itemEncoding: false unevaluatedProperties: false encoding: From 1a133aa155ee1dc5be21446e54350bf08c2fa294 Mon Sep 17 00:00:00 2001 From: "Henry H. Andrews" Date: Sun, 18 May 2025 07:44:34 -0700 Subject: [PATCH 2/7] Fix formatting in Encoding guidance --- src/oas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/oas.md b/src/oas.md index 68af364bd1..419dfbe7f6 100644 --- a/src/oas.md +++ b/src/oas.md @@ -1795,7 +1795,7 @@ Note that there are significant restrictions on what headers can be used with `m Note also that `Content-Transfer-Encoding` is deprecated for `multipart/form-data` ([RFC7578](https://www.rfc-editor.org/rfc/rfc7578.html#section-4.7)) where binary data is supported, as it is in HTTP. Using `contentEncoding` for a multipart field is equivalent to specifying an [Encoding Object](#encoding-object) with a `headers` field containing `Content-Transfer-Encoding` with a schema that requires the value used in `contentEncoding`. -+If `contentEncoding` is used for a multipart field that has an Encoding Object with a `headers` field containing `Content-Transfer-Encoding` with a schema that disallows the value from `contentEncoding`, the result is undefined for serialization and parsing. +If `contentEncoding` is used for a multipart field that has an Encoding Object with a `headers` field containing `Content-Transfer-Encoding` with a schema that disallows the value from `contentEncoding`, the result is undefined for serialization and parsing. Note that as stated in [Working with Binary Data](#working-with-binary-data), if the Encoding Object's `contentType`, whether set explicitly or implicitly through its default value rules, disagrees with the `contentMediaType` in a Schema Object, the `contentMediaType` SHALL be ignored. Because of this, and because the Encoding Object's `contentType` defaulting rules do not take the Schema Object's`contentMediaType` into account, the use of `contentMediaType` with an Encoding Object is NOT RECOMMENDED. From 3b9286782fb14cea6dae573bbd207b171eccb8c8 Mon Sep 17 00:00:00 2001 From: "Henry H. Andrews" Date: Tue, 27 May 2025 13:29:13 -0700 Subject: [PATCH 3/7] Clarify multipart preamble/epilogue --- src/oas.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/oas.md b/src/oas.md index 419dfbe7f6..acefd4edb5 100644 --- a/src/oas.md +++ b/src/oas.md @@ -107,6 +107,7 @@ Some examples of sequential media types (including some that are not IANA-regist In the first three above, the repeating structure is any [JSON value](https://tools.ietf.org/html/rfc8259#section-3). The fourth repeats `application/geo+json`-structured values, while `text/event-stream` repeats a custom text format related to Server-Sent Events. The final media type listed above, `multipart/mixed`, provides an ordered list of documents of any media type, and is sometimes streamed. +Note that while `multipart` formats technically allow a preamble and an epilogue, the RFC directs that they are to be ignored, making the effectively comments, and this specification does not model them. Implementations MUST support mapping sequential media types into the JSON Schema data model by treating them as if the values were in an array in the same order. From aeddb7464b13441de6ec002ff00a22395a6bde7e Mon Sep 17 00:00:00 2001 From: "Henry H. Andrews" Date: Tue, 27 May 2025 13:30:41 -0700 Subject: [PATCH 4/7] Clarify lack of nested multipart support. --- src/oas.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/oas.md b/src/oas.md index acefd4edb5..3aeabb0230 100644 --- a/src/oas.md +++ b/src/oas.md @@ -1686,6 +1686,8 @@ Determining how to handle a `type` value of `null` depends on how `null` values If `null` values are entirely omitted, then the `contentType` is irrelevant. See [Appendix B](#appendix-b-data-type-conversion) for a discussion of data type conversion options. +It is not currently possible to model nested `multipart` media types. + ###### Fixed Fields for RFC6570-style Serialization | Field Name | Type | Description | From be6ba1d736261270fbe4a56d66b1cf711e1dfd94 Mon Sep 17 00:00:00 2001 From: "Henry H. Andrews" Date: Fri, 30 May 2025 15:14:59 -0700 Subject: [PATCH 5/7] Fix typo Thanks to @thecheatah for catching this. --- src/oas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/oas.md b/src/oas.md index 3aeabb0230..1c65c8d069 100644 --- a/src/oas.md +++ b/src/oas.md @@ -107,7 +107,7 @@ Some examples of sequential media types (including some that are not IANA-regist In the first three above, the repeating structure is any [JSON value](https://tools.ietf.org/html/rfc8259#section-3). The fourth repeats `application/geo+json`-structured values, while `text/event-stream` repeats a custom text format related to Server-Sent Events. The final media type listed above, `multipart/mixed`, provides an ordered list of documents of any media type, and is sometimes streamed. -Note that while `multipart` formats technically allow a preamble and an epilogue, the RFC directs that they are to be ignored, making the effectively comments, and this specification does not model them. +Note that while `multipart` formats technically allow a preamble and an epilogue, the RFC directs that they are to be ignored, making them effectively comments, and this specification does not model them. Implementations MUST support mapping sequential media types into the JSON Schema data model by treating them as if the values were in an array in the same order. From ccd95a0ea83184a1ceca8f371188572172bd13ca Mon Sep 17 00:00:00 2001 From: Henry Andrews Date: Fri, 30 May 2025 19:24:15 -0700 Subject: [PATCH 6/7] Apply suggestions from code review Co-authored-by: Jeremy Fiel <32110157+jeremyfiel@users.noreply.github.com> --- src/oas.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/oas.md b/src/oas.md index 1c65c8d069..1fe5596dfa 100644 --- a/src/oas.md +++ b/src/oas.md @@ -1351,7 +1351,7 @@ The `itemEncoding` field can also be used with `itemSchema` to support streaming The `prefixEncoding` field can be used with any `multipart` content to require a fixed part order. This includes `multipart/form-data`, for which the Encoding Object's `headers` field MUST be used to provide the `Content-Disposition` and part name, as no property names exist to provide the names automatically. -Prior versions of this specifications advised using the `name` [`Content-Disposition` parameter](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-2) of the `form-data` [`Content-Disposition` value](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-1) with `multipart` media types other than `multipart/form-data` in order to work around the limitations of the `encoding` field. +Prior versions of this specification advised using the `name` [`Content-Disposition` parameter](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-2) of the `form-data` [`Content-Disposition` value](https://www.iana.org/assignments/cont-disp/cont-disp.xhtml#cont-disp-1) with `multipart` media types other than `multipart/form-data` in order to work around the limitations of the `encoding` field. Implementations MAY choose to support this workaround, but as this usage is not common, implementations of non-`form-data` `multipart` media types are unlikely to support it. ##### Media Type Examples @@ -1905,7 +1905,8 @@ multipart/mixed: schema: prefixItems: - # default content type for objects - # is `application/json`type: object + # is `application/json` + type: object properties: author: type: string From 20e80421aba66ecf8480c23dc531299151732791 Mon Sep 17 00:00:00 2001 From: Henry Andrews Date: Sun, 1 Jun 2025 15:35:48 -0700 Subject: [PATCH 7/7] Fix typo Co-authored-by: Jeremy Fiel <32110157+jeremyfiel@users.noreply.github.com> --- src/schemas/validation/schema.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schemas/validation/schema.yaml b/src/schemas/validation/schema.yaml index efe32ee7da..5589956a05 100644 --- a/src/schemas/validation/schema.yaml +++ b/src/schemas/validation/schema.yaml @@ -542,7 +542,7 @@ $defs: allOf: - $ref: '#/$defs/examples' - $ref: '#/$defs/specification-extensions' - - dependentSchema: + - dependentSchemas: encoding: properties: prefixEncoding: false