proposal: net/http: add support for the upcoming "Structured Field Values for HTTP" RFC

[dunglas]

Structured Field Values for HTTP is an upcoming RFC from the HTTP Wording Group defining a set of well-defined data types to use in HTTP headers and trailers.

This new format will improve the interoperability and the safety of HTTP by allowing to create generic parsers and serializers suitable for all HTTP headers. It is already used in the wild, for instance for the new security headers supported by Google Chrome (Sec-Fetch-Dest etc), the Signature proposal in Prefer-Push or in Vulcain.

After the RFC publication, it would be nice to be able to parse and generate such headers directly using the standard library.

I proposed a patch adding support for this spec in #41045. The code is also available as a standalone library: https://github.com/dunglas/httpsfv

[gopherbot]

Change https://golang.org/cl/250837 mentions this issue: net/http: add a package to parse and serialize Structured Field Values

[bradfitz]

I think this is probably best outside the standard library until the RFC is accepted. Once it's in the standard library it's pretty frozen and we can't change APIs or break behavior easily.

[dunglas]

For sure!

[rsc]

@dunglas, your README has a broken link. Should be https://pkg.go.dev/github.com/dunglas/httpsfv.

[rsc]

Also, given your comment above (For sure!) it sounds like you are saying to close this proposal until at least the RFC is accepted?

[dunglas]

@rsc thanks, link fixed.

We should at least wait for the Internet-Draft to become a RFC before merging the patch related to this proposal. However, it should happen soon. The specification is in last stages of standardization. The algorithms described in the I-D and implemented in the patch will most likely not change anymore.

The proposal can probably be kept open, and we can use the time window before the publication as a RFC to review and improve the patch.

[rsc]

@dunglas, I looked at at httpsfv and my first thought is: can we make this simpler? It seems like a huge amount of new API surface for HTTP.

[dunglas]

@rsc we can maybe reduce the API surface a bit, but probably not much. The spec defines many data structures, and several of them cannot be implemented using only Go's builtins: https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#types

The main difference is that SFV's maps (Dictionary and Params) are ordered while Go's map isn't. Much of the API surface is related to these two types and the related methods. We can maybe remove the Del() methods from them, but I'm not sure if it's worth it (having the ability to remove elements from maps is convenient in some cases).

Another option would be to use interface{} in most places. This would allow to merge all Unmarshal* methods and maybe to also merge Dictionary and Params, but it will make the library less safe and less practical to use.

However I may have missed opportunities to reduce the API surface. I'm open to any suggestion to simplify this.

[rsc]

It would be nice to see how widely this is adopted. Being an RFC is one thing; being widely used is another.
It might be that x/net/http/sfv would be a decent starting place?
Even if we do put something in x/net I think we'd need to do a very careful API review to try to reduce the amount here.

/cc @neild

[rsc]

It seems like we should put this on hold for now. We are probably going to have to rethink a bit for HTTP/3, and since there are not many uses of structured field values yet, it might make sense to wait until that work is going on too, to try to deal with potential API changes all at the same time.

For now, we've already seen that nothing prevents using a third-party package for this functionality.

[dunglas]

Structured Field Values are now officially an RFC: https://www.rfc-editor.org/rfc/rfc8941.html

Should we consider adding this implementation in x/net/http/sfv or should we wait for more headers to adopt it?

[mitar]

@dunglas I looked at https://github.com/dunglas/httpsfv and I must say that I agree with @rsc that the API is large and not very Go idiomatic. My proposal would be (for standard library):

• This should go under encoding/sfv namespace.
• It should be made to marshal/unmarshal standard structs and maps and slices like JSON marshaling does, using sfv.Marshal and sfv.Unamarshal functions.
• It should use struct tags for controlling sfv specific behavior (e.g., parameter handling).

For example, I made a simpler implementation now (others: feel free to request to make it a stand-alone package if anyone else finds it useful as well) which just takes a map[string]interface{} as input and marshals it. It does not support parameters, but if I want to convert only from Go to SFV it works great to encode and send some values.

So my API proposal would be:

• Marshal function which takes interface{} as input and returns string, error.
• Unmarshal should take []string as input, because SFV strings could be split among multiple headers and in general http.Header values are []string so you do not have to think about which value to pick.
• Custom types can implement MarshalSFV or UnmarshalSFV method to control their own serialization.
• We should support marshaling regular maps and slices.
• Struct tags:
  • "sfv:name" - under which dictionary key should this be marshaled to/unmarshaled from
  • "sfv:,display" - for strings, to encode it as (still proposed) display string - we can add it later if/when that draft becomes a standard
  • "sfv:name,params" - this makes the field provide and get params for the key with name name
  • "sfv:,value" and "sfv:,params" - for list items it tells them where the list value gets stored and where the list value's params go

So my proposal to support parameters is that one value maps to two Go fields in a struct (when there is a struct), one for the value and one for params. If target is not a struct, params are dropped (e.g., when using map[string]interface{} or []interface{} as target type).

We should probably also look at JSON v2 design doc by @dsnet and see if we can pick some ideas there for the API. For example, the signature of MarshalSFV and UnmarshalSFV should probably be worked out so that they can get struct tags to tune their behavior.

Two more points:

• SFV cares about field order. I think this is fine and structs have the order defined. If you store data in maps or use them when marshaling, then it means that you do not care about order and Go implementation should just use alphabetical order of keys.
• If you dislike the idea of two fields (one for value and one for params), then another option is to define WithParams[T any] generic wrapper struct which stores both the value and params. If you use that as field type then parser stores both the value and params. With other types just value. You could also make map[string]WithParams[interface{}] to store both values and params in a map.

Thank you for starting this work and bringing it at all to the Go ecosystem!

[shogo82148]

FYI, my implementation is here: https://github.com/shogo82148/go-sfv

[mitar]

@shogo82148: Also it does not look very idiomatic? Why InnerList and not simple a slice? Dictionary instead of simply any struct?

[micahhausler]

RFC 9421 (HTTP Message Signatures) recently was recently moved to the standards track, and relies on RFC 8941. I understand that RFC 8941 is not widely adopted yet, but would strongly prefer to build on either an /x/net or standard lib implementation in an RFC 9421 library.

[seankhliao]

The current iteration is RFC 9651, which obsoletes RFC 8941.

RFC 9651 is currently only referenced by drafts https://datatracker.ietf.org/doc/rfc9651/referencedby/ though it is backwards compatible with RFC 8941, which is referenced by several proposed standards https://datatracker.ietf.org/doc/rfc8941/referencedby/ plus RFC 9110 / STD 97 and RFC 9205 / BCP 56

[dunglas]

I'll plan to update my implementation soon. This should be easy, there are few changes.

[dunglas]

Version 1.1 of my implementation now fully implements RFC 9651.

[neild]

RFC 9651 defines a common language for specifying the syntax of HTTP header values (and other parsable texts). For example, it permits defining a header as "the value of the Foo header is a List of Integers (RFC 9651)", which removes the need to describe the integer format, the list separator, and so forth.

As a concrete example, RFC 9211 defines a particular entity as:

The value of "hit" is a Boolean

...thereby saying that the value may be "?0" or "?1" to indicate a false or true value.

RFC 9651 may be used for trivial field definitions (e.g., a string) or complicated ones (e.g., a list of (integer, string) tuples where each entry may also have an associated set of key-value pairs).

I don't think there is any need at this time for us to add a package for working with RFC 9651 structured values to either the standard library or an x/ repo.

This is something which can be supported by third-party packages, as https://github.com/dunglas/httpsfv and demonstrate https://github.com/shogo82148/go-sfv demonstrate.

There is no clearly correct API for an SFV package. github.com/dunglas/httpsfv and github.com/shogo82148/go-sfv have substantially different APIs. #41046 (comment) proposes an entirely different approach using reflection and struct tags. RFC 9651 permits defining complex nested data types, so a comprehensive implementation will not be small. Adding a implementation at this time runs a high risk of us regretting the API in the future.

Many users of structured field values will not require any API at all. strconv.ParseInt suffices to parse an Integer SFV field, for example.

One justification for adding a package to the standard library is to enable interoperability. (For example, a key justification for #53435 was that it enabled interoperability between third-party multierrror implementations.) I don't believe that applies here: A package which parses an HTTP header with one SFV implementation can coexist happily with a package that parses a different header with a different SFV implementation.

In summary, I think this is something that's best left to third-party packages for now.