Relation between AMQP Format and Transport Binding is unclear

[deissnerk]

I've been reviewing the AMQP related parts of the spec, and I'm having trouble figuring out how the data section of the AMQP format relates to the AMQP transport binding. I found this:

All implementations of this specification MUST support the JSON event format as well as the AMQP event format for the application-properties section

This restricts the usage of the AMQP event format to the application-properties section, which excludes the handling of data.

In section 2.2 I found:

An application is free to hold the information in any in-memory representation of its choosing, but as the value is transposed into AMQP as defined in this specification, the assumption is that the data value is made available as a sequence of bytes.

Finally section 3.1.2 simply states:

The data byte-sequence is used as the AMQP application-data section.

So the handling of data as described in the AMQP format with its usage of AMQP values would not take place. As I don't think the AMQP format would make sense for the structured encoding, either, this pretty much excludes that this rule from the AMQP format is applied:

For other types (non-binary data without a datacontenttype attribute), the implementation MUST translate the data value into an AMQP type system value and the value MUST be stored in an AMQP value section.

All this really confuses me. In the examples section of the JSON format there is this example:

{
    "specversion" : "1.0-rc1",
    "type" : "com.example.someevent",
    "source" : "/mycontext",
    "id" : "C234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "application/json",
    "data" : {
        "appinfoA" : "abc",
        "appinfoB" : 123,
        "appinfoC" : true
    }
}

How would an SDK transform this into AMQP format/binary encoding? What would happen, if the datacontenttype was omitted? I suppose, according to the AMQP format some kind of AMQP value would be created. Would it be an AMQP Map?

[alanconway]

Here's my understanding:

Structured format (e.g. JSON):
AMQP { content-type: "application/cloudevents+json"; data-section=}

Binary format where event has non-empty datacontenttype, e.g. text/XML
AMQP{ content-type="text/XML"; data-section=

Binary format where event has missing/empty datacontenttype: in this case the event data MUST be a CloudEvents type; this is where the AMQP format rules come in.
AMQP{ value-section=}

I'm working on an AMQP binding for cloudevents/sdk-go right now along these lines.
The structured and non-empty content-type cases are straightforward - just set the AMQP content-type and a single data section.

In the empty datacontentype case: the Go SDK holds the event Data as a Go value corresponding to one of the CE types (int32, string, []byte, time.Time) etc. The AMQP binding looks at this and converts to a corresponding AMQP type for the AMQP value section (amqp long, timestamp, binary etc.)

[deissnerk]

@alanconway That totally makes sense, but I could not find this in the spec for binary.
As I wrote in cloudevents/sdk-go#217 (comment), my understanding in case of a missing datacontenttype is that the spec allows arbitrary types:

When this attribute [datacontenttype] is omitted, data simply follows the event format's encoding rules. For the JSON event format, the data value can therefore be a JSON object, array, or value.

[alanconway]

@deissnerk I agree that needs clarity. The way I read it "the event format encoding rules" only cover CE types - the formatting rules are explicitly about CE types. Therefore it's implied (unclearly) that if there's no datacontentype (or no AMQP content-type) then we must be dealing with a CE type. Otherwise there's no way to know what to do. The spec does need clearer language.

[deissnerk]

@alanconway So, if I remove the datacontenttype from the example above, it would not be a valid CloudEvent?

{
    "specversion" : "1.0-rc1",
    "type" : "com.example.someevent",
    "source" : "/mycontext",
    "id" : "C234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "data" : {
        "appinfoA" : "abc",
        "appinfoB" : 123,
        "appinfoC" : true
    }
}

[alanconway]

On Wed, Oct 2, 2019 at 1:35 PM Klaus Deißner ***@***.***> wrote: @alanconway <https://github.com/alanconway> So, if I remove the datacontenttype from the example above, it would not be a valid CloudEvent? { "specversion" : "1.0-rc1", "type" : "com.example.someevent", "source" : "/mycontext", "id" : "C234-1234-1234", "time" : "2018-04-05T17:31:00Z", "comexampleextension1" : "value", "comexampleothervalue" : 5, "data" : { "appinfoA" : "abc", "appinfoB" : 123, "appinfoC" : true } }

There is residual confusion here because Map used to be a CloudEvents type, it was removed precisely because of the encoding confusion. I can't swear to the wording in the spec, here's what I believe should happen (and the spec should be changed till I believe it ;) Your example is a valid *JSON* representation of a cloud event . It is not a valid way to write an *abstract* cloud event, because you cannot understand the "data" attribute without knowing that we are talking JSON. A JSON decoder (cloudevents/sdk-go is now going this way) can tell that the "data" value is not a CE type, so it cannot decode this as an *abstract* event with no DataContentType. However: it knows it's own default encoding is the well-known "application/json" so it can create this equivalent abstract event: Event{ Data: "{\"appinfoA\" : \"abc\",...}"; DataContentType: "application/json"} Now that abstract event can be encoded by any other binding, the content remains forever and explicitly "application/json" The only time an *abstract* event is allowed to have an empty content type is when the data is a CE type, so for example this JSON: { "specversion"... etc, "data": 123 } Can be legally decoded in two ways: 1. Event{ Data: int32(123); DataContentType: nil } 2. Event{ Data: "123", DataContentType: "application/json" } Now if we were to encode these 2 abstract Events in AMQP we would get different results: 1. amqp{ content-type:"", value-section: amqp-long(123) } 2. amqp{ content-type:"application/json", data-section: "123" } In the first case (no datacontentype) we encode into the default AMQP encoding - an AMQP long int. If we decoded this we round-trip back to an event with no datatype and an int32Data. In the second case, the encoding got pinned to "application/json" when we decoded the event from JSON, and it will stay that way forever. There is no spec requirement to be able to decode arbitrary AMQP types (phew!) only the CE types. However a fancy AMQP decoder *might* chose to transcode complex AMQP types into JSON, XML or whatever. For example the AMQP management spec defines 2 forms for management data - an AMQP map or a JSON object - a smart AMQP encoder/decoder might want to automatically do the translation (or no, I leave it to your discretion how "smart" this would be :) I would be happier if datacontenttype was *required* and the payload was always bytes in a known MIME format, but it would be less convenient for people typing JSON events by hand so I guess that ship has sailed. Side note: IMO this is why AMQP failed to take over the world - if AMQP messages only carried bytes with a known content type the burden on implementers would be halved, the loss to users would be almost nothing, and there would have been no motivation for MQTT. We might not even be doing cloudevents, just AMQP over HTTP. It was a huge mistake to bundle a complex type system into a messaging transport. Thankfully the CE type system only has a handful of scalar types so the mistake is much smaller.

…

[deissnerk]

I am not sure, if your understanding regarding data and the CE type system matches with what the others (@clemensv?) intended in #484. In this PR the notion of data as an attribute is removed. The idea seems to be that people do not want application/json as datacontenttype. If, as you implemented it, an intermediary can set the datacontenttype to application/json when forwarding the event using a different format, this can work for JSON and becomes just a convenience feature. For the AMQP format it is worse. As it does not define an envelope, it would always require the transformation of an object graph into something else.

So having the support of the AMQP format mandatory for a compliant implementation is problematic.

It is confusing that the semantics of datacontenttype not being present differs depending on transport bindings and event formats:

• For HTTP binary you can only treat the payload as an undefined binary.
• In the JSON format it is implicitly assumed to be application/json, if datais used. It should then also be set to application/json when forwarding the event in a different format. Even that is not specified. So someone emitting and event in JSON format leaving out the datacontenttype for convenience, must fear that the JSON object in data will be transcoded several times in a lossy way.
• If in JSON data_base64 is used, datacontenttype again is assumed to be unspecified.

[alanconway]

On Thu, Oct 3, 2019 at 10:43 AM Klaus Deißner ***@***.***> wrote: I am not sure, if your understanding regarding data and the CE type system matches with what the others ***@***.*** <https://github.com/clemensv>?) intended in #484 <#484>. In this PR the notion of data as an attribute is removed. The idea seems to be that people do not want application/json as datacontenttype. If, as you implemented it, an intermediary can set the datacontenttype to application/json when forwarding the event using a different format, this can work for JSON and becomes just a convenience feature. For the AMQP format it is worse. As it does not define an envelope, it would always require the transformation of an object graph into something else.

Not exactly true. The AMQP spec defines a binary encoding. If there was a standard AMQP MIME type we could have data=<encoded-amqp-value> and datacontentype="application/amqp-value". The real difference is that JSON encoding is popular and widely used, AMQP encoding isn't used outside AMQP connections. For that reason, I don't think CE should or does define a translation of arbitrary AMQP values to anything else.

So having the support of the AMQP format mandatory for a compliant implementation is problematic.

I do think that "mandatory" is odd. However supporting AMQP is not problematic if you read the requirement as "be able to transport cloudevents over AMQP" It only becomes problematic if you read it as "be able to represent arbitrary AMQP values in cloudevents" I don't think the latter was ever intended.

It is confusing that the semantics of datacontenttype not being present differs depending on transport bindings and event formats:

+1 it is a dreadful rathole, we should try to make it as small as possible - hence my "only CE types" interpretation. I accept that my interpretation may not be justified by the spec or shared by others :) Here's an alternative, which I actually like better: Every event *does* have a datacontenttype, but it MAY be implicit in *some* bindings. Bindings can only have a "default encoding" if there is a valid datacontentype associated with that encoding (e.g. "application/json"). Bindings like HTTP and AMQP have no such default - they MUST have an explicit datacontentype. AMQP value-section messages are not allowed as events unless/until we declare "application/amqp-value" to be a valid datacontentype. (NB: I'm only talking about data here, we still have a CE-AMQP mapping for extension attributes) So a JSON event {... "data": {"foo": "bar" }} is just shorthand for {... "datacontenttype":"application/json", "data": {"foo": "bar" }} Now we have no interop problems. A decoder to abstract Event can fill in the implied datacontentype if it's not explicit. An encoder can optionally drop the datacontentype if it matches the target format. Whenever there's a mismatch of bindings/formats there will always be an explicit datacontentype. And for formats like JSON, we still have the shorthand with no datacontentype. I like that a LOT better - any takers?

…

- For HTTP binary you can only treat the payload as an undefined binary. - In the JSON format it is implicitly assumed to be application/json, if datais used. It should then also be set to application/json when forwarding the event in a different format. Even that is not specified. So someone emitting and event in JSON format leaving out the datacontenttype for convenience, must fear that the JSON object in data will be transcoded several times in a lossy way. - If in JSON data_base64 is used, datacontenttype again is assumed to be unspecified. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#519?email_source=notifications&email_token=AB3LUXROPBPA4MMEPXET5XLQMYAHLA5CNFSM4I4UPPE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAIN5DQ#issuecomment-537976462>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB3LUXRWFYDSI3VFNT3B3SLQMYAHLANCNFSM4I4UPPEQ> .

[deissnerk]

I like the proposal. As a consequence we could simply remove the section Data from the AMQP format.

[duglin]

late to the party...

Can be legally decoded in two ways:

1. Event{ Data: int32(123); DataContentType: nil }
2. Event{ Data: "123", DataContentType: "application/json" }

I don't think that's right. I think the only interpretation of it is:

3. Event { Data: number(123); DataContentType: "application/json" }

I say this because:

• w/o a datacontenttype we implicitly get app/json from the event itself
• 123 isn't wrapped in quotes so in JSON its a number, not a string

[alanconway]

late to the party...

Can be legally decoded in two ways:

1. Event{ Data: int32(123); DataContentType: nil }
2. Event{ Data: "123", DataContentType: "application/json" }

I don't think that's right. I think the only interpretation of it is:

1. Event { Data: number(123); DataContentType: "application/json" }

I say this because:

• w/o a datacontenttype we implicitly get app/json from the event itself
• 123 isn't wrapped in quotes so in JSON its a number, not a string

This is confusion between my initial daft idea about using the CE type system when datacontentype is absent and the more sensible idea expressed in #521. I think if we can hammer out #521 then we're set on this one also.

The Go SDK has to deal with multiple representations of Data, since the JSON-encoded "number" 123 is actually the byte sequence {49,50,51}, alias the string "123", also represented in Go as the 32-bit binary integer int32(123). Plus there are a bunch of convertible numeric types. So probably my use of Go notation above is not helping to clarify :)

[deissnerk]

I really like #521, but even with this PR being accepted the original question of this issue remains unanswered. So far the AMQP protocol binding references the AMQP format only to clarify how CE types for CE attributes map to AMQP types. When it comes to data, the protocol binding describes structured and binary mode. As the AMQP format does not specify a default datacontenttype, the attribute becomes mandatory in AMQP. So section 3 becomes obsolete. But if we remove section 3, it will not be a format any more, as it does not "specify how to serialize a CloudEvent as a sequence of bytes". A logical consequence would be to get rid of the AMQP format and move the type mapping for CE attributes over to the protocol binding.
@clemensv Any thoughts on that?

[clemensv]

The AMQP format was really only meant to provide a proper type mapping such that attributes can be mapped to AMQP properties.

It wasn't meant to fully express events in the AMQP type system because the AMQP type system is not really being used as a general purpose data serialization format.

The idea was that AMQP messages generally carry events that are using an encoding that's useful for end-to-end payloads.

[deissnerk]

Yes, I understand that. I think the confusion was caused by some changes we waved through in August. With data not being an attribute anymore, the AMQP format got a section describing how data would be encoded. That was different from what the transport binding specified, hence I created this issue and the whole discussion started.
With @alanconway's #521 we get a definition of event format as something that "specifies how to serialize a CloudEvent as a sequence of bytes". The AMQP format as it was originally intended does not match to this definition. This could be resolved by moving the relevant parts of the AMQP format into the transport/protocol binding spec.

[alanconway]

@deissnerk +1 - I'd prefer to reserve the term "format" for encodings that have a "application/cloudevents+" MIME type and can be used to format structured events over any protocol. The JSON format fits that bill, but the AMQP format doesn't.
(It could if the AMQP value encoding had a MIME type and there was interest in using it outside of AMQP protocol connections, but I don't think that is likely)

[deissnerk]

@alanconway Please have a look at #529.