Kafka transport binding v2 #337

bluemonk3y · 2018-11-01T13:47:14Z

Kafka transport binding for CloudEvents, similar to the HTTP binding and proposed NATS, MQTT, AMQP bindings.

Follow-on PR from #300

Signed-off-by: Neil Avery neil@confluent.io

duglin · 2018-11-01T14:35:42Z

CI error is ok since it'll be automagically fixed once merged. But the DCO issue is real.

kafka-transport-binding.md

duglin · 2018-11-01T14:42:47Z

kafka-transport-binding.md

+
+The receiver of the event can distinguish between the two content modes by
+inspecting the `cloudEvents_contentType` property of the Kafka message. If the
+value is prefixed with the CloudEvents media type `application/cloudevents`, indicating the use of a known [event


duglin · 2018-11-01T14:44:45Z

kafka-transport-binding.md

+
+#### 3.1.3. Metadata Headers
+
+All [CloudEvents][CE] attributes with exception of `data` MUST be individually


should we include extensions in this too?

bluemonk3y · 2018-11-01T16:16:42Z

CI error is ok since it'll be automagically fixed once merged. But the DCO issue is real.

@duglin - yep I will fix that tomorrow

kafka-transport-binding.md

matzew · 2018-11-07T12:39:34Z

kafka-transport-binding.md

+
+##### 3.1.3.1 Property Names
+
+Cloud Event attributes are prefixed with "cloudEvents_" for use in the


cloudEvents_ ? instead of "cloudEvents_" ?

I wonder if we should use some sort of fqn: io.cloudevents. to avoid potential clashes ?

We are using cloudEvents_ as it has been generally adopted by other transport bindings. I believe we also need to avoid '.' characters. We should be consistent across all bindings.
@duglin - your thoughts on this?

I like consistency, but I think we need to be consistent with that users of the transport expect. I did a quick search and came across this: https://streamsets.com/documentation/controlhub/3.5.0/help/pdesigner/datacollector/UserGuide/Origins/KConsumer.html
and it talks about things like ssl.truststore.location and com.sun.security.auth.module.Krb5LoginModule. So Kafka does seem to be ok with '.'.
I don't know enough about Kafka to know if FQNs would be more "expected" or not, I'll leave that to others - I'd be ok with either.

@clemensv - what is your opinion on this (above)? Do we adopt language packaging/namespace semantics like io.cloudevents or cloudEvents_ It does raise the question about event propagation between transports.

Just for the record, Apache Kafka never managed to decide on whether "." or "_" are preferred as separators. Configuration parameters are always "." separated (as the Streamset doc shows), but topics are often namespaced with "_" (for example, the internal "__consumer_offsets" topic). I've definitely seen both used in "the wild" (a fact that was sometimes inconvenient as the project had to deal with converting topic names to metric names...)

Either separator will be acceptable by the Apache Kafka community (or rather, either will be objected to, by different sets of people).

(duglin: I edited the comment to escape the underscore since it wasn't showing up and was being treated as the "turn on italics" symbol)

matzew · 2018-11-07T12:45:54Z

kafka-transport-binding.md

+
+    * `eventTime` maps to `cloudEvents_eventTime`
+    * `eventID` maps to `cloudEvents_eventID`
+    * `cloudEventsVersion` maps to `cloudEvents_cloudEventsVersion`


+1 on the general concept of this.

Our group has been doing exactly the same:
https://github.com/rh-event-flow/jcloudevents/blob/master/kafka/src/main/java/io/streamzi/cloudevents/kafka/util/KafkaHeaderUtil.java#L42

Ok, biggest diff is ours is on internal RecordHeader(s) API, and no prefix.

Perhaps it should be based on the Header(s) interface API

s/cloudEventsVersion/specversion/

kafka-transport-binding.md

matzew · 2018-11-07T13:30:48Z

@bluemonk3y we have some POC code for this. and it would be nice to get this into the Cloudevents-sdk for java.

See my proposal here:
cloudevents/sdk-java#5

looking forward working with you on that

kafka-transport-binding.md

gunnarmorling · 2018-11-07T13:42:52Z

kafka-transport-binding.md

+
+------------------- key ----------------------
+
+Key: mykey


The Kafka message key is where things are still a bit blurry to me. How will its value be obtained from the CloudEvents event? Have you foreseen some extractor function or similar?

Thinking more about that one, perhaps a header key should be defined whose value, if present, will be propagated as key of the Kafka message, e.g. cloudEvents_messageKey. Alternatively, the binding, wherever it's running, could be configured with some kind of path expression which gets applied to the data value to extract the value, e.g. /customer/id.

Yep, there is a PR for this: #218

Ah yes, thanks. I've commented over there, too. Seems that'd need resolution first.

The conflict we have here is that the partition key is a transport specific concern (and the requirements may vary across different transport options) and that the publisher ought not to care about what transport the event route gets bound to. We may also have the situation that an event gets first published via MQTT from a little device and then gets put on Kafka by a device gateway.

I think this binding needs to define a rule by which a key is constructed from the event rather than expecting that the event brings it along.

It seems to me that the natural way to get a key for a Kafka message is to use the source.

I'm not sure if source is the right place - I'd expect all events produced by same application to have identical source, why partitioning typically applies to events produced by same source.

I like the proposal from @clemensv in #218 :
" instead of putting the burden of producing an event key on the client and then only having one, any transport that requires particular constructs such as keys such define a mechanism by which you can harvest/synthesize a key from an incoming CloudEvent as some sort of transform. The spec doesn't need to prescribe how -- it just needs to say that that's how the key materializes. A transform that just cooks up a random key might also be valid if that's what you want."

KafkaConnect already has "key extraction" transformation, exactly because external records require mapping to Kafka keys, and the logic for doing so varies between use-cases.

@gwenshap @clemensv that would mean that the kafka transport spec would leave the determination of the "key" as an exercise for the implementer (or admin configuring the transport) ?

I would hope a reasonable implementation of transport would allow plugging in the "key selection" logic, since we can't know in advance what it will be. (Although I'm still quite new to CloudEvents, so take my opinions with a bit of salt)

Just added a comment on this over at #218. The idea being that instead of having fully pluggable logic for retrieving the key instead there could be a mechanism to just select the key from a given event field when sending a cloud event to a Kafka Topic.

clemensv · 2018-11-08T16:25:28Z

The header prefixes might be a bit generous. I think we'll end up collapsing them to "ce" across the board.

matzew · 2018-12-12T17:46:58Z

I think we'll end up collapsing them to "ce" across the board.

+1

kafka-transport-binding.md

JemDay · 2019-03-19T22:31:07Z

It seems this PR is a bit behind the 0.2 spec version ...

duglin · 2019-05-10T02:09:47Z

ping @hschaffner - please review and add any comments w.r.t. why the partitionkey extension should not be needed.

ping @clemensv - please add comments about the call-back mechanism you mentioned in PR #429

@bluemonk3y now that #429 is merged, can you rebase this one to deal with the merge conflicts? Also, you may need to edit some of the text to deal with recent spec changes - such as the names of the attributes being changed (or lower-cased).

thanks everyone for your patience.

bluemonk3y · 2019-05-10T08:57:34Z

@duglin - yep as soon as I get a chance.

duglin · 2019-05-28T17:58:23Z

@bluemonk3y sorry to pester :-) but any chance of getting an update of this one? I get the sense we're nearing a pretty big milestone and this is one of the bigger outstanding items for us.

bluemonk3y · 2019-05-29T07:40:29Z

@duglin - sure, I'm currently away but will pick it up on Monday.

gunnarmorling · 2019-05-29T08:30:48Z

now that #429 is merged

@duglin, I reckon you mean #218?

Could you (or someone else) perhaps give a summary of the latest state of the discussion? The closed PR #218 adds the partitionkey attribute, but it actually might be removed again as per #430?

What @clemensv describes in #218 (comment) makes sense to me: make this a concern of the transport binding which would allow for a callback function for retrieving the partition/message key on an per-event basis. The spec-defined partitionkey attribute would still remain in place as basis for the default behavior in case no custom callback has been configured for the transport and a value can be defined by the producer.

One open question to me is how the callback would be defined. What exactly is it? E.g. a (Java) class implementing a certain interface for extracting the key based on an incoming event? Or would at the spec level just be defined that there is a callback mechanism but it'd then be up to specific implementations of this to define the concrete mechanism?

Thanks!

duglin · 2019-06-04T23:40:29Z

@bluemonk3y the travis errors seem real. Also, can you sign your commits?

duglin · 2019-06-10T01:31:35Z

Even though I didn't tag this as "v1.0" (it's "try-for-v1.0"), it's been lingering for a while so it'll be first on the PR review list this week - please look it over when you get a chance.

clemensv · 2019-06-13T12:56:22Z

kafka-transport-binding.md

+
+This example shows the *binary* mode mapping of an event into the
+Kafka message. All other CloudEvents attributes
+are mapped to Kafka Header fields with prefix `cloudEvents_`.


ce_ might just be sufficient.

clemensv · 2019-06-13T12:56:37Z

kafka-transport-binding.md

+
+Examples:
+
+    * `eventTime` maps to `cloudEvents_eventTime`


needs update

s/eventTime/time/

clemensv · 2019-06-13T12:59:41Z

kafka-transport-binding.md

+Example for the [JSON format][JSON-format]:
+
+``` text
+content-type: application/cloudevents+json; charset=UTF-8


This shows no prefix

Is this missing a ce_ or is the sentence in the first paragraph of this section incorrect?

I think the intro paragraph is wrong

duglin · 2019-06-13T15:46:38Z

kafka-transport-binding.md

+placed into the Kafka message value section
+using an [event format](#14-event-formats).
+
+In the *binary* content mode, the value of the event `data` attribute MUST be


would it make sense to move this paragraph up so it's next to the one on line 52, that also talks about "binary" mode?

duglin · 2019-06-13T15:50:55Z

kafka-transport-binding.md

+Examples:
+
+    * `eventTime` maps to `cloudEvents_eventTime`
+    * `eventID` maps to `cloudEvents_eventID`


s/eventID/id/

duglin · 2019-06-13T15:54:35Z

@hschaffner did you have any comments on this one? In particular around the key stuff

duglin · 2019-06-17T16:56:55Z

@bluemonk3y rebase needed - and there are some outstanding comments

kafka-transport-binding.md

duglin · 2019-06-17T18:34:47Z

kafka-transport-binding.md

-cloudEvents_eventTime: "2018-04-05T03:56:24Z"
-cloudEvents_eventID: "1234-1234-1234"
-cloudEvents_source: "/mycontext/subcontext"
+ce_specVersion: "0.1"


s/specVersion/specversion/

and "0.4-wip"

README.md

json-format.md

kafka-transport-binding.md

duglin · 2019-06-17T18:40:14Z

kafka-transport-binding.md

+------------------- value --------------------
+
+{
+    "specVersion" : "0.1",


s/specVersion/specversion/
s/0.1/0.4-wip/

I believe I fixed the wonky rebase!

duglin · 2019-06-17T18:41:03Z

I think the rebase got a little wonky

Kafka transport binding for CloudEvents, similar to the HTTP binding and proposed NATS, MQTT, AMQP bindings. Signed-off-by: Neil Avery <neil@confluent.io>

duglin · 2019-06-19T20:00:38Z

kafka-transport-binding.md

+### 2.1. data Attribute
+
+The `data` attribute is assumed to contain opaque application data that is
+encoded as declared by the `contentType` attribute.


contentType or "datacontenttype" ?

hmm I think I might be wrong, but I wanted to verify.

duglin · 2019-06-19T20:07:30Z

kafka-transport-binding.md

+here.
+
+The receiver of the event can distinguish between the two content modes by
+inspecting the `ce_contentType` [Header][Kafka-Message-Header] of the 


s/ce_contentType/ce_datacontenttype/ I think (but it's the "T" that jumped out at me)

kafka-transport-binding.md

duglin · 2019-06-19T20:11:34Z

kafka-transport-binding.md

+
+#### 3.3.1. Kafka Content-Type
+
+The [Kafka `ce_contentType`] property field MUST be set to the media


Actually, I think the "ce_" here is incorrect since this isn't a CE property. Right?

duglin · 2019-06-19T20:12:58Z

kafka-transport-binding.md

+
+------------------ headers -------------------
+
+ce_contentType: application/cloudevents+json; charset=UTF-8


I thin the "ce_" here is incorrect.

agree - removed

duglin · 2019-06-19T20:13:20Z

proprietary-specs.md

@@ -1,4 +1,4 @@
-# CloudEvent Specs for Proprietary Protocols and Encodings
+### CloudEvent Specs for Proprietary Protocols and Encodings


I think this change is invalid

duglin · 2019-06-19T20:13:39Z

spec.md

@@ -58,6 +58,14 @@ that does not support that feature will then silently ignore that part of the
 message. The sender needs to be prepared for the situation where a receiver
 ignores that feature.

+For clarity, when a feature is marked as "OPTIONAL" this means that it is


I think the edits in this doc are invalid

@duglin - I believe I have applied the changes correctly ;) - would you mind reviewing?

Kafka transport binding for CloudEvents, similar to the HTTP binding and proposed NATS, MQTT, AMQP bindings. Signed-off-by: Neil Avery <neil@confluent.io>

duglin · 2019-06-21T21:39:16Z

@bluemonk3y looks good to me.

Can I get one more LGTM and then I'll merge!!

duglin · 2019-06-27T08:09:55Z

Approved on the 6/20 call with the minor rebase fixes

duglin · 2019-06-27T17:03:18Z

Approved, again :-) , on the 6/28 call - but this time with the rebase fixes

duglin · 2019-06-27T17:03:44Z

@bluemonk3y et al... thanks a ton for your patience and hard work on this one!!!

duglin mentioned this pull request Nov 1, 2018

Kafka transport binding #300

Closed

duglin reviewed Nov 1, 2018

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

duglin reviewed Nov 1, 2018

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

duglin reviewed Nov 1, 2018

View reviewed changes

duglin reviewed Nov 2, 2018

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

matzew reviewed Nov 7, 2018

View reviewed changes

gunnarmorling reviewed Nov 7, 2018

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

kafka-transport-binding.md Outdated Show resolved Hide resolved

matzew mentioned this pull request Nov 7, 2018

Kafka transport cloudevents/sdk-java#5

Closed

gunnarmorling reviewed Nov 7, 2018

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

gunnarmorling reviewed Nov 7, 2018

View reviewed changes

gunnarmorling mentioned this pull request Nov 8, 2018

Proposal for an event key field as an extension #218

Merged

matzew mentioned this pull request Dec 12, 2018

WIP: Adding initial support for some simple Kafka "mapping cloudevents/sdk-java#26

Closed

kevinbader reviewed Jan 30, 2019

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

duglin mentioned this pull request May 10, 2019

Revisit PR 218 to see if partitionkey extension is still needed #430

Closed

duglin added the needs work label May 30, 2019

duglin added the try-for-v1.0 label Jun 8, 2019

duglin removed the needs work label Jun 8, 2019

clemensv reviewed Jun 13, 2019

View reviewed changes

duglin reviewed Jun 13, 2019

View reviewed changes

duglin reviewed Jun 17, 2019

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

duglin reviewed Jun 17, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

duglin reviewed Jun 17, 2019

View reviewed changes

json-format.md Outdated Show resolved Hide resolved

duglin reviewed Jun 17, 2019

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

duglin reviewed Jun 17, 2019

View reviewed changes

Follow-on PR from #300

27eeee4

Kafka transport binding for CloudEvents, similar to the HTTP binding and proposed NATS, MQTT, AMQP bindings. Signed-off-by: Neil Avery <neil@confluent.io>

duglin reviewed Jun 19, 2019

View reviewed changes

kafka-transport-binding.md Outdated Show resolved Hide resolved

duglin reviewed Jun 19, 2019

View reviewed changes

addressing remaining cloudEvent attribute names

2c2d22d

Kafka transport binding for CloudEvents, similar to the HTTP binding and proposed NATS, MQTT, AMQP bindings. Signed-off-by: Neil Avery <neil@confluent.io>

duglin merged commit 7c3bdb1 into cloudevents:master Jun 27, 2019


		#### 3.1.3. Metadata Headers

		All [CloudEvents][CE] attributes with exception of `data` MUST be individually


		##### 3.1.3.1 Property Names

		Cloud Event attributes are prefixed with "cloudEvents_" for use in the


		#### 3.3.1. Kafka Content-Type

		The [Kafka `ce_contentType`] property field MUST be set to the media


		------------------ headers -------------------

		ce_contentType: application/cloudevents+json; charset=UTF-8

		@@ -1,4 +1,4 @@
		# CloudEvent Specs for Proprietary Protocols and Encodings
		### CloudEvent Specs for Proprietary Protocols and Encodings

Kafka transport binding v2 #337

Kafka transport binding v2 #337

Conversation

bluemonk3y commented Nov 1, 2018 • edited Loading

duglin commented Nov 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bluemonk3y commented Nov 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gwenshap Nov 20, 2018 • edited by duglin Loading

Choose a reason for hiding this comment

matzew Nov 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matzew commented Nov 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clemensv commented Nov 8, 2018

matzew commented Dec 12, 2018

JemDay commented Mar 19, 2019

duglin commented May 10, 2019

bluemonk3y commented May 10, 2019

duglin commented May 28, 2019

bluemonk3y commented May 29, 2019

gunnarmorling commented May 29, 2019

duglin commented Jun 4, 2019

duglin commented Jun 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Jun 13, 2019

duglin commented Jun 17, 2019

Choose a reason for hiding this comment

duglin Jun 17, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Jun 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duglin commented Jun 21, 2019

duglin commented Jun 27, 2019

duglin commented Jun 27, 2019

duglin commented Jun 27, 2019

bluemonk3y commented Nov 1, 2018 •

edited

Loading

gwenshap Nov 20, 2018 •

edited by duglin

Loading

matzew Nov 7, 2018 •

edited

Loading

duglin Jun 17, 2019 •

edited

Loading