Add ability to Write to GCP PubSub with an orderingKey

[damccorm]

There was a prior email dev thread about PubSub ordering key[1] but that was in reference to reads. It should be possible to support pubsub Write with ordering key. We might be able to have users supply a serializable function to extra the ordering key from each message, for example?

 

 [1] http://mail-archives.apache.org/mod_mbox/beam-dev/202010.mbox/%3CCAGAbUe93TcwnC+fyAkE3MmtNANi-oFUXPQbg-gDyP6oB8eLTkA@mail.gmail.com%3E

Imported from Jira BEAM-13148. Original Jira may contain additional context.
Reported by: egalpin.

[codertimu]

Any updates on this? Python api has this but Java api lacks it. Can't it be prioritised?

[damccorm]

@egalpin may have context

[egalpin]

Yes publishing messages with an ordering key is now supported in the Java SDK as well, added by #22216. I believe this should be included in SDK v2.43.0 and higher.

Using ordering key is supported by employing PubsubIO.writeMessages[1], where messages will need to be instantiated using the appropriate constructor[2] that accepts ordering keys. It’s also worth noting that pubsub write can make use of regional endpoints[3], which might be required for your use case to fully ensure ordering.

[1] https://beam.apache.org/releases/javadoc/2.43.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html#writeMessages--
[2] https://beam.apache.org/releases/javadoc/2.43.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.html
[3] https://beam.apache.org/releases/javadoc/2.43.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Write.html#withPubsubRootUrl-java.lang.String-

[codertimu]

Interesting, I used 2.43 with the appropriate constructor that takes messageId and orderingKey but no messageId and orderingKey was in Pubsub. I am using dataflow runner.

Could it be related to #23525 ?

[egalpin]

Yes good call out, this is exactly the issue of #23525. You can safely and properly circumvent the issue by using setCoder on the PCollection of pubsub messages which have ordering keys, specifying PubsubMessageWithAttributesAndMessageIdAndOrderingKeyCoder as the coder

[egalpin]

Note that, as far as I understand, any messageId set on a pubsub message to be published will be dropped before being published. This is because message IDs are assigned on the server side of pubsub (again, this is my own understanding and may not be factual)