Cloudevent spec fields are lost when publish to a NATS subject

[omgbebebe]

Problem
When coudevent is published to a NATS subject, all fields except the data field are lost. This behavior is more like a gateway than a transport between NATS and Cloudevent systems. This forces us to carry all metadata in the payload to be able to differentiate events by source or subject. I don't know about the design decisions in this project, and maybe it works as intended, but for our purposes it is a real blocker.

Persona:
I believe it's an Event producer

Exit Criteria

• send Cloudevent to a NATS subject
• receive it on another subject after natss processing
• be able to determine event source, subject, etc. including any Cloudevent extension metadata fields

Time Estimate (optional):
I can't say

Additional context (optional)
in our project we have a quick'n'dirty patch to address this issue but I really not so good in goland and I strongly believe that this can be coded in a more elegant way. I also haven't found a way to get extension headers, if any, from Cloudevent without major codebase refactoring. Maybe I'm missing something obvious.

Right now we're able to receive a NATS message with proper Cloudevent metadata. Like:

[#69] Received on "user-pion.wrkeys-broker-kne-trigger._knative" with reply "_INBOX.jGwt371eVlLFs73MCfDJ9j.m0CJ3Sx5"
traceparent: 00-6e8a9765de2e0722bcf5baf17ba479dc-bfd7b95b2001615c-01
Nats-Msg-Id: a9ed644024cf41ecbf52f6237ce4c7ac
ce-datacontenttype: text/plain; charset=utf-8
ce-id: a9ed644024cf41ecbf52f6237ce4c7ac
ce-source: org.mindwm.cyan.knfunc.pong
ce-specversion: 1.0
ce-subject: org.mindwm.pion.wrkeys.tmux.L3RtcC90bXV4LTEwMDAvZGVmYXVsdA==.e3f65957-a3d9-7c45-13b7-9e0a4c61bc0c.23.36
ce-type: org.mindwm.v1.pong

{"type":"org.mindwm.v1.pong","uuid":"fff0669ffb64456c95b16671633e8c7d"}

without our patch it looks like

[#69] Received on "user-pion.wrkeys-broker-kne-trigger._knative" with reply "_INBOX.jGwt371eVlLFs73MCfDJ9j.m0CJ3Sx5"
Nats-Msg-Id: a9ed644024cf41ecbf52f6237ce4c7ac

{"type":"org.mindwm.v1.pong","uuid":"fff0669ffb64456c95b16671633e8c7d"}

This is definitely not enough to properly route the message further down the processing bus.

[astelmashenko]

Hey, is it the latest 1.15 version? I know about this bug, I fixed in a PR which adds also experimental pull-consumers https://github.com/knative-extensions/eventing-natss/pull/587/files#diff-c369cb6bcf8ad0f92682328cc07052862bafb7a7000324c3a66a24fbbf71040dR389
see this commit
209c0cb
Please check dispatcher.go the line of code below:
ctx = ce.WithEncodingStructured(ctx)

I will take a look on the next week if it is present in 1.15 version and patch-fix it there.

[omgbebebe]

Hi,

I'm talking about this code
When the dispatcher send cloudevent to the NATS subject it set only Nats-Msg-Id as a message header but drop all other ce-* headers. Our approach is to fetch all ce-headers and attach it to the NATS message as headers.
The transformer looks like this.

func (a *CloudeventExtractorTransformer) Transform(reader binding.MessageMetadataReader, _ binding.MessageMetadataWriter) error {
    spec_fields := []spec.Kind{
        spec.ID,
        spec.Source,
        spec.SpecVersion,
        spec.Type,
        spec.DataContentType,
        spec.DataSchema,
        spec.Subject,
        spec.Time,
    }
    // fetch spec
    for _, f := range spec_fields {
	    _, ty := reader.GetAttribute(f)
	    if ty != nil {
		    tyParsed, err := types.ToString(ty)
		    if err != nil {
			    return err
		    }
            (*a)[fmt.Sprintf("ce-%s", f.String())] = tyParsed
        }
    }

PS. right now we're working on latest commit of the main branch

[astelmashenko]

with the fix I mentioned you will get data and headers as nats message data, so called cloud event structured mode

A "structured-mode message" is one where the entire event (attributes and data) are encoded in the message body, according to a specific event format.

I'll provide a fix on this week and you will be able to try it.

A possible problem in your code would be missing extension headers, e.g. you want to add your custom CE header ce-myheader.

[astelmashenko]

Hey, @omgbebebe , please check #599 , the issue is fixed in the PR.