pipeline: support a TriggerCutoff event

[adamdecaf]

This is a proposal to introduce a new event called TriggerCutoff to achgateway. This event will function the same as a manual/automatic cutoff trigger but allows us to keep state consistent across multiple instances of achgateway.

Imagine a kafka topic with messages file1, file2, triggerCutoff[09:00], file3:

Each instance of achgateway will queue file1 and file2 for a cutoff. Then the triggerCutoff event is consumed which initiates leadership election and processing. Only after completed cutoff processing is file3 queued for later upload.

Currently instances of achgateway can get out of sync due to clock skew which might let file3 be consumed by some instances and not others. This produces inconsistent state.

The service producing QueueACHFile events could produce this. Often the producing service needs knowledge of cutoff times to allow for setting EffectiveEntryDate to a proper value. Duplication of cutoff time configuration is not ideal.

[darwinz]

For this to work with Kafka, it will require using only a single partition to guarantee linear message ordering. Otherwise, RabbitMQ might be a good option for this particular case.

[adamdecaf]

@InfernoJJ Do you have any ideas?

[InfernoJJ]

You don’t need one partition to make it linear. It’s the key that makes it linear.

…

On Thu, Feb 24, 2022 at 7:31 PM Adam Shannon ***@***.***> wrote: @InfernoJJ <https://github.com/InfernoJJ> Do you have any ideas? — Reply to this email directly, view it on GitHub <#88 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFMTIS5B72KNZPIDWH6VYTU43LVZANCNFSM5PISZMTQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[InfernoJJ]

I think you can do a event to signal the export.

…

On Thu, Feb 24, 2022 at 7:33 PM JJ Philipp ***@***.***> wrote: You don’t need one partition to make it linear. It’s the key that makes it linear. On Thu, Feb 24, 2022 at 7:31 PM Adam Shannon ***@***.***> wrote: > @InfernoJJ <https://github.com/InfernoJJ> Do you have any ideas? > > — > Reply to this email directly, view it on GitHub > <#88 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAFMTIS5B72KNZPIDWH6VYTU43LVZANCNFSM5PISZMTQ> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[InfernoJJ]

You can have 100 partitions on a topic but if all of the events use the same key they will all end up on the same partition.

…

On Thu, Feb 24, 2022 at 7:34 PM JJ Philipp ***@***.***> wrote: I think you can do a event to signal the export. On Thu, Feb 24, 2022 at 7:33 PM JJ Philipp ***@***.***> wrote: > You don’t need one partition to make it linear. It’s the key that makes > it linear. > > On Thu, Feb 24, 2022 at 7:31 PM Adam Shannon ***@***.***> > wrote: > >> @InfernoJJ <https://github.com/InfernoJJ> Do you have any ideas? >> >> — >> Reply to this email directly, view it on GitHub >> <#88 (comment)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AAFMTIS5B72KNZPIDWH6VYTU43LVZANCNFSM5PISZMTQ> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

[InfernoJJ]

Rabbit would add to the problems if you have multiple endives consuming it it will serve too what service is asking and breaking linearity. This is not an issue with Kafka because only one service can consume one partition.

…

On Thu, Feb 24, 2022 at 7:37 PM JJ Philipp ***@***.***> wrote: You can have 100 partitions on a topic but if all of the events use the same key they will all end up on the same partition. On Thu, Feb 24, 2022 at 7:34 PM JJ Philipp ***@***.***> wrote: > I think you can do a event to signal the export. > > On Thu, Feb 24, 2022 at 7:33 PM JJ Philipp ***@***.***> wrote: > >> You don’t need one partition to make it linear. It’s the key that makes >> it linear. >> >> On Thu, Feb 24, 2022 at 7:31 PM Adam Shannon ***@***.***> >> wrote: >> >>> @InfernoJJ <https://github.com/InfernoJJ> Do you have any ideas? >>> >>> — >>> Reply to this email directly, view it on GitHub >>> <#88 (comment)>, >>> or unsubscribe >>> <https://github.com/notifications/unsubscribe-auth/AAFMTIS5B72KNZPIDWH6VYTU43LVZANCNFSM5PISZMTQ> >>> . >>> You are receiving this because you were mentioned.Message ID: >>> ***@***.***> >>> >>

[InfernoJJ]

You don't really need to do the leader election either as Kafka already time one to determine what service reads the partitions. Just need impotency of checking if the file was already uploaded if for instance a network failure causes it to not commit the offsets.

The event to trigger to cutoff is the best way I can think of having consistent flow of messages and knowing exactly what events ended up in the file.

[adamdecaf]

Yea. Doing this would require the partition keys are managed by producers in a way that they understand when cutoffs are triggered. We have a bunch of UUIDs mapped to "ODFI_1" and "ODFI_2", so the partition keys would need to be "ODFI_1" and "ODFI_2". Then each achgateway instance will get a consistent set of files to upload.

https://kafka.apache.org/documentation/#intro_concepts_and_terms