Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use unused field in LogEntryDescriptor for non-processed marker #11504

Closed
Tracked by #11416
, #11539
...
ChrisKujawa opened this issue Jan 31, 2023 · 1 comment · Fixed by #11509
Closed
Tracked by #11416
, #11539
...

Use unused field in LogEntryDescriptor for non-processed marker #11504

ChrisKujawa opened this issue Jan 31, 2023 · 1 comment · Fixed by #11509
Assignees
@lenaschoenburg @ChrisKujawa
Labels
component/stream-platform kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. version:8.2.0-alpha4 Marks an issue as being completely or in parts released in 8.2.0-alpha4 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0

Comments

@ChrisKujawa
Copy link
Member

Description

Part of #11416

In order to implement batch processing with batch size limit handling we need to make sure that we can somehow mark records as processed and non-processed. For that we can use an unused field in the LogEntryDescriptor https://github.com/camunda/zeebe/blob/main/logstreams/src/main/java/io/camunda/zeebe/logstreams/impl/log/LogEntryDescriptor.java#L94

See related POC #11469

AT:

  • We can set a boolean value on the existing offset, which marks the entry as processed
  • Default value is non-processed
@ChrisKujawa ChrisKujawa added kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. component/stream-platform labels Jan 31, 2023
@ChrisKujawa ChrisKujawa mentioned this issue Jan 31, 2023
Closed
12 tasks
@ChrisKujawa
Copy link
Member Author

@oleschoenburg and I decided to use https://github.com/camunda/zeebe/blob/main/logstreams/src/main/java/io/camunda/zeebe/logstreams/impl/log/LogEntryDescriptor.java#L76 instead https://github.com/camunda/zeebe/blob/main/logstreams/src/main/java/io/camunda/zeebe/logstreams/impl/log/LogEntryDescriptor.java#L94

Since after the Metadata length to store data felt a bit weird. We implemented it in a way that we can extend it further with more boolean flags/markers.

@lenaschoenburg lenaschoenburg mentioned this issue Jan 31, 2023
Merged
ghost pushed a commit that referenced this issue Jan 31, 2023
@ChrisKujawa @lenaschoenburg
merge: #11509
53cf41c 
11509: feat: allow marking records as processed r=Zelldon a=oleschoenburg

Uses a [previously unused](#11504 (comment)) byte in the `LogEntryDescriptor` as a field for arbitrary boolean flags, first used to mark records as already processed.

This allows the stream processor to mark new records as processed and read the `isProcessed` flag back during processing which is useful for batch processing.

By default, entries are assumed to be not processed yet.

closes #11504

Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
@ghost ghost closed this as completed in c4b14e7 Jan 31, 2023
@ChrisKujawa ChrisKujawa mentioned this issue Feb 3, 2023
Closed
19 tasks
@koevskinikola koevskinikola added the version:8.2.0-alpha4 Marks an issue as being completely or in parts released in 8.2.0-alpha4 label Feb 8, 2023
@npepinpe npepinpe added the version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0 label Apr 5, 2023
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lenaschoenburg lenaschoenburg

@ChrisKujawa ChrisKujawa

Labels
component/stream-platform kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. version:8.2.0-alpha4 Marks an issue as being completely or in parts released in 8.2.0-alpha4 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: allow marking records as processed camunda/camunda
4 participants
@npepinpe @lenaschoenburg @koevskinikola @ChrisKujawa