Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize new JsonEventWriters when layouting in JsonLayout #246

Merged
merged 3 commits into from
Jul 2, 2024

Conversation

knittl
Copy link
Contributor

@knittl knittl commented Jun 29, 2024

The doLayout method used a single JsonEventWriter field in a non-thread-safe way. Consequently, writing logs from multiple threads would corrupt its buffer and send malformed JSON to Loki.

This patch initializes a new JsonEventWriter instance for every event that is layouted, similar to how Logback's PatternLayout is implemented. That way, each thread has its own writer instance (and buffer), guaranteeing that the buffer's content remains valid JSON. Allocated memory of the writer and buffer can be freed immediately after the method has returned.

Alternatives include but are not limited to:

  • Synchronizing the method, but this has turned out to be quite slow.
  • Using a ThreadLocal to allocate one writer instance per thread. This makes the lifetime of the writers a bit fuzzy and it is not clear when their memory will be freed. With long-lived thread pools, this will keep one writer (with buffer) per thread.
  • Using a pool of writers, allowing parallelism of up to N threads (could be made configurable via logback.xml) requiring a constant amount of memory overhead (higher parallelism incurs higher overhead).

Closes #245

@knittl knittl force-pushed the jsonlayout-threadsafe branch from 54ce92d to 79e9a5e Compare June 30, 2024 05:45
Copy link
Contributor

@nehaev nehaev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution, @knittl! Could you please address the comments above?

@knittl knittl force-pushed the jsonlayout-threadsafe branch from 4e16f32 to e37115b Compare July 2, 2024 06:15
knittl added 3 commits July 2, 2024 08:22
The doLayout method used a single JsonEventWriter field in a
non-thread-safe way. Consequently, writing logs from multiple threads
would corrupt its buffer and send malformed JSON to Loki.

This patch initializes a new JsonEventWriter instance for every event
that is layouted, similar to how Logback's `PatternLayout` is
implemented. That way, each thread has its own writer instance (and
buffer), guaranteeing that the buffer's content remains valid JSON.
Allocated memory of the writer and buffer can be freed immediately after
the method has returned.
@knittl knittl force-pushed the jsonlayout-threadsafe branch from e37115b to d037a5c Compare July 2, 2024 06:22
Copy link
Contributor

@nehaev nehaev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@nehaev nehaev merged commit 72eacb1 into loki4j:main Jul 2, 2024
5 of 7 checks passed
nehaev pushed a commit that referenced this pull request Jul 2, 2024
@knittl knittl changed the title Synchronize JsonLayout Initialize new JsonEventWriters when layouting in JsonLayout Jul 2, 2024
nehaev pushed a commit that referenced this pull request Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Corrupted JSON logs on concurrent write
2 participants