Option for dropping the log body if log was parsed

[a-thaler]

Description
A LogPipeline tries to parse a log payload if the payload is in a structured JSON format and then enriches the log record with the parsed entries using the same attribute keys as in the JSON, see also https://kyma-project.io/docs/kyma/latest/01-overview/telemetry/telemetry-02-logs#kubernetes-filter-json-parser

So if the payload is not in JSON you will have the log attribute with the payload.
If the payload is in JSON you will have the log attribute with the payload and additional all parsed JSON root attributes, including the actual log message.

With that approach, you will have the content of the payload actually duplicated for the advantage of having the same log attribute consistently available across all log records.

If that behaviour is not wanted in order to save storage/bandwith, there should be a new flag in the LogPipeline API to disable the duplication, for example like:

spec:
  output:
      http:
        keepBody: true

To stay compatible, the flag should be introduced first being enabled, so that all existing clusters will have it persisted. In a follow-up ticket the flag will get switched to be disabled by default.

Criterias

• As a user I have an option per pipeline to disable keeping the log body
• When setting the flag, the log body will not be part of the log data anymore in case it was successfully parsed as JSON document
• The flag is documented
• The flag is available in the dashboard UI

Reasons
Log payloads are being duplicated when they are in JSON. That causes unneeded bandwith/storage/performance increase and leads to polluted log records.

Attachments

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[hisarbalik]

Decisions:

The new LogPipeline attribute was named keepRawBody to clarify the purpose.

The new attribute moved from the Output section to the Input section to make it possible for the custom output type to change the configuration if necessary.