Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON stringify unmapped fields #2459

Open
2 tasks
kfirpeled opened this issue Aug 28, 2024 · 0 comments
Open
2 tasks

JSON stringify unmapped fields #2459

kfirpeled opened this issue Aug 28, 2024 · 0 comments
Labels
enhancement New feature or request Team:Cloud Security Cloud Security team related

Comments

@kfirpeled
Copy link
Contributor

kfirpeled commented Aug 28, 2024

Motivation

Today we don't map resource.raw, result.evidence and result.expected because we these are sent as part of the source, if we will apply dynamic mappings on these fields we reach to the default max limit of number of fields being mapped and we experienced mapping collision which drops the document from being indexed.

So our solution at this time was not to map them, so we at least be able to present these values to the user because they are part of _source.

However, there could be a better solution for this problem which is to map these fields as text and their value will be a stringified JSON

  1. It keeps the same behavior from user perspective as it is today
  2. This is our only use of _source in kibana, which blocks us from using synthetic source which optimizes the storage of these indices
  3. It will allow the user (and us) to query these fields as text which is an improvement
  4. In the future, es|ql might support querying these fields with jq kind of query language (reference)

Definition of done

  • Add mapping as text to cloud_security_package to the fields mentioned above
  • Cloudbeat should send this values as a valid stringified JSON

Out of scope

Related tasks/epics

@kfirpeled kfirpeled added enhancement New feature or request Team:Cloud Security Cloud Security team related labels Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Team:Cloud Security Cloud Security team related
Projects
None yet
Development

No branches or pull requests

1 participant