[AWS][Kinesis] Add dimension fields for TSDB support #5891
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,6 +5,7 @@ | |
| type: group | ||
| fields: | ||
| - name: StreamName | ||
| dimension: true | ||
|
There was a problem hiding this comment. Please add here the reason for adding the specific field as a dimension field.Adding the reason is among the best practices for TSDB enablement. There was a problem hiding this comment. By "here" you mean in the manifest @agithomas? It's explained in "Details" in the PR description. There was a problem hiding this comment. It can be added as the inline comment. Reference There was a problem hiding this comment. I have certain thoughts around a better way to handle aws dimensions. We have a length limitation in dimension and AWS permits 30 dimensions If all 30 names and values are fully used to max limit, the 32KB dimension field length limitation would reach. Can we have fingerprint processor applied on all aws dimensions and use the new field (having fingerprint) used as a dimension field? There was a problem hiding this comment. Ok, I will try to add it to the document without being too confused. I don't understand the other part though, there are only 3 fields set as dimension, why would we have the need for a processor? @agithomas There was a problem hiding this comment. Please validate the proposal against
There was a problem hiding this comment. Sorry, I don't understand: is there a reason to create a new dimension using the There was a problem hiding this comment. The advantages are mentioned here. There was a problem hiding this comment. The above proposal is based only on the convenience of TSDB. Please compare and choose the best approach. There was a problem hiding this comment. @agithomas from my understanding, aws dimensions shouldn't be a concern at least for this data_stream, since we set There was a problem hiding this comment. @constanca-m how this name is defined? as I see this name is not set in configuration what if 2 kinesis data_stream in the same region will be created? There was a problem hiding this comment. When we connect to the AWS, it fetches data from the existent data streams. We don't create any data stream when we add the integration. The stream name is unique per region, and since region is a dimension, it shouldn't be a problem @tetianakravchenko |
||
| type: keyword | ||
| description: The name of the Kinesis stream. All available statistics are filtered by StreamName. | ||
| - name: kinesis.metrics | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here was a suggestion to align on list of fields - elastic/ecs#2172 and was suggested to use
cloud.project.id. Is this field available for AWS?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't checked, but if we added that as a dimension it would be redundant, as we don't really need it set as a dimension @tetianakravchenko
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we checked with @agithomas that
cloud.project.iddoes not exist for all cloud providers, it only present for gcp.