-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703
Conversation
Pinging @elastic/integrations-platforms (Team:Platforms) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are data.json
files updated?
"s3": { | ||
"metrics": { | ||
"NumberOfObjects": { | ||
"avg": 57828 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is rename not being applied here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah it's applied when I run metricbeat for testing but not applied when I tried to create the event for data.json. Is there any way I can make the rename processor work with TestData
function?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh ok, maybe processors are not initialized on testing, if that is the case we should fix that, but not on this PR. Could you please open an issue about that?
But interestingly in the iis module there is a rename that seems to be applied 🤔
beats/x-pack/metricbeat/module/iis/website/manifest.yml
Lines 36 to 40 in 3e5a167
- rename: | |
ignore_missing: true | |
fields: | |
- from: "iis.website.instance" | |
to: "iis.website.name" |
"name": "Default Web Site" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. @narph I saw you generated the data.json file for iis website. Does the field iis.website.name
generated automatically by TestData
function? For my case, TestData
function ignores the rename processor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am setting the iis.website.name
using the rename processor inside the manifest.yml file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, but we should check why processors are not being executed on TestData
.
"s3": { | ||
"metrics": { | ||
"NumberOfObjects": { | ||
"avg": 57828 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh ok, maybe processors are not initialized on testing, if that is the case we should fix that, but not on this PR. Could you please open an issue about that?
But interestingly in the iis module there is a rename that seems to be applied 🤔
beats/x-pack/metricbeat/module/iis/website/manifest.yml
Lines 36 to 40 in 3e5a167
- rename: | |
ignore_missing: true | |
fields: | |
- from: "iis.website.instance" | |
to: "iis.website.name" |
"name": "Default Web Site" |
* upstream/master: (29 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
* upstream/master: (216 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
* upstream/master: (156 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
* upstream/master: (156 commits) [CI] Enable build stage for arm architecture (elastic#21284) [BUILD][CI] fetch dependencies with retry (elastic#21614) Add proxy metricset (elastic#21751) [Filebeat][httpjson] Fix date_cursor validation (elastic#21756) Update endpoint-related terminology in Elastic Agent docs (elastic#21458) Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703) [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602) [Elastic Agent] Don't perform install until after enroll (elastic#21772) [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804) Change cloud.provider from googlecloud to gcp (elastic#21775) Fix stress test package list (elastic#21719) [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694) [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338) Fix non-windows fields on system/filesystem (elastic#21758) disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750) Use badger code from upstream repository (elastic#21705) Disable writes sync in persistent cache (elastic#21754) Make API address and Shard ID required in Cloud Foundry settings (elastic#21759) [CI] Support skip-ci label (elastic#21377) Increase recommended memory when deploying in Cloud foundry (elastic#21755) ...
What does this PR do?
This PR is to move s3_daily_storage and s3_request metricsets to use cloudwatch input. In order to make it not a breaking change,
rename
processor is used in bothmanifest.yml
files.Why is it important?
This way we only need to maintain code in cloudwatch input instead of maintaining code in each metricset.
Checklist
I have commented my code, particularly in hard-to-understand areasI have made corresponding changes to the documentationI have made corresponding change to the default configuration filesCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.