Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

Merged
merged 3 commits into from
Oct 14, 2020
Merged

Move s3_daily_storage and s3_request metricsets to use cloudwatch input #21703

merged 3 commits into from
Oct 14, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Oct 9, 2020

What does this PR do?

This PR is to move s3_daily_storage and s3_request metricsets to use cloudwatch input. In order to make it not a breaking change, rename processor is used in both manifest.yml files.

Why is it important?

This way we only need to maintain code in cloudwatch input instead of maintaining code in each metricset.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 9, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Oct 9, 2020
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. review Team:Platforms Label for the Integrations - Platforms team labels Oct 9, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 9, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 9, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21703 updated]

  • Start Time: 2020-10-13T20:30:51.104+0000

  • Duration: 65 min 4 sec

Test stats 🧪

Test Results
Failed 0
Passed 2629
Skipped 597
Total 3226

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are data.json files updated?

"s3": {
"metrics": {
"NumberOfObjects": {
"avg": 57828
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is rename not being applied here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's applied when I run metricbeat for testing but not applied when I tried to create the event for data.json. Is there any way I can make the rename processor work with TestData function?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh ok, maybe processors are not initialized on testing, if that is the case we should fix that, but not on this PR. Could you please open an issue about that?

But interestingly in the iis module there is a rename that seems to be applied 🤔

- rename:
ignore_missing: true
fields:
- from: "iis.website.instance"
to: "iis.website.name"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. @narph I saw you generated the data.json file for iis website. Does the field iis.website.name generated automatically by TestData function? For my case, TestData function ignores the rename processor.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am setting the iis.website.name using the rename processor inside the manifest.yml file

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but we should check why processors are not being executed on TestData.

"s3": {
"metrics": {
"NumberOfObjects": {
"avg": 57828
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh ok, maybe processors are not initialized on testing, if that is the case we should fix that, but not on this PR. Could you please open an issue about that?

But interestingly in the iis module there is a rename that seems to be applied 🤔

- rename:
ignore_missing: true
fields:
- from: "iis.website.instance"
to: "iis.website.name"

@kaiyan-sheng kaiyan-sheng merged commit 99b4172 into elastic:master Oct 14, 2020
@kaiyan-sheng kaiyan-sheng deleted the s3_metriccsets branch October 14, 2020 19:30
@kaiyan-sheng kaiyan-sheng added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Oct 14, 2020
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
* upstream/master: (29 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
* upstream/master: (216 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
* upstream/master: (156 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 15, 2020
* upstream/master: (156 commits)
  [CI] Enable build stage for arm architecture (elastic#21284)
  [BUILD][CI] fetch dependencies with retry (elastic#21614)
  Add proxy metricset (elastic#21751)
  [Filebeat][httpjson] Fix date_cursor validation (elastic#21756)
  Update endpoint-related terminology in Elastic Agent docs (elastic#21458)
  Move s3_daily_storage and s3_request metricsets to use cloudwatch input (elastic#21703)
  [Agent] Remove heartbeat from bundled beats in agent. (elastic#21602)
  [Elastic Agent] Don't perform install until after enroll (elastic#21772)
  [Elastic Agent] Update path from policy change for Kibana connection. (elastic#21804)
  Change cloud.provider from googlecloud to gcp (elastic#21775)
  Fix stress test package list (elastic#21719)
  [Elastic Agent] Remove the service installer scripts from packaging (elastic#21694)
  [CI] Support Windows-2012 in pipeline 2.0 (elastic#21338)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  ...
kaiyan-sheng added a commit that referenced this pull request Oct 15, 2020
…csets to use cloudwatch input (#21821)

(cherry picked from commit 99b4172)

* backport #21828
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review Team:Platforms Label for the Integrations - Platforms team v7.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants