Cherry-pick #19423 to 7.x: [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions #19519

kaiyan-sheng · 2020-06-30T15:14:10Z

Cherry-pick of PR #19423 to 7.x branch. Original message:

What does this PR do?

This PR is to add a parameter aws_partition for metricbeat aws cloudwatch module, so metricbeat can support aws china and aws us-gov account.

Why is it important?

Because there are three partitions aws, aws-cn, aws-us-gov of aws regions, now metricbeat can only support aws global region, here is a hardcode.

beats/x-pack/metricbeat/module/aws/aws.go

Lines 95 to 100 in 5ba7026

 
 // Get IAM account name 

 awsConfig.Region = "us-east-1" 

 svcIam := iam.New(awscommon.EnrichAWSConfigWithEndpoint( 

 config.AWSConfig.Endpoint, "iam", "", awsConfig)) 

 req := svcIam.ListAccountAliasesRequest(&iam.ListAccountAliasesInput{})

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

add aws_partition param in aws module config

metricbeat.modules:
  - module: aws
    aws_partition: aws-cn
    period: 300s
    access_key_id: 
    secret_access_key:

run metricbeat , open debug log, then you can see it can retrive metrics from correct regions

Related issues

Closes [Metricbeat] Unable to get account id in China regions #18850

Use cases

no side affect
user can use metricbeat to get metrics from aws china and aws us-gov account except aws global account

Screenshots

aws global account, set aws_partition: aws in aws module config or remove the param

aws china account, set aws_partition: aws-cn in aws module config

…regions (#19423) * add params `aws_partition` to support other aws region: aws-cn, aws-us-gov (cherry picked from commit 1ea8c84)

elasticmachine · 2020-06-30T15:14:13Z

Pinging @elastic/integrations-platforms (Team:Platforms)

elasticmachine · 2020-06-30T15:36:27Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Expand to view the summary

Build stats

Build Cause: [Pull request #19519 updated]
Reason: Aborted from #6
Start Time: 2020-07-01T13:47:20.258+0000
Duration: 111 min 26 sec
Commit: 1d53741

Test stats 🧪

Test	Results
Failed	0
Passed	3680
Skipped	512
Total	4192

Log output

Expand to view the last 100 lines of log output

[2020-07-01T15:04:22.791Z] WARNING: You are using pip version 19.2.3, however version 20.1.1 is available.
[2020-07-01T15:04:22.791Z] You should consider upgrading via the 'python -m pip install --upgrade pip' command.
[2020-07-01T15:04:22.791Z] S..
[2020-07-01T15:04:22.791Z] [success] 74.30% test_base.Test.test_export_function: 0.3363s
[2020-07-01T15:04:22.791Z] [success] 25.70% test_base.Test.test_export_function_invalid_conf: 0.1163s
[2020-07-01T15:04:22.791Z] ----------------------------------------------------------------------
[2020-07-01T15:04:22.791Z] Ran 3 tests in 0.454s
[2020-07-01T15:04:22.791Z] 
[2020-07-01T15:04:22.791Z] OK (SKIP=1)
[2020-07-01T15:04:22.791Z] >> python test: Unit Testing Complete
[2020-07-01T15:04:22.861Z] Recording test results
[2020-07-01T15:04:25.241Z] Stashed 2 file(s)
[2020-07-01T15:04:25.251Z] Archiving artifacts
[2020-07-01T15:38:02.174Z] .......Sending interrupt signal to process
[2020-07-01T15:38:04.906Z] Sending interrupt signal to process
[2020-07-01T15:38:17.170Z] Click here to forcibly terminate running steps
[2020-07-01T15:38:19.823Z] failed to run compiled magefile: signal: terminated
[2020-07-01T15:38:19.831Z] script returned exit code 255
[2020-07-01T15:38:19.912Z] Recording test results
[2020-07-01T15:38:21.596Z] Stashed 3 file(s)
[2020-07-01T15:38:21.605Z] Archiving artifacts
[2020-07-01T15:38:22.409Z] + curl -sSLo codecov https://codecov.io/bash
[2020-07-01T15:38:22.669Z] + FILE=auditbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=filebeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=heartbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=libbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=metricbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=packetbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-01T15:38:22.669Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-01T15:38:22.669Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-01T15:38:24.052Z] Failed in branch Filebeat x-pack
[2020-07-01T15:38:24.207Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:24.522Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-01T15:38:24.533Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Lint
[2020-07-01T15:38:24.612Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-01T15:38:24.686Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-01T15:38:24.760Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-01T15:38:24.845Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-01T15:38:24.928Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-01T15:38:25.002Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-01T15:38:25.080Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-01T15:38:25.167Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-01T15:38:25.244Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-01T15:38:25.319Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-01T15:38:25.394Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-01T15:38:25.471Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.548Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.633Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-01T15:38:25.710Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-01T15:38:25.800Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-01T15:38:25.873Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-01T15:38:25.957Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-01T15:38:26.324Z] + cat
[2020-07-01T15:38:26.324Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-01T15:38:26.324Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-01T15:38:32.926Z] runbld>>> runbld started
[2020-07-01T15:38:32.926Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-01T15:38:33.868Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19519' in order of occurrence in the config (last value wins).
[2020-07-01T15:38:35.257Z] runbld>>> Debug logging enabled.
[2020-07-01T15:38:35.257Z] runbld>>> Storing result
[2020-07-01T15:38:35.257Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-01T15:38:35.257Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701153834-479C418D
[2020-07-01T15:38:35.257Z] runbld>>> Adding system facts.
[2020-07-01T15:38:36.201Z] runbld>>> Adding vcs info for the latest commit:  bc31a955805ee892da6a6ebd2b5ee3a322506299
[2020-07-01T15:38:36.201Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-01T15:38:36.201Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-01T15:38:36.201Z] Processing JUnit reports with runbld...
[2020-07-01T15:38:36.201Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-01T15:38:36.461Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-01T15:38:36.461Z] runbld>>> DURATION: 18ms
[2020-07-01T15:38:36.461Z] runbld>>> STDOUT: 40 bytes
[2020-07-01T15:38:36.461Z] runbld>>> STDERR: 49 bytes
[2020-07-01T15:38:36.461Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-01T15:38:36.461Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:37.403Z] runbld>>> Storing build metadata: 
[2020-07-01T15:38:37.403Z] runbld>>> Adding test report.
[2020-07-01T15:38:37.403Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519/src/github.com/elastic/beats
[2020-07-01T15:38:38.343Z] runbld>>> Found 30 test output files
[2020-07-01T15:38:39.283Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 4042 Skipped: 475
[2020-07-01T15:38:39.283Z] runbld>>> Storing result
[2020-07-01T15:38:39.283Z] runbld>>> FAILURES: 0
[2020-07-01T15:38:39.543Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-01T15:38:39.543Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701153834-479C418D
[2020-07-01T15:38:39.543Z] runbld>>> Email notification disabled by environment variable.
[2020-07-01T15:38:39.543Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-01T15:38:45.157Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19519
[2020-07-01T15:38:45.302Z] [INFO] getVaultSecret: Getting secrets
[2020-07-01T15:38:45.388Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-01T15:38:46.500Z] + chmod 755 generate-build-data.sh
[2020-07-01T15:38:46.500Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5 ABORTED 6685973
[2020-07-01T15:38:46.500Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5/steps/?limit=10000 -o steps-info.json
[2020-07-01T15:38:47.843Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19519/runs/5/tests/?status=FAILED -o tests-errors.json

kaiyan-sheng · 2020-06-30T16:09:03Z

CI failure should be fixed by #19521

kaiyan-sheng · 2020-06-30T21:09:23Z

run tests please

ycombinator

LGTM.

[MetricBeat] add param aws_partition to support aws-cn, aws-us-gov …

9fd31ae

…regions (#19423) * add params `aws_partition` to support other aws region: aws-cn, aws-us-gov (cherry picked from commit 1ea8c84)

kaiyan-sheng added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jun 30, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 30, 2020

kaiyan-sheng self-assigned this Jun 30, 2020

update go.mod

810887f

ycombinator approved these changes Jul 1, 2020

View reviewed changes

kaiyan-sheng added 2 commits July 1, 2020 07:46

Merge remote-tracking branch 'upstream/7.x' into backport_19423_7.x

1d53741

update changelog

f082f55

kaiyan-sheng merged commit 60ea971 into elastic:7.x Jul 1, 2020

kaiyan-sheng deleted the backport_19423_7.x branch July 1, 2020 16:20

zube bot added [zube]: Done and removed [zube]: In Review labels Jul 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #19423 to 7.x: [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions #19519

Cherry-pick #19423 to 7.x: [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions #19519

kaiyan-sheng commented Jun 30, 2020 •

edited by zube bot

Loading

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020 •

edited

Loading

Build stats

Test stats 🧪

kaiyan-sheng commented Jun 30, 2020

kaiyan-sheng commented Jun 30, 2020

ycombinator left a comment


	// Get IAM account name
	awsConfig.Region = "us-east-1"
	svcIam := iam.New(awscommon.EnrichAWSConfigWithEndpoint(
	config.AWSConfig.Endpoint, "iam", "", awsConfig))
	req := svcIam.ListAccountAliasesRequest(&iam.ListAccountAliasesInput{})

Cherry-pick #19423 to 7.x: [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions #19519

Cherry-pick #19423 to 7.x: [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions #19519

Conversation

kaiyan-sheng commented Jun 30, 2020 • edited by zube bot Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020 • edited Loading

❕ Build Aborted

Build stats

Test stats 🧪

Log output

kaiyan-sheng commented Jun 30, 2020

kaiyan-sheng commented Jun 30, 2020

ycombinator left a comment

Choose a reason for hiding this comment

Cherry-pick #19423 to 7.x: [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions #19519

Cherry-pick #19423 to 7.x: [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions #19519

kaiyan-sheng commented Jun 30, 2020 •

edited by zube bot

Loading

elasticmachine commented Jun 30, 2020 •

edited

Loading