Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automatic retries and exponential backoff to Filebeat httpjson input #18956

Merged
merged 9 commits into from
Jun 19, 2020
Merged

Add automatic retries and exponential backoff to Filebeat httpjson input #18956

merged 9 commits into from
Jun 19, 2020

Conversation

alakahakai
Copy link

@alakahakai alakahakai commented Jun 3, 2020

Add automatic retries and exponential backoff to Filebeat httpjson input. If an HTTP error is returned by the client, e.g., connection errors, or if a 500-range response code is received (except 501), then a retry is automatically invoked after a defined wait period.

go test result:

=== RUN   TestProviderCanonical
--- PASS: TestProviderCanonical (0.00s)
=== RUN   TestGetProviderIsCanonical
--- PASS: TestGetProviderIsCanonical (0.00s)
=== RUN   TestIsEnabled
--- PASS: TestIsEnabled (0.00s)
=== RUN   TestGetTokenURL
--- PASS: TestGetTokenURL (0.00s)
=== RUN   TestGetTokenURLWithAzure
--- PASS: TestGetTokenURLWithAzure (0.00s)
=== RUN   TestGetEndpointParams
--- PASS: TestGetEndpointParams (0.00s)
=== RUN   TestGetEndpointParamsWithAzure
--- PASS: TestGetEndpointParamsWithAzure (0.00s)
=== RUN   TestConfigValidationCase1
--- PASS: TestConfigValidationCase1 (0.00s)
=== RUN   TestConfigValidationCase2
--- PASS: TestConfigValidationCase2 (0.00s)
=== RUN   TestConfigValidationCase3
--- PASS: TestConfigValidationCase3 (0.00s)
=== RUN   TestConfigValidationCase4
--- PASS: TestConfigValidationCase4 (0.00s)
=== RUN   TestConfigValidationCase5
--- PASS: TestConfigValidationCase5 (0.00s)
=== RUN   TestConfigValidationCase6
--- PASS: TestConfigValidationCase6 (0.00s)
=== RUN   TestConfigValidationCase7
--- PASS: TestConfigValidationCase7 (0.00s)
=== RUN   TestConfigOauth2Validation
=== RUN   TestConfigOauth2Validation/can't_set_oauth2_and_api_key_together
=== RUN   TestConfigOauth2Validation/can_set_oauth2_and_api_key_together_if_oauth2_is_disabled
=== RUN   TestConfigOauth2Validation/can't_set_oauth2_and_authentication_scheme
=== RUN   TestConfigOauth2Validation/token_url_and_client_credentials_must_be_set
=== RUN   TestConfigOauth2Validation/must_fail_with_an_unknown_provider
=== RUN   TestConfigOauth2Validation/azure_must_have_either_tenant_id_or_token_url
=== RUN   TestConfigOauth2Validation/azure_must_have_only_one_of_token_url_and_tenant_id
=== RUN   TestConfigOauth2Validation/azure_must_have_client_credentials_set
=== RUN   TestConfigOauth2Validation/azure_config_is_valid
=== RUN   TestConfigOauth2Validation/google_can't_have_token_url_or_client_credentials_set
=== RUN   TestConfigOauth2Validation/google_must_fail_if_no_ADC_available
=== RUN   TestConfigOauth2Validation/google_must_fail_if_credentials_file_not_found
=== RUN   TestConfigOauth2Validation/google_must_fail_if_ADC_is_wrongly_set
=== RUN   TestConfigOauth2Validation/google_must_work_if_ADC_is_set_up
=== RUN   TestConfigOauth2Validation/google_must_work_if_credentials_file_is_correct
=== RUN   TestConfigOauth2Validation/google_must_work_if_jwt_file_is_correct
=== RUN   TestConfigOauth2Validation/google_must_work_if_credentials_json_is_correct
=== RUN   TestConfigOauth2Validation/google_must_fail_if_credentials_json_is_not_a_valid_JSON
=== RUN   TestConfigOauth2Validation/google_must_fail_if_the_provided_credentials_file_is_not_a_valid_JSON
--- PASS: TestConfigOauth2Validation (0.00s)
    --- PASS: TestConfigOauth2Validation/can't_set_oauth2_and_api_key_together (0.00s)
    --- PASS: TestConfigOauth2Validation/can_set_oauth2_and_api_key_together_if_oauth2_is_disabled (0.00s)
    --- PASS: TestConfigOauth2Validation/can't_set_oauth2_and_authentication_scheme (0.00s)
    --- PASS: TestConfigOauth2Validation/token_url_and_client_credentials_must_be_set (0.00s)
    --- PASS: TestConfigOauth2Validation/must_fail_with_an_unknown_provider (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_either_tenant_id_or_token_url (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_only_one_of_token_url_and_tenant_id (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_must_have_client_credentials_set (0.00s)
    --- PASS: TestConfigOauth2Validation/azure_config_is_valid (0.00s)
    --- PASS: TestConfigOauth2Validation/google_can't_have_token_url_or_client_credentials_set (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_no_ADC_available (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_credentials_file_not_found (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_ADC_is_wrongly_set (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_ADC_is_set_up (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_credentials_file_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_jwt_file_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_work_if_credentials_json_is_correct (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_credentials_json_is_not_a_valid_JSON (0.00s)
    --- PASS: TestConfigOauth2Validation/google_must_fail_if_the_provided_credentials_file_is_not_a_valid_JSON (0.00s)
=== RUN   TestGetNextLinkFromHeader
--- PASS: TestGetNextLinkFromHeader (0.00s)
=== RUN   TestCreateRequestInfoFromBody
--- PASS: TestCreateRequestInfoFromBody (0.00s)
=== RUN   TestGetRateLimitCase1
--- PASS: TestGetRateLimitCase1 (0.00s)
=== RUN   TestGetRateLimitCase2
--- PASS: TestGetRateLimitCase2 (0.00s)
=== RUN   TestGetRateLimitCase3
--- PASS: TestGetRateLimitCase3 (0.00s)
=== RUN   TestGET
2020-06-16T20:53:14.947-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53858"}
2020-06-16T20:53:14.947-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53858
--- PASS: TestGET (0.00s)
=== RUN   TestGetHTTPS
2020-06-16T20:53:14.948-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53860"}
2020-06-16T20:53:14.948-0700	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-16T20:53:14.948-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53860
--- PASS: TestGetHTTPS (0.00s)
=== RUN   TestRateLimitRetry
2020-06-16T20:53:14.949-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53862"}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53862
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson]	httpjson/input.go:325	HTTP request failed	{"url": "http://127.0.0.1:53862", "http.response.status_code": 429, "http.response.body": ""}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson]	httpjson/input.go:262	Rate Limit: No need to apply rate limit.	{"url": "http://127.0.0.1:53862"}
2020-06-16T20:53:14.949-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53862
--- PASS: TestRateLimitRetry (0.00s)
=== RUN   TestErrorRetry
2020-06-16T20:53:14.950-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53865"}
2020-06-16T20:53:14.950-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53865
2020-06-16T20:53:14.950-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:600	[DEBUG] GET http://127.0.0.1:53865 (status: 506): retrying in 1s (5 left)
2020-06-16T20:53:15.951-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:600	[DEBUG] GET http://127.0.0.1:53865 (status: 508): retrying in 2s (4 left)
--- PASS: TestErrorRetry (3.01s)
=== RUN   TestArrayResponse
2020-06-16T20:53:17.955-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53900"}
2020-06-16T20:53:17.956-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53900
--- PASS: TestArrayResponse (0.00s)
=== RUN   TestPOST
2020-06-16T20:53:17.957-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53902"}
2020-06-16T20:53:17.957-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53902
--- PASS: TestPOST (0.00s)
=== RUN   TestRepeatedPOST
2020-06-16T20:53:17.958-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:17.958-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:20.963-0700	INFO	[httpjson]	httpjson/input.go:438	Process another repeated request.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:20.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:23.962-0700	INFO	[httpjson]	httpjson/input.go:438	Process another repeated request.	{"url": "http://127.0.0.1:53904"}
2020-06-16T20:53:23.962-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53904
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:435	Context done.	{"url": "http://127.0.0.1:53904"}
--- PASS: TestRepeatedPOST (6.01s)
=== RUN   TestRunStop
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:133	httpjson input worker has started.	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53970
2020-06-16T20:53:23.963-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:551	[ERR] GET http://127.0.0.1:53970 request failed: Get http://127.0.0.1:53970: context canceled
2020-06-16T20:53:23.963-0700	ERROR	[httpjson]	httpjson/input.go:138	failed to execute http client.Do: Get http://127.0.0.1:53970: context canceled	{"url": "http://127.0.0.1:53970"}
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:139	httpjson input worker has stopped.	{"url": "http://127.0.0.1:53970"}
--- PASS: TestRunStop (0.00s)
=== RUN   TestOAuth2
2020-06-16T20:53:23.963-0700	INFO	[httpjson]	httpjson/input.go:123	Initialized httpjson input.	{"url": "http://127.0.0.1:53972"}
2020-06-16T20:53:23.964-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] POST http://127.0.0.1:53971
2020-06-16T20:53:23.964-0700	DEBUG	[httpjson.retryablehttp]	go-retryablehttp@v0.6.6/client.go:502	[DEBUG] GET http://127.0.0.1:53972
--- PASS: TestOAuth2 (0.00s)
PASS
ok  	github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson	9.214s
(base)

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 3, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-06-18T15:15:42.525+0000

  • Duration: 124 min 41 sec

Test stats 🧪

Test Results
Failed 0
Passed 8464
Skipped 1553
Total 10017

Steps errors

Expand to view the steps failures

  • Name: Make -C libbeat testsuite
    • Description: make -C libbeat testsuite

    • Duration: 6 min 7 sec

    • Start Time: 2020-06-18T15:39:31.209+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution.cancel(TimeoutStepExecution.java:165)
[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution.access$100(TimeoutStepExecution.java:38)
[2020-06-18T17:18:49.067Z] 	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$1.run(TimeoutStepExecution.java:139)
[2020-06-18T17:18:49.067Z] 	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-06-18T17:18:49.067Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-06-18T17:18:49.067Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-06-18T17:18:49.067Z] 
[2020-06-18T17:18:49.083Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-18T17:18:49.267Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Lint
[2020-06-18T17:18:49.747Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-18T17:18:50.074Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-18T17:18:50.342Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-18T17:18:50.827Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-18T17:18:51.350Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-18T17:18:51.718Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-18T17:18:52.066Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-18T17:18:52.456Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Libbeat-oss
[2020-06-18T17:18:52.742Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-06-18T17:18:53.092Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-18T17:18:53.590Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-18T17:18:53.996Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-18T17:18:54.467Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-18T17:18:54.841Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-18T17:18:55.109Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-18T17:18:55.339Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-18T17:18:55.661Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-18T17:18:55.908Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-18T17:18:56.187Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-18T17:18:56.551Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-18T17:18:56.775Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-18T17:18:56.956Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-18T17:18:57.239Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-18T17:18:57.441Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-18T17:18:57.659Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-18T17:18:57.848Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-oss
[2020-06-18T17:18:58.060Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-18T17:18:58.245Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-18T17:18:58.463Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-18T17:18:58.697Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-18T17:18:58.999Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-18T17:18:59.400Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-18T17:18:59.607Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-18T17:18:59.769Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-18T17:19:00.030Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-18T17:19:00.197Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-18T17:19:00.372Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-18T17:19:00.922Z] + cat
[2020-06-18T17:19:00.922Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-18T17:19:00.922Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-18T17:19:07.692Z] runbld>>> runbld started
[2020-06-18T17:19:07.692Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-06-18T17:19:08.663Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18956' in order of occurrence in the config (last value wins).
[2020-06-18T17:19:10.102Z] runbld>>> Debug logging enabled.
[2020-06-18T17:19:10.102Z] runbld>>> Storing result
[2020-06-18T17:19:10.102Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-18T17:19:10.102Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200618171909-160AC007
[2020-06-18T17:19:10.102Z] runbld>>> Adding system facts.
[2020-06-18T17:19:10.692Z] runbld>>> Adding vcs info for the latest commit:  4657cbfc603647101b87550a45d0124e94ccbf92
[2020-06-18T17:19:10.692Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-18T17:19:10.971Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-18T17:19:10.971Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-18T17:19:10.971Z] Processing JUnit reports with runbld...
[2020-06-18T17:19:11.262Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-18T17:19:11.262Z] runbld>>> DURATION: 29ms
[2020-06-18T17:19:11.262Z] runbld>>> STDOUT: 40 bytes
[2020-06-18T17:19:11.262Z] runbld>>> STDERR: 49 bytes
[2020-06-18T17:19:11.262Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-18T17:19:11.262Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats
[2020-06-18T17:19:12.691Z] runbld>>> Storing build metadata: 
[2020-06-18T17:19:12.691Z] runbld>>> Adding test report.
[2020-06-18T17:19:12.691Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats
[2020-06-18T17:19:14.103Z] runbld>>> Found 105 test output files
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-18T17:19:15.018Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-18T17:19:15.648Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-18T17:19:15.648Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-18T17:19:16.148Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 9867 Skipped: 1316
[2020-06-18T17:19:16.148Z] runbld>>> Storing result
[2020-06-18T17:19:16.148Z] runbld>>> FAILURES: 0
[2020-06-18T17:19:16.415Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-18T17:19:16.415Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200618171909-160AC007
[2020-06-18T17:19:16.415Z] runbld>>> Email notification disabled by environment variable.
[2020-06-18T17:19:16.415Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-18T17:19:22.124Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18956
[2020-06-18T17:19:22.285Z] [INFO] getVaultSecret: Getting secrets
[2020-06-18T17:19:22.358Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-18T17:19:23.479Z] + chmod 755 generate-build-data.sh
[2020-06-18T17:19:23.479Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18 ABORTED 7420646
[2020-06-18T17:19:23.479Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/steps/?limit=10000 -o steps-info.json
[2020-06-18T17:19:25.962Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/tests/?status=FAILED -o tests-errors.json
[2020-06-18T17:19:27.305Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18956/runs/18/log/ -o pipeline-log.txt

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@alakahakai
Copy link
Author

jenkins, test this

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please update the documentation for the httpjson input to include these new options.

x-pack/filebeat/input/httpjson/config.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/httpjson/input.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/httpjson/input.go Outdated Show resolved Hide resolved
@andrewkroh andrewkroh requested a review from marc-gr June 8, 2020 23:03
@alakahakai
Copy link
Author

@andrewkroh Please review again. thanks

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alakahakai alakahakai merged commit 0dcb3df into elastic:master Jun 19, 2020
@andrewkroh andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Jun 30, 2020
@andrewkroh
Copy link
Member

andrewkroh commented Jun 30, 2020

Please backport this to 7.8 and 7.x.

marc-gr pushed a commit to marc-gr/beats that referenced this pull request Jul 13, 2020
…put (elastic#18956)

- Add automatic retries and exponential backoff to Filebeat httpjson input (elastic#18956)
- Add test cases
- Support the new OAuth2 functionality

(cherry picked from commit 0dcb3df)
marc-gr added a commit that referenced this pull request Jul 13, 2020
…put (#18956) (#19538)

- Add automatic retries and exponential backoff to Filebeat httpjson input (#18956)
- Add test cases
- Support the new OAuth2 functionality

(cherry picked from commit 0dcb3df)

Co-authored-by: Lei Qiu <lei.qiu@elastic.co>
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…put (elastic#18956)

- Add automatic retries and exponential backoff to Filebeat httpjson input (elastic#18956)
- Add test cases
- Support the new OAuth2 functionality
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants