[Filebeat] Integration tests in CI for AWS-S3 input #27491
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
|
Pinging @elastic/integrations (Team:Integrations) |
botelastic
bot
removed
the
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
|
there was an error in the directory name and terraform apply failed: anyway the CI status was green cc @v1v |
jsoriano
reviewed
Aug 19, 2021
x-pack/filebeat/Jenkinsfile.yml
Outdated
| withModule: true ## run the ITs only if the changeset affects a specific module. | ||
| dirs: ## run the cloud tests for the given modules. | ||
| - "x-pack/filebeat/input/awss3/_meta/terraform" # needed by startCloudTestEnv in order to apply terraform | ||
| - "x-pack/filebeat/input/awss3" # the actual dir where tests reside |
There was a problem hiding this comment.
This looks a bit hacky, I would prefer if we don't need to add both directories.
I see some options:
- Add a
terraformDirconfig to thiscloudstep, so we can explicitly configure where is the terraform scenario and where the tests. This would have the advantage of allowing to use the same scenario for tests residing in multiple scenarios. - Add a terraform file where the tests reside, so we don't have to add both directories. This terraform file could use
_meta/terraformas a module. - odify
startCloudTestEnvto use./_meta/terraformif there are no terraform files in the
fixes: This version of Terraform has an outdated GPG key and is unable to verify new provider releases
botelastic
bot
added
the
Team:Automation
|
v1v
reviewed
Aug 23, 2021
| @@ -876,6 +876,7 @@ def startCloudTestEnv(Map args = [:]) { | |||
| // If it failed then cleanup without failing the build | |||
| sh(label: 'Terraform Cleanup', script: ".ci/scripts/terraform-cleanup.sh ${folder}", returnStatus: true) | |||
| } | |||
| error('startCloudTestEnv: terraform apply failed.') | |||
There was a problem hiding this comment.
Enforce the error is reported after the catch closure
|
not sure it is correct: I can see that if I can define |
|
@aspacca , we can syncup if needed, please poke me. 5bf9090 is half done, there are a couple of changes to be done:
|
cachedout
approved these changes
Aug 26, 2021
v1v
reviewed
Aug 26, 2021
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
|
@v1v everything seems fine: we don't need to introduce As far as I can see only problem left is that the step is triggered when github label It would be valuable in my opinion to add something like |
v1v
reviewed
Sep 2, 2021
Comment on lines
+835
to
+838
| // Allow AWS credentials when the build was configured to do so with: | ||
| // - the cloudtests build parameters | ||
| // - the aws github label | ||
| if (params.allCloudTests || params.awsCloudTests || matchesPrLabel(label: 'aws')) { |
There was a problem hiding this comment.
This should help to configure the AWS credentials when the aws github label was assigned to the Pull Request as explained in #27491 (comment)
|
There are some failures when applying the terraform, see the below screenshot
As far as I see it's related to this terraform resource: No idea whether it's a temporary glitch or something else Let me rerun the build again |
|
/test |
|
/test |
|
|
|
/test |
|
@v1v all good, let's merge? |
aspacca
added
backport-v7.15.0
mergify bot
pushed a commit
that referenced
this pull request
Sep 6, 2021
* [Filebeat] Integration tests in CI for AWS-S3 input (cherry picked from commit 9d4597a)
mergify bot
pushed a commit
that referenced
this pull request
Sep 6, 2021
* [Filebeat] Integration tests in CI for AWS-S3 input (cherry picked from commit 9d4597a)
aspacca
pushed a commit
that referenced
this pull request
Sep 6, 2021
aspacca
pushed a commit
that referenced
this pull request
Sep 6, 2021
mdelapenya
added a commit
to mdelapenya/beats
that referenced
this pull request
Sep 9, 2021
* master: (39 commits) [Heartbeat] Move JSON tests from python->go (elastic#27816) docs: simplify permissions for Dockerfile COPY (elastic#27754) Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789) [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774) warn log entry and no validation failure when both queue_url and buck… (elastic#27612) libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178) [Heartbeat] Add httpcommon options to ZipURL (elastic#27699) Add a header round tripper option to httpcommon (elastic#27509) [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779) Rename dashboards according to module.yml files for master (elastic#27749) Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726) Accept syslog dates with leading 0 (elastic#27775) [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727) [Filebeat] Threatintel compatibility updates (elastic#27323) Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707) [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491) Fix flakyness of TestFilestreamEmptyLine (elastic#27705) [Filebeat] kafka v2 using parsers (elastic#27335) Update Kafka version parsing / supported range (elastic#27720) Update Sarama to 1.29.1 (elastic#27717) ...
Icedroid
pushed a commit
to Icedroid/beats
that referenced
this pull request
Nov 1, 2021
* [Filebeat] Integration tests in CI for AWS-S3 input
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhancement
What does this PR do?
#27199 introduced new integration tests that rely on terraform (https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/awss3/input_integration_test.go)
We will run them in CI, similarly to what happens for Metricbeat (https://github.com/elastic/beats/blob/master/x-pack/metricbeat/Jenkinsfile.yml#L35-L47)
Why is it important?
Avoiding regression
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
Related issues
Closes #27469
Use cases
Screenshots
Logs