Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat Azure - Add missing var definitions in the manifest files #16468

Merged
merged 7 commits into from
Feb 25, 2020

Conversation

narph
Copy link
Contributor

@narph narph commented Feb 20, 2020

Should fix #16270.

Due to new checks on the configuration file keys in v7.6.0, when running:

filebeat setup --module azure

users get:

Exiting: Error getting config for fileset azure/activitylogs: Error interpreting the template of the input: template: text:2:22: executing "text" at <.connection_string>: map has no entry for key "connection_string"

due to missing to define some of the vars in the manifest file.
This PR does this.

@narph narph requested a review from a team as a code owner February 20, 2020 19:33
@narph narph self-assigned this Feb 20, 2020
@narph narph added Filebeat Filebeat Team:Integrations Label for the Integrations team Team:Platforms Label for the Integrations - Platforms team [zube]: In Review labels Feb 20, 2020
- name: connection_string
- name: storage_account
- name: storage_account_key

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: extra line here.

@exekias
Copy link
Contributor

exekias commented Feb 20, 2020

Thank you for fixing this so quickly! I think we should have a system test to make sure this doesn’t happen again. This would probably caught it, right? https://github.com/elastic/beats/blob/master/filebeat/tests/system/test_setup.py, probably we can load it also from x-pack Filebeat.

Please update the description, even if the original issue has lots of details, it’s nice to have a good summary here explaining the change.

Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@narph Under configs in each azure fileset like https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/azure/activitylogs/config/azure-eventhub.yml, the parameters that are not required (like connection_string, storage_account) needs to be checked to see if they are defined first. For example,

{{ if .connection_string }}
connection_string: {{ .connection_string }}
{{ end }}

Please see Andrew's comment in a similar PR: #15656 (comment)

@andrewkroh
Copy link
Member

andrewkroh commented Feb 21, 2020

@exekias I figured that the existing module tests would have caught this case. But what I realized is that because we only use the file input in tests that the separate config template azure-eventhub.yml is never exercised in tests. So we'd need a test that minimally instantiates the module with that input. Just enough to verify that it can start.

Or maybe the the fileset package can expose an easy to use way to evaluate a module's config templates. Then you could run assertions on the generated config output based on various input settings.

@exekias
Copy link
Contributor

exekias commented Feb 25, 2020

SGTM, let's get this in for the next release and work on the testing part later

@narph
Copy link
Contributor Author

narph commented Feb 25, 2020

@narph Under configs in each azure fileset like https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/azure/activitylogs/config/azure-eventhub.yml, the parameters that are not required (like connection_string, storage_account) needs to be checked to see if they are defined first. For example,

{{ if .connection_string }}
connection_string: {{ .connection_string }}
{{ end }}

Please see Andrew's comment in a similar PR: #15656 (comment)

@kaiyan-sheng , all the 3 params are required in the azure module for each fileset in this case. Two of these options might not be mandatory in the azure-eventhub input in the future so I would not add the {{ if }} in order for the template to fail.

@narph
Copy link
Contributor Author

narph commented Feb 25, 2020

jenkins test this

@exekias exekias added the needs_backport PR is waiting to be backported to other branches. label Feb 25, 2020
@narph
Copy link
Contributor Author

narph commented Feb 25, 2020

jenkins test this

@narph narph merged commit da3584d into elastic:master Feb 25, 2020
@narph narph deleted the azure-declare-var branch February 25, 2020 15:21
narph added a commit to narph/beats that referenced this pull request Feb 25, 2020
…lastic#16468)

* Add vars

* changelog

* temp

* line

* update

* take test out

(cherry picked from commit da3584d)
narph added a commit to narph/beats that referenced this pull request Feb 25, 2020
…lastic#16468)

* Add vars

* changelog

* temp

* line

* update

* take test out

(cherry picked from commit da3584d)
narph added a commit that referenced this pull request Feb 25, 2020
…t files (#16573)

* Filebeat Azure - Add missing var definitions in the manifest files (#16468)

* Add vars

* changelog

* temp

* line

* update

* take test out

(cherry picked from commit da3584d)

* changelog
narph added a commit that referenced this pull request Feb 26, 2020
…16468) (#16574)

* Add vars

* changelog

* temp

* line

* update

* take test out

(cherry picked from commit da3584d)
@narph narph added test-plan Add this PR to be manual test plan v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 27, 2020
@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Mar 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat Team:Integrations Label for the Integrations team Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filebeat Azure module pipeline setup failing in 7.6.0
5 participants