[Metricbeat] Add Airflow module in xpack #26220

aspacca · 2021-06-09T08:01:25Z

What does this PR do?

Adding Airflow monitoring as lightweight module based on Statsd (https://airflow.apache.org/docs/apache-airflow/1.10.15/metrics.html)
Introduced statsd.mapping to filter and transform Airflow mapping (it's anyway generic and could be applied to any other Statsd lightweigth module in the future).

It uses eventMapping(...) function according to https://github.com/elastic/beats/blob/23e4403ae093fcc8f7905345cad2c7ad256976d8/docs/devguide/metricset-details.asciidoc#data-transformation. It is anyway in the Run() method and not in the Fetch() one, but there is where we call reporter.Event(*e) for the module. It has to be clarified if it's correct.

Why is it important?

Because it is my first PR :)

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

$ ./metricbeat modules enable airflow
$ ./metricbeat -e # (Start metricbeat according your preferred setup)
$ echo "dagrun.duration.failed.dagid:200|ms" > /dev/udp/127.0.0.1/8126 # (with any of the metrics that can be found at https://airflow.apache.org/docs/apache-airflow/1.10.15/metrics.html)

Related issues

Closes #26197

Use cases

  Scenario: Andrea monitors Airflow
    Given the Andrea has an Airflow instance to monitor
    When the Airflow module is enabled
    And statsd configuration points to it in Airflow
    Then Andrea would be able to see Airflow metrics in Kibana

Screenshots

Logs

2021-06-08T18:08:47.126+0200    INFO    udp/udp.go:80   Started listening for UDP on: 127.0.0.1:8126

elasticmachine · 2021-06-09T08:01:27Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-06-09T08:19:41Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #26220 updated
Start Time: 2021-07-01T07:59:45.915+0000
Duration: 94 min 45 sec
Commit: 2b32fdc

Test stats 🧪

Test	Results
Failed	0
Passed	9473
Skipped	2595
Total	12068

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9473
Skipped	2595
Total	12068

jsoriano

Great first PR @aspacca! I think the new mapping support in the statsd module offers many possibilities. I have added some comments, but mostly about Metricbeat specifics.

This change will need two changelog entries, one for the new mapping in the statsd module and another one for the new airflow module. And this makes me think that this PR might be splitted into two, this will make it easier to separate the specific discussions for each feature, and we can focus first in merging the new mapping for the statsd module. What do you think?

x-pack/metricbeat/module/airflow/_meta/Dockerfile

x-pack/metricbeat/module/airflow/statsd/_meta/data.json

x-pack/metricbeat/module/airflow/statsd/_meta/fields.yml

x-pack/metricbeat/module/airflow/statsd/_meta/docs.asciidoc

x-pack/metricbeat/module/statsd/server/server.go

x-pack/metricbeat/module/statsd/server/mapping.go

x-pack/metricbeat/module/statsd/server/data.go

aspacca · 2021-06-10T10:07:12Z

This change will need two changelog entries, one for the new mapping in the statsd module and another one for the new airflow module.

just adding the entries under Metricbeat in CHANGELOG.next.asciidoc, is it?

And this makes me think that this PR might be splitted into two, this will make it easier to separate the specific discussions for each feature, and we can focus first in merging the new mapping for the statsd module. What do you think?

I'm fine having both discussions in this PR, but I understand your "concern", it's up about what is the standard for such cases in the team.
in case the second PR should be based on this branch (removing the part strictly related to Airflow), is it fine?

jsoriano

Thanks for addressing most of the comments! Adding some extra suggestions.

x-pack/metricbeat/module/airflow/_meta/fields.yml

x-pack/metricbeat/module/statsd/server/data_test.go

x-pack/metricbeat/module/statsd/server/data.go

x-pack/metricbeat/module/statsd/server/server.go

x-pack/metricbeat/module/airflow/_meta/Dockerfile

mergify · 2021-06-23T13:55:14Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-airflow-module upstream/add-airflow-module
git merge upstream/master
git push upstream add-airflow-module

jsoriano

Added some minor comments, nothing serious. Good job!

x-pack/metricbeat/module/statsd/server/data_test.go

x-pack/metricbeat/module/statsd/server/server.go

CHANGELOG.next.asciidoc

x-pack/metricbeat/module/airflow/_meta/README.md

x-pack/metricbeat/module/statsd/server/data.go

x-pack/metricbeat/module/statsd/server/data_test.go

jsoriano · 2021-06-24T16:35:43Z

/test

jsoriano

Thanks for addressing all the comments! I have added some extra nitpicking, but I'd say this is ready to merge.

x-pack/metricbeat/module/airflow/statsd/data_test.go

CHANGELOG.next.asciidoc

x-pack/metricbeat/module/statsd/server/server.go

jsoriano · 2021-06-28T17:35:32Z

One failing test seems related, the other one is unrelated (#26537).

aspacca · 2021-06-29T07:42:40Z

on win images windows firewall seems to block the udp statsd server created by the module. that's the reason why the TestData is failing.

jsoriano · 2021-06-29T10:05:59Z

on win images windows firewall seems to block the udp statsd server created by the module. that's the reason why the TestData is failing.

Ok, then skip these tests in windows by now, and maybe in the future we can configure the firewall in these images to allow local UDP connections.

x-pack/metricbeat/module/airflow/statsd/data_test.go

mergify · 2021-06-29T20:20:22Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-airflow-module upstream/add-airflow-module
git merge upstream/master
git push upstream add-airflow-module

Airflow module is a lightweight module based on Statsd module. It filters, renames and adds metrics according to mappings defined in the manifest.yml

mergify · 2021-07-01T07:53:44Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add-airflow-module upstream/add-airflow-module
git merge upstream/master
git push upstream add-airflow-module

* [Metricbeat] Add Airflow module Airflow module is a lightweight module based on Statsd module. It filters, renames and adds metrics according to mappings defined in the manifest.yml (cherry picked from commit 3d01b5b)

…stage-failed-within-same-build * upstream/master: (36 commits) Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617)" (elastic#26704) Packaging: linux/armv7 is not supported (elastic#26706) Cyberarkpas: Link to official docs on how to setup TLS (elastic#26614) Make network_direction, registered_domain and convert processors compatible with ES older than 7.13.0 (elastic#26676) Disable armv7 packaging (elastic#26679) [Heartbeat] use --params flag for synthetics (elastic#26674) Update dependent package to avoid downloading a suspicious file (elastic#26406) [mergify] set title and allow bp in any direction (elastic#26648) Fix memory leak in SQL helper when database is not available (elastic#26607) [CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617) [mergify] automate PRs that change the backport rules (elastic#26641) [Metricbeat] Add Airflow module in xpack (elastic#26220) chore: add-backport-next (elastic#26620) [metricbeat] Add state_job metricset (elastic#26479) CI: jenkins labels are less time consuming now (elastic#26613) [MetricBeat] [AWS] Fix aws metric tags with resourcegroupstaggingapi paginator (elastic#26385) (elastic#26443) Move openmetrics module to oss (elastic#26561) Skip flaky test TestFilestreamMetadataUpdatedOnRename (elastic#26609) [filebeat][fortinet] Use default add_locale for fortinet.firewall (elastic#26524) Enroll proxy settings (elastic#26514) ...

* [Metricbeat] Add Airflow module Airflow module is a lightweight module based on Statsd module. It filters, renames and adds metrics according to mappings defined in the manifest.yml

aspacca added enhancement Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Jun 9, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 9, 2021

jsoriano assigned aspacca Jun 10, 2021

jsoriano self-requested a review June 10, 2021 08:20

jsoriano requested changes Jun 10, 2021

View reviewed changes

jsoriano reviewed Jun 14, 2021

View reviewed changes

jsoriano reviewed Jun 24, 2021

View reviewed changes

jsoriano approved these changes Jun 28, 2021

View reviewed changes

jsoriano approved these changes Jun 29, 2021

View reviewed changes

x-pack/metricbeat/module/airflow/statsd/data_test.go Show resolved Hide resolved

Andrea Spacca added 9 commits June 30, 2021 09:38

[Metricbeat] Add Airflow module in xpack

e878203

Airflow module is a lightweight module based on Statsd module. It filters, renames and adds metrics according to mappings defined in the manifest.yml

Added dataa.json

65933c9

Use eventMapping function

ea91c9d

mage update and integration test

c0458a6

fix ci

a998547

added system test, added data.json generation

d088ef1

added statsd.mapping documentation, regex factory for mappings

a676af0

import processors for test data

0236af3

python lint

3a3cb56

Andrea Spacca added 12 commits June 30, 2021 09:38

statsd.mapping validation

85c60e0

finalised

f67b1d7

fix fields.yml

bd57e95

cr fixes

416cb73

move TestData, specify statsd_prefix conf in airflow

ff6bdab

make update

1929bb6

floating test on windows

26c927a

approved

41df9eb

win debug

3c39a41

skip testdata on win

a4a18d7

revert last testdata not failing on linux

fc7e5d4

remove changes to unrelated files

80738ea

jsoriano approved these changes Jun 30, 2021

View reviewed changes

jsoriano added the backport-v7.14.0 Automated backport with mergify label Jun 30, 2021

Merge remote-tracking branch 'upstream/master' into add-airflow-module

2b32fdc

aspacca merged commit 3d01b5b into elastic:master Jul 1, 2021

mergify bot mentioned this pull request Jul 1, 2021

[Metricbeat] Add Airflow module in xpack (backport #26220) #26647

Closed

aspacca added backport-v7.15.0 Automated backport with mergify and removed backport-v7.14.0 Automated backport with mergify labels Jul 5, 2021

mergify bot mentioned this pull request Jul 5, 2021

[7.x](backport #26220) [Metricbeat] Add Airflow module in xpack #26705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add Airflow module in xpack #26220

[Metricbeat] Add Airflow module in xpack #26220

aspacca commented Jun 9, 2021 •

edited

Loading

elasticmachine commented Jun 9, 2021

elasticmachine commented Jun 9, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

jsoriano left a comment

aspacca commented Jun 10, 2021 •

edited

Loading

jsoriano left a comment

mergify bot commented Jun 23, 2021

jsoriano left a comment

jsoriano commented Jun 24, 2021

jsoriano left a comment

jsoriano commented Jun 28, 2021

aspacca commented Jun 29, 2021

jsoriano commented Jun 29, 2021

mergify bot commented Jun 29, 2021

mergify bot commented Jul 1, 2021

[Metricbeat] Add Airflow module in xpack #26220

[Metricbeat] Add Airflow module in xpack #26220

Conversation

aspacca commented Jun 9, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Jun 9, 2021

elasticmachine commented Jun 9, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

jsoriano left a comment

Choose a reason for hiding this comment

aspacca commented Jun 10, 2021 • edited Loading

jsoriano left a comment

Choose a reason for hiding this comment

mergify bot commented Jun 23, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Jun 24, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Jun 28, 2021

aspacca commented Jun 29, 2021

jsoriano commented Jun 29, 2021

mergify bot commented Jun 29, 2021

mergify bot commented Jul 1, 2021

aspacca commented Jun 9, 2021 •

edited

Loading

elasticmachine commented Jun 9, 2021 •

edited by jenkins-beats-ci bot

Loading

aspacca commented Jun 10, 2021 •

edited

Loading