Remove GCP support from Functionbeat #28253

kvch · 2021-10-05T13:30:15Z

What does this PR do?

This PR removes support for Google Cloud Functions in Functionbeat. The Beat approach is replaced in favour of a unified serverless agent in https://github.com/elastic/elastic-serverless-agent

Why is it important?

GCP support has been broken for a long time, and we do not intend to support it anymore because the Beats approach in not good for such data sources. E.g. there is no need for position tracking or ACK handling.

Checklist

~~- [ ] My code follows the style guidelines of this project~~
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
~~- [ ] I have made corresponding changes to the documentation~~

I have made corresponding change to the default configuration files
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-10-05T13:30:31Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mergify · 2021-10-05T13:30:39Z

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-10-05T13:50:29Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-11T13:53:59.329+0000
Duration: 68 min 46 sec
Commit: a6b4b69

Test stats 🧪

Test	Results
Failed	0
Passed	20462
Skipped	1519
Total	21981

Steps errors

Expand to view the steps failures

`filebeat-goIntegTest - mage goIntegTest`

Took 5 min 26 sec . View more details here
Description: mage goIntegTest

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 11 min 29 sec . View more details here
Description: make -C generator/_templates/metricbeat test test-package

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 6 min 49 sec . View more details here
Description: make -C generator/_templates/metricbeat test test-package

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

Took 6 min 48 sec . View more details here
Description: make -C generator/_templates/metricbeat test test-package

`Archive system tests files`

Took 0 min 0 sec . View more details here
Description: mage packageSystemTests

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 12 min 52 sec . View more details here
Description: make -C generator/_templates/beat test test-package

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 6 min 52 sec . View more details here
Description: make -C generator/_templates/beat test test-package

`generator-beat-test - make -C generator/_templates/beat test test-package`

Took 6 min 52 sec . View more details here
Description: make -C generator/_templates/beat test test-package

`Archive system tests files`

Took 0 min 0 sec . View more details here
Description: mage packageSystemTests

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 2'

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

mergify · 2021-10-06T12:49:05Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-functionbeat-remove-gcp-support upstream/feature-functionbeat-remove-gcp-support
git merge upstream/master
git push upstream feature-functionbeat-remove-gcp-support

x-pack/functionbeat/docs/getting-started.asciidoc

Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

kvch · 2021-10-11T08:05:26Z

The PR is ready for review. The documentation updates are missing, but I would like to add those in a follow-up PR.

…eat-remove-gcp-support

(cherry picked from commit 3754124) # Conflicts: # NOTICE.txt

* upstream/master: (73 commits) Remove GCP support from Functionbeat (elastic#28253) Move labels and annotations under kubernetes.namespace. (elastic#27917) Update go release version 1.17.1 (elastic#27543) Osquerybeat: Runner and Fetcher unit tests (elastic#28290) Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) seccomp: allow clone3 syscall for x86 (elastic#28117) packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) [cloud][docker] use the private docker namespace (elastic#28286) Update aws-lambda-go library version to 1.13.3 (elastic#28236) Deprecate common.Float (elastic#28280) Filebeat: Change compatibility test stage to test against previous minor instead of 7.11 (elastic#28274) x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (elastic#28124) Explicitly pass http config to doppler consumer (elastic#28277) processors/actions/add_fields: Do not panic if event.Fields is nil map (elastic#28219) Resolved timestamp for defender atp (elastic#28272) [Winlogbeat] Tolerate faults when Windows Event Log session is interrupted (elastic#28191) [elastic-agent] proxy requests to subprocesses to their metrics endpoints (elastic#28165) Build cloud docker images for elastic-agent (elastic#28134) Upgrade k8s go-client library (elastic#28228) ...

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (#28236) * [cloud][docker] use the private docker namespace (#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (#28297) * seccomp: allow clone3 syscall for x86 (#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (#28253) * Fix build tags for Go 1.17 (#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (#28325) * Redis: remove deprecated fields (#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (#27878) partial fix for #27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

(cherry picked from commit 3754124)

(cherry picked from commit 3754124) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

* singleton sysinfo host to avoid frequently collecting host info * add Host object to Stats object * update changelog * set procStats.host to nil if any error calling sysinfo.Host() * Update aws-lambda-go library version to 1.13.3 (elastic#28236) * [cloud][docker] use the private docker namespace (elastic#28286) * [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288) * packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297) * seccomp: allow clone3 syscall for x86 (elastic#28117) clone3 is a linux syscall that is now used by glibc starting version 2.34. It is used when pthread_create() gets called. Current seccomp filters do not allow this syscall leading to crashes like runtime/cgo: pthread_create failed: Operation not permitted See elastic/apm-server#6238 for more details * Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289) Previously the osquery.autoload file was overwritten every time on osquerybeat start and stamped with our extension. After the change we check the content of the file and do not overwrite it on each osquerybeat start. This allows the user to deploy their own extensions if their want and start osquery with that. * Osquerybeat: Runner and Fetcher unit tests (elastic#28290) * Runner and Fetcher unit tests * Fix header formatting * Tweak test * Update go release version 1.17.1 (elastic#27543) * format of conditional build tags has changed * matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123 * iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests * go.mod file contains separate require directive for transitive dependencies * Move labels and annotations under kubernetes.namespace. (elastic#27917) * Move labels and annotations under kubernetes.namespace. * Remove GCP support from Functionbeat (elastic#28253) * Fix build tags for Go 1.17 (elastic#28338) * [Elastic Agent] Add ability to communicate with Kibana through service token (elastic#28096) * Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand. * Add changelog entry. * Fix go fmt. * Add username to ASA Security negotiation log (elastic#26975) * Add username to ASA Security negotiation log I added the username user.name field to ASA Security negotiation log line. * adding support for both formats * adding changelog entry * updating geo fields in expected output files * reverse formatting * reverting to older version of file * reverting formatting again * regenrate golden files again * remove formatting, ready for review * fixing missing message due to no newline * fix dissect pattern to fit correctly Co-authored-by: Marius Iversen <marius.iversen@elastic.co> * x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (elastic#28325) * Redis: remove deprecated fields (elastic#28246) * Redis: remove deprecated fields * Disable generator tests temporarily (elastic#28362) * Windows/perfmon metricset - remove deprecated perfmon.counters configuration (elastic#28282) * remove deprecated config * changelog * [Filebeat] - S3 Input - Add support for only iterating/accessing only… (elastic#28252) * [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (elastic#28230) * Breaking change for 8.0, namespace_annotations replaced by namespace.annotations * Take care of namespace being nil * [Heartbeat] Setuid to regular user / lower capabilities when possible (elastic#27878) partial fix for elastic#27648 , this PR: Detects if the user is running as root then: Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present Attempts to Setuid , Setgid and Setgroups to that user / groups Invokes setcap to drop all privileges except NET_RAW+ep This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that. With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root. * mage fmt Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Ugo Sangiorgi <ugo.sangiorgi@elastic.co> Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Arnaud Lefebvre <a.lefebvre@outlook.fr> Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: apmmachine <58790750+apmmachine@users.noreply.github.com> Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: LaZyDK <dennisperto@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co> Co-authored-by: Mariana Dima <mariana@elastic.co> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>

chipzzz · 2022-01-12T21:26:42Z

@bmorelli25 @faec , can you point to any docs relating to GCP support with elastic serverless agent. There I only see AWS support.

Ref : "The Beat approach is replaced in favour of a unified serverless agent in https://github.com/elastic/elastic-serverless-agent"

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 5, 2021

kvch added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 5, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 5, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 5, 2021

kvch force-pushed the feature-functionbeat-remove-gcp-support branch from 29084b1 to 77f7920 Compare October 6, 2021 07:51

kvch added 3 commits October 6, 2021 14:50

Remove GCP support from Functionbeat

d94ffb7

rm unnecessary dep

67b057d

update notice

cfba4f4

kvch force-pushed the feature-functionbeat-remove-gcp-support branch from 2817eb6 to cfba4f4 Compare October 6, 2021 12:53

jlind23 added the 7.16-candidate label Oct 6, 2021

bmorelli25 reviewed Oct 6, 2021

View reviewed changes

x-pack/functionbeat/docs/getting-started.asciidoc Show resolved Hide resolved

remove test

792ee44

kvch force-pushed the feature-functionbeat-remove-gcp-support branch from 4e7ba65 to 792ee44 Compare October 7, 2021 08:06

Update x-pack/functionbeat/docs/getting-started.asciidoc

752e1a5

Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

kvch added backport-v7.16.0 Automated backport with mergify needs_docs and removed backport-skip Skip notification from the automated backport with mergify labels Oct 7, 2021

add changelog entry

8d742c8

faec approved these changes Oct 11, 2021

View reviewed changes

Merge remote-tracking branch 'upstream/master' into feature-functionb…

a6b4b69

…eat-remove-gcp-support

kvch merged commit 3754124 into elastic:master Oct 11, 2021

mergify bot pushed a commit that referenced this pull request Oct 11, 2021

Remove GCP support from Functionbeat (#28253)

0e4aab2

(cherry picked from commit 3754124) # Conflicts: # NOTICE.txt

mergify bot mentioned this pull request Oct 11, 2021

[7.16](backport #28253) Remove GCP support from Functionbeat #28339

Merged

newly12 pushed a commit to newly12/beats that referenced this pull request Oct 13, 2021

Remove GCP support from Functionbeat (elastic#28253)

056c4d7

kvch added a commit that referenced this pull request Oct 25, 2021

Remove GCP support from Functionbeat (#28253)

7c391f2

(cherry picked from commit 3754124)

kvch added a commit that referenced this pull request Oct 25, 2021

Remove GCP support from Functionbeat (#28253) (#28339)

61452a8

(cherry picked from commit 3754124) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

Remove GCP support from Functionbeat (elastic#28253)

b7ced82

chipzzz mentioned this pull request Jan 24, 2022

Functionbeat GCP pub/sub requiring storage entry point when only pub/sub is enabled #29967

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove GCP support from Functionbeat #28253

Remove GCP support from Functionbeat #28253

kvch commented Oct 5, 2021 •

edited

Loading

elasticmachine commented Oct 5, 2021

mergify bot commented Oct 5, 2021

elasticmachine commented Oct 5, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

`filebeat-goIntegTest - mage goIntegTest`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`Archive system tests files`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`Archive system tests files`

`Error signal`

mergify bot commented Oct 6, 2021

kvch commented Oct 11, 2021

chipzzz commented Jan 12, 2022

Remove GCP support from Functionbeat #28253

Remove GCP support from Functionbeat #28253

Conversation

kvch commented Oct 5, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Oct 5, 2021

mergify bot commented Oct 5, 2021

elasticmachine commented Oct 5, 2021 • edited by jenkins-beats-ci bot Loading

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

filebeat-goIntegTest - mage goIntegTest

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package

Archive system tests files

generator-beat-test - make -C generator/_templates/beat test test-package

generator-beat-test - make -C generator/_templates/beat test test-package

generator-beat-test - make -C generator/_templates/beat test test-package

Archive system tests files

Error signal

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Oct 6, 2021

kvch commented Oct 11, 2021

chipzzz commented Jan 12, 2022

kvch commented Oct 5, 2021 •

edited

Loading

elasticmachine commented Oct 5, 2021 •

edited by jenkins-beats-ci bot

Loading

`filebeat-goIntegTest - mage goIntegTest`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package`

`Archive system tests files`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`generator-beat-test - make -C generator/_templates/beat test test-package`

`Archive system tests files`

`Error signal`