Add new `shipper` output type #31281

rdner · 2022-04-13T13:19:33Z

What does this PR do?

This will allow us to start experimenting with the new shipper architecture.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

THERE ARE NO TESTS WHATSOEVER

The tests for the actual gRPC communication will be added later.

Closes elastic/elastic-agent-shipper#22

This will allow us to start experimenting with the new shipper architecture.

elasticmachine · 2022-04-13T13:37:25Z

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

libbeat/outputs/shipper/config.go

This reverts commit 066954a.

libbeat/outputs/shipper/config.go

This reverts commit 16ef370.

elasticmachine · 2022-04-14T12:19:46Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

rdner · 2022-04-14T13:35:39Z

/test

go.mod

libbeat/outputs/shipper/docs/shipper.asciidoc

belimawr

LGTM

libbeat/outputs/shipper/shipper.go

kvch

Please do not expose backoff settings and convert the documentation page to a README.md file.

belimawr

LGTM, but see @kvch's comment about not exposing the backoff settings.

libbeat/outputs/shipper/README.md

rdner · 2022-04-26T17:33:11Z

@belimawr I addressed all comments including the one about the backoff configuration. I planned to re-request reviews after the CI passed.

rdner · 2022-04-26T18:27:23Z

/test

rdner · 2022-04-27T07:51:24Z

/test

rdner · 2022-04-27T11:18:46Z

/test

rdner · 2022-04-27T14:26:09Z

I have strange failures in the e2e tests, I'm pretty sure my added feature should not break anything existing:


[2022-04-27T13:22:15.695Z] --- Failed steps:
[2022-04-27T13:22:15.695Z] 
[2022-04-27T13:22:15.695Z]   Scenario: Logs collection from running pod # features/elastic_agent_standalone.feature:9
[2022-04-27T13:22:15.695Z]     Then "elastic-agent" collects events with "kubernetes.pod.name:a-pod" # features/elastic_agent_standalone.feature:12
[2022-04-27T13:22:15.695Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.695Z] 
[2022-04-27T13:22:15.695Z]   Scenario: Logs collection from a pod with an init container # features/elastic_agent_standalone.feature:14
[2022-04-27T13:22:15.695Z]     Then "elastic-agent" collects events with "kubernetes.container.name:init-container" # features/elastic_agent_standalone.feature:17
[2022-04-27T13:22:15.695Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.695Z] 
[2022-04-27T13:22:15.695Z]   Scenario: Logs collection from short-living cronjobs # features/elastic_agent_standalone.feature:22
[2022-04-27T13:22:15.695Z]     Then "elastic-agent" collects events with "kubernetes.container.name:cronjob-container" # features/elastic_agent_standalone.feature:26
[2022-04-27T13:22:15.695Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z]   Scenario: Logs collection from failing pod # features/elastic_agent_standalone.feature:28
[2022-04-27T13:22:15.696Z]     Then "elastic-agent" collects events with "kubernetes.pod.name:a-failing-pod" # features/elastic_agent_standalone.feature:31
[2022-04-27T13:22:15.696Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z]   Scenario: Metrics collection configured from targeted Redis Pod # features/elastic_agent_standalone.feature:33
[2022-04-27T13:22:15.696Z]     Then "elastic-agent" collects events with "kubernetes.pod.name:redis" # features/elastic_agent_standalone.feature:36
[2022-04-27T13:22:15.696Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z] 5 scenarios (5 failed)
[2022-04-27T13:22:15.696Z] 22 steps (16 passed, 5 failed, 1 skipped)

@cmacknz would you mind if I merged it regardless the failures?

cmacknz · 2022-04-27T14:28:58Z

Some of the failing E2E tests (https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/main/876/testReport/) are listed as flaky: https://github.com/elastic/e2e-testing/issues

The packaging step is being triggered because of the changes to go.mod here.

cmacknz · 2022-04-27T14:30:40Z

The exact same test set is failing on main so I don't think they are caused by your change: https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/main/876/#showFailuresLink

cmacknz · 2022-04-27T14:31:39Z

I think it is safe to merge this given your change didn't introduce the E2E test failures.

This will allow us to start experimenting with the new shipper architecture.

Add new shipper output type

73aecb0

This will allow us to start experimenting with the new shipper architecture.

rdner added the 8.3-candidate label Apr 13, 2022

rdner self-assigned this Apr 13, 2022

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2022

Add more logging for debugging

453a82a

rdner added v8.3.0 and removed 8.3-candidate labels Apr 13, 2022

Fix checks

0f0ab35

kvch reviewed Apr 13, 2022

View reviewed changes

libbeat/outputs/shipper/config.go Outdated Show resolved Hide resolved

cmacknz requested a review from faec April 13, 2022 14:17

cmacknz added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Apr 13, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2022

rdner added 4 commits April 13, 2022 16:31

Make the default config a function

d333e5c

Update the shipper package version

d28c2f0

Fix dependencies

066954a

Revert "Fix dependencies"

ad039ce

This reverts commit 066954a.

kvch reviewed Apr 14, 2022

View reviewed changes

libbeat/outputs/shipper/config.go Outdated Show resolved Hide resolved

rdner added 4 commits April 14, 2022 10:34

Update mage version

ac378fd

Fix test

16ef370

Revert "Fix test"

6f4446a

This reverts commit 16ef370.

Fix error message test

982106c

rdner marked this pull request as ready for review April 14, 2022 12:19

rdner requested a review from a team as a code owner April 14, 2022 12:19

rdner requested review from leehinman and removed request for a team April 14, 2022 12:19

rdner requested a review from kvch April 14, 2022 12:20

cmacknz reviewed Apr 14, 2022

View reviewed changes

go.mod Outdated Show resolved Hide resolved

cmacknz reviewed Apr 14, 2022

View reviewed changes

libbeat/outputs/shipper/docs/shipper.asciidoc Outdated Show resolved Hide resolved

belimawr approved these changes Apr 14, 2022

View reviewed changes

libbeat/outputs/shipper/shipper.go Outdated Show resolved Hide resolved

kvch suggested changes Apr 21, 2022

View reviewed changes

rdner added 6 commits April 26, 2022 17:02

Convert the doc into README.md and remove backoff config

b8db15e

Fix incorrect log message

6e1422f

Copy gRPC API dependency instead of referencing

cb515e6

Add license headers

66c85a9

Exclude the generated gRPC code from linting

aa6e6ea

Revert the license override

f5748f7

belimawr approved these changes Apr 26, 2022

View reviewed changes

libbeat/outputs/shipper/README.md Show resolved Hide resolved

kvch approved these changes Apr 27, 2022

View reviewed changes

cmacknz approved these changes Apr 27, 2022

View reviewed changes

rdner merged commit 755cb8a into elastic:main Apr 27, 2022

rdner deleted the shipper-client branch April 27, 2022 14:36

kush-elastic pushed a commit to kush-elastic/beats that referenced this pull request May 2, 2022

Add new shipper output type (elastic#31281)

fe95a3c

This will allow us to start experimenting with the new shipper architecture.

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

Add new shipper output type (#31281)

ba40ad6

This will allow us to start experimenting with the new shipper architecture.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new `shipper` output type #31281

Add new `shipper` output type #31281

rdner commented Apr 13, 2022 •

edited

Loading

elasticmachine commented Apr 13, 2022 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Apr 14, 2022

rdner commented Apr 14, 2022

belimawr left a comment

kvch left a comment

belimawr left a comment

rdner commented Apr 26, 2022

rdner commented Apr 26, 2022

rdner commented Apr 27, 2022

rdner commented Apr 27, 2022

rdner commented Apr 27, 2022

cmacknz commented Apr 27, 2022

cmacknz commented Apr 27, 2022

cmacknz commented Apr 27, 2022

Add new shipper output type #31281

Add new shipper output type #31281

Conversation

rdner commented Apr 13, 2022 • edited Loading

What does this PR do?

Checklist

elasticmachine commented Apr 13, 2022 • edited by jenkins-beats-ci bot Loading

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Apr 14, 2022

rdner commented Apr 14, 2022

belimawr left a comment

Choose a reason for hiding this comment

kvch left a comment

Choose a reason for hiding this comment

belimawr left a comment

Choose a reason for hiding this comment

rdner commented Apr 26, 2022

rdner commented Apr 26, 2022

rdner commented Apr 27, 2022

rdner commented Apr 27, 2022

rdner commented Apr 27, 2022

cmacknz commented Apr 27, 2022

cmacknz commented Apr 27, 2022

cmacknz commented Apr 27, 2022

Add new `shipper` output type #31281

Add new `shipper` output type #31281

rdner commented Apr 13, 2022 •

edited

Loading

elasticmachine commented Apr 13, 2022 •

edited by jenkins-beats-ci bot

Loading