[Fleet] Support for remote ES output #169252

juliaElastic · 2023-10-18T14:06:51Z

Summary

Opening up for review, the feature flag is off for now, and the TODO items can come in follow up prs.

TODO:

make service_token a secret field in output - depends on [Fleet] Output secrets storage #157458
should link to remote elasticsearch docs in UI - depends on Document sending monitoring data to a remote Elasticsearch cluster ingest-docs#530
remote es connection check and report on UI - depends on fleet-server to report unhealthy status if can't access the remote ES cluster
enable feature flag when feature is ready

Added Remote ES output type, support to generate service token for fleet-server-remote account, support to create and edit remote es output.
Added validation to disallow making remote ES output as default for integration data.

How to test locally?

Enable feature flag by adding this to kibana.dev.yml:

xpack.fleet.enableExperimental: ['remoteESOutput']

See e2e test instructions here: elastic/fleet-server#3051

Generate service token

Create remote service token API:

POST kbn:/api/fleet/service_tokens
{
  "remote": true
}

// kibana logs out
[2023-10-19T16:22:05.776+02:00][DEBUG][plugins.fleet] Creating service token for account elastic/fleet-server-remote

Add/Edit output flyout:

Add output flyout:

Edd output flyout:

Remote ES output not allowed to be set as integrations data output in agent policies, only as monitoring output:

Example API call to create/update output:

POST kbn:/api/fleet/outputs
{"name":"remote1","type":"remote-elasticsearch","hosts":["http://localhost:9200"],"is_default":false,"is_default_monitoring":false,"config_yaml":"","service_token":"token1","proxy_id":null}

PUT kbn:/api/fleet/outputs/39168010-6db8-11ee-9bf3-ed5492034535
{"name":"remote2","type":"remote-elasticsearch","hosts":["http://localhost:9200"],"is_default":false,"is_default_monitoring":false,"config_yaml":"","service_token":"token2","proxy_id":null}

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

apmmachine · 2023-10-18T14:07:11Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…te --fix'

...et/public/applications/fleet/components/fleet_server_instructions/hooks/use_service_token.ts

juliaElastic · 2023-10-19T14:36:23Z

...ublic/applications/fleet/components/fleet_server_instructions/steps/create_service_token.tsx

        serviceToken={serviceToken}
        generateServiceToken={generateServiceToken}
        isLoadingServiceToken={isLoadingServiceToken}
      />
    ),
  };
 };
-
-const ServiceTokenStepContent: React.FunctionComponent<{


moved out and renamed this component to GenerateServiceTokenComponent to reuse in generating remote token

At the end I haven't used this component somewhere else, but I think the refactor can be left in.

juliaElastic · 2023-10-19T14:37:04Z

...applications/fleet/sections/agents/agent_requirements_page/fleet_server_requirement_page.tsx

@@ -42,28 +42,7 @@ export const FleetServerRequirementPage: React.FunctionComponent<
  const startService = useStartServices();
  const deploymentUrl = startService.cloud?.deploymentUrl;

-  const [isPermissionsLoading, setIsPermissionsLoading] = useState<boolean>(false);
-  const [permissionsError, setPermissionsError] = useState<string>();


moved this to its own hook, useCheckPermissions

juliaElastic · 2023-10-19T14:42:39Z

...ns/fleet/public/applications/fleet/sections/settings/components/edit_output_flyout/index.tsx

-            }
-          >
-            <EuiComboBox
+          {isRemoteESOutput ? null : (


hiding the proxy for remote ES output

juliaElastic · 2023-10-23T13:47:04Z

...c/applications/fleet/sections/agent_policy/components/agent_policy_advanced_fields/hooks.tsx

-      ...outputsRequest.data.items.map((item) => {
-        const isOutputTypeUnsupported = !allowedOutputTypes.includes(item.type);
+      ...outputsRequest.data.items
+        .filter((item) => item.type !== outputType.RemoteElasticsearch)


remote ES can't be used as integrations data output:

Do not allow remote elasticsearch output to be selected as integrations data output (should not be shown in dropdown)

juliaElastic · 2023-10-24T15:28:35Z

x-pack/plugins/fleet/server/services/agent_policies/full_agent_policy.ts

-    if (output && output.type === outputType.Elasticsearch) {
+    if (
+      output &&
+      (output.type === outputType.Elasticsearch || output.type === outputType.RemoteElasticsearch)


I found output_permissions are needed for remote-elasticsearch output, though it was not mentioned in the definition.

… src/core/server/integration_tests/ci_checks'

elasticmachine · 2023-10-25T12:17:33Z

Pinging @elastic/fleet (Team:Fleet)

x-pack/plugins/fleet/server/saved_objects/index.ts

… src/core/server/integration_tests/ci_checks'

pgayvallet

LGTM

nchaulet · 2023-10-30T13:08:33Z

x-pack/plugins/fleet/server/types/rest_spec/app.ts

+
+export const GenerateServiceTokenRequestSchema = {
+  query: schema.object({
+    remote: schema.boolean({ defaultValue: false }),


nit: I personally found the query parameter for POST request confusing and not consistent with our other endpoints, should we move this to a body parameter?

done, also replaced the callout

x-pack/plugins/fleet/common/constants/output.ts

x-pack/plugins/fleet/public/applications/fleet/hooks/use_check_permissions.ts

nchaulet

Thanks for your changes, I just tested the UI locally and works as expected 🚀

nchaulet · 2023-10-30T15:27:00Z

x-pack/plugins/fleet/server/types/rest_spec/app.ts

+import { schema } from '@kbn/config-schema';
+
+export const GenerateServiceTokenRequestSchema = {
+  body: schema.object({


@juliaElastic looks like it break normal service tokens creation probably should be wrapped in a maybe

{ "error": "Bad Request", "message": "[request body]: expected a plain object value, but found [null] instead.", "statusCode": 400 }

Unfortunately maybe didn't work, I tried a few more combinations (nullable, oneOf, literal) but no luck. At the end I could solve it with a custom validator function.

kibana-ci · 2023-10-30T17:59:32Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: fcaa3e8

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #68 / core plugins application using ScopedHistory.block "before each" hook for "allows navigation if user click confirm on the confirmation dialog"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	941	944	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.2MB	1.2MB	+3.4KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	44	45	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	148.3KB	149.3KB	+1.0KB

History

💔 Build #171929 failed 813846d
💔 Build #171890 failed 54c3900
💛 Build #171137 was flaky d9257fb
💔 Build #171093 failed 43c864c
💛 Build #170799 was flaky bc2e10b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

## Summary Resolves elastic#104986 Opening up for review, the feature flag is off for now, and the TODO items can come in follow up prs. TODO: - make service_token a secret field in output - depends on elastic#157458 - should link to remote elasticsearch docs in UI - depends on elastic/ingest-docs#530 - remote es connection check and report on UI - depends on fleet-server to report unhealthy status if can't access the remote ES cluster - enable feature flag when feature is ready Added Remote ES output type, support to generate service token for `fleet-server-remote` account, support to create and edit remote es output. Added validation to disallow making remote ES output as default for integration data. ## How to test locally? Enable feature flag by adding this to `kibana.dev.yml`: ``` xpack.fleet.enableExperimental: ['remoteESOutput'] ``` See e2e test instructions here: elastic/fleet-server#3051 ## Generate service token Create remote service token API: ``` POST kbn:/api/fleet/service_tokens { "remote": true } // kibana logs out [2023-10-19T16:22:05.776+02:00][DEBUG][plugins.fleet] Creating service token for account elastic/fleet-server-remote ``` ## Add/Edit output flyout: Add output flyout: <img width="675" alt="image" src="https://github.com/elastic/kibana/assets/90178898/dafc7d0e-05be-467f-871c-c4256fc833f6"> Edd output flyout: <img width="660" alt="image" src="https://github.com/elastic/kibana/assets/90178898/0d58fcfb-8c22-4e27-8719-db86ecba2e8d"> Remote ES output not allowed to be set as integrations data output in agent policies, only as monitoring output: <img width="690" alt="image" src="https://github.com/elastic/kibana/assets/90178898/675279cd-1c89-4069-9e07-e448aa796885"> <img width="683" alt="image" src="https://github.com/elastic/kibana/assets/90178898/6f67179d-b971-497f-9b04-3d3db5a42976"> Example API call to create/update output: ``` POST kbn:/api/fleet/outputs {"name":"remote1","type":"remote-elasticsearch","hosts":["http://localhost:9200"],"is_default":false,"is_default_monitoring":false,"config_yaml":"","service_token":"token1","proxy_id":null} PUT kbn:/api/fleet/outputs/39168010-6db8-11ee-9bf3-ed5492034535 {"name":"remote2","type":"remote-elasticsearch","hosts":["http://localhost:9200"],"is_default":false,"is_default_monitoring":false,"config_yaml":"","service_token":"token2","proxy_id":null} ``` ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

remote es output

c98403e

juliaElastic added the release_note:feature Makes this part of the condensed release notes label Oct 18, 2023

juliaElastic self-assigned this Oct 18, 2023

kibanamachine and others added 3 commits October 18, 2023 14:27

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

e49de06

…te --fix'

generate service token from UI

22db8e2

moved service token generation to settings

97cf572

juliaElastic added the ci:cloud-deploy Create or update a Cloud deployment label Oct 19, 2023

juliaElastic commented Oct 19, 2023

View reviewed changes

...et/public/applications/fleet/components/fleet_server_instructions/hooks/use_service_token.ts Outdated Show resolved Hide resolved

juliaElastic commented Oct 19, 2023

View reviewed changes

juliaElastic added 2 commits October 19, 2023 16:44

removed unused translations

f153d18

hide remote es output from agent policy data output dropdown

7cb9c22

juliaElastic commented Oct 23, 2023

View reviewed changes

include output_permissions and service_token for remote output

1fa0426

juliaElastic mentioned this pull request Oct 24, 2023

Support for remote ES output elastic/fleet-server#3051

Merged

8 tasks

juliaElastic commented Oct 24, 2023

View reviewed changes

juliaElastic and others added 9 commits October 25, 2023 10:45

removed generate token from settings and added callout to flyout

f0e8070

fix unit test

9ef33e4

fix test

fd02cfd

Merge remote-tracking branch 'upstream/main' into remote-es-output

f42f1c6

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

11b8cc1

… src/core/server/integration_tests/ci_checks'

fix type after merge conflict

b818b90

turn feature flag off for now

9e0621b

Merge branch 'main' into remote-es-output

2700ab6

added tests

200a227

juliaElastic marked this pull request as ready for review October 25, 2023 12:17

juliaElastic requested review from a team as code owners October 25, 2023 12:17

updated callout

bc2e10b

pgayvallet reviewed Oct 25, 2023

View reviewed changes

x-pack/plugins/fleet/server/saved_objects/index.ts Show resolved Hide resolved

added mappings_addition to SO config

0315d01

juliaElastic requested a review from pgayvallet October 26, 2023 08:18

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

43c864c

… src/core/server/integration_tests/ci_checks'

pgayvallet approved these changes Oct 26, 2023

View reviewed changes

Merge branch 'main' into remote-es-output

d9257fb

juliaElastic added ci:cloud-deploy Create or update a Cloud deployment and removed ci:cloud-deploy Create or update a Cloud deployment labels Oct 26, 2023

nchaulet self-requested a review October 30, 2023 12:42

nchaulet reviewed Oct 30, 2023

View reviewed changes

x-pack/plugins/fleet/common/constants/output.ts Outdated Show resolved Hide resolved

nchaulet reviewed Oct 30, 2023

View reviewed changes

x-pack/plugins/fleet/public/applications/fleet/hooks/use_check_permissions.ts Outdated Show resolved Hide resolved

juliaElastic force-pushed the remote-es-output branch from bb2f627 to ea22138 Compare October 30, 2023 14:04

review comments

d9eb33f

juliaElastic force-pushed the remote-es-output branch from ea22138 to d9eb33f Compare October 30, 2023 14:04

Merge branch 'main' into remote-es-output

54c3900

juliaElastic requested a review from nchaulet October 30, 2023 14:06

nchaulet approved these changes Oct 30, 2023

View reviewed changes

juliaElastic removed the ci:cloud-deploy Create or update a Cloud deployment label Oct 30, 2023

fix tests

813846d

nchaulet reviewed Oct 30, 2023

View reviewed changes

fixing schema validation

fcaa3e8

juliaElastic merged commit a732dea into elastic:main Nov 1, 2023
29 checks passed

kibanamachine added v8.12.0 backport:skip This commit does not require backporting labels Nov 1, 2023

kilfoyle mentioned this pull request Jan 23, 2024

Fix remote ES cluster output docs elastic/ingest-docs#848

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Support for remote ES output #169252

[Fleet] Support for remote ES output #169252

juliaElastic commented Oct 18, 2023 •

edited

Loading

apmmachine commented Oct 18, 2023

juliaElastic Oct 19, 2023

juliaElastic Oct 25, 2023

juliaElastic Oct 19, 2023

juliaElastic Oct 19, 2023

juliaElastic Oct 23, 2023

juliaElastic Oct 24, 2023

elasticmachine commented Oct 25, 2023

pgayvallet left a comment

nchaulet Oct 30, 2023

juliaElastic Oct 30, 2023

nchaulet left a comment

nchaulet Oct 30, 2023

juliaElastic Oct 30, 2023 •

edited

Loading

kibana-ci commented Oct 30, 2023

[Fleet] Support for remote ES output #169252

[Fleet] Support for remote ES output #169252

Conversation

juliaElastic commented Oct 18, 2023 • edited Loading

Summary

How to test locally?

Generate service token

Add/Edit output flyout:

Checklist

apmmachine commented Oct 18, 2023

🤖 GitHub comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 25, 2023

pgayvallet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliaElastic Oct 30, 2023 • edited Loading

Choose a reason for hiding this comment

kibana-ci commented Oct 30, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Public APIs missing exports

Page load bundle

History

juliaElastic commented Oct 18, 2023 •

edited

Loading

juliaElastic Oct 30, 2023 •

edited

Loading