[Ingest Manager] support multiple kibana urls #75712

neptunian · 2020-08-21T22:35:46Z

#72731

Support letting the user have multiple kibana urls in their global Settings and notify the agent when they change.

Changes

validation in the settings flyout and the /settings endpoint that paths and protocol or a list of kibana urls cannot differ
the enroll command in the Add Agent flyout will return the first item in the list of kibana URLS.
the ingest_manager_settings saved object's kibana_url attribute is now an array of strings type
the PUT /api/ingest_manager/settings was changed from allowing only a string url to allowing only an array of string urls. GET always returns kibana_url as an array of strings.
the kibana configuration xpack.ingestManager.fleet.kibana.host now accepts both a string url or an array of string urls.
PUT /ingest_manager/settings, regardless of what was updated, will bump all agent configuration saved objects revision, triggering a CONFIG_CHANGE action
the Kibana URL in the settings flyout is now a EUI combo box that allows for multiple URLS
changes all instances of kibana_url of type string to kibana_urls of type string[] and adds migration
updates the view agent policy flyout to handle errors from /{agentPolicyId}/full endpoint
Before:

After:

Test

integration and unit tests should run
in your Kibana configuration yaml file try a list of kibana urls where the path and protocols don't match. It should fail. eg: xpack.ingestManager.fleet.kibana.host: ["http://localhost:5601/", "https://localhost:5603/"]
In the Settings flyout, try adding a list of kibana urls where the path and protocols don't match. It should fail. On a successful save, all agent policy saved objects revision should be updated. Look at the Full Agent Policy yaml file through the UI, notice there is now a list of kibana host urls under "fleet".
hit the PUT /ingest_manager/settings endpoint, all agent policy saved objects revision should be updated. If an agent is running, notice CONFIG_CHANGE action after a settings update. Also try to to update kibana_url with mismatched paths or protocol through this endpoint. It should fail.

Note

I do not check what in the settings was updated or whether kibana_url array has changed or not. The agent configs are always bumped when settings are updated.
have left the agent policy SO attribute kibana_url but we should probably change it to kibana_urls. Would like to do this change and SO migration in a separate PR.

x-pack/plugins/ingest_manager/server/services/agent_policy.ts

elasticmachine · 2020-08-27T20:03:43Z

Pinging @elastic/ingest-management (Team:Ingest Management)

x-pack/plugins/ingest_manager/server/services/agent_policy.ts

nchaulet

Tested and it work as expected

There is maybe just one think that we can improve (can be done in a follow up PR)
I think the config yaml flyout in the policy details should show the standalone config and not the full one, (it's here https://github.com/neptunian/kibana/blob/72731-allow-multiple-kibana-urls/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_policy/components/agent_policy_yaml_flyout.tsx/#L36)

…cies when revision is bumped

jonathan-buttner · 2020-09-01T14:37:58Z

x-pack/test/security_solution_endpoint/apps/endpoint/policy_details.ts

@@ -186,6 +196,11 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
              type: 'elasticsearch',
            },
          },
+          fleet: {


These look fine to me. @paul-tavares do we need to make changes in the UI code that creates the policy?

@jonathan-buttner no, I don't think this impacts the Endpoint Integration Policy because its outside of the Integration input - that's the only one we manipulate in the Endpoint Policy Details page.

jonathan-buttner

Security changes look good.

…ana_urls: string[] and add migration

jen-huang · 2020-09-01T17:25:30Z

x-pack/plugins/ingest_manager/server/saved_objects/migrations/to_v7_10_0.ts

+  },
+  Settings
+> = (settingsDoc) => {
+  settingsDoc.attributes.kibana_urls = [settingsDoc.attributes.kibana_url];


is it possible for the old settingsDoc to have undefined for settingsDoc.attributes.kibana_url? if so, we may need to wrap this line in a conditional so that we don't set [undefined] as a value for the new doc

It's only possible for there to be no settings if they haven't started kibana. Settings along with the kibana_url gets created during setup and they cannot remove it through the settings api

...nager/public/applications/ingest_manager/components/enrollment_instructions/manual/index.tsx

x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/settings_flyout.tsx

…ix types

neptunian · 2020-09-01T19:25:47Z

x-pack/plugins/ingest_manager/server/services/settings.ts

-        GLOBAL_SETTINGS_SAVED_OBJECT_TYPE,
-        newData
-      );
+      const defaultSettings = createDefaultSettings();


@nchaulet since during setup we create the Settings saved object with default attributes, I moved those into a function and they will be added when the saved object (if missing for some reason) is created here as well.

kibanamachine · 2020-09-01T21:11:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 3577435

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
ingestManager	519	+1	518

async chunks size

id	value	diff	baseline
ingestManager	1.1MB	+750.0B	1.1MB

page load bundle size

id	value	diff	baseline
ingestManager	468.0KB	+567.0B	467.5KB

distributable file count

id	value	diff	baseline
total	45448	+1	45447

History

💚 Build #71656 succeeded bb7eb07
💔 Build #71636 failed b644e08
💚 Build #71416 succeeded 1e47f95
💚 Build #71399 succeeded 1591640
💔 Build #71386 failed 75108cf

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* let the user specify multiple kibana urls * add validation to kibana urls so paths and protocols cannot differ * update i18n message * only send the first url to the instructions * udpate all agent configs' revision when settings is updated * fix jest test * update endpoint full agent policy test * fix type * dont add settings if standalone mode * fix ui not handling errors from /{agentPolicyId}/full endpoint * fix formatted message id * only return needed fields * fill in updated_by and updated_at attributes of the ingest-agent-policies when revision is bumped * throw error if kibana_urls not set and update tests * change ingest_manager_settings SO attribute kibana_url: string to kibana_urls: string[] and add migration * leave instructions single kibana url * make kibana_url and other attributes created during setup required, fix types

ghost · 2020-09-14T12:46:53Z

Hi @EricDavisX

We have validated the ticket by setting up 03 different 7.10.0-SNAPSHOT Kibana cloud environments.

Steps Followed:

Login to Kibana Cloud environment with 'elastic' user.
Click on the 'Settings' flyout.
Add 03 kibana cloud urls in the 'Kibana URL' field and click on 'Save Settings' button.
Click on Policies->Default policy->Actions->View policy.
Click on Fleet->Online Host->Activity log

Observations:

Kibana URL in the settings flyout is a EUI combo box that allows for multiple URLs.
03 Kibana urls are updated in policy under fleet->kibana->hosts section.
'CONFIG_CHANGE' action is displayed in agent activity logs after kibana urls are added in Settings.
Enroll command in the 'Add Agent' flyout returns the first item in the list of kibana URLs on Settings flyout.

Moreover, we have created and executed 05 testcases under Support multiple kibana urls TestRun.

Queries:

On saving kibana url with incorrect http protocol in 'Settings' flyout, no error message occurs and 'CONFIG_CHANGE' action is displayed in agent activity logs.
Screenshot:

As per #75712 description under Test section and our understanding, error message should be displayed for the same. Please share your feedback.

(i) On executing the PUT /ingest_manager/settings ES query under Management->Dev Tools->Console, following error message is displayed.
Screenshot:

(ii) On executing the POST /ingest_manager/settings ES query under Management->Dev Tools->Console, following error message is displayed.
Screenshot:

However as per #75712 description under Test section, all agent policy saved objects revision should be updated on executing PUT ES query . Please share your feedback.

Error is displayed on enrolling the agent with different kibana enroll command than kibana logged-in into.
Steps Followed:
(a) Login into Kibana cloud environment (say https://1c238ddda25743e297c3b47d3c5de1fc.us-central1.gcp.foundit.no:9243)
(b) Click on Settings flyout and update the 'Kibana URL' field by adding https://e4eb0f3ad19f4043aecd7ab95d9294a8.us-central1.gcp.foundit.no:9243 and then https://1c238ddda25743e297c3b47d3c5de1fc.us-central1.gcp.foundit.no:9243 kibana url.
(c) Click on Fleet->Add Agent flyout
(d) Copy the following enroll command and enroll the agent.
.\elastic-agent enroll https://e4eb0f3ad19f4043aecd7ab95d9294a8.us-central1.gcp.foundit.no:9243 RGE0Z2kzUUJ2U2xwNlVodEt5S1o6RzVpTUY5X1hRUHE0NUdoQlByeUpjQQ==
.\install-service-elastic-agent.ps1

Screenshot:

Please share your feedback.

Could you please let us know the business use-case, functionality impact from end-user point of view.
On following the below steps, elastic-agent is enrolled only on the logged-in Kibana environment.
Steps Followed:
(i) RDP to the windows10 endpoint and download the 7.10.0-SNAPSHOT elastic-agent.
(ii) Edit the elastic-agent.yml file to include 03 kibana cloud environments under fleet: section (as per #72731_comment)

elastic-agent_beforeEnroll.zip
(iii) Login to the first 7.10.0-SNAPSHOT Kibana cloud environment(say https://0f19b73ebc5c47ab846337ff293239bc.us-central1.gcp.foundit.no:9243)
(iv) Click on Ingest Manager->Fleet to navigate to Fleet page.
(v) Click on Add Agent and copy the enroll command for windows(say):-

.\elastic-agent enroll https://0f19b73ebc5c47ab846337ff293239bc.us-central1.gcp.foundit.no:443 UlowSWZIUUJrVTF5TDFqOEtsUVA6TGpQRHpaYlJSWnVITkh6Q01YRDZNdw==

.\install-service-elastic-agent.ps1

(vi) Enroll the agent on Windows10 using above command.

Observation:
(a) Agent is enrolled only on the Kibana cloud environment from which enroll command was copied(here first Kibana cloud environment).
(b) elastic-agent.yml file is modified and does not contain the above 03 kibana cloud environments.

elastic-agent_afterEnroll.zip

Screenshot:

Please let us know if it is the expected behavior.

'xpack.ingestManager.fleet.kibana.host' field is not accepted in kibana.yml of cloud environment(#75712 description->Test section->point2). Please let us know if it is the expected behavior.

(a)Error is displayed on adding following string url setting in kibana.yml file on cloud environment:
xpack.ingestManager.fleet.kibana.host: "http://0f19b73ebc5c47ab846337ff293239bc.us-central1.gcp.foundit.no:9243/"

Screenshot:

(b) Error is displayed on adding following string array of urls setting in kibana yml file on cloud environment:-
xpack.ingestManager.fleet.kibana.host: ["http://0f19b73ebc5c47ab846337ff293239bc.us-central1.gcp.foundit.no:9243", "http://5a3e547c36a3471fb9b0273088b89428.us-central1.gcp.foundit.no:1234/", "http://a687c886912145408f636ff9b92f28bc.us-central1.gcp.foundit.no:6789/"]

Screenshot:

(c) Error is displayed on adding suggested host.0 setting in kibana yml file on cloud environment::-
xpack.ingestManager.fleet.kibana.host.0: "http://0f19b73ebc5c47ab846337ff293239bc.us-central1.gcp.foundit.no:9243/"

Screenshot:

Please let us know if anything is missing from our end.

EricDavisX · 2020-09-14T19:29:04Z

@rahulgupta-qasource I'm sending you a DM, lets talk testing in a new test ticket instead of this closed pr. Thanks for kicking this off!

neptunian self-assigned this Aug 21, 2020

neptunian mentioned this pull request Aug 21, 2020

[Ingest Manager] Support for multiple kibana urls #75710

Closed

neptunian force-pushed the 72731-allow-multiple-kibana-urls branch from 21e41f1 to 543f456 Compare August 25, 2020 17:58

neptunian added 5 commits August 27, 2020 14:31

let the user specify multiple kibana urls

aa7ac6b

add validation to kibana urls so paths and protocols cannot differ

11382f1

update i18n message

b81c7ba

only send the first url to the instructions

b11aaab

udpate all agent configs' revision when settings is updated

c71e883

neptunian force-pushed the 72731-allow-multiple-kibana-urls branch from ebe2dfb to c71e883 Compare August 27, 2020 18:31

neptunian added release_note:enhancement v7.10.0 v8.0.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Aug 27, 2020

fix jest test

7f9f138

neptunian commented Aug 27, 2020

View reviewed changes

x-pack/plugins/ingest_manager/server/services/agent_policy.ts Outdated Show resolved Hide resolved

neptunian mentioned this pull request Aug 27, 2020

[Ingest Manager] Rename ingest_manager_settings SO attribute kibana_url to kibana_urls #76143

Closed

neptunian marked this pull request as ready for review August 27, 2020 20:03

neptunian requested a review from a team August 27, 2020 20:03

nchaulet reviewed Aug 27, 2020

View reviewed changes

x-pack/plugins/ingest_manager/server/services/agent_policy.ts Show resolved Hide resolved

nchaulet reviewed Aug 27, 2020

View reviewed changes

x-pack/plugins/ingest_manager/server/services/agent_policy.ts Outdated Show resolved Hide resolved

update endpoint full agent policy test

4eb6a85

neptunian requested review from a team as code owners August 28, 2020 13:29

fix type

279bde0

neptunian mentioned this pull request Aug 31, 2020

Changes to Ingest settings (ex ES URL) do not propagate down to running Agents with Elastic Endpoint Security running #76136

Closed

neptunian added 4 commits August 31, 2020 14:58

dont add settings if standalone mode

9e1fbd1

fix ui not handling errors from /{agentPolicyId}/full endpoint

6745e68

Merge branch 'master' into 72731-allow-multiple-kibana-urls

75108cf

fix formatted message id

1591640

only return needed fields

69e76f4

nchaulet approved these changes Aug 31, 2020

View reviewed changes

fill in updated_by and updated_at attributes of the ingest-agent-poli…

1e47f95

…cies when revision is bumped

neptunian mentioned this pull request Sep 1, 2020

[Ingest Manager] Add support for multiples Kibana hosts. #72731

Closed

jonathan-buttner reviewed Sep 1, 2020

View reviewed changes

jonathan-buttner approved these changes Sep 1, 2020

View reviewed changes

neptunian added 2 commits September 1, 2020 12:48

throw error if kibana_urls not set and update tests

f157c43

change ingest_manager_settings SO attribute kibana_url: string to kib…

b644e08

…ana_urls: string[] and add migration

jen-huang reviewed Sep 1, 2020

View reviewed changes

neptunian added 2 commits September 1, 2020 14:02

leave instructions single kibana url

bb7eb07

make kibana_url and other attributes created during setup required, f…

3577435

…ix types

neptunian commented Sep 1, 2020

View reviewed changes

neptunian merged commit a656b96 into elastic:master Sep 2, 2020

neptunian mentioned this pull request Sep 2, 2020

[7.x] [Ingest Manager] support multiple kibana urls (#75712) #76491

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] support multiple kibana urls #75712

[Ingest Manager] support multiple kibana urls #75712

neptunian commented Aug 21, 2020 •

edited

Loading

elasticmachine commented Aug 27, 2020

nchaulet left a comment

jonathan-buttner Sep 1, 2020

paul-tavares Sep 1, 2020

jonathan-buttner left a comment

jen-huang Sep 1, 2020

neptunian Sep 1, 2020

neptunian Sep 1, 2020

kibanamachine commented Sep 1, 2020

ghost commented Sep 14, 2020

EricDavisX commented Sep 14, 2020

[Ingest Manager] support multiple kibana urls #75712

[Ingest Manager] support multiple kibana urls #75712

Conversation

neptunian commented Aug 21, 2020 • edited Loading

Changes

Test

Note

elasticmachine commented Aug 27, 2020

nchaulet left a comment

Choose a reason for hiding this comment

jonathan-buttner Sep 1, 2020

Choose a reason for hiding this comment

paul-tavares Sep 1, 2020

Choose a reason for hiding this comment

jonathan-buttner left a comment

Choose a reason for hiding this comment

jen-huang Sep 1, 2020

Choose a reason for hiding this comment

neptunian Sep 1, 2020

Choose a reason for hiding this comment

neptunian Sep 1, 2020

Choose a reason for hiding this comment

kibanamachine commented Sep 1, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

page load bundle size

distributable file count

History

ghost commented Sep 14, 2020

EricDavisX commented Sep 14, 2020

neptunian commented Aug 21, 2020 •

edited

Loading