Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

Conversation

paul-tavares
Copy link
Contributor

@paul-tavares paul-tavares commented Aug 4, 2020

Summary

When the Endpoint generator is run, in addtion to indexing endpoint host metadata, it should also create the integrations needed for the endpoints in Ingest, which are:

  • Create an Endpoint Package Configuration (and associated Agent Configuration)
  • Enroll an agent in Fleet for the endpoint Host being indexed
  • Check the agent in with Fleet (to remove it from enrolling)
  • Have the Agent send an ack for a configuration change (tells fleet that the agent configuration was applied and lists the Agent in Fleet as a user of endpoint package)

TODO

  • Add CLI option to "skip ingest integration"
  • Find better way to call Fleet APIs that require ApiKey
  • (?) move Ingest/Fleet interaction methods to a separate lib/service for reusability
  • Can the --delete option be supported for the Ingest/Fleet created docs?
  • Caching of created agent private API keys (so that we don't keep creating them if we need subsquent updates to the agent)

Fleet showing an Agent that was created via the Endpoint data generator:

image

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@paul-tavares paul-tavares self-assigned this Aug 4, 2020
@kevinlog
Copy link
Contributor

@elasticmachine merge upstream

@paul-tavares paul-tavares mentioned this pull request Sep 3, 2020
7 tasks
@parkiino parkiino force-pushed the task/endpoint-host-data-generator-create-policy branch from ead9c3a to dfb9627 Compare September 17, 2020 20:55
@parkiino parkiino marked this pull request as ready for review September 21, 2020 17:01
@parkiino parkiino requested review from a team as code owners September 21, 2020 17:01
@parkiino parkiino added Feature:Endpoint Elastic Endpoint feature release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Management v7.10.0 v8.0.0 labels Sep 21, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

@elasticmachine
Copy link
Contributor

Pinging @elastic/endpoint-management (Team:Endpoint Management)

@@ -880,6 +883,14 @@ export interface PolicyConfig {
};
}

export interface AdvancedFields {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this in for the changes that @jmiller263 is making for Advanced policy?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

she's using this updated generator code for her changes, but its not explicitly in her pr

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer that we make changes to the Policy types when we add those new features. @jmiller263 could use the type changes locally until it's time to merge those features.

Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KbnClient changes LGTM

Copy link
Contributor Author

@paul-tavares paul-tavares left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I added a few more comments, but I think they are ok to be worked on after this goes in. :)

Thanks for picking this up.

// Enroll an agent for the Host
const body: PostAgentEnrollRequest['body'] = {
type: 'PERMANENT',
metadata: {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a better metadata message, which will no longer cause the Fleet agent list to show - for host:

{
    //...
    metadata: {
            local: {
                "elastic": {
                  "agent": {
                    "version": versionNumber
                  }
                },
                "host": {
                  "architecture": "x86_64",
                  "hostname": `artifact-downloader.${Date.now()}.elastic.co`,
                  "name": "artifact-downloader",
                  "id": "1c032ec0-3a94-4d54-9ad2-c5610c0eaba4",
                  "ip": [
                    "fe80::703b:b9e6:887d:7f5/64",
                    "10.0.2.15/24",
                    "::1/128",
                    "127.0.0.1/8"
                  ],
                  "mac": [
                    "08:00:27:d8:c5:c0"
                  ]
                },
                "os": {
                  "family": "windows",
                  "kernel": "10.0.19041.388 (WinBuild.160101.0800)",
                  "platform": "windows",
                  "version": "10.0",
                  "name": "Windows 10 Pro",
                  "full": "Windows 10 Pro(10.0)"
                }
            },
}

change local.elastic.agent.version to 8.0.0 (although, we should address that as well by using the kbnClient to query Kibana to get its version number, but we can do that another time (can you open an issue to track? :) )
Also - you will want to change the host.hostname ++ host.name to be endpointHost.host

import { Client, ClientOptions } from '@elastic/elasticsearch';
import { ResponseError } from '@elastic/elasticsearch/lib/errors';
import { KbnClient, ToolingLog } from '@kbn/dev-utils';
import { AxiosResponse } from 'axios';
import { KibanaConfig } from '@kbn/dev-utils/target/kbn_client/kbn_client_requester';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check if this can be imported directly from @kbn/dev-utils? if so, then just add it to the import above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think it can be

Copy link
Contributor

@kevinlog kevinlog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -98,3 +198,262 @@ async function indexAlerts(
await client.bulk({ body, refresh: true });
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paul-tavares this is a significant code change to the core class. I think it may be good to separate this code out. And from a cursory look it may be difficult to maintain and change. Perhaps a follow up PR with that view in mind. Because if there is any changes in the system we have to change this code also.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nnamdifrankie Agreed - some of this needs to be broken out into smaller set of classes/namespaces. but: could you clarify what class you are referring to? (or did you mean the entire module index_data.ts?)

Also agree that if the system changes, this will have to be updated, but I think it valuable to have the ability to create a fake set of data across the entire system (minus the actual endpoint). Lets disucss if perhaps you feel strongly that we should not be creating this type of "full feature" tooling 😄

also - just FYI: my idea is to ultimately create a FakeAgent class that will allows us to enroll and control agents against fleet without actually having an agent running (just sending messages to fleet - some of that will land in this external tool). The ability to do this will significantly improve our ability to do CI end-to-end testing within Kibana (Functional testing) - especially for our Endpoint (hosts) List page which is driven by a combination of both endpoint metadata and Fleet/Agent data (maybe even create a FakeEndpoint class to mimic Endpoints sending data to elastic).

Copy link
Contributor

@marshallmain marshallmain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great overall, just a few questions and type nits

);
// eslint-disable-next-line no-process-exit
process.exit(1);
const setupResponse = (await kbnClient.request({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 kbnClient looks like a much cleaner way to do this

agentPolicyId: string
): Promise<undefined | PostAgentEnrollResponse['item']> => {
// Get Enrollement key for host's applied policy
const enrollmentApiKey = await kbnClient
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could consider making this request.then.then.catch its own function called getEnrollmentApiKey or similar

@@ -32,12 +32,11 @@ import {
} from '../../../ingest_manager/common';
import { factory as policyConfigFactory } from './models/policy_config';
import { HostMetadata } from './types';
import { KbnClientWithApiKeySupport } from '../../scripts/endpoint/resolver_generator_script';
Copy link
Contributor

@kevinlog kevinlog Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import from resolver_generator_script.ts is causing a cyclic dependency since that file also imports from here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts#L13

PostIngestSetupResponse,
} from '../../../ingest_manager/common/types/rest_spec';

export class KbnClientWithApiKeySupport extends KbnClient {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you move this to another file and import it from the files you need it, you should get rid of the cyclic dependency

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

distributable file count

id value diff baseline
default 45807 +1 45806

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@parkiino parkiino merged commit 15e7623 into elastic:master Sep 30, 2020
@parkiino parkiino deleted the task/endpoint-host-data-generator-create-policy branch September 30, 2020 20:08
phillipb added a commit to phillipb/kibana that referenced this pull request Sep 30, 2020
…aly-detection-partition-field

* 'master' of github.com:elastic/kibana: (37 commits)
  Fixes for the Ticket 78375 (elastic#79004)
  [Security] Alert Telemetry for the Security app (elastic#77200)
  [Search bar] Remove duplicate `popoverProps` (elastic#79025)
  [Security Solution][Detections] Add rule overrides for single event EQL rules (elastic#78876)
  [SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest (elastic#74305)
  remove file accidentally checked in (elastic#79005)
  [ML] DF Analytics creation wizard: replace select input with job type cards with icons (elastic#78872)
  [Design] A couple fixes for 7.10 (elastic#78801)
  Fix KQL autocomplete value suggestions (elastic#78676)
  [Security Solution][Resolver] New mock with cursor (elastic#78863)
  Embeddables: basic documentation (elastic#78900)
  [security solution] only import beat_schema when needed (elastic#78708)
  [Reporting] API Integration tests: fix flaky tests for Spaces CSV formatting (elastic#78849)
  [Actions] Adds a "Test Connector" button on the Connectors List to make discovery of the Test tab easier (elastic#78746)
  [Discover] Fix functional time picker test permissions (elastic#78564)
  [ML] Fixing module datafeed overrides (elastic#78925)
  Adds some missing licenses to the CSV export (elastic#78719)
  [dev/cli] ensure plugins/ and all watch source dirs exist (elastic#78973)
  [Lens] Stop using scripted metric to collect telemetry (elastic#78687)
  [Lens] fix wrong message in fields accordion (elastic#78924)
  ...
parkiino added a commit that referenced this pull request Oct 1, 2020
… also integrate with Ingest (#74305) (#79028)

* Endpoint generator connects host with a real policy and enrolls agent

Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
Co-authored-by: kevinlog <kevin.logan@elastic.co>
Co-authored-by: Candace Park <candace.park@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Endpoint Elastic Endpoint feature release_note:skip Skip the PR/issue when compiling release notes v7.10.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants