Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

Merged
merged 31 commits into from
Sep 30, 2020
Merged

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
293eeaf
Create Policies for each generated host
paul-tavares Aug 3, 2020
2b5ab65
Refactor Ingest setup to also setup Fleet
paul-tavares Aug 3, 2020
ad39d54
Rename prop name
paul-tavares Aug 3, 2020
c2aea57
Add generic response type to KbnClient.request + support for headers
paul-tavares Aug 3, 2020
b71e438
first attempt at adding fleet agent registration
paul-tavares Aug 4, 2020
73a0754
a little closer with fleet integration
paul-tavares Aug 4, 2020
a8221bb
SUCCESS. Able to enroll agent and set it to online
paul-tavares Aug 4, 2020
e3ef509
update names to be policy
kevinlog Aug 23, 2020
4deebbf
policy generator has advanced types in endpoint confit
parkiino Aug 31, 2020
0e61d00
use KbnClientWithRequestAPI
parkiino Sep 16, 2020
dfb9627
fix typecheck errors
parkiino Sep 17, 2020
76db23b
add flag to skip fleet
parkiino Sep 17, 2020
a6b6d3f
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 21, 2020
848eec5
fix tests
parkiino Sep 21, 2020
4caaf96
make advanced field type optional
parkiino Sep 22, 2020
02cd8fa
remove advanced fields
parkiino Sep 22, 2020
58ef6d5
another advanced removal
parkiino Sep 22, 2020
7b82e05
revert experimental change
parkiino Sep 22, 2020
e8ead7f
shorten urls that use requestWithApiKey
parkiino Sep 22, 2020
51730fa
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 22, 2020
65596ec
remove excess quaotes
parkiino Sep 23, 2020
047925b
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 23, 2020
c6ee44e
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 23, 2020
2f4d043
fix ci errors
parkiino Sep 23, 2020
c3d9282
agent enrollment fix
parkiino Sep 28, 2020
02d122f
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 28, 2020
321546a
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 28, 2020
56d2c99
marshall's changes + edits to hostname
parkiino Sep 29, 2020
edd7bde
fetch kibana version
parkiino Sep 30, 2020
575282e
Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…
parkiino Sep 30, 2020
369318c
remove circular dependency
parkiino Sep 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 21 additions & 10 deletions x-pack/plugins/security_solution/common/endpoint/index_data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,11 @@ import {
} from '../../../ingest_manager/common';
import { factory as policyConfigFactory } from './models/policy_config';
import { HostMetadata } from './types';
import { KbnClientWithApiKeySupport } from '../../scripts/endpoint/resolver_generator_script';
Copy link
Contributor

@kevinlog kevinlog Sep 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import from resolver_generator_script.ts is causing a cyclic dependency since that file also imports from here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts#L13


export async function indexHostsAndAlerts(
client: Client,
kbnClient: KbnClient & {
requestWithApiKey: (path: string, init?: RequestInit | undefined) => Promise<Response>;
},
kbnClient: KbnClientWithApiKeySupport,
seed: string,
numHosts: number,
numDocs: number,
Expand Down Expand Up @@ -85,9 +84,7 @@ function delay(ms: number) {
async function indexHostDocs(
numDocs: number,
client: Client,
kbnClient: KbnClient & {
requestWithApiKey: (path: string, init?: RequestInit | undefined) => Promise<Response>;
},
kbnClient: KbnClientWithApiKeySupport,
realPolicies: Record<string, CreatePackagePolicyResponse['item']>,
epmEndpointPackage: GetPackagesResponse['response'][0],
metadataIndex: string,
Expand Down Expand Up @@ -273,9 +270,7 @@ const getEndpointPackageInfo = async (
};

const fleetEnrollAgentForHost = async (
kbnClient: KbnClient & {
requestWithApiKey: (path: string, init?: RequestInit | undefined) => Promise<Response>;
},
kbnClient: KbnClientWithApiKeySupport,
endpointHost: HostMetadata,
agentPolicyId: string
): Promise<undefined | PostAgentEnrollResponse['item']> => {
Expand Down Expand Up @@ -321,17 +316,33 @@ const fleetEnrollAgentForHost = async (
return;
}

const kibanaVersion = await kbnClient.fetchKibanaVersion().number;
// Enroll an agent for the Host
const body: PostAgentEnrollRequest['body'] = {
type: 'PERMANENT',
metadata: {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a better metadata message, which will no longer cause the Fleet agent list to show - for host:

{
    //...
    metadata: {
            local: {
                "elastic": {
                  "agent": {
                    "version": versionNumber
                  }
                },
                "host": {
                  "architecture": "x86_64",
                  "hostname": `artifact-downloader.${Date.now()}.elastic.co`,
                  "name": "artifact-downloader",
                  "id": "1c032ec0-3a94-4d54-9ad2-c5610c0eaba4",
                  "ip": [
                    "fe80::703b:b9e6:887d:7f5/64",
                    "10.0.2.15/24",
                    "::1/128",
                    "127.0.0.1/8"
                  ],
                  "mac": [
                    "08:00:27:d8:c5:c0"
                  ]
                },
                "os": {
                  "family": "windows",
                  "kernel": "10.0.19041.388 (WinBuild.160101.0800)",
                  "platform": "windows",
                  "version": "10.0",
                  "name": "Windows 10 Pro",
                  "full": "Windows 10 Pro(10.0)"
                }
            },
}

change local.elastic.agent.version to 8.0.0 (although, we should address that as well by using the kbnClient to query Kibana to get its version number, but we can do that another time (can you open an issue to track? :) )
Also - you will want to change the host.hostname ++ host.name to be endpointHost.host

local: {
host: endpointHost.host,
elastic: {
agent: {
version: '8.0.0',
},
},
host: {
architecture: 'x86_64',
hostname: endpointHost.host,
name: endpointHost.host,
id: '1c032ec0-3a94-4d54-9ad2-c5610c0eaba4',
ip: ['fe80::703b:b9e6:887d:7f5/64', '10.0.2.15/24', '::1/128', '127.0.0.1/8'],
mac: ['08:00:27:d8:c5:c0'],
},
os: {
family: 'windows',
kernel: '10.0.19041.388 (WinBuild.160101.0800)',
platform: 'windows',
version: '10.0',
name: 'Windows 10 Pro',
full: 'Windows 10 Pro(10.0)',
},
},
user_provided: {
dev_agent_version: '0.0.1',
Expand Down
5 changes: 4 additions & 1 deletion x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import {
PostIngestSetupResponse,
} from '../../../ingest_manager/common/types/rest_spec';

class KbnClientWithApiKeySupport extends KbnClient {
export class KbnClientWithApiKeySupport extends KbnClient {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you move this to another file and import it from the files you need it, you should get rid of the cyclic dependency

private kibanaUrlNoAuth: string;
constructor(log: ToolingLog, kibanaConfig: KibanaConfig) {
super(log, kibanaConfig);
Expand All @@ -31,6 +31,9 @@ class KbnClientWithApiKeySupport extends KbnClient {
? matches[1] + matches[3].replace('/', '')
: kibanaUrl.replace('/', '');
}
/**
* The fleet api to enroll and agent requires an api key when you mke the request, however KbnClient currently does not support sending an api key with the request. This function allows you to send an api key with a request.
*/
requestWithApiKey(path: string, init?: RequestInit | undefined): Promise<Response> {
marshallmain marked this conversation as resolved.
Show resolved Hide resolved
return (fetch(
`${this.kibanaUrlNoAuth}${path}`,
Expand Down