[APM] Adds 'Anomaly detection' settings page to create ML jobs per environment #70560

ogupte · 2020-07-02T08:26:02Z

Closes #70422.
Adds 'Anomaly detection' settings page along with required API endpoints to list and create the APM anomaly detection jobs per environment.

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

formgeist

Just a quick review for the copy mainly as I had updated it yesterday.

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/index.tsx

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/jobs_list.tsx

formgeist · 2020-07-02T09:29:13Z

@bmorelli25 Can you give the general copy a quick look and give your suggestion for improvements. This was written very quickly by yours truly, but I feel like it needs professional help 😊

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

to list and create the apm anomaly detection jobs per environment. Some test data is hardcoded while the the required changes in the ML plugin are in flight.

in groups array. Also adds random token to the job ID to prevent collisions for job ids where diffferent environment names convert to the same string

…ML module - Implements job list in settings by reading from `custom_settings.job_tags['service.environment']` - Add ML module method `createModuleItem` for job configuration - Don't allow user to type in duplicate environments

…ction/add_environments.tsx Co-authored-by: Casper Hübertz <casper@formgeist.com>

…ction/index.tsx Co-authored-by: Casper Hübertz <casper@formgeist.com>

- makes the 'all' environment name ALL_OPTION_VALUE agent configuration-specific - replace field literals with constants

elasticmachine · 2020-07-04T07:22:23Z

Pinging @elastic/apm-ui (Team:apm)

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

sorenlouv

Looks really good. Just a bunch of nits

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

x-pack/plugins/apm/server/routes/settings/anomaly_detection.ts

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

x-pack/plugins/apm/server/lib/anomaly_detection/get_anomaly_detection_jobs.ts

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/jobs_list.tsx

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

x-pack/plugins/apm/typings/anomaly_detection.ts

…-detection-settings

bmorelli25

A few text suggestions

x-pack/plugins/apm/public/components/app/Settings/index.tsx

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/jobs_list.tsx

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/index.tsx

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx

… 'Not defined' option is disabled if it already exists.

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

documents without service.environment set

… indices

ogupte · 2020-07-07T04:47:12Z

@elasticmachine merge upstream

sorenlouv

Lgtm

x-pack/plugins/ml/common/types/anomaly_detection_jobs/job.ts

jgowdyelastic

ML changes LGTM

kibanamachine · 2020-07-07T11:22:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: f42e553

Build metrics

✅ unchanged

History

💚 Build #59255 succeeded 680ea4b
💔 Build #59218 failed 12c55d0
💚 Build #59076 succeeded 69bcc2c
💚 Build #59060 succeeded 9b86faa
💚 Build #59044 succeeded e3b2cec

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: Rename legacy ES mock accessors (elastic#70432) [APM] Adds 'Anomaly detection' settings page to create ML jobs per environment (elastic#70560) Forbid timezones not working in Elasticsearch (elastic#70780)

…rbac * alerting/consumer-based-rbac: Rename legacy ES mock accessors (elastic#70432) [APM] Adds 'Anomaly detection' settings page to create ML jobs per environment (elastic#70560) Forbid timezones not working in Elasticsearch (elastic#70780) [ML] Adding peak_model_bytes to model size stats type (elastic#70825)

…vironment (elastic#70560) * Adds 'Anomaly detection' settings page along with require API endpoints to list and create the apm anomaly detection jobs per environment. Some test data is hardcoded while the the required changes in the ML plugin are in flight. * Converts the environment name to a compatible ML id string and persist in groups array. Also adds random token to the job ID to prevent collisions for job ids where diffferent environment names convert to the same string * - Improve job creation with latest updates for the `apm_transaction` ML module - Implements job list in settings by reading from `custom_settings.job_tags['service.environment']` - Add ML module method `createModuleItem` for job configuration - Don't allow user to type in duplicate environments * Update x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx Co-authored-by: Casper Hübertz <casper@formgeist.com> * Update x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/index.tsx Co-authored-by: Casper Hübertz <casper@formgeist.com> * UX feedback, adds i18n, and handles failed state for ML jobs fetch. * - Moves get_all_environments from agent_configuration dir to common dir - makes the 'all' environment name ALL_OPTION_VALUE agent configuration-specific - replace field literals with constants * PR feedback * Adds support to create jobs for environment which are not defined. * Fixes description copy, rearranges settings links, and makes sure the 'Not defined' option is disabled if it already exists. * Only show "Not defined" in environment selector if there are actually documents without service.environment set * get the indexPatternName for the ML job from the set of user-definned indices * updated job_tags type definition Co-authored-by: Casper Hübertz <casper@formgeist.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…vironment (#70560) (#70945)

sorenlouv · 2020-07-07T22:25:24Z

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

+};
+
+export function convertToMLIdentifier(value: string) {
+ return value.replace(/\s+/g, '_').toLowerCase();


@ogupte we should improve this

// remove non-alphanumeric characters retur value.toLowerCase().replace(/[^a-z0-9+]+/g, '_');

i simplified this in another PR:

kibana/x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts

Line 86 in 55b1e91

prefix: `${ML_GROUP_NAME_APM}-${snakeCase(environment)}-${randomToken}-`,

to just use the lodash function snakeCase. This also has the effect of deburring and removing all invalid chars

sorenlouv · 2020-07-07T22:28:46Z

x-pack/plugins/apm/server/lib/anomaly_detection/get_anomaly_detection_jobs.ts

+ return [];
+ }
+ try {
+ const { jobs } = await ml.anomalyDetectors.jobs(ML_GROUP_NAME_APM);


It just occurs to me that we are retrieving jobs without specifying that we are filtering by groups. Will this return for any field that matches ML_GROUP_NAME_APM?

I'd expect something like:

const { jobs } = await ml.anomalyDetectors.jobs({group: ML_GROUP_NAME_APM });

Similarly, if we want to retrieve jobs by job id:

const { jobs } = await ml.anomalyDetectors.job({ id: 'some-id' });

Use "apm-*" to match the new job IDs added in elastic#70560.

* [APM] Update ML job ID in data telemetry tasks Use "apm-*" to match the new job IDs added in #70560. * additional fix * Remove unused import

* [APM] Update ML job ID in data telemetry tasks Use "apm-*" to match the new job IDs added in elastic#70560. * additional fix * Remove unused import

Use "apm-*" to match the new job IDs added in #70560.

ogupte added release_note:enhancement blocked v7.9.0 labels Jul 2, 2020

ogupte mentioned this pull request Jul 2, 2020

[ML] Updates APM Module to Work with Service Maps #70361

Merged

jgowdyelastic reviewed Jul 2, 2020

View reviewed changes

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts Outdated Show resolved Hide resolved

formgeist reviewed Jul 2, 2020

View reviewed changes

formgeist requested a review from bmorelli25 July 2, 2020 09:28

sorenlouv reviewed Jul 2, 2020

View reviewed changes

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts Outdated Show resolved Hide resolved

ogupte removed the blocked label Jul 2, 2020

ogupte force-pushed the apm-70422-anomaly-detection-settings branch from 2ce5235 to 0c17927 Compare July 2, 2020 22:03

ogupte and others added 7 commits July 3, 2020 23:54

Adds 'Anomaly detection' settings page along with require API endpoints

67fa184

to list and create the apm anomaly detection jobs per environment. Some test data is hardcoded while the the required changes in the ML plugin are in flight.

Converts the environment name to a compatible ML id string and persist

03cc6a0

in groups array. Also adds random token to the job ID to prevent collisions for job ids where diffferent environment names convert to the same string

Update x-pack/plugins/apm/public/components/app/Settings/anomaly_dete…

cbe5c3b

…ction/add_environments.tsx Co-authored-by: Casper Hübertz <casper@formgeist.com>

Update x-pack/plugins/apm/public/components/app/Settings/anomaly_dete…

89dec0c

…ction/index.tsx Co-authored-by: Casper Hübertz <casper@formgeist.com>

UX feedback, adds i18n, and handles failed state for ML jobs fetch.

925b600

- Moves get_all_environments from agent_configuration dir to common dir

cb9a8ad

- makes the 'all' environment name ALL_OPTION_VALUE agent configuration-specific - replace field literals with constants

ogupte force-pushed the apm-70422-anomaly-detection-settings branch from 417d01d to cb9a8ad Compare July 4, 2020 07:21

ogupte marked this pull request as ready for review July 4, 2020 07:22

ogupte requested review from a team as code owners July 4, 2020 07:22

botelastic bot added the Team:APM All issues that need APM UI Team support label Jul 4, 2020

sorenlouv reviewed Jul 5, 2020

View reviewed changes

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts Outdated Show resolved Hide resolved

sorenlouv approved these changes Jul 5, 2020

View reviewed changes

ogupte added 3 commits July 6, 2020 08:54

Merge remote-tracking branch 'upstream/master' into apm-70422-anomaly…

8a2e968

…-detection-settings

PR feedback

e3b2cec

Adds support to create jobs for environment which are not defined.

9b86faa

bmorelli25 reviewed Jul 6, 2020

View reviewed changes

Fixes description copy, rearranges settings links, and makes sure the…

69bcc2c

… 'Not defined' option is disabled if it already exists.

sorenlouv reviewed Jul 6, 2020

View reviewed changes

x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx Outdated Show resolved Hide resolved

ogupte mentioned this pull request Jul 6, 2020

[APM] Add anomaly detection high/low bounds per bucket for transaction duration chart #70859

Closed

ogupte commented Jul 6, 2020

View reviewed changes

x-pack/plugins/apm/server/lib/anomaly_detection/create_anomaly_detection_jobs.ts Outdated Show resolved Hide resolved

ogupte added 2 commits July 6, 2020 16:46

Only show "Not defined" in environment selector if there are actually

2dca952

documents without service.environment set

get the indexPatternName for the ML job from the set of user-definned…

12c55d0

… indices

Merge branch 'master' into apm-70422-anomaly-detection-settings

680ea4b

sorenlouv approved these changes Jul 7, 2020

View reviewed changes

jgowdyelastic reviewed Jul 7, 2020

View reviewed changes

x-pack/plugins/ml/common/types/anomaly_detection_jobs/job.ts Outdated Show resolved Hide resolved

updated job_tags type definition

f42e553

jgowdyelastic approved these changes Jul 7, 2020

View reviewed changes

sorenlouv merged commit 7d44d02 into elastic:master Jul 7, 2020

sorenlouv mentioned this pull request Jul 7, 2020

[7.x] [APM] Adds 'Anomaly detection' settings page to create ML jobs per environment (#70560) #70945

Merged

sorenlouv added a commit that referenced this pull request Jul 7, 2020

[APM] Adds 'Anomaly detection' settings page to create ML jobs per en…

607ee02

…vironment (#70560) (#70945)

sorenlouv reviewed Jul 7, 2020

View reviewed changes

smith added a commit to smith/kibana that referenced this pull request Jul 8, 2020

[APM] Update ML job ID in data telemetry tasks

8dbb0aa

Use "apm-*" to match the new job IDs added in elastic#70560.

smith mentioned this pull request Jul 8, 2020

[APM] Update ML job ID in data telemetry tasks #71044

Merged

smith added a commit that referenced this pull request Jul 9, 2020

[APM] Update ML job ID in data telemetry tasks (#71044)

1f5a1fe

* [APM] Update ML job ID in data telemetry tasks Use "apm-*" to match the new job IDs added in #70560. * additional fix * Remove unused import

smith added a commit that referenced this pull request Jul 9, 2020

[APM] Update ML job ID in data telemetry tasks (#71044) (#71185)

014dce0

Use "apm-*" to match the new job IDs added in #70560.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Adds 'Anomaly detection' settings page to create ML jobs per environment #70560

[APM] Adds 'Anomaly detection' settings page to create ML jobs per environment #70560

ogupte commented Jul 2, 2020 •

edited

Loading

formgeist left a comment

formgeist commented Jul 2, 2020

elasticmachine commented Jul 4, 2020

sorenlouv left a comment

bmorelli25 left a comment

ogupte commented Jul 7, 2020

sorenlouv left a comment

jgowdyelastic left a comment

kibanamachine commented Jul 7, 2020

sorenlouv Jul 7, 2020

ogupte Jul 8, 2020

sorenlouv Jul 7, 2020 •

edited

Loading

[APM] Adds 'Anomaly detection' settings page to create ML jobs per environment #70560

[APM] Adds 'Anomaly detection' settings page to create ML jobs per environment #70560

Conversation

ogupte commented Jul 2, 2020 • edited Loading

formgeist left a comment

Choose a reason for hiding this comment

formgeist commented Jul 2, 2020

elasticmachine commented Jul 4, 2020

sorenlouv left a comment

Choose a reason for hiding this comment

bmorelli25 left a comment

Choose a reason for hiding this comment

ogupte commented Jul 7, 2020

sorenlouv left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

kibanamachine commented Jul 7, 2020

💚 Build Succeeded

Build metrics

History

sorenlouv Jul 7, 2020

Choose a reason for hiding this comment

ogupte Jul 8, 2020

Choose a reason for hiding this comment

sorenlouv Jul 7, 2020 • edited Loading

Choose a reason for hiding this comment

ogupte commented Jul 2, 2020 •

edited

Loading

sorenlouv Jul 7, 2020 •

edited

Loading