Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] AIOps: Adds Pattern analysis tab in Discover #178916

Merged

Conversation

jgowdyelastic
Copy link
Member

@jgowdyelastic jgowdyelastic commented Mar 18, 2024

Closes #178534

Replaces the pattern analysis flyout in Discover with a tab which sits alongside the Documents and Field statistics tabs.

image

Field selection
Lists all of the text fields in the index. Auto selects message, then error.message, then event.original and if none of these fields are available, it just selects the first field in the list.

image

The Options menu provides some configuration options:

Minimum time range
Sets the minimum time range used for the pattern analysis search. The pattern matching results results will be more accurate the more data it sees, so if the user has selected e.g. last 15mins in the time picker, this settings will ensure a wider time range is used to improve the accuracy of the patterns. If the time picker has a larger time range than this setting, the larger time range will be used.

Random sampling
Improves the search performance by using a random sampler. This is the same setting as before.

image

@jgowdyelastic
Copy link
Member Author

/ci

@jgowdyelastic
Copy link
Member Author

/ci

@jgowdyelastic
Copy link
Member Author

/ci

@qn895
Copy link
Member

qn895 commented May 17, 2024

Tested latest changes and ML-changes LGTM. Please feel free to merge this one ahead of the ES|QL field stats in Discover PR 👍 . I'll wait to merge it after this one is in.

import { METRIC_TYPE, UiCounterMetricType } from '@kbn/analytics';
import { type EmbeddablePatternAnalysisInput } from '@kbn/aiops-log-pattern-analysis/embeddable';
import { pick } from 'lodash';
import { LogCategorizationEmbeddableProps } from '@kbn/aiops-plugin/public/components/log_categorization/log_categorization_for_embeddable/log_categorization_for_embeddable';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: import type

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 1aaa89f

@andreadelrio andreadelrio self-requested a review May 20, 2024 04:19
@andreadelrio
Copy link
Contributor

Hi @jgowdyelastic sending design comments below:

  • Add a heading to the Pattern analysis settings popover. For reference, this should look like the Keyboard shortcuts popover in the Document tab.
Frame 1
  • Increase the padding on the left side of the Patterns tab to achieve something similar to what we have in the Documents tab.
Frame 9

Long term and as I mentioned to Aris, it would be beneficial to consider using EuiDataGrid for this view as well. It would match nicely with the Documents tab. On that note, given the amount of information you're displaying when expanding the row, I would suggest considering moving it to a flyout. This would be similar to how in Documents tab the user presses the expand icon and the push flyout appears. The pattern shown in the flyout would be highlighted in the main Patterns table. Here's a quick mockup of how that flyout could look like:

Flyout

cc @kertal @davismcphee @ryankeairns for awareness.

@jgowdyelastic
Copy link
Member Author

@andreadelrio

  • Add a heading to the Pattern analysis settings popover. For reference, this should look like the Keyboard shortcuts popover in the Document tab.

Updated in 13e4b30

  • Increase the padding on the left side of the Patterns tab to achieve something similar to what we have in the Documents tab.

Updated in. 13e4b30

Long term and as I mentioned to Aris, it would be beneficial to consider using EuiDataGrid for this view as well. It would match nicely with the Documents tab. On that note, given the amount of information you're displaying when expanding the row, I would suggest considering moving it to a flyout. This would be similar to how in Documents tab the user presses the expand icon and the push flyout appears. The pattern shown in the flyout would be highlighted in the main Patterns table. Here's a quick mockup of how that flyout could look like:

Yeah, I think this would be an improvement to the expanded rows.
I think we would also need to navigation controls similar to the document flyout to allow users to easily move between patterns. e.g.
image
I can look into this for a follow up PR.

Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested latest changes and LGTM.

Copy link
Contributor

@jughosta jughosta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great feature! And thanks for addressing the comments!

Some things to consider improving as a follow up:

  • What are these warnings 0% contain 3 or more tokens about? The shown patterns seem to be fine:
Screenshot 2024-05-22 at 10 52 20
  • Noticed that cancelling a patterns request does not always work right away. Is it expected?

  • Once canceled, should there be a button to restart?

Screenshot 2024-05-22 at 10 41 07

Style nits:

  • The label is to close to the side:
Screenshot 2024-05-22 at 10 39 58
  • Mobile view is a little off:
Screenshot 2024-05-22 at 10 37 38 Screenshot 2024-05-22 at 10 57 08 Screenshot 2024-05-22 at 10 56 28

@jgowdyelastic
Copy link
Member Author

Follow on tasks have been added to this meta issue #183922

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
aiops 610 618 +8
apm 1806 1799 -7
discover 910 906 -4
eventAnnotationListing 613 606 -7
lens 1477 1470 -7
logsExplorer 600 593 -7
securitySolution 5497 5490 -7
slo 846 839 -7
total -38

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
aiops 4 9 +5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 520.7KB 547.1KB +26.4KB
dataVisualizer 757.9KB 757.9KB +28.0B
discover 809.4KB 810.0KB +616.0B
lens 1.5MB 1.5MB -1.5KB
ml 4.2MB 4.2MB +56.0B
securitySolution 15.1MB 15.1MB -1.5KB
slo 870.2KB 868.7KB -1.5KB
transform 479.4KB 479.4KB +28.0B
total +22.5KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
@kbn/unified-field-list 10 9 -1
aiops 1 2 +1
total -0

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
aiops 7.1KB 7.5KB +400.0B
discover 35.6KB 35.6KB +14.0B
savedSearch 11.5KB 11.5KB +27.0B
total +441.0B
Unknown metric groups

API count

id before after diff
@kbn/ml-date-picker 48 50 +2
aiops 67 74 +7
total +9

async chunk count

id before after diff
aiops 24 27 +3

ESLint disabled line counts

id before after diff
@kbn/unified-field-list 17 16 -1
aiops 26 29 +3
total +2

Total ESLint disabled count

id before after diff
@kbn/unified-field-list 17 16 -1
aiops 26 29 +3
total +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

@jgowdyelastic jgowdyelastic merged commit a69f24b into elastic:main May 22, 2024
19 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label May 22, 2024
@szabosteve szabosteve changed the title [ML] [AIOps] Pattern analysis tab in Discover [ML] [AIOps] Adds Pattern analysis tab in Discover Jul 31, 2024
@szabosteve szabosteve changed the title [ML] [AIOps] Adds Pattern analysis tab in Discover [ML] AIOps: Adds Pattern analysis tab in Discover Jul 31, 2024
@kertal kertal added the Feature:Discover Discover Application label Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Discover Discover Application Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml release_note:feature Makes this part of the condensed release notes v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ML] [AIOps] Add pattern analysis tab to Discover