Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] AIOps: 8.15 Log Rate Analysis improvements. #181111

Closed
11 tasks done
walterra opened this issue Apr 18, 2024 · 2 comments
Closed
11 tasks done

[ML] AIOps: 8.15 Log Rate Analysis improvements. #181111

walterra opened this issue Apr 18, 2024 · 2 comments
Assignees
@walterra @alvarezmelissa87
Labels
Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis Meta :ml v8.15.0

Comments

@walterra
Copy link
Contributor

walterra commented Apr 18, 2024 •
edited
Loading

Follow up to #172981.

API
Preview Give feedback
  1. [ML] AIOps: Remove v1 of log rate analysis API. #181803
    :ml Feature:ML/AIOps backport:skip release_note:skip v8.15.0
    walterra
  2. [ML] AIOps: Adds cardinality check to Log Rate Analysis #181129
    :ml Feature:ML/AIOps backport:skip release_note:enhancement v8.15.0
    walterra
  3. [ML] AIOps Log Rate Analysis: Merges fetch queue for keyword and text field candidates #183649
    :ml Feature:ML/AIOps backport:skip release_note:enhancement v8.15.0
    walterra
  4. [ML] AIOps Log Rate Analysis: Fixes text field selection #186176
    :ml Feature:ML/AIOps bug release_note:fix v8.14.2 v8.15.0
    walterra

UI
Preview Give feedback
  2. [ML] AIOps: Reduces rerenders when streaming analysis results #182793
    :ml Feature:ML/AIOps backport:skip release_note:enhancement v8.15.0
    walterra
  3. [ML] Adds redux toolkit example for response_stream to developer examples. #182690
    :ml backport:skip release_note:skip v8.15.0
    walterra
  4. [ML] AIOps Log Rate Analysis: Replace custom global state management with Redux Toolkit. #180969
    :ml Feature:ML/AIOps backport:skip release_note:skip v8.15.0
    walterra

Text field pattern support
Preview Give feedback
  2. [ML] AIOps: Fix the text field icon margins in the log rate analysis results table. #186520
    :ml Feature:ML/AIOps backport:skip release_note:skip v8.15.0
    walterra

Observability AI Assistant Context
Preview Give feedback
  1. [ML] AIOps: Adds AI Assistant contextual insight to Log Rate Analysis page in ML plugin in Observability serverless #186509
    :ml Feature:ML/AIOps backport:skip release_note:feature v8.15.0
    walterra
@walterra walterra added Meta :ml Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis v8.15.0 labels Apr 18, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

This was referenced Apr 18, 2024
Merged
Closed
@walterra walterra mentioned this issue Apr 26, 2024
Merged
2 tasks
walterra added a commit that referenced this issue Apr 26, 2024
@walterra
[ML] AIOps: Remove v1 of log rate analysis API. (#181803)
a79d201 
## Summary

Follow up to #170274. Part of #181111 and #181603.

We had v1 and v2 of the log rate analysis API for a while now (Nov 23).
Originally we did this to practice API versioning for serverless. Enough
time has passed so we can remove v1 which this PR does.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
@alvarezmelissa87 alvarezmelissa87 mentioned this issue May 6, 2024
Closed
2 tasks
@walterra walterra mentioned this issue May 7, 2024
Merged
3 tasks
This was referenced May 15, 2024
Merged
Merged
Merged
walterra added a commit that referenced this issue May 21, 2024
@walterra
[ML] AIOps: Adds cardinality check to Log Rate Analysis (#181129)
d43c8f9 
## Summary

Part of #181111.

This filters field/value items from the results if the field has a
cardinality of just 1 since it wouldn't be useful as a result.

Before (you can easily spot fields with cardinality of 1 because the
mini histogram in the table is just one color which means the value is
present in all logs):


![image](https://github.com/elastic/kibana/assets/230104/2904c026-5a69-43b7-b80f-87923368b506)

After:


![image](https://github.com/elastic/kibana/assets/230104/7a9bffae-9991-4584-91f1-cff9fdc1eaf1)

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
walterra added a commit that referenced this issue May 21, 2024
@walterra
[ML] AIOps Log Rate Analysis: Merge fetch queue for keyword and text …
4c9d65b 
…field candidates. (#183649)

## Summary

Part of #181111.

So far we identified significant `text` field candidates before
`keyword` field candidates. There were 2 problems with this: 1) The
loading behaviour for `text` fields was not considered in the progress
bar. 2) Analysing a `text` field can be quite slow we weren't taking
advantage of the same queue we used to fetch `keyword` fields. If it was
taking quite a while to analysis the `text` fields, it would appear the
progress bar was "stuck". And we had to wait to finish those slow
queries before we started fetching p-values for `keyword` fields.

This PR improves the behaviour by adding `text` field fetching to the
same queue that fetches `keyword` fields. This means while a maybe
slower `text` field is analysed, we can continue to fetch `keyword`
based fields in parallel given the max concurrency.

It's a bit tricky to test because with the local datasets the analysis
might be so fast you won't notice a difference. So for testing you could
do the following: In the file `significant_items_handler.ts` at the top
add:

```
const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
```

And further down around line `110` add the following delay:

```
   const pValuesQueue = queue(async function (payload: Candidate) {
      await delay(Math.random() * 10000);

      if (isFieldCandidate(payload)) {
 ...
```

This will randomly delay the next call for the text/keyword field so you
should see the progress bar moving. The difference to before would be
that the progress bar would not move while it analyses the text fields.

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
walterra added a commit that referenced this issue May 21, 2024
@walterra
[ML] AIOps Log Rate Analysis: Fix date picker refresh button. (#183768)
63441fd 
## Summary

Part of #181111.

The refresh button wasn't working as expected. It would refetch the date
histogram, but if you had a time set like `Last 15 minutes`, the time
range on the page wasn't updated. This PR adds a fix to trigger a
refresh of active bounds. It fixes a problem with the deviation brush
not properly updating too. Without the fix, the deviation brush would
not move when the time range changes.


[aiops-lra-refresh-fix-0001.webm](https://github.com/elastic/kibana/assets/230104/70d51b1c-e831-4971-b385-3c455632b8eb)

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
walterra added a commit that referenced this issue May 22, 2024
@walterra
[ML] Adds redux toolkit example for response_stream to developer ex…
5345e34 
…amples. (#182690)

## Summary

Follow up to #132590.
Part of #181111.

This updates the developer examples for `@kbn/ml-response-stream` to
include a variant with a full Redux Toolkit setup. For this case, the
`@kbn/ml-response-stream` now includes a generic slice `streamSlice`
that can be used. This allows the actions created to be streamed via
NDJSON to be shared across server and client.

Functional tests for the examples were added too. To run these tests you
can use the following commands:

```
# Start the test server (can continue running)
node scripts/functional_tests_server.js --config test/examples/config.js
# Start a test run
node scripts/functional_test_runner.js --config test/examples/config.js
```

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
@walterra walterra mentioned this issue Jun 13, 2024
Merged
2 tasks
@peteharverson peteharverson mentioned this issue Jun 19, 2024
Merged
9 tasks
This was referenced Jun 20, 2024
Merged
Merged
walterra added a commit that referenced this issue Jun 21, 2024
@walterra
[ML] AIOps: Fix the text field icon margins in the log rate analysis …
695d382 
…results table. (#186520)

## Summary

Part of #181111.

Fixes the regression where the text field icon margins in the log rate
analysis results table were not rendered. It turned out the css applied
to `EuiIconTip` was not picked up correctly. Wrapping it in a `span`
applies it correctly.

Before:

<img width="1094" alt="image"
src="https://github.com/elastic/kibana/assets/230104/fd1a3e09-1815-490c-9c6a-ded3af71cf15">


After:

<img width="1098" alt="image"
src="https://github.com/elastic/kibana/assets/230104/4522f85a-82b5-4009-9ac5-ad207ac7b3e3">

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
bhapas pushed a commit to bhapas/kibana that referenced this issue Jun 24, 2024
@walterra @bhapas
[ML] AIOps: Fix the text field icon margins in the log rate analysis …
1ac8752 
…results table. (elastic#186520)

## Summary

Part of elastic#181111.

Fixes the regression where the text field icon margins in the log rate
analysis results table were not rendered. It turned out the css applied
to `EuiIconTip` was not picked up correctly. Wrapping it in a `span`
applies it correctly.

Before:

<img width="1094" alt="image"
src="https://github.com/elastic/kibana/assets/230104/fd1a3e09-1815-490c-9c6a-ded3af71cf15">


After:

<img width="1098" alt="image"
src="https://github.com/elastic/kibana/assets/230104/4522f85a-82b5-4009-9ac5-ad207ac7b3e3">

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
walterra added a commit that referenced this issue Jul 3, 2024
@walterra
[ML] AIOps: Add AI Assistant contextual insight to Log Rate Analysis …
c60efaf 
…page in ML plugin in Observability serverless. (#186509)

## Summary

Part of #181111.

This adds O11y AI Assistant's contextual insight to the Log Rate
Analysis page in the ML plugin for O11y serverless projects..

Note the code is almost an exact copy of what's currently used in the
O11y UI itself to render the contextual insight when log rate analysis
gets embedded on alert details pages. As a follow up we will be able to
consolidate some code.

[aiops-ai-assistant-contextual-insight-0001.webm](https://github.com/elastic/kibana/assets/230104/08269a40-f45a-4eb9-bd4d-a249cdd46266)

To test this you need to set up AI Assistant in your `kibana.dev.yml`:

```yml
xpack.actions.preconfigured:
  my-gen-ai:
    name: Preconfigured Azure OpenAi
    actionTypeId: .gen-ai
    config:
      apiUrl: <YOUR-API-URL>
      apiProvider: 'Azure OpenAI'
    secrets:
      apiKey: <YOUR-API-KEY>
```

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
@walterra walterra changed the title [ML] AIOps: Log Rate Analysis improvements. [ML] AIOps: 8.15 Log Rate Analysis improvements. Jul 5, 2024
@walterra walterra mentioned this issue Jul 5, 2024
Open
@walterra walterra mentioned this issue Jul 5, 2024
Closed
@walterra
Copy link
Contributor Author

walterra commented Jul 5, 2024 •
edited
Loading

Closing, created a v8.16.0 follow up #187684 and a general backlog issue #187683.

@walterra walterra closed this as completed Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walterra walterra

@alvarezmelissa87 alvarezmelissa87

Labels
Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis Meta :ml v8.15.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@walterra @alvarezmelissa87 @elasticmachine