[Dataset quality] Support failure store #199806

yngrdyn · 2024-11-12T14:46:59Z

Closes https://github.com/elastic/logs-dev/issues/183 and https://github.com/elastic/logs-dev/issues/184

Summary

This PR aims to show to the user failed docs % in dataset quality table. The following acceptance criteria items were resolved

Dataset quality page

A column for Failed docs is included in the table
A tooltip is placed in the title of the column
A % of documents inside Failure store is calculated for every dataStream
If % is lesser than 0.0001 but greater than 0 we should show ⚠ symbol next to the ~0 value (as we do with degraded docs)
Failed docs percentages greater than 0 should link to discover

Dataset details page

A metric, Failed docs, is included in the Overview panel under Data set quality. This metric includes the number of documents inside the failure store for the specific dataStream.
A tooltip is placed in the title of the Failed docs metric with message: The percentage of docs sent to failure store due to an issue during ingestion.
Degraded docs graph section is transformed to Document trends allowing the users to switch between Degraded docs and Failed docs trends over time.
A new chart for failed documents is created with links to discover/Logs explorer using the right dataView

🎥 Demo

Screen.Recording.2024-11-12.at.11.53.02.mov

Specific cases

Failed documents less than 0.001%

Quality column is shown after degraded docs and failed docs is loaded (Quality is now calculated using both values)

Screen.Recording.2024-11-18.at.15.55.05.mov

Screen.Recording.2024-11-18.at.16.01.03.mov

Users can still see Quality even when they don't have access to stats or something went wrong with that endpoint

Screen.Recording.2024-11-18.at.16.02.47.mov

The selected chart is kept in the URL

Screen.Recording.2024-11-19.at.16.57.54.mov

Missing

The following acceptance criteria are missing

Failed docs percentages greater than 0 should link to discover
A new chart for failed documents is created with links to discover/Logs explorer using the right dataView

because es changes are not finalised

elasticmachine · 2024-11-12T14:47:06Z

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

Click to trigger kibana-pull-request for this PR!
Click to trigger kibana-deploy-project-from-pr for this PR!

yngrdyn · 2024-11-12T14:47:59Z

/ci

yngrdyn · 2024-11-12T15:57:11Z

x-pack/plugins/observability_solution/dataset_quality/common/constants.ts

@@ -26,7 +26,7 @@ export const NONE = 'none';
 export const DEFAULT_TIME_RANGE = { from: 'now-24h', to: 'now' };
 export const DEFAULT_DATEPICKER_REFRESH = { value: 60000, pause: false };

-export const DEFAULT_DEGRADED_DOCS = {
+export const DEFAULT_QUALITY_DOC_STATS = {


Note
This is now reused by degradedDocs and failedDocs.

yngrdyn · 2024-11-12T15:58:08Z

...plugins/observability_solution/dataset_quality/common/data_streams_stats/data_stream_stat.ts

    };

    return new DataStreamStat(dataStreamStatProps);
  }

-  public static fromDegradedDocStat({
+  public static fromQualityStats({


Note
We can extend this method with more qualityStats and construct a Dataset from them

yngrdyn · 2024-11-12T15:59:25Z

x-pack/plugins/observability_solution/dataset_quality/common/utils/dataset_name.ts

@@ -40,7 +40,7 @@ export const indexNameToDataStreamParts = (dataStreamName: string) => {
 };

 export const extractIndexNameFromBackingIndex = (indexString: string): string => {
-  const pattern = /.ds-(.*?)-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-[0-9]{6}/;
+  const pattern = /.(?:ds|fs)-(.*?)-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-[0-9]{6}/;


Important
Failure store is at the moment a backing index that starts with .fs

x-pack/plugins/observability_solution/dataset_quality/common/utils/quality_helpers.ts

yngrdyn · 2024-11-12T16:05:23Z

@mdbirnstiehl I need some help from your side

The tooltip for the quality column should now tell the users that quality is calculated by degradedDocs and failedDocs percentages. I came up with

How can we explain users what is a Failed doc? I came up with

In the summary we should also reference the new calculation of quality (DegradedDocs and failedDocs)

yngrdyn · 2024-11-12T16:08:34Z

x-pack/plugins/observability_solution/dataset_quality/public/hooks/use_summary_panel.ts

-  const datasetsQuality = {
-    percentages: filteredItems.map((item) => item.degradedDocs.percentage),
-  };
+  const datasetsQuality = countBy(filteredItems.map((item) => item.quality)) as Record<


Quality is now part of the dataset, so we don't need this calculation anymore. Instead I just created a map that will hold how many datasets fall into a quality status. e.g.

{ 'poor': 1, 'degraded': 2, 'good': 1, }

yngrdyn · 2024-11-12T18:56:50Z

/ci

mdbirnstiehl · 2024-11-12T19:26:46Z

@mdbirnstiehl I need some help from your side

The tooltip for the quality column should now tell the users that quality is calculated by degradedDocs and failedDocs percentages.

This one looks good.

How can we explain users what is a Failed doc? I came up with

I'm not sure if this is exactly how it works, but is there some way we could be more specific here? Like, "Percentage of docs sent to failure store due to an issue during ingestion."

I'm not sure "failure store" itself will be meaningful for users.

In the summary we should also reference the new calculation of quality (DegradedDocs and failedDocs)

This tool tip can probably match the tool tip for the data set quality column in the table.

A side note, maybe this would need to be a different issue, but this heading:

Should probably be Data Set Quality, not sets. It also looks like the table column headers are in title case, and should be in sentence case to be consistent with the other pages in Stack Management. For example, "Failed docs" instead of "Failed Docs"

yngrdyn · 2024-11-12T19:38:23Z

I'm not sure if this is exactly how it works, but is there some way we could be more specific here? Like, "Percentage of docs sent to failure store due to an issue during ingestion."

That's exactly how it works 👍🏼

Should probably be Data Set Quality, not sets. It also looks like the table column headers are in title case, and should be in sentence case to be consistent with the other pages in Stack Management. For example, "Failed docs" instead of "Failed Docs"

I'll address those in this PR as well. This is how everything looks now

Screen.Recording.2024-11-12.at.16.42.32.mov

yngrdyn · 2024-11-12T19:46:37Z

/ci

yngrdyn · 2024-11-13T10:40:23Z

/ci

mohamedhamed-ahmed · 2024-11-14T11:40:52Z

I started testing quickly and got 2 issues:

When I make the failed_docs request fail from the network tab, I still get the results somehow not sure whats happening there but need to take a further look.

Screen.Recording.2024-11-14.at.12.38.24.mov

When I open any dataset I get an error as the mapPercentageToQuality is expecting an array but gets a single value in this case

Screen.Recording.2024-11-14.at.12.39.03.mov

When I make the failed_docs request fail from the network tab, I still get the results somehow not sure whats happening there but need to take a further look.

This works fine you can ignore it, tried it and seems good. Only the other problem is existing

mohamedhamed-ahmed

Did first round of code review and got some small comments

x-pack/plugins/observability_solution/dataset_quality/common/utils/quality_helpers.ts

...s/observability_solution/dataset_quality/public/components/dataset_quality/table/columns.tsx

...tion/dataset_quality/public/components/dataset_quality/table/failed_docs_percentage_link.tsx

...plugins/observability_solution/dataset_quality/public/hooks/use_dataset_details_telemetry.ts

x-pack/plugins/observability_solution/dataset_quality/common/translations.ts

mohamedhamed-ahmed · 2024-11-19T12:34:43Z

...plugins/observability_solution/dataset_quality/server/routes/data_streams/get_failed_docs.ts

+export async function getFailedDocsPaginated(options: {
+  esClient: ElasticsearchClient;
+  types: DataStreamType[];
+  datasetQuery?: string;


nit: maybe renam datasetQuery to just datasetNames as its a bit confusing since its not really a query

what do you think about renaming it to indexPattern? or maybe just datasetPattern?

Sounds good to me 👍

…seEffect

yngrdyn · 2024-11-26T16:04:26Z

/ci

yngrdyn · 2024-11-26T16:04:49Z

/ci

yngrdyn · 2024-11-26T18:36:49Z

/ci

yngrdyn · 2024-11-26T19:55:49Z

/ci

…rt-for-failure-store

yngrdyn · 2024-11-27T13:11:12Z

/ci

…-quality-support-for-failure-store-flyout

elasticmachine · 2024-11-27T13:39:20Z

💔 Build Failed

Buildkite Build
Commit: 895110f

Failed CI Steps

Check Types

History

cc @yngrdyn

…e-store-flyout [Dataset quality] Dataset quality support for failure store flyout

Closes elastic/logs-dev#183, elastic/logs-dev#184 and elastic/logs-dev#185. ## Summary This PR aims to support failure store in dataset quality page. The following acceptance criteria items were resolved ### Dataset quality page - [x] A column for Failed docs is included in the table - [x] A tooltip is placed in the title of the column - [x] A % of documents inside Failure store is calculated for every dataStream - [x] If % is lesser than 0.0001 but greater than 0 we should show ⚠ symbol next to the ~0 value (as we do with degraded docs) - [x] Failed docs percentages greater than 0 should link to discover 🎥 Demo https://github.com/user-attachments/assets/6d9e3f4c-02d9-43ab-88cb-ae70716b05d9 ### Dataset details page - [x] A metric, Failed docs, is included in the Overview panel under Data set quality. This metric includes the number of documents inside the failure store for the specific dataStream. - [x] A tooltip is placed in the title of the Failed docs metric with message: `The percentage of docs sent to failure store due to an issue during ingestion.` - [x] Degraded docs graph section is transformed to Document trends allowing the users to switch between Degraded docs and Failed docs trends over time. - [x] A new chart for failed documents is created with links to discover/Logs explorer using the right dataView 🎥 Demo https://github.com/user-attachments/assets/6a3a1f09-2668-4e83-938e-ecdda798c199 ### Failed docs ingestion issue flyout - [x] Whenever documents are found in failure store we should list Document indexing failed in Quality issues table - [x] User should be able to expand Document indexing failed and see more information in the flyout - [x] The flyout will show Docs count, an aggregation of the number of documents inside failure store for the selected timeframe - [x] The flyout will show Last ocurrence, the datetime registered for the most recent document in the failure store. - [x] The flyout will contain a section called Error messages where a list of unique error messages should be shown, exposing Content (error message) and Type (Error Type). - [x] Type should contain a tooltip where message (`Error message category`) explain users how we are categorising the errors. - [x] Other issues inside Quality issues table will be appended by field ignored and the field will be shown in bold. https://github.com/user-attachments/assets/94dc81f0-9720-4596-b256-c9d289cefd94 Note: This PR was reconstructed from #199806 which it supersedes. ## How to test 1. Execute `failed_logs` synthtrace scenario 2. Open dataset quality page ## Follow ups - Enable in serverless - Deployment agnostic tests cannot be added until we enable this in serverless - FTR tests will be added as part of elastic/logs-dev#182 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Closes elastic/logs-dev#183, elastic/logs-dev#184 and elastic/logs-dev#185. ## Summary This PR aims to support failure store in dataset quality page. The following acceptance criteria items were resolved ### Dataset quality page - [x] A column for Failed docs is included in the table - [x] A tooltip is placed in the title of the column - [x] A % of documents inside Failure store is calculated for every dataStream - [x] If % is lesser than 0.0001 but greater than 0 we should show ⚠ symbol next to the ~0 value (as we do with degraded docs) - [x] Failed docs percentages greater than 0 should link to discover 🎥 Demo https://github.com/user-attachments/assets/6d9e3f4c-02d9-43ab-88cb-ae70716b05d9 ### Dataset details page - [x] A metric, Failed docs, is included in the Overview panel under Data set quality. This metric includes the number of documents inside the failure store for the specific dataStream. - [x] A tooltip is placed in the title of the Failed docs metric with message: `The percentage of docs sent to failure store due to an issue during ingestion.` - [x] Degraded docs graph section is transformed to Document trends allowing the users to switch between Degraded docs and Failed docs trends over time. - [x] A new chart for failed documents is created with links to discover/Logs explorer using the right dataView 🎥 Demo https://github.com/user-attachments/assets/6a3a1f09-2668-4e83-938e-ecdda798c199 ### Failed docs ingestion issue flyout - [x] Whenever documents are found in failure store we should list Document indexing failed in Quality issues table - [x] User should be able to expand Document indexing failed and see more information in the flyout - [x] The flyout will show Docs count, an aggregation of the number of documents inside failure store for the selected timeframe - [x] The flyout will show Last ocurrence, the datetime registered for the most recent document in the failure store. - [x] The flyout will contain a section called Error messages where a list of unique error messages should be shown, exposing Content (error message) and Type (Error Type). - [x] Type should contain a tooltip where message (`Error message category`) explain users how we are categorising the errors. - [x] Other issues inside Quality issues table will be appended by field ignored and the field will be shown in bold. https://github.com/user-attachments/assets/94dc81f0-9720-4596-b256-c9d289cefd94 Note: This PR was reconstructed from elastic#199806 which it supersedes. ## How to test 1. Execute `failed_logs` synthtrace scenario 2. Open dataset quality page ## Follow ups - Enable in serverless - Deployment agnostic tests cannot be added until we enable this in serverless - FTR tests will be added as part of elastic/logs-dev#182 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 511f77c) # Conflicts: # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/overview/document_trends/degraded_docs/index.tsx # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/overview/summary/panel.tsx # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/quality_issue_flyout/degraded_field/field_info.tsx # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/quality_issue_flyout/index.tsx # x-pack/platform/plugins/shared/dataset_quality/public/hooks/use_redirect_link.ts

# Backport This will backport the following commits from `main` to `8.x`: - [[Dataset quality] Failure store support (#206758)](#206758)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Closes elastic/logs-dev#183, elastic/logs-dev#184 and elastic/logs-dev#185. ## Summary This PR aims to support failure store in dataset quality page. The following acceptance criteria items were resolved ### Dataset quality page - [x] A column for Failed docs is included in the table - [x] A tooltip is placed in the title of the column - [x] A % of documents inside Failure store is calculated for every dataStream - [x] If % is lesser than 0.0001 but greater than 0 we should show ⚠ symbol next to the ~0 value (as we do with degraded docs) - [x] Failed docs percentages greater than 0 should link to discover 🎥 Demo https://github.com/user-attachments/assets/6d9e3f4c-02d9-43ab-88cb-ae70716b05d9 ### Dataset details page - [x] A metric, Failed docs, is included in the Overview panel under Data set quality. This metric includes the number of documents inside the failure store for the specific dataStream. - [x] A tooltip is placed in the title of the Failed docs metric with message: `The percentage of docs sent to failure store due to an issue during ingestion.` - [x] Degraded docs graph section is transformed to Document trends allowing the users to switch between Degraded docs and Failed docs trends over time. - [x] A new chart for failed documents is created with links to discover/Logs explorer using the right dataView 🎥 Demo https://github.com/user-attachments/assets/6a3a1f09-2668-4e83-938e-ecdda798c199 ### Failed docs ingestion issue flyout - [x] Whenever documents are found in failure store we should list Document indexing failed in Quality issues table - [x] User should be able to expand Document indexing failed and see more information in the flyout - [x] The flyout will show Docs count, an aggregation of the number of documents inside failure store for the selected timeframe - [x] The flyout will show Last ocurrence, the datetime registered for the most recent document in the failure store. - [x] The flyout will contain a section called Error messages where a list of unique error messages should be shown, exposing Content (error message) and Type (Error Type). - [x] Type should contain a tooltip where message (`Error message category`) explain users how we are categorising the errors. - [x] Other issues inside Quality issues table will be appended by field ignored and the field will be shown in bold. https://github.com/user-attachments/assets/94dc81f0-9720-4596-b256-c9d289cefd94 Note: This PR was reconstructed from elastic#199806 which it supersedes. ## How to test 1. Execute `failed_logs` synthtrace scenario 2. Open dataset quality page ## Follow ups - Enable in serverless - Deployment agnostic tests cannot be added until we enable this in serverless - FTR tests will be added as part of elastic/logs-dev#182 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

yngrdyn and others added 2 commits November 12, 2024 11:42

Fetching failed docs and showing them in the table

81ad3fa

Merge branch 'elastic:main' into main

301a53a

yngrdyn self-assigned this Nov 12, 2024

yngrdyn commented Nov 12, 2024

View reviewed changes

x-pack/plugins/observability_solution/dataset_quality/common/utils/quality_helpers.ts Show resolved Hide resolved

yngrdyn commented Nov 12, 2024

View reviewed changes

fixing tests

2ad742f

yngrdyn added 2 commits November 12, 2024 16:44

Addressing copies across table and page

4fe638e

Merge branch 'main' into dataset-quality-support-for-failure-store

56a1793

Column names changed in tests

e11fabf

mohamedhamed-ahmed self-requested a review November 13, 2024 11:52

Merge branch 'main' into dataset-quality-support-for-failure-store

a9f173c

Adding failed info to dataset details

41d249a

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 41e4f61 to 41d249a Compare November 14, 2024 13:17

mohamedhamed-ahmed reviewed Nov 14, 2024

View reviewed changes

Addresing Pr cmments

91e2c3e

mohamedhamed-ahmed reviewed Nov 19, 2024

View reviewed changes

yngrdyn added 2 commits November 19, 2024 12:12

extracting percentage calculator to an util

f753581

merging useDocsChart hooks for quality issues + removing unnecesary u…

74df05c

…seEffect

fixing e2e tests

076cb6c

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 0bb1ac6 to 076cb6c Compare November 26, 2024 16:04

Merge branch 'main' into dataset-quality-support-for-failure-store

0057ecb

fixing tests

a33ca8f

fixing tests

f10abac

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from a33ca8f to f10abac Compare November 26, 2024 19:55

yngrdyn and others added 3 commits November 27, 2024 10:04

fixing some tests

f9ba989

Merge branch 'elastic:main' into main

c6846eb

Merge remote-tracking branch 'origin/main' into dataset-quality-suppo…

895110f

…rt-for-failure-store

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 45e0213 to 895110f Compare November 27, 2024 13:11

Merge branch 'dataset-quality-support-for-failure-store' into dataset…

18577c5

…-quality-support-for-failure-store-flyout

yngrdyn and others added 7 commits December 4, 2024 11:18

Adding failed documents to quality issues table

ca2e876

creating a new url version for dataset details

8a2b8ee

cleaning the PR

db560e0

Merge pull request #1 from yngrdyn/dataset-quality-support-for-failur…

2d188e2

…e-store-flyout [Dataset quality] Dataset quality support for failure store flyout

adjusting failed_logs scenario with the new changes for failure store

394320d

using new failure store selector syntax

5289ba5

support failures store syntax in links and logs explorer locator

1b56786

yngrdyn mentioned this pull request Jan 15, 2025

[Dataset quality] Failure store support #206758

Merged

16 tasks

yngrdyn closed this Jan 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dataset quality] Support failure store #199806

[Dataset quality] Support failure store #199806

yngrdyn commented Nov 12, 2024 •

edited

Loading

elasticmachine commented Nov 12, 2024

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn commented Nov 12, 2024

mdbirnstiehl commented Nov 12, 2024 •

edited

Loading

yngrdyn commented Nov 12, 2024 •

edited

Loading

yngrdyn commented Nov 12, 2024

yngrdyn commented Nov 13, 2024

mohamedhamed-ahmed commented Nov 14, 2024 •

edited

Loading

mohamedhamed-ahmed left a comment

mohamedhamed-ahmed Nov 19, 2024

yngrdyn Nov 19, 2024 •

edited

Loading

mohamedhamed-ahmed Nov 20, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

[Dataset quality] Support failure store #199806

[Dataset quality] Support failure store #199806

Conversation

yngrdyn commented Nov 12, 2024 • edited Loading

Summary

Dataset quality page

Dataset details page

🎥 Demo

Specific cases

Missing

elasticmachine commented Nov 12, 2024

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn commented Nov 12, 2024

mdbirnstiehl commented Nov 12, 2024 • edited Loading

yngrdyn commented Nov 12, 2024 • edited Loading

yngrdyn commented Nov 12, 2024

yngrdyn commented Nov 13, 2024

mohamedhamed-ahmed commented Nov 14, 2024 • edited Loading

mohamedhamed-ahmed left a comment

Choose a reason for hiding this comment

mohamedhamed-ahmed Nov 19, 2024

Choose a reason for hiding this comment

yngrdyn Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

mohamedhamed-ahmed Nov 20, 2024

Choose a reason for hiding this comment

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

💔 Build Failed

Failed CI Steps

History

yngrdyn commented Nov 12, 2024 •

edited

Loading

mdbirnstiehl commented Nov 12, 2024 •

edited

Loading

yngrdyn commented Nov 12, 2024 •

edited

Loading

mohamedhamed-ahmed commented Nov 14, 2024 •

edited

Loading

yngrdyn Nov 19, 2024 •

edited

Loading