[8.x] [Dataset quality] Failure store support (#206758) #208051
Merged
+3,349
−1,191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closes elastic/logs-dev#183, elastic/logs-dev#184 and elastic/logs-dev#185. ## Summary This PR aims to support failure store in dataset quality page. The following acceptance criteria items were resolved ### Dataset quality page - [x] A column for Failed docs is included in the table - [x] A tooltip is placed in the title of the column - [x] A % of documents inside Failure store is calculated for every dataStream - [x] If % is lesser than 0.0001 but greater than 0 we should show ⚠ symbol next to the ~0 value (as we do with degraded docs) - [x] Failed docs percentages greater than 0 should link to discover 🎥 Demo https://github.com/user-attachments/assets/6d9e3f4c-02d9-43ab-88cb-ae70716b05d9 ### Dataset details page - [x] A metric, Failed docs, is included in the Overview panel under Data set quality. This metric includes the number of documents inside the failure store for the specific dataStream. - [x] A tooltip is placed in the title of the Failed docs metric with message: `The percentage of docs sent to failure store due to an issue during ingestion.` - [x] Degraded docs graph section is transformed to Document trends allowing the users to switch between Degraded docs and Failed docs trends over time. - [x] A new chart for failed documents is created with links to discover/Logs explorer using the right dataView 🎥 Demo https://github.com/user-attachments/assets/6a3a1f09-2668-4e83-938e-ecdda798c199 ### Failed docs ingestion issue flyout - [x] Whenever documents are found in failure store we should list Document indexing failed in Quality issues table - [x] User should be able to expand Document indexing failed and see more information in the flyout - [x] The flyout will show Docs count, an aggregation of the number of documents inside failure store for the selected timeframe - [x] The flyout will show Last ocurrence, the datetime registered for the most recent document in the failure store. - [x] The flyout will contain a section called Error messages where a list of unique error messages should be shown, exposing Content (error message) and Type (Error Type). - [x] Type should contain a tooltip where message (`Error message category`) explain users how we are categorising the errors. - [x] Other issues inside Quality issues table will be appended by field ignored and the field will be shown in bold. https://github.com/user-attachments/assets/94dc81f0-9720-4596-b256-c9d289cefd94 Note: This PR was reconstructed from elastic#199806 which it supersedes. ## How to test 1. Execute `failed_logs` synthtrace scenario 2. Open dataset quality page ## Follow ups - Enable in serverless - Deployment agnostic tests cannot be added until we enable this in serverless - FTR tests will be added as part of elastic/logs-dev#182 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 511f77c) # Conflicts: # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/overview/document_trends/degraded_docs/index.tsx # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/overview/summary/panel.tsx # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/quality_issue_flyout/degraded_field/field_info.tsx # x-pack/platform/plugins/shared/dataset_quality/public/components/dataset_quality_details/quality_issue_flyout/index.tsx # x-pack/platform/plugins/shared/dataset_quality/public/hooks/use_redirect_link.ts
botelastic
bot
added
ci:project-deploy-observability
16 tasks
|
Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services) |
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
weltenwort
approved these changes
Jan 23, 2025
|
just to be clear: I didn't review, just approved to unblock the manual backport |
💚 Build Succeeded
Metrics [docs]Module Count
Public APIs missing comments
Async chunks
Page load bundle
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
History
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.x:Questions ?
Please refer to the Backport tool documentation