[Dataset Quality]Add logic to display degraded fields in DQ Flyout

[achyutjhunjhunwala]

Summary

Closes #172265

Description

This PR adds the Degraded Fields Table to the Dataset Quality Flyout for individual Data Stream. Following tasks were done as part of this PR

1. A new server side endpoint created which queries the datastream directly and aggregates _ignored fields for that data stream during the given time range and also adds a sub aggregation for last occurence.
2. On the UI Side, the table was added with 3 columns as mentioned in the Original ticket - Field, Count and Last Occurrence
3. The UI currently supports clients side sorting and pagination. We can move this to server side pagination sorting if required in the future.
4. The Flyout also supports sync with the URL which means user can navigate to the Dataset Quality page where the flyout would be open and Sorting and Pagination would be pre-applied
5. API Tests
6. Stateful and Serverless FTR tests

Screenshot

Scenario

1. Spin up a 8.14-snapshot instance
2. Ingest degraded docs
3. Upgrade to 8.15-snapshot
4. Open Data Set Quality Flyout and see how the page looks like

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[achyutjhunjhunwala]

/ci

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[tonyghiani]

I did a first pass and left some notes, I think the new API is not versioned and it requires to update the implementation to comply with the specs, good job overall 👌

[flash1293]

It seems like the "last occurrence" is not formatted properly:

[achyutjhunjhunwala]

@flash1293 Thank you for pointing it out. I have fixed it now

[flash1293]

Thanks @achyutjhunjhunwala ! One other note about the user experience - should we indicate somehow if the list of ignored fields is not exhaustive?

In the simplest way it would just be a mention in the tooltip: https://github.com/elastic/kibana/pull/183934/files#diff-e14f87ebe930d154e4d8be9da8d99811c8a756fca476f339edaaa62009c5c257R147

[achyutjhunjhunwala]

Thanks @achyutjhunjhunwala ! One other note about the user experience - should we indicate somehow if the list of ignored fields is not exhaustive?

In the simplest way it would just be a mention in the tooltip: https://github.com/elastic/kibana/pull/183934/files#diff-e14f87ebe930d154e4d8be9da8d99811c8a756fca476f339edaaa62009c5c257R147

@flash1293 Sounds good to me. I will proceed with something like this "Set of degraded fields in the dataset. Please not that this list may not be exhaustive."

@mdbirnstiehl Can you help me here with a better copy ?

[tonyghiani]

LGTM, thanks for addressing the changes!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 6f06c20
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183934-6f06c20a1d51
• Observability Deployment

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
datasetQuality	187	193	+6

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
observability	689	691	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
datasetQuality	179.4KB	183.5KB	+4.1KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
module count	-	5412	+5412
total size	-	8.8MB	+8.8MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
datasetQuality	36.7KB	38.1KB	+1.4KB
observabilityLogsExplorer	16.3KB	16.3KB	+49.0B
total			+1.5KB

Unknown metric groups

API count

id	before	after	diff
observability	698	700	+2

History

• 💚 Build #212768 succeeded 10abcbc
• 💛 Build #212597 was flaky e24a79c
• 💛 Build #212504 was flaky 75113c7
• 💔 Build #212428 failed 803f067

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @achyutjhunjhunwala

[mdbirnstiehl]

Looks like there's a typo here. "Please note..." instead of "Please not..." You can probably remove that part anyway and just say "This list may not be exhaustive." We could also change the tooltip a little more and say, "A partial list of degraded fields found in your dataset." to imply that it's not exhaustive.

[achyutjhunjhunwala]

Thank @mdbirnstiehl I will go with "A partial list of degraded fields found in your dataset." and update the tooltip in my next PR