[Infra UI] Merge InfraOps feature branch #24068
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Fix CI branch fallback * Rename files to adhere to snake case rule * Align dependency versions
This changes the graphql generation such that * the generated types are located in `common/graphql/types.ts` * the imported code does not involve the resolvers * there are no imports from `common` to `public` or `server` * the root schema is shared between client and server * a `public/graphql` file collects all future client-side schemas
# New Configuration This PR adds new Kibana configuration options under the key `xpack.infra`. Any settings given in Kibana's configuration file will be overlaid on top of the following defaults (as discussed in #20428): ```yaml xpack.infra: sources: default: metricAlias: 'xpack-infra-default-metrics' logAlias: 'xpack-infra-default-logs' fields: message: ['message', '@message'] hostname: 'beat.hostname' pod: 'kubernetes.pod.name' container: 'docker.container.name' timestamp: '@timestamp' tiebreaker: '_doc' query: partitionSize: 75 partitionFactor: 1.2 ``` # New GraphQL Queries Based on the configuration introduced above, the following endpoints have been added: ## `source(id)` ```graphql extend type Query { "Get an infrastructure data source by id" source("The id of the source" id: ID!): InfraSource! } ``` ### Example ```graphql query { source(id: "default") { id configuration { metricAlias logAlias fields { container hostname message pod tiebreaker timestamp } } } } ``` ## `allSources` ```graphql extend type Query { "Get a list of all infrastructure data sources" allSources: [InfraSource!]! } ``` ### Example ```graphql query { allSources { id } } ``` # Other Notes This also distributes most of the types from `infra_types.ts` into the `adapter_types.ts` of the respective adapters. It also moves functions and types specific to the "nodes" domain into the nodes adapter directory to increase local cohesion. The source defaults are applied inside of the source lib instead of the configuration schema to maintain separation of concerns and because the configuration schema is Kibana-specific. # Follow-up Tasks * move `map()` query type inside of `source()` * use source configuration for log fetching * add `source().indices` field to diagnose matching indices * add `source().capabilities` field to communicate detected data characteristics
…21077) This merges Kibana `master` into the feature branch and fixes some minor breakages due to Kibana api changes.
## New GraphQL field `InfraSource.status`
The new status field of the `InfraSource` type provides some insight into the indices that are contained in the metric and log aliases.
```graphql
extend type InfraSource {
"The status of the source"
status: InfraSourceStatus!
}
"The status of an infrastructure data source"
type InfraSourceStatus {
"Whether the configured metric alias exists"
metricAliasExists: Boolean!
"Whether the configured log alias exists"
logAliasExists: Boolean!
"The list of indices in the metric alias"
metricIndices: [String!]!
"The list of indices in the log alias"
logIndices: [String!]!
}
```
### Example
```graphql
query {
source(id: "default") {
id
status {
metricIndices
metricAliasExists
logIndices
logAliasExists
}
}
}
```
## Moved GraphQL field `InfraSource.map(timerange, filters)`
The `map(indexPattern, timerange, filters)` query is now available as `InfraSource.map(timerange, filters)`. The queried indices are now derived from the configuration of the source this is located in, thereby making the indexPattern argument unnecessary.
```graphql
extend type InfraSource {
"A hierarchy of hosts, pods, containers, services or arbitrary groups"
map(timerange: InfraTimerange!, filters: [InfraFilter!]): InfraResponse
}
```
### Example
```graphql
{
source(id: "default") {
id
map(timerange: {interval: "5m", from: 1000000000000, to: 9999999999999}) {
hosts {
name
}
}
}
}
```
## New `InfraSources` lib
The new `InfraSources` lib is now consuming the `InfraConfigurationSourcesAdapter`. In doing so, the adapter-independent `get(sourceId: string)` method was moved from the adapter interface to the lib. The new lib takes the place of the adapter in the composed `libs` object.
## New `InfraSourceStatus` lib
Using the new `InfraSourceStatus` lib the resolvers can query Elasticsearch (via the `InfraElasticsearchSourceStatusAdapter`) for existence and matching indices of the metric and log aliases.
## New resolver helper types
The framework adapter provides several new helper types that can derive a more complete `InfraResolver` type from the resolver types that are automatically generated. The `InfraResolver` differs from the auto-generated resolver in several ways:
* It has a well-defined `parent` type.
* It has a Infra UI-specific `context` type.
* It allows for fields of objects returned from the resolver to be async functions as [mentioned in the graphql-tools docs](https://www.apollographql.com/docs/graphql-tools/resolvers.html#Default-resolver).
To derive such an `InfraResolver` from an automatically generated resolver type, these helper types are made available:
* `InfraResolverOf<Resolver, Parent, Context>` adds the `Parent` and `Context` types to the resolver function arguments.
* `InfraResolverWithFields<Resolver, Parent, Context, IncludedFields extends string>` behaves like `InfraResolverOf` but also filters out any fields that are not specified in `IncludedFields`. This is useful because the value resolution can be distributed across several layers of the resolver map while the auto-generated resolver types assume monolithic resolvers.
* `InfraResolverWithoutFields<Resolver, Parent, Context, ExcludedFields extends string>` behaves like `InfraResolverOf` but also filters out any fields that are specified in `IncludedFields`.
## Other Notes
Since node types are not defined as unions or interfaces, the unnecessary `__resolveType` resolvers of the node types were removed.
The Infra UI project uses styled-components for styling.
While working on the porting the WaffleMap over to the Infra UI I noticed that the map endpoint would need to allow for the group by and metrics to be programmatically created. The query schema did not allow for this. This PR changes to the map query schema to allow for the user to define the grouping and metrics as request variable. Grouping are renamed to paths and the last value on the path argument is the "node" type. When the documents are returned each node will have a unique path; the path will be used to generate the groups and squares (nodes) on the waffle map.
#### Example Test Query:
```graphql
{
source(id: "default") {
id
map(timerange: { interval: "1m", to:1532549840415, from: 1532546240416}, filters: []) {
nodes(path: [
{ id: "1", type: terms, field: "metricset.module" },
{ id: "2", type: terms, field: "metricset.name" },
{ id: "3", type: hosts }
]) {
path {
value
}
metrics(metrics: [{ type: count }]) {
name
value
}
}
}
}
}
```
#### Example Response
```json
{
"data": {
"source": {
"id": "default",
"map": {
"nodes": [
{
"path": [
{ "value": "system" },
{ "value": "process" },
{ "value": "demo-stack-mysql-01" }
],
"metrics": [ { "name": "count", "value": 5971 } ]
},
{
"path": [
{ "value": "system" },
{ "value": "diskio" },
{ "value": "demo-stack-mysql-01" }
],
"metrics": [ { "name": "count", "value": 5971 } ]
}
]
}
}
}
}
```
* Re-commit waffle map * Moving files around to amek more sense; adding context menu * Remane the maps query and use the predefined types * Remove IDs as much as possible in map response * Update apollo-client to fix broken fetchPolicy * Removing unnecessary exports * Fixing typo * Updating component names to be more specific
* Upgrade prettier to 1.14.0 and apply new format to upstream code * Apply new prettier formatting * Change log entries state to use the new graphql api * Fix query filename after glob pattern change in script * Fix "compator" typo
## New GraphQL field `InfraSource.logSummaryBetween()` The new `logSummaryBetween()` field of the `InfraSource` type provides a sequence of buckets with a given size between two given timestamps. The field is used by the "Minimap" in the logging ui. ## Simplified log summary state The client-side log summary state has been simplified to use the graphql helpers, a simpler bucket size type and much a simpler management epic. ## Fix race condition in type generation script The type generation script contained a race condition, because the underlying library wrote the output files asynchronously even after the returned promises were resolved. The impact of this has been mitigated by not having the two generation steps read each other's output. ## Removed old log summary http API The old api at `/api/logging/summary` has been removed. The functionality is now provided by the GraphQL field described above.
This adds a query bar to the top of the logging UI. The query language is Kuery as developed by the Kibana team. The bar also features auto-completion based on the x-pack kuery autocompleter.
…2539) In order to reduce the initial setup effort by the user until we have a configuration UI, the default source should use wildcard-based index patterns instead of aliases. closes #22518 ## Known Limitations Using index patterns like this leads to the source status api reporting that the alias does not exist and that no indices match. Since that api is not used on the client side yet, this has no further effect right now.
💔 Build Failed |
|
jenkins, test this |
💔 Build Failed |
* [Infra UI] Add log rate to metrics on Waffle Map This PR adds the log rate metric to the waffle map * Changing abvNumber to abbreviatedNumber * adding support for normalized_value * Fixing normalized_value issue in inbound/outbound traffic * Removing support for multiple metrics form the waffle interface * refactor map endpoint to remove the info parsing code * Updating PR with changes upstream
💔 Build Failed |
* Add and use WithCapabilities container. * Remove unused 'requires' on layouts.
💔 Build Failed |
💚 Build Succeeded |
weltenwort
added a commit
to weltenwort/kibana
that referenced
this pull request
Oct 17, 2018
weltenwort
added a commit
that referenced
this pull request
Oct 17, 2018
| @@ -73,6 +73,8 @@ | |||
| "@kbn/pm": "link:packages/kbn-pm", | |||
| "@kbn/test-subj-selector": "link:packages/kbn-test-subj-selector", | |||
| "@kbn/ui-framework": "link:packages/kbn-ui-framework", | |||
| "@types/mustache": "^0.8.31", | |||
There was a problem hiding this comment.
Can you please move this to devDependencies so we don't ship in the build?
There was a problem hiding this comment.
#24165 (sorry about that)
There was a problem hiding this comment.
Oh no worries, thanks for the quick turn around.
|
@weltenwort it looks there is almost no test coverage for any of this code. Is there any concern with that? |
|
yes, that is not a new concern. #24163 lists improvements in that regard |
tylersmalley
pushed a commit
to tylersmalley/kibana
that referenced
this pull request
Oct 25, 2019
Was added as part of elastic#24068 but not actually ever used. Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley
pushed a commit
to tylersmalley/kibana
that referenced
this pull request
Oct 25, 2019
Originally added in elastic#24068, however, it doesn't appear to have ever been used. Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley
pushed a commit
that referenced
this pull request
Oct 25, 2019
Originally added in #24068, however, it doesn't appear to have ever been used. Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley
pushed a commit
to tylersmalley/kibana
that referenced
this pull request
Oct 25, 2019
Originally added in elastic#24068, however, it doesn't appear to have ever been used. Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley
pushed a commit
that referenced
this pull request
Oct 25, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This merges the InfraOps UI currently developed in the branch
feature-infra-uiintomaster. It is intended to be backported to 6.5.