[Infra UI] Add graphql query to retrieve log entries from a configured source

[weltenwort]

New GraphQL field InfraSource.logEntriesAround()

The new logEntriesAround() field of the InfraSource type allows for retrieval of a chunk of consecutive log entries preceding and following a certain position in the event stream.

  extend type InfraSource {
    "A consecutive span of log entries following a point in time"
    logEntriesAfter(
      "The sort key that corresponds to the point in time"
      key: InfraTimeKeyInput!
      "The maximum number of preceding to return"
      countBefore: Int = 0
      "The maximum number of following to return"
      countAfter: Int = 0
      "The query to filter the log entries by"
      filterQuery: String
      "The query to highlight the log entries with"
      highlightQuery: String
    ): InfraLogEntryInterval!
  }

Example

{
  source(id: "default") {
    id
    logEntriesAround(key: {time: 1531836161000, tiebreaker: 949}, countBefore: 3, countAfter: 3) {
      entries {
        gid
        key {
          time
          tiebreaker
        }
        message {
          ... on InfraLogMessageFieldSegment {
            field
            value
          }
          ... on InfraLogMessageConstantSegment {
            constant
          }
        }
      }
    }
  }
}

New GraphQL field InfraSource.logEntriesBetween()

The new logEntriesBetween() field of the InfraSource type allows for retrieval of a chunk of consecutive log entries between two positions in the event stream (exclusively).

  extend type InfraSource {
    "A consecutive span of log entries within an interval"
    logEntriesBetween(
      "The sort key that corresponds to the start of the interval"
      startKey: InfraTimeKeyInput!
      "The sort key that corresponds to the end of the interval"
      endKey: InfraTimeKeyInput!
      "The query to filter the log entries by"
      filterQuery: String
      "The query to highlight the log entries with"
      highlightQuery: String
    ): InfraLogEntryInterval!
  }

Example

{
  source(id: "default") {
    id
    logEntriesBetween(startKey: {time: 1531836159000, tiebreaker: 947}, endKey: {time: 1531836161000, tiebreaker: 953}) {
      entries {
        gid
        key {
          time
          tiebreaker
        }
        message {
          ... on InfraLogMessageFieldSegment {
            field
            value
          }
          ... on InfraLogMessageConstantSegment {
            constant
          }
        }
      }
    }
  }
}

New InfraLogEntriesDomain lib

InfraLogEntriesDomain is a domain lib, that provides methods to retrieve log entries given a source id, time information, filters and highlights. In the Kibana server it is currently backed by the InfraKibanaLogEntriesAdapter, which retrieves events from the Elasticsearch cluster that the Kibana instance belongs to.

Built-in message formatting

The domain lib mentioned above comes with a set of built-in rules to derive a message from a few message document types:

• system.syslog.message as produced by the filebeat system module
• system.auth.message as produced by the filebeat system module
• message as produced by plain filebeat
• @message as produced by popular logstash configurations

This list of rules will be expanded and made extensible by the user. The rules are currently specified in a small declarative DSL as described in #21204 and compiled at runtime to a composite javascript formatting function.

Testing

For manually testing the new queries the GraphiQL interface at /api/infra/graphql/graphiql can be used.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[weltenwort]

@skh while implementing the follow-up that uses the api I came up with a structure for the query type that is better suited for the way it is going to be used. Hope that doesn't throw your review off - the change quite superficial 😇

I combined logEntriesBefore and logEntriesAfter into logEntriesAround, because on the initial load and after jumping, both will be queried in tandem anyway. The old queries can still be emulated by setting countBefore or countAfter to 0 (or leaving them out).

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[skh]

I can confirm the example queries work and the data returned looks good to me, 👍 from me.

[weltenwort]

I'll wait for #21871 to be merged so I can resolve the conflicts in the generated types here.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[weltenwort]

looks like an unrelated error in the region map tests again