Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Logs UI] Create ML module for log analysis #42593

Closed
weltenwort opened this issue Aug 5, 2019 · 1 comment · Fixed by #42872
Closed

[Logs UI] Create ML module for log analysis #42593

weltenwort opened this issue Aug 5, 2019 · 1 comment · Fixed by #42872
Assignees
Labels
Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0

Comments

@weltenwort
Copy link
Member

weltenwort commented Aug 5, 2019

Summary

The job that produces the data displayed on the log analysis screen (#41881) should be defined in and deployed via an ML module along with the corresponding datafeed.

Acceptance criteria

  • A new ML module is created that is owned by the Logs UI.
  • The module contains an ML job with the following characteristics:
    • The job id is deterministically derived from the space id and the source id, so it can be easily discovered later. It could be something like kibana-logs-ui-${spaceId}-${sourceId}-log-entry-rate.
    • The job is a time series analysis job with the time field being based on the source configuration.
    • The detector uses the count function.
    • The model plot functionality is enabled.
    • The bucket span is chosen as a compromise between storage size and precision. The ML team's recommendations are about 15 minutes, but not lower than that.
  • The module contains a corresponding datafeed:
    • The datafeed id is also deterministically derived from the space id and the source id.
    • The indices are based on the logs indices from source configuration.

Implementation notes

Example job and datafeed configurations

⚠️ these are just examples and do not necessarily match the final configurations

{
  "job_id": "kibana-logs-ui-testspace-default-log-entry-rate",
  "analysis_config": {
    "bucket_span": "15m",
    "summary_count_field_name": "doc_count",
    "detectors": [
      {
        "detector_description": "count",
        "function": "count",
        "detector_index": 0
      }
    ],
    "influencers": []
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": true
  }
}
{
  "datafeed_id": "datafeed-kibana-logs-ui-testspace-default-log-entry-rate",
  "job_id": "kibana-logs-ui-testspace-default-log-entry-rate",
  "indices": [
    "filebeat-*"
  ]
}
@weltenwort weltenwort added [zube]: Ready Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0 labels Aug 5, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/infra-logs-ui

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants