Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Anomaly Explorer fails to load after update to 7.9.0 #74935

Closed
walterra opened this issue Aug 13, 2020 · 8 comments · Fixed by #74953
Closed

[ML] Anomaly Explorer fails to load after update to 7.9.0 #74935

walterra opened this issue Aug 13, 2020 · 8 comments · Fixed by #74953
Assignees
@walterra
Labels
bug Fixes for quality problems that affect the customer experience Feature:Anomaly Detection ML anomaly detection :ml regression v7.9.0

Comments

@walterra
Copy link
Contributor

walterra commented Aug 13, 2020 •
edited by droberts195
Loading

After an update to 7.9.0, the Anomaly Explorer might fail to load. The root cause is that the mappings for the index for annotations were not properly updated.

To reproduce the original problem:

  • Before the upgrade, in 7.8.1, there are bunch of ml jobs which have annotations
  • Upgrade to 7.9.0,
  • After the upgrade, from the UI, check the annotations, there is nothing wrong so far.
  • now, create/run a new AD job, No issues from API. job finished without errors
  • now, go back to the UI, expand any job from Anomaly detection jobs list, click annotation tab. it shows Error loading the list of annotations for this job
  • it has error code 400 in dev console:

image

To reproduce the issue without the update progress:

When pointing the annotations index aliases to another index without the proper mapping, we can recreate the issue for testing and development purposes. In Dev Console do:

PUT /.ml-annotations-6-wrong-mapping
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "field1": { "type": "text" }
    }
  }
}

POST /_aliases
{
  "actions" : [
    { "add" : { "index" : ".ml-annotations-6-wrong-mapping", "alias" : ".ml-annotations-read", "is_hidden": true } },
    { "remove" : { "index" : ".ml-annotations-6", "alias" : ".ml-annotations-read" } },
    { "add" : { "index" : ".ml-annotations-6-wrong-mapping", "alias" : ".ml-annotations-write", "is_hidden": true } },
    { "remove" : { "index" : ".ml-annotations-6", "alias" : ".ml-annotations-write" } }
  ]
}

After creating the next annotation you should be able to reproduce the issue in Anomaly Explorer.

To restore the original behavior call:

POST /_aliases
{
  "actions" : [
    { "add" : { "index" : ".ml-annotations-6", "alias" : ".ml-annotations-read", "is_hidden": true } },
    { "remove" : { "index" : ".ml-annotations-6-wrong-mapping", "alias" : ".ml-annotations-read" } },
    { "add" : { "index" : ".ml-annotations-6", "alias" : ".ml-annotations-write", "is_hidden": true } },
    { "remove" : { "index" : ".ml-annotations-6-wrong-mapping", "alias" : ".ml-annotations-write" } }
  ]
}

Workaround

Since you cannot change mappings of an existing index, if you don’t have anything critically important in the existing annotations index then the easiest way to fix it is just to delete the .ml-annotations-6 index

If you do have annotations that are important then you need to do some reindexing:

  1. Create a new index, say temp-ml-annotations - it doesn't matter about settings or mappings for this index - defaults are fine
  2. Reindex .ml-annotations-6 into temp-ml-annotations
  3. Delete the .ml-annotations-6 index
  4. Wait for .ml-annotations-6 to be recreated
  5. Reindex temp-ml-annotations into .ml-annotations-6
  6. Delete the temp-ml-annotations index

Bugfix / Desired outcome for the UI

The UI should handle the error loading annotations more gracefully with a meaningful error message instead of ending up with a blank page.
@walterra walterra added bug Fixes for quality problems that affect the customer experience regression :ml Feature:Anomaly Detection ML anomaly detection v8.0.0 v7.10.0 v7.9.1 labels Aug 13, 2020
@walterra walterra self-assigned this Aug 13, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@droberts195
Copy link
Contributor

  • Upgrade to 7.9.0 BC9

I don't understand why this is only happening in BC9. As far as I can see the same problem was present in BC1 - BC8 too.

@walterra
Copy link
Contributor Author

walterra commented Aug 13, 2020 •
edited
Loading

I updated the description and removed the reference to BC9. BC9 was just the version where we discovered the issue.

@walterra walterra mentioned this issue Aug 13, 2020
Merged
3 tasks
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Aug 13, 2020
@droberts195
[ML] Ensure annotations index mappings are up to date
90155af 
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Relates elastic/kibana#74935
@droberts195 droberts195 mentioned this issue Aug 13, 2020
Merged
@droberts195
Copy link
Contributor

The underlying problem will be fixed by elastic/elasticsearch#61107

droberts195 added a commit to elastic/elasticsearch that referenced this issue Aug 13, 2020
@droberts195
[ML] Ensure annotations index mappings are up to date (#61107)
a78d951 
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Relates elastic/kibana#74935
@droberts195 droberts195 mentioned this issue Aug 14, 2020
Open
@droberts195
Copy link
Contributor

One way to insure against suffering this problem at all is to manually adjust the mappings that are required in 7.9.0 before upgrading to 7.9.0 (it doesn't hurt to apply them to an earlier version if the .ml-annotations-6 index already exists):

PUT .ml-annotations-6/_mapping
{
  "properties": {
    "event" : {
      "type" : "keyword"
    },
    "detector_index" : {
      "type" : "integer"
    },
    "partition_field_name" : {
      "type" : "keyword"
    },
    "partition_field_value" : {
      "type" : "keyword"
    },
    "over_field_name" : {
      "type" : "keyword"
    },
    "over_field_value" : {
      "type" : "keyword"
    },
    "by_field_name" : {
      "type" : "keyword"
    },
    "by_field_value" : {
      "type" : "keyword"
    }
  }
}

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Aug 17, 2020
@droberts195
[DOCS] Add 7.9.0 known issue for problems with ML index mappings
570cc62 
Relates elastic#61157
Relates elastic/kibana#74935
@droberts195 droberts195 mentioned this issue Aug 17, 2020
Merged
@wwang500
Copy link

Workaround

Since you cannot change mappings of an existing index, if you don’t have anything critically important in the existing annotations index then the easiest way to fix it is just to delete the .ml-annotations-6 index

If you do have annotations that are important then you need to do some reindexing:

  1. First create an index .ml-annotations-saved - it should pick up the new template
  2. Reindex .ml-annotations-6 into .ml-annotations-saved
  3. Delete the .ml-annotations-6 index
  4. Wait for .ml-annotations-6 to be recreated
  5. Add alias .ml-annotations-read to .ml-annotations-saved - then it will span both indices

@walterra and @droberts195 the above workaround successfully brought our UI back, and new annotations can be created without problem. However, the annotations being created before the upgrade are still missing. I am wondering if there is one extra step to manually create .ml-annotations-saved index with new mapping (before step 2)?
image

@droberts195
Copy link
Contributor

Thanks for testing it @wwang500. You are correct that there's a flaw in the workaround, which is that there is no template for .ml-annotations* - the mappings are always applied programmatically.

So the instructions need to be different:

  1. Create a new index, say temp-ml-annotations - it doesn't matter about settings or mappings for this index - defaults are fine
  2. Reindex .ml-annotations-6 into temp-ml-annotations
  3. Delete the .ml-annotations-6 index
  4. Wait for .ml-annotations-6 to be recreated
  5. Reindex temp-ml-annotations into .ml-annotations-6
  6. Delete the temp-ml-annotations index

Sorry for the confusion. I will adjust the original issue description to avoid confusion for people who just read that.

droberts195 added a commit to elastic/elasticsearch that referenced this issue Aug 18, 2020
@droberts195
[DOCS] Add 7.9.0 known issue for problems with ML index mappings (#61195
60e1cec 
)

Relates #61157
Relates elastic/kibana#74935
@wwang500
Copy link

Thanks for testing it @wwang500. You are correct that there's a flaw in the workaround, which is that there is no template for .ml-annotations* - the mappings are always applied programmatically.

So the instructions need to be different:

  1. Create a new index, say temp-ml-annotations - it doesn't matter about settings or mappings for this index - defaults are fine
  2. Reindex .ml-annotations-6 into temp-ml-annotations
  3. Delete the .ml-annotations-6 index
  4. Wait for .ml-annotations-6 to be recreated
  5. Reindex temp-ml-annotations into .ml-annotations-6
  6. Delete the temp-ml-annotations index

Sorry for the confusion. I will adjust the original issue description to avoid confusion for people who just read that.

I can confirm the above workaround is a valid working one. thanks @droberts195

@yoli1888 yoli1888 mentioned this issue Sep 17, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walterra walterra

Labels
bug Fixes for quality problems that affect the customer experience Feature:Anomaly Detection ML anomaly detection :ml regression v7.9.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Anomaly Explorer / Single Metric Viewer: Fix error reporting for annotations. walterra/kibana
4 participants
@walterra @droberts195 @elasticmachine @wwang500