Support to remove aliases from index mappings

[ruflin]

Today it's possible to add an alias to an index mapping:

PUT foo
{
  "mappings": {
    "_doc": {
      "properties": {
        "a": {
          "type": "keyword"
        }
      }
    }
  }
}

After adding an alias it's not possible AFAIK to remove it. This would be useful if someone applies aliases to many indices and then realises it was the wrong field and wants to change it.

To my surprise it is possible to modify existing aliases by adding the same mapping again with a different path. This is great and already allows to handle the case where the wrong path was set by mistake but the alias name is correct.

[elasticmachine]

Pinging @elastic/es-core-features

[colings86]

I'm not sure about allowing this. Mapping updates are today only additive and this prevents a lot of issues (we actually used to have the ability to remove a mapping type but that caused issues and so we removed it). The problems this could cause are not just limited to within Elasticsearch either. Applications using Elasticsearch rely on mappings and would now need to account for the fact that a field may disappear from the mapping at any time meaning that caching the fields for an index (Kibana for example does this in its index patterns) would be a lot harder.

[bleskes]

I'm not per se supportive of the issue and I can see the points that @colings86 made but I want to add some color to this, to make sure the impact is understood (@ruflin please correct me if I misread where you're coming from).

The current plan is to heavily rely on field aliases to help the migration from 6.x non-ECS fields (Elastic Common Schema) to 7.x ECS based mappings. Currently we plan to offer people an opt-in BWC layer that will added 6.x compatible field aliases to 7.x so people can query both 7.x and 6.x indices in one query/agg. The ability to remove aliases will allow for us to fix bugs / change things in existing aliases if needed.

[ruflin]

It seems mappings are not purely additive as they can also be edited. Field alias mapping already today supports changing the path of an existing alias. I think that solves part of the problem that @bleskes described above. In case a user introduced an alias a and pointed it to b and realises it should have been pointed to c, it can be updated. The problem left is what if the user realises he doesn't need alias a anymore.

One use I have in mind here is that users have the 7.x aliases around during the time they have 6.x and 7.x data and at one point when they only have 7.x data want to get rid of all the aliases as they are not needed anymore.

@stacey-gammon I wonder if changing the path of an alias could have an affect on Kibana?

[jpountz]

We discussed this issue in FixitFriday. In general we don't want to remove fields from the mappings without removing them as well from the index. We had some attempts in the past (eg. upon merge) but we eventually decided against removing fields due to the complexity and how it gave room for bugs including data corruption.

Field aliases are a bit different due to the fact that they don't store any data in the index, which might make removing aliases acceptable. Yet that would still come with a number of challenges, for instance it would be an issue for saved searches, eg. the percolator, since queries are indexed depending on how a field is mapped. If an alias was changed to point to a field of a different type, it would never match (either correctly or at all) again.

This might also be a high hanging fruit as I suspect that there are lots of places that rely on the fact that mappings are additive internally.

[ruflin]

It is already possible today changing the path of an alias even if the field has a different type (see example below). Wouldn't that cause the same issue as described above?

PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "a": {
          "type": "keyword"
        },
        "b": {
          "type": "long"
        },
        "c": {
          "type": "alias",
          "path": "a"
        }
      }
    }
  }
}

GET test

PUT test/_mapping/_doc
{
  "properties": {
    "c": {
      "type": "alias",
      "path": "b"
    }
  }
}

GET test

The alias first points to a keyword field and then to a long field on the second GET.

[jpountz]

Indeed.

[elasticmachine]

Pinging @elastic/es-search

[jtibshirani]

I'm removing the team-discuss label here, because I think we already discussed the considerations around deleting field aliases.

I opened a bug report for the percolator issue in #37212. I suggest we use that issue for a focused discussion around whether the path of a field alias should be updatable.

[javanna]

I think this one is no longer a high hanging fruit and would be automatically fixed by resolving #87969 .

[jtibshirani]

@javanna sorry if I'm missing something, how will making field aliases a runtime field (#87969) automatically let aliases be deleted?

[javanna]

@jtibshirani fair question, maybe I should have expanded a bit. I was thinking that once we have a good answer for aliasing a field with a runtime field that does not require a script, given that runtime fields can be removed this issue is solved. But it is solved only for users that are using the new way and it remains for folks that have field aliases in their mappings. We should then discuss what to do about those.

Does this make more sense now?

[jtibshirani]

Yes thanks, I didn't understand that runtime fields could be easily removed. Should we close this out in favor of #87969? It feels like the best path forward for field aliases.

[javanna]

makes sense, closing this in favour of #87969 .