Z values support to geo_point and geo_shape

[sherry-ger]

Describe the feature:

Add the ability to store and retrieve z values with geo_points and geo_shapes, not necessarily for the analytics portion of things. Similarly, add the ability to store and retrieve M (measure) and W (weight) values with point and shape vertices. The values are needed (stored and retrieved) when rendering in 3D and performing 3D analytics on the client side after the es query. Although it is possible to store the values in separate field(s), it is complex and difficult to work with.

[clintongormley]

@nknize what are your thoughts about this one?

[hanoch]

@nknize will be good to have this enhancement included with the next release (v5.4)

[jpountz]

Could you explain what in particular makes it complex and difficult to work with? I'm a bit torn about making it possible to provide some data that would systematically be ignored. Do you only care about being able to do it in documents or also in queries?

For the record, it is possible to have stored-only values by mapping them as objects with enabled:false:

PUT index 
{
  "mappings": {
    "doc": {
      "properties": {
        "my_source_field": {
          "type": "object",
          "enabled": false
        }
      }
    }
  }
}

PUt index/doc/1
{
  "my_source_field": [[[-45, 30, 100], [45, 30, 75], [45, -30, 77], [-45, -30, 101], [-45, 30, 100]]]
}

And in case you need to duplicate fields so that you can both store the 3D shape and index the 2D shape, you can have ingest automatically take care of it for you, for instance the following script processor:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "source": "List removeZFromPoint(List coords) {return coords.subList(0, 2);} List removeZFromPoly(List poly) { return poly.stream().map(this::removeZFromPoint).collect(Collectors.toList()); } List removeZFromShape(List shape) { return shape.stream().map(this::removeZFromPoly).collect(Collectors.toList()); }  ctx.shape_index = removeZFromShape(ctx.shape)"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "shape": [
          [
            [
              -45,
              30,
              100
            ],
            [
              45,
              30,
              75
            ],
            [
              45,
              -30,
              77
            ],
            [
              -45,
              -30,
              101
            ],
            [
              -45,
              30,
              100
            ]
          ]
        ]
      }
    }
  ]
}

returns

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_type": "_type",
        "_id": "_id",
        "_source": {
          "shape": [
            [
              [
                -45,
                30,
                100
              ],
              [
                45,
                30,
                75
              ],
              [
                45,
                -30,
                77
              ],
              [
                -45,
                -30,
                101
              ],
              [
                -45,
                30,
                100
              ]
            ]
          ],
          "shape_index": [
            [
              [
                -45,
                30
              ],
              [
                45,
                30
              ],
              [
                45,
                -30
              ],
              [
                -45,
                -30
              ],
              [
                -45,
                30
              ]
            ]
          ]
        },
        "_ingest": {
          "timestamp": "2017-08-01T15:58:54.722Z"
        }
      }
    }
  ]
}

In that case, you'll just need to make sure to query the shape_index field instead of the shape field.

[hanoch]

We are aware of the workaround to store the other n-dimension (Z, M etc.) of the geo_point/geo_shape (polygon and poylline vertices) in another field, but the product we develop is a general purpose product, where field names are provided by the product users, and can collide with any other auto-generated field names (such as shape or shape_index).

It is much more straight forward to allow ES to be able to store (and retrieve) all the point/shape-vertices n-dimnesions, for example Z, on top of the X, Y dimension. It stream-lines queries against those geometry fields, returning all the geometry field dimensions in a staight-forward way, rather than trying to use workarounds and other type of tricks to try and join the returned X, Y query results with Z values, for example.

Not being able to store and retrieve geometry Z values (as an example) with the geometry type fields can be compared to not being able to store and retrieve the seconds value portion from a Date/Time field type, and instead needing to store the seconds portion in a second non-date field, or the entire date/time value including the seconds value in a second non-date numeric or text field. Imagine that that was a limitation of the date type fields, not being able to store the seconds value. We can work with our time values stored in two fields, and manage to get the functionality implemented, working against two fields rather than the one date field, needing to merge and massage the query results, but it makes the usage less intuitive, custom, harder to maintain, and more errors prone. Isn't it much easier and staright forward to have the seconds portion of the date field value already included in the field value?

Also, think about the need to update the geometry field. Having to manage two fields forces more custom implementation/workarounds, and is more errors prone.

On top of all of that, it results in a non-standard index, which can only be queried, edited and managed (e.g. backup/restore, etc.) by custom code/scripts/implementation.

This is what we mean by "complex and difficult to work with".

[jpountz]

the product we develop is a general purpose product, where field names are provided by the product users, and can collide with any other auto-generated field names (such as shape or shape_index)

I understand you'd like to not have to modify your document at all, but as a workaround to this particular issue you could reserve some namespace for the user document and then deal with your fields with more freedom, eg. { "original_source": { /* user data is here */ }, /* fields that you need to index or add doc values to are here */ }.

It is much more straight forward to allow ES to be able to store (and retrieve) all the point/shape-vertices n-dimnesions, for example Z, on top of the X, Y dimension.

I don't disagree it might make things more straight forward in your case. But ignoring everything but the x and y dimensions will be trappy to other users as well. So from our perspective, there is a trade-off between predictability and ease-of-use, and it is not obvious to me that we should favour the latter over the former.

I have been thinking about accepting z coordinates at index time but rejecting them on the query side to not make users feel like we might use it somehow. But it would still annoy me a bit to not have the same parsing logic on both ends, which is bug-prone. Also, what should we do if the query is built using a pre-indexed shape that has a z dimension in that case?

Not being able to store and retrieve geometry Z values (as an example) with the geometry type fields can be compared to not being able to store and retrieve the seconds value portion from a Date/Time field type, and instead needing to store the seconds portion in a second non-date field, or the entire date/time value including the seconds value in a second non-date numeric or text field. Imagine that that was a limitation of the date type fields

Actually, we do have that limitation on date fields, and some users need to store micros or nanos in a different field: #10005.

[hanoch]

We already had long discussions with Shane, Sherry, Nick and others from Elastic, explaining the need to support Z values with geometries, and going into depth and more details about the different use cases.
This requirement is not specific to our use cases, having geometries include Z values is basic DBs (and search engines) requirement, when supporting geometry filed types. I don't think we have any disagreement here.

To reiterate some points, and to make sure we are on the same page:

1. we are using ES as a DB, and need to be able store and retrieve geometries including X, Y, Z (and possibly also M & W) along with the geometry field.
2. this functionality is supported by other DBs that support geometries.
3. It will be nice to also be able to do geometry Z-value (on top of the already exiting Lucene X and Y index capabilities) analytics while querying/searching, but this is not a must for now. We can do post query analysis if need to, but the main need to store and retrieve Z values is to be able to store them in the DB and use their values when we query for the geometry field(s).
4. We understand that there are two levels of support - being able to store-and-retrieve Z values with the geometry fields, and being able to actually index and include those Z-values with the query/search/analysis (this portion needs to be supported down in Lucene). It will be nice to get both capabilities, but having Z values search-and-retrieve capabilities without the extra Z value analysis capabilities goes a long way, and completes the ES requirement to be a DB that can support geometry field types, like all the other DBs that support geometry field types.

As far as your comment about my comparison/analogy of the geometry Z value to a date field seconds component, perhaps my analogy was not representative of the basic need of geometry field types to support Z values. a better analogy would probably be to imagine that the date fields in ES can only hold the date component (dd/MM/yyyy) but not the time component (HH:mm:ss.sss). This is basic date field capability. The requirement of having the date field value support micro or nano seconds precision is more about the ability of the field to store higher precision. From basic vs. advanced requirement point of view, it might be compared to the need of the geometry fields to support n-dimensions above X,Y,Z, for example M (measure), W (weight), etc. Z values support is basic geometry field type requirement.

I think we agree that Z value support is something ES should have out of the box, the question is how can we roll in that support. Should ES:

1. start with store-and-retrieve Z values support and document the limitation of that support, until Lucene can support the analysis portion, or -
2. wait until Z values are fully supported down in Lucene to be able to support ALL the search/query/analysis cases ES already support with X and Y values. I am not sure that will ever be fully completed.

With my discussions with Shane and others from Elastic, I was under the impression that approach 1 was chosen - start with the ability to store-and-retrieve Z values with the geometry field types.

[jpountz]

Agreed with you about the usefulness of support for Z values. I'm just wondering about potential side-effects on other users and it would be a pity that eg. malformed inputs are not detected anymore because of the fact there is an optional 3rd dimension. Or that some users think that the z dimension will be taken into account for their queries while it's not. Even though I tend to dislike adding options to mappings, I'm wondering whether this is something we should only do when an option is turned on (maybe something like coerce on numeric fields) so that it can be enabled when you know what you are doing but it is not trappy to the majority of users.

[gabi939]

Hey,

According to documentation there is support to Z values is included.
https://www.elastic.co/guide/en/elasticsearch/reference/7.7/geo-point.html

I deployed docker image in the following way:
sudo docker run --name es01 --net elastic -p 9200:9200 -it docker.elastic.co/elasticsearch/elasticsearch:7.7.0

I have created an index:

But when I try to upload documents in the following format:

I receive the following exception:

Is the feature works for elasticsearch 7.7.0? Do I need to activate anything in my container?