Reindex confict: "proceed" throws error when casting malformed data to geo_point

[g00fy-]

When reindexing malformed object to geo_point the conflict parameter is ignorred and stops the task

example doc

{"geo":{"lat":null,"lon":null}}

with mapping:

{"properties":{"geo":{"properties":{"lat":{"type":long}, "lon":{"type":long}}}}

reindexing to

{"properties":{"geo":{"type":"geo_point", "ignore_malformed": true}}}

exception:

{
   "took": 24483,
   "timed_out": false,
   "total": 5211,
   "updated": 699,
   "created": 2100,
   "batches": 28,
   "version_conflicts": 0,
   "noops": 0,
   "retries": 0,
   "failures": [
      {
         "index": "search.locations",
         "type": "location",
         "id": "86d20ff9-ea24-448c-95a4-f060a30a80ad",
         "cause": {
            "type": "mapper_parsing_exception",
            "reason": "failed to parse",
            "caused_by": {
               "type": "parse_exception",
               "reason": "latitude must be a number"
            }
         },
         "status": 400
      }
   ]
}

Elasticsearch version:
docker:latest (2.3)

example query

POST /_reindex
{
   "conflicts": "proceed",
   "source": {
      "index": "raw.locations"
   },
   "dest": {
      "index": "search.locations"
   }
}

[nik9000]

That doesn't look like a bug to me. Maybe a documentation bug.
conflicts=proceed only works on version conflicts.

If you want to skip the ones with invalid Geo points maybe you can craft
the query to do so? You could probably also use a script to try to make
them valid. But reindex doesn't have support for skipping arbitrary errors.
On Apr 8, 2016 6:58 AM, "Piotrek Majewski" notifications@github.com wrote:

When reindexing malformed object to geo_point the conflict parameter is
ignorred and stops the task

example doc

{"geo":{"lat":null,"lon":null}}

with mapping:

{"properties":{"geo":{"properties":{"lat":{"type":long}, "lon":{"type":long}}}}

reindexing to

{"properties":{"geo":{"type":"geo_point", "ignore_malformed": true}}}

exception:

{
"took": 24483,
"timed_out": false,
"total": 5211,
"updated": 699,
"created": 2100,
"batches": 28,
"version_conflicts": 0,
"noops": 0,
"retries": 0,
"failures": [
{
"index": "search.locations",
"type": "location",
"id": "86d20ff9-ea24-448c-95a4-f060a30a80ad",
"cause": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "parse_exception",
"reason": "latitude must be a number"
}
},
"status": 400
}
]
}

Elasticsearch version:
docker:latest (2.3)

example query

POST /_reindex
{
"conflicts": "proceed",
"source": {
"index": "raw.locations"
},
"dest": {
"index": "search.locations"
}
}

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#17617

[eskibars]

We probably should clarify a bit further in the documentation what "conflicts" means. Its a bit implicit by the fact that the return result says "version_conflicts", but I could see how somebody could miss that and assume this will skip more problem types

[henningandersen]

I tend to think there is a bug with geo_point here. Since the mapping was marked ignore_malformed : true, the index request should have succeeded. Nothing to do with the "conflicts": "proceed" _reindex setting though.

I tried reproducing this on master by indexing a doc with null lat/lon. It gave me a different error from above:

put localhost:9200/x/_doc/3?pretty
{"geo":{"lat":null,"lon":null}}
{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Malformed content, found extra data after parsing: END_OBJECT"
    }
  },
  "status" : 400
}

It seems the ignore_malformed handling has been put in place, but the parsing then mismatches the curly braces. This PR: #16833 seems to have added the ignore_malformed handling.

[henningandersen]

Fixed above mentioned bug in geo_point parsing and clarified meaning of conflicts: proceed in reindex. Closing this issue.