Add import_filter to MongoDBRiverDefinition.

[bernd]

This allows filtering of the initial import.

I noticed that the initial import ignores the defined filter in the MongoDBRiverDefinition. I first tried to feed the configured filter to slurpedCollection.find(), but that does not work because the oplog filter needs the o. prefix. And that does not work with the regular find() on the collection. So I created another filter option. (import_filter)

Not sure if you want to merge this as it is. Maybe it would be nicer to only have one filter definition that works for the initial import and the oplog filter. (by prepending the o. prefix automatically or something like that)

It works for me and helps me to filter the initial import so I wanted to share it.

Attention: I don't know Java, so please double check this code. ;)

[bernd]

Usage example:

curl -XPUT "localhost:9200/_river/foo-de/_meta" -d '
{
  "type": "mongodb",
  "mongodb": { 
    "servers":
    [
      { "host": "localhost", "port": 27017 }
    ],
    "db": "foo",
    "collection": "data",
    "import_filter": "{\"lang\":\"de\"}",
    "filter": "{\"o.lang\":\"de\"}"
  }, 
  "index": { 
    "name": "foo", 
    "type": "de"
  }
}'

[richardwilly98]

@bernd Thank you for reporting the issue.
I will take a look this week at your PR and will se if we can use only one filter definition.

[benmccann]

I agree one filter definition would be nicer. Should be able to just check if the filter starts with "o." then strip off the first two characters

[richardwilly98]

@bernd it is now possible to set the filter like this:

curl -XPUT "localhost:9200/_river/foo-de/_meta" -d '
{
  "type": "mongodb",
  "mongodb": { 
    "servers":
    [
      { "host": "localhost", "port": 27017 }
    ],
    "db": "foo",
    "collection": "data",
    "filter": "{\"o.lang\":\"de\"}"
  }, 
  "index": { 
    "name": "foo", 
    "type": "de"
  }
}'

or

curl -XPUT "localhost:9200/_river/foo-de/_meta" -d '
{
  "type": "mongodb",
  "mongodb": { 
    "servers":
    [
      { "host": "localhost", "port": 27017 }
    ],
    "db": "foo",
    "collection": "data",
    "filter": "{\"lang\":\"de\"}"
  }, 
  "index": { 
    "name": "foo", 
    "type": "de"
  }
}'

The river definition will provide 2 BDObjects for filtering one without the o. for the initial import and one with the prefix for oplog.rs

[bernd]

Awesome, thank you very much! :)

I'm closing this one.