languages not filtered when using custom tokenizer

[sjhewitt]

What version of Dgraph are you using?

master

Have you tried reproducing the issue with the latest release?

yes

Steps to reproduce the issue (command/config used to run Dgraph).

• run dgraph using the anagram tokenizer: dgraph alpha --custom_tokenizers anagram.so
• schema:

word: string @index(anagram) @lang .

• data:

[
	{ "word@en": "airmen", "word@fr": "no match" },
	{ "word@en": "no match", "word@fr": "marine" }
]

• query:

{
  q(func: allof(word@en, anagram, "remain")) {
		word: word@en
	}
}

Expected behaviour and actual result.

Actual Result - results where any language strings match are returned:

{
  "data": {
    "q": [
      {
        "word": "airmen"
      },
      {
        "word": "no match"
      }
    ]
  }
}

Expected Result - only nodes where strings from the specified language match are returned:

{
  "data": {
    "q": [
      {
        "word": "airmen"
      }
    ]
  }
}