the api of detect_full_text() return incorrect results when the texts is chinese characters

[HopLiu]

now i use detect_full_text() to detect words in sdk way. there are two issues:

1. the detect_full_text cannot return correct language code and words info when the texts in picture is chinese.
2. meanwhile, the detect_text() can return correct info(words and bounds info).

since i need the info of each words and bounds, i have to use the first api. Any ideas or is this an exist issue of this api?

[lukesneeringer]

Hi @HopLiu,
Thank you for reporting this issue.

I do not think the client library does anything differently with regard to character encodings between detect_text and detect_full_text, although I intend to double check.

Could you provide me with a reproduction case? (Basically, what is an image I can use to observe this behavior?) I expect it is a problem on the vision backend, but I would like to confirm that before passing the report on.

[HopLiu]

use this attached jpg file(8E27CE646909307503C8C5D16.jpg) to test.

1. "python detect_chn.py fulltext resources/8E27CE646909307503C8C5D16.jpg "
   2."python detect_chn.py text resources/8E27CE646909307503C8C5D16.jpg"
   the detect_chn.py is the same as detect.py in "python-docs-samples-master/vision/cloud-client" except i add a little encodings in order to print on the console.
   

[daspecster]

Thanks @HopLiu! I can test this out real quick @lukesneeringer if you want.

[lukesneeringer]

@daspecster Sure. I have faith in the reproduction case, what I really want to know is whether it is our bug or the backend API's bug.

[daspecster]

@lukesneeringer so I'm still looking into this but for detect_full_text, under pages, I'm getting results like the following, but there are no properly encoded Chinese characters in the results AFAICT.

(Pdb) full_text.text
u'Eitute : WIHFW PGA TOUR DRAFX HESO 9: 274400\nFBIE ST9 : 0530-3560885 400-607 1001 (AFE HOiE)\n1$US49: 0530-3560898\n'

symbols {
    property {
      detected_languages {
        language_code: "cy"
      }
    }
    bounding_box {
      vertices {
        x: 141
        y: 442
      }
      vertices {
        x: 156
        y: 442
      }
      vertices {
        x: 156
        y: 462
      }
      vertices {
        x: 141
        y: 462
      }
    }
    text: "U"
}

Example

(Pdb) full_text.pages[0].blocks[0].paragraphs[0].words[0].symbols[0].text
u'E'
(Pdb) full_text.pages[0].blocks[0].paragraphs[0].words[0].symbols[1].text
u'i'
(Pdb) full_text.pages[0].blocks[0].paragraphs[0].words[0].symbols[2].text
u't'
(Pdb) full_text.pages[0].blocks[0].paragraphs[0].words[0].symbols[3].text
u'u'
(Pdb) full_text.pages[0].blocks[0].paragraphs[0].words[0].symbols[4].text
u't'

[daspecster]

Also, I tried passing languageHints (which currently isn't supported well in the library), but I get this...

"message": "image-annotator::error(12): Image processing error!"

I don't get that error if I leave languageHints out of the request.

I think we need some backend confirmation of what exactly is supported.

[daspecster]

@gguuss would you have any insight on this?

[lukesneeringer]

This sounds like a backend issue to me.

[gguuss]

The API accepts an optional parameter, the image context, which needs to specify the language. I am going to see if I can determine how to specify this in our Python Cloud client.

[daspecster]

So this may be related to my #3132 issue then.

@gguuss do you know if all annotation API's support ImageContext right now? If I need to support ImageContext for all annotation types then I can do that, but if it's only one or two types(as it was in the past) then I tried to make adding that information to the API call more direct.

[gguuss]

The ImageContext is not used by all features. I think the context configuration is used for crop hints, language features, and landmark / entities / labels. Do we currently have a way of setting it?

[gguuss]

I authored a a web-based proof of concept that correctly detects Chinese text so it is definitely not a backend issue. Passing the image context does not appear to have side effects when extra parameters are passed, for example, landmark detection still works with language set to zh.

[daspecster]

@gguuss your example uses this library?

[daspecster]

@gguuss I think I missed adding that. I would have sworn I had that for the EntityAnnotations but it appears that I'm not passing that along.

I do use ImageContext for crop hints but I think that might be the only one.

@lukesneeringer I can get to work on this if there aren't other priorities?

[lukesneeringer]

Go for it.

[gguuss]

@daspecster My example is Apiary on JavaScript (insert joke about merely being a front-end developer and microservices on Python scaring / frightening me).

[daspecster]

No jokes here! I hail from the frontend as well..but a long time ago in a galaxy far far away.

[gguuss]

Btw, crop hints works well, maybe we can similarly accept an optional parameter on detect_text and detext_fulltext for language.