Protobuf messages can no longer be deserialized after migration to 2.0.0

[alexcombessie]

Environment details

• OS type and version: macOS 10.14.6 - also reproduced with Linux CentOS 7
• Python version: 3.6.10
• pip version: 18.1
• google-cloud-vision version: 2.0.0
• protobuf version: 3.13.0

Steps to reproduce

1. Run client.annotate_image(request)=> SUCCESS (request is valid and you can print the output)
2. Try to deserialized message => FAILURE AttributeError: 'DESCRIPTOR'

Code example

from google.protobuf.json_format import MessageToDict
message = client.annotate_image(request)
print(message)  # this works
MessageToDict(message)  # this fails

Stack trace

[12:09:25] [INFO] [dku.utils]  - responses {
[12:09:25] [INFO] [dku.utils]  -   localized_object_annotations {
[12:09:25] [INFO] [dku.utils]  -     mid: "/m/02dl1y"
[12:09:25] [INFO] [dku.utils]  -     name: "Hat"
[12:09:25] [INFO] [dku.utils]  -     score: 0.509366512298584
[12:09:25] [INFO] [dku.utils]  -     bounding_poly {
[12:09:25] [INFO] [dku.utils]  -       normalized_vertices {
[12:09:25] [INFO] [dku.utils]  -         x: 0.005275096278637648
[12:09:25] [INFO] [dku.utils]  -         y: 0.008623555302619934
[12:09:25] [INFO] [dku.utils]  -       }
[12:09:25] [INFO] [dku.utils]  -       normalized_vertices {
[12:09:25] [INFO] [dku.utils]  -         x: 0.9973958134651184
[12:09:25] [INFO] [dku.utils]  -         y: 0.008623555302619934
[12:09:25] [INFO] [dku.utils]  -       }
[12:09:25] [INFO] [dku.utils]  -       normalized_vertices {
[12:09:25] [INFO] [dku.utils]  -         x: 0.9973958134651184
[12:09:25] [INFO] [dku.utils]  -         y: 0.9851147532463074
[12:09:25] [INFO] [dku.utils]  -       }
[12:09:25] [INFO] [dku.utils]  -       normalized_vertices {
[12:09:25] [INFO] [dku.utils]  -         x: 0.005275096278637648
[12:09:25] [INFO] [dku.utils]  -         y: 0.9851147532463074
[12:09:25] [INFO] [dku.utils]  -       }
[12:09:25] [INFO] [dku.utils]  -     }
[12:09:25] [INFO] [dku.utils]  -   }
[12:09:25] [INFO] [dku.utils]  - }
[12:09:25] [INFO] [dku.utils]  - responses {
[12:09:25] [INFO] [dku.utils]  - *************** Recipe code failed **************
[12:09:25] [INFO] [dku.utils]  - }
[12:09:25] [INFO] [dku.utils]  - Begin Python stack
[12:09:25] [INFO] [dku.utils]  - responses {
[12:09:25] [INFO] [dku.utils]  - Traceback (most recent call last):
[12:09:25] [INFO] [dku.utils]  - }
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/code-envs/python/plugin_google-cloud-vision_managed_2/lib/python3.6/site-packages/proto/message.py", line 520, in __getattr__
[12:09:25] [INFO] [dku.utils]  -     pb_type = self._meta.fields[key].pb_type
[12:09:25] [INFO] [dku.utils]  - KeyError: 'DESCRIPTOR'
[12:09:25] [INFO] [dku.utils]  - During handling of the above exception, another exception occurred:
[12:09:25] [INFO] [dku.utils]  - Traceback (most recent call last):
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/jobs/TESTGOOGLEVISIONPLUGIN/Build_cloud_content__me__2020-10-13T10-09-12.241/compute_cloud_content_me/custom-python-recipe/pyouthtnvsUANhkXA/python-exec-wrapper.py", line 206, in <module>
[12:09:25] [INFO] [dku.utils]  -     exec(f.read())
[12:09:25] [INFO] [dku.utils]  -   File "<string>", line 58, in <module>
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/plugins/dev/google-cloud-vision/python-lib/api_parallelizer.py", line 201, in api_parallelizer
[12:09:25] [INFO] [dku.utils]  -     api_results.append(f.result())
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/.pyenv/versions/3.6.10/lib/python3.6/concurrent/futures/_base.py", line 425, in result
[12:09:25] [INFO] [dku.utils]  -     return self.__get_result()
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/.pyenv/versions/3.6.10/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
[12:09:25] [INFO] [dku.utils]  -     raise self._exception
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/.pyenv/versions/3.6.10/lib/python3.6/concurrent/futures/thread.py", line 56, in run
[12:09:25] [INFO] [dku.utils]  -     result = self.fn(*self.args, **self.kwargs)
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/plugins/dev/google-cloud-vision/python-lib/api_parallelizer.py", line 109, in api_call_batch
[12:09:25] [INFO] [dku.utils]  -     batch = batch_api_response_parser(batch=batch, response=response, api_column_names=api_column_names)
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/plugins/dev/google-cloud-vision/python-lib/google_vision_api_client.py", line 78, in batch_api_response_parser
[12:09:25] [INFO] [dku.utils]  -     response_dict = MessageToDict(response)
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/code-envs/python/plugin_google-cloud-vision_managed_2/lib/python3.6/site-packages/google/protobuf/json_format.py", line 175, in MessageToDict
[12:09:25] [INFO] [dku.utils]  -     return printer._MessageToJsonObject(message)
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/code-envs/python/plugin_google-cloud-vision_managed_2/lib/python3.6/site-packages/google/protobuf/json_format.py", line 209, in _MessageToJsonObject
[12:09:25] [INFO] [dku.utils]  -     message_descriptor = message.DESCRIPTOR
[12:09:25] [INFO] [dku.utils]  -   File "/Users/alexandrecombessie/Library/DataScienceStudio/dss_home/code-envs/python/plugin_google-cloud-vision_managed_2/lib/python3.6/site-packages/proto/message.py", line 525, in __getattr__
[12:09:25] [INFO] [dku.utils]  -     raise AttributeError(str(ex))
[12:09:25] [INFO] [dku.utils]  - AttributeError: 'DESCRIPTOR'

I read other issues, and it seems this bug is linked to 2 other issues: #60 #58

[alexcombessie]

Hi there,

Are there any news on this topic? It's a blocking issue for my use case. I will not be able to use the new version until this is fixed.

Thanks,

Alex

[tubli]

I've encountered the same issue. My config:
Ubuntu 20
pip 20.0.2
python 3.8.5

I tried different permutations of library versions and this is what I found:

has issue
google-cloud-vision 2.0.0, protobuf 3.13.0
google-cloud-vision 2.0.0, protobuf 3.12.0

no issue
google-cloud-vision 1.0.0, protobuf 3.13.0

Happy to run other tests as well if it helps get this issue resolved.

Alex M

[busunkim96]

Hi folks,

The 2.0.0 uses messages provided by proto-plus, which wraps the raw protobuf messages. See this issue for examples of how to serialize and deserialize messages.

[tubli]

For anyone who comes across this thread in the future or is still trying to figure how how to get this working, here is some code and notes.

import io, json
from google.cloud import vision_v1
from google.cloud.vision_v1 import AnnotateImageResponse

with io.open('000048.jpg', 'rb') as image_file:
    content = image_file.read()

image = vision_v1.Image(content=content)
client = vision_v1.ImageAnnotatorClient()
response = client.document_text_detection(image=image)

# serialize / deserialize proto (binary)
serialized_proto_plus = AnnotateImageResponse.serialize(response)
response = AnnotateImageResponse.deserialize(serialized_proto_plus)
print(response.full_text_annotation.text)

# serialize / deserialize json
response_json = AnnotateImageResponse.to_json(response)
response = json.loads(response_json)
print(response['fullTextAnnotation']['text'])

Note 1: proto-plus doesn't support converting to snake_case names, which is supported in protobuf with "preserving_proto_field_name=True". So currently there is no way around the field names being converted from response['full_text_annotation'] to response['fullTextAnnotation']
There is an open feature request for this: googleapis/proto-plus-python#109

Note 2: The google vision api doesn't return an x coordinate if x=0. If x doesn't exist, the protobuf will default x=0. In python vision 1.0.0 using MessageToJson(), these x values weren't included in the json, but now with python vision 2.0.0 and .To_Json() these values are included as x:0

[alexcombessie]

Thanks @tubli @busunkim96. I managed to make it work with the following code snippet from @TobKed which does that for any protobuf message: googleapis/python-memcache#19 (comment)

I think an update of the google-cloud-vision Python library documentation would really help others encountering the same issue.

[software-dov]

@tubli This response is spot on. We'll update the documentation to make it less opaque and frustrating to folks upgrading to client library versions that use proto plus.