MessageToDict and ParseDict fail on Instance

[TobKed]

ipdb> result
name: "projects/polidea-airflow/locations/europe-north1/instances/test-memorystore-memcached-1"
labels {
  key: "airflow-version"
  value: "v2-0-0-dev0"
}
authorized_network: "projects/project-name/global/networks/default"
zones: "europe-north1-a"
zones: "europe-north1-b"
zones: "europe-north1-c"
node_count: 1
node_config {
  cpu_count: 1
  memory_size_mb: 1024
}
memcache_version: MEMCACHE_1_5
parameters {
}
memcache_nodes {
  node_id: "node-c-1"
  zone: "europe-north1-c"
  state: READY
  host: "######"
  port: 11211
  parameters {
  }
}
create_time {
  seconds: 1602506221
  nanos: 813043718
}
update_time {
  seconds: 1602506527
  nanos: 277737323
}
state: READY
memcache_full_version: "memcached-1.5.16"
discovery_endpoint: "######:11211"

ipdb> type(result)
<class 'google.cloud.memcache_v1beta2.types.cloud_memcache.Instance'>
ipdb> MessageToDict(result)
*** AttributeError: 'DESCRIPTOR'

[2020-09-25 12:32:12,847] {cloud_memorystore.py:680} INFO - Instance not exists.
[2020-09-25 12:32:12,849] {taskinstance.py:1337} ERROR - 'DESCRIPTOR'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/proto/message.py", line 520, in __getattr__
    pb_type = self._meta.fields[key].pb_type
KeyError: 'DESCRIPTOR'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/airflow/airflow/models/taskinstance.py", line 1076, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/airflow/airflow/models/taskinstance.py", line 1198, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/airflow/airflow/models/taskinstance.py", line 1243, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/airflow/airflow/providers/google/cloud/operators/cloud_memorystore.py", line 1291, in execute
    metadata=self.metadata,
  File "/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 373, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/providers/google/cloud/hooks/cloud_memorystore.py", line 683, in create_instance
    instance = ParseDict(instance, cloud_memcache.Instance())
  File "/usr/local/lib/python3.7/site-packages/google/protobuf/json_format.py", line 452, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/usr/local/lib/python3.7/site-packages/google/protobuf/json_format.py", line 476, in ConvertMessage
    message_descriptor = message.DESCRIPTOR
  File "/usr/local/lib/python3.7/site-packages/proto/message.py", line 525, in __getattr__
    raise AttributeError(str(ex))
AttributeError: 'DESCRIPTOR

• OS type and version: Debian GNU/Linux 10 (buster) (docker)
• Python version:Python 3.7.9`
• pip version: 20.2.3
• google-cloud-memcache version: 0.2.0

Other relate packages:

googleapis-common-protos           1.52.0
proto-plus                         1.10.0
protobuf                           3.13.0

[busunkim96]

Hi,

proto-plus adds a wrapper around the raw protobuf types. If you want to convert a message to the protobuf dict you can use the to_json and from_json methods. https://proto-plus-python.readthedocs.io/en/latest/messages.html#serialization

[TobKed]

Thanks @busunkim96 !
It can be done as you suggested, however should not ParseDict and MessageToDict to work as well?
In Apache Airflow it is used e.g in with Memorystore Redis (google.cloud.redis_v1.types.Instance): https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/cloud_memorystore.py#L160
and works without any problems.

Exactly same thing is tried to be done here with google.cloud.memcache_v1beta2.types.cloud_memcache.Instance (@tanjinP PR):
https://github.com/apache/airflow/blob/aaca868f36e08e405e75a39f1774f16e9b731c48/airflow/providers/google/cloud/hooks/cloud_memorystore.py#L687
and it fails as I described in the issue description.

[busunkim96]

ParseDict and MessageToDict are methods provided by protobuf. Since proto-plus wraps the proto messages those protobuf methods can no longer be used directly.

It looks like the first file is written for the 1.x release of google-cloud-redis. At that point we were still providing protobuf types directly. When that code uses the 2.x release you will see the same issue

https://proto-plus-python.readthedocs.io/en/latest/messages.html#usage

Creating an Instance:

from google.cloud import memcache

# via kwargs
instance = memcache.Instance(name="projects/project/locations/location/instances/instance")

# from dict
instance = memcache.Instance({"name": "projects/project/locations/location/instances/instance"})

# from a JSON string
instance = memcache.Instance.from_json('{"name":"projects/project/locations/location/instances/instance"}')

Converting into other formats

import json

from google.cloud import memcache

# to json
instance = memcache.Instance(name="projects/project/locations/location/instances/instance")
instance_json = memcache.Instance.to_json(instance)
# to dict
instance_dict = json.loads(instance_json)

# to bytes
instance_bytes = memcache.Instance.serialize(instance)

[TobKed]

It makes sense.

I made helper function which seems to solve my problem with conversion to dictionary:

import json

import proto

def proto_message_to_dict(message: proto.Message) -> dict:
    """Helper method to parse protobuf message to dictionary."""
    return json.loads(message.__class__.to_json(message))

Thank you for comprehensive answer @busunkim96!

[busunkim96]

@TobKed Good to hear! I also opened googleapis/proto-plus-python#151 to see if we can add a to_dict method to the messages.

[busunkim96]

For anyone looking at this now, there is a to_dict method in the latest release. https://proto-plus-python.readthedocs.io/en/latest/messages.html#serialization

Similarly, messages can be converted into dictionaries via the to_dict() helper method. There is no from_dict() method because the Message constructor already allows construction from mapping types.

song_dict = Song.to_dict(song)

new_song = Song(song_dict)