datastore: Nested entity parsing fails

[pdknsk]

https://cloud.google.com/appengine/docs/python/ndb/properties#structured

In order to write entities with structured properties, I tried to read them first, to figure out the format.

First StructuredProperty.

class Item_Meta(ndb.Model):
    xy = ndb.IntegerProperty(repeated = True)

class Item(ndb.Model):
    meta = ndb.StructuredProperty(Item_Meta)

item = Item()
item.meta = Item_Meta(xy = [100, 200])
item.put() # 5710239819104256

>>> datastore.api.get([datastore.Key('Item', 5710239819104256)])
[<Entity[{'kind': u'Item', 'id': 5710239819104256L}] {u'meta.xy': [100L, 200L]}>]

Next LocalStructuredProperty.

Although a StructuredProperty can be repeated and a StructuredProperty can contain another StructuredProperty, beware: if one structured property contains another, only one of them can be repeated. A work-around is to use LocalStructuredProperty, which does not have this constraint.

class Item_Meta(ndb.Model):
    xy = ndb.IntegerProperty(repeated = True)

class Item(ndb.Model):
    meta = ndb.LocalStructuredProperty(Item_Meta, repeated = True)

item = Item()
item.meta = [Item_Meta(xy = [100, 200])]
item.put() # 6217263929622528

>>> datastore.api.get([datastore.Key('Item', 6217263929622528)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/api.py", line 229, in get
    entities.append(helpers.entity_from_protobuf(entity_pb))
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 53, in entity_from_protobuf
    value = _get_value_from_property_pb(property_pb)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 237, in _get_value_from_property_pb
    return _get_value_from_value_pb(property_pb.value)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 217, in _get_value_from_value_pb
    result = [_get_value_from_value_pb(x) for x in value_pb.list_value]
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 214, in _get_value_from_value_pb
    result = entity_from_protobuf(value_pb.entity_value)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 48, in entity_from_protobuf
    key = key_from_protobuf(pb.key)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/helpers.py", line 105, in key_from_protobuf
    return Key(*path_args, namespace=namespace, dataset_id=dataset_id)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/key.py", line 76, in __init__
    self._path = self._combine_args()
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/key.py", line 133, in _combine_args
    child_path = self._parse_path(self._flat_path)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud/datastore/key.py", line 94, in _parse_path
    raise ValueError('Key path must not be empty.')
ValueError: Key path must not be empty.

[dhermes]

@pdknsk Are you sure that's the correct stacktrace?

Also, we don't intend to support LocalStructruredProperty in gcloud.datastore but are hoping to put ndb in gcloud.datastore at some point. (See #557.)

I expect the request to return a bytes object of the local structured property serialized via pickle.

[pdknsk]

Yes, it's correct.

According to ndb source the value of LocalStructuredProperty uses the standard "protocol buffer" encoding.

[dhermes]

1. I didn't read closely enough, the stacktrace is from parsing the response.
2. After posting the pickle comment I immediately wondering if the protocol buffer was in fact the underlying format. My bad.

I will try to spin up an App Engine app to test this, but in the mean time you could run this code (at 0.4.0) and post your findings

from gcloud.datastore import _implicit_environ
from gcloud import datastore

datastore.set_defaults()

key_pb = datastore.Key('Item', 6217263929622528).to_protobuf()

cnxn = _implicit_environ.CONNECTION
dataset_id = _implicit_environ.DATASET_ID

results, missing, deferred = cnxn.lookup(dataset_id, [key_pbs])

print 'Results:'
for r in results:
    print r

print 'Missing:'
for m in missing:
    print m

print 'Deferred:'
for d in deferred:
    print d

Then the protobufs in results have __repr__ defined so they print quite nicely.

[pdknsk]

Looks good, I think.

Results:
key {
  partition_id {
    dataset_id: "s~boxheart-net"
  }
  path_element {
    kind: "Item"
    id: 5707702298738688
  }
}
property {
  name: "meta"
  value {
    list_value {
      entity_value {
        property {
          name: "xy"
          value {
            list_value {
              integer_value: 100
            }
            list_value {
              integer_value: 200
            }
          }
        }
      }
      indexed: false
    }
  }
}

Missing:
Deferred:

[dhermes]

Debugging now, recreated the entity in https://gist.github.com/dhermes/94746659780f25e362d4

[dhermes]

@pdknsk Can you confirm something for me?

# Same code as above
# ...
results, missing, deferred = cnxn.lookup(dataset_id, [key_pbs])

entity_pb = results[0]
print entity_pb.property[0].value.list_value[0].entity_value.HasField('key')

I'm fairly certain this will be False and we can avoid a thorny fix.

Thanks a ton for finding and reporting this!

PS I renamed the issue to describe the more generic issue at hand.

[pdknsk]

Thanks, reading works great, writing too.

I have a minor enhancement suggestion, which I'm not sure can be done. When you forget to exclude meta from indexes, writing fails and the error message seems a bit unrelated. Perhaps the user could be given a better error message here. Or better yet, because embedded entities are always unindexed, this could be detected and done automatically.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.local/lib/python2.7/site-packages/gcloud-0.4.0-py2.7.egg/gcloud/datastore/api.py", line 266, in put
    current.commit()
  File "/home/user/.local/lib/python2.7/site-packages/gcloud-0.4.0-py2.7.egg/gcloud/datastore/batch.py", line 207, in commit
    response = self.connection.commit(self._dataset_id, self.mutation)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud-0.4.0-py2.7.egg/gcloud/datastore/connection.py", line 313, in commit
    datastore_pb.CommitResponse)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud-0.4.0-py2.7.egg/gcloud/datastore/connection.py", line 93, in _rpc
    data=request_pb.SerializeToString())
  File "/home/user/.local/lib/python2.7/site-packages/gcloud-0.4.0-py2.7.egg/gcloud/datastore/connection.py", line 70, in _request
    raise make_exception(headers, content)
  File "/home/user/.local/lib/python2.7/site-packages/gcloud-0.4.0-py2.7.egg/gcloud/exceptions.py", line 165, in make_exception
    content = json.loads(content)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[dhermes]

@pdknsk That's a bug in how we print error messages, thanks for finding it!

I filed #626 and we can discuss more there.

In #532 we are discussing when to step in for disallowed things (like indexed entity values) and when to let the backend deliver the same message.