Size/length limits for Datastore properties are enforced differently than NDB

[eric-optimizely]

The Datastore/NDB documentation states that BlobProperty (and other blob property types) can be of unlimited length (given that an Entity can still only be 1MB).
https://cloud.google.com/appengine/docs/python/ndb/properties#types

When using an unindexed JsonProperty (which is a blob type), the gcloud API enforces a 1500 byte limit, which creates a discrepancy between the GAE Datastore API, documentation, and our expectations.

Here's a stacktrace snippet:

Traceback (most recent call last):
  File "/script/datastore_helpers/gcloud_populate_datastore.py", line 266, in <module>
    populate_with_contents_of_file(path, args.profile)
  File "/script/datastore_helpers/gcloud_populate_datastore.py", line 229, in populate_with_contents_of_file
    api.put(gae_entities)
  File "/lib/python2.7/site-packages/gcloud/datastore/api.py", line 272, in put
    current.commit()
  File "/lib/python2.7/site-packages/gcloud/datastore/batch.py", line 207, in commit
    response = self.connection.commit(self._dataset_id, self.mutation)
  File "/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 324, in commit
    datastore_pb.CommitResponse)
  File "/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 108, in _rpc
    data=request_pb.SerializeToString())
  File "/lib/python2.7/site-packages/gcloud/datastore/connection.py", line 85, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.BadRequest: 400 The value of property &quot;variations.actions&quot; is longer than 1500 bytes.

FWIW: We're still using version 0.4.3 of this gcloud library, but since this response comes from your API, upgrading would not appear to resolve it.

Is there a way to work around this? Should we expect the Cloud Datastore API to have the same limits as the GAE Datastore/NDB API?

[dhermes]

@eric-optimizely Thanks for filing. API issues may be better off in https://github.com/GoogleCloudPlatform/google-cloud-datastore/, since we can't tweak it in gcloud-python. (You're correct that upgrading from 0.4.3 won't change this fact, though I strongly recommend you upgrade.)

/cc @pcostell For this issue

[dhermes]

@eric-optimizely Are you setting this property in exclude_from_indexes?

The key issue may be that ndb.BlobProperty defaults to indexed=False whereas our API doesn't provided any such defaults.

FWIW we are hoping to integrate ndb as in #557

[pcostell]

There are a few key difference between Cloud Datastore and the App Engine Datastore API, however string size is not one of them. It looks like @dhermes is correct and exclude_from_indexes is not getting set.

An incomplete list of difference, just to give you an idea:

• GAE specific things (transactional tasks, user properties)
• "Old" API usage (sequential ID support, invalid UTF-8 strings)
• Things we are still working on:
  • GeoPoints
  • Non-cursor based query continuation
  • Indexes (list of ones used by query, build status)
  • Other (there is some more nuanced stuff)

[dhermes]

Any potential issue here seems to be similar to #1206 which was closed by #1241. @eric-optimizely feel free to re-open if you think there's more to discuss.