Implement datastore v1beta3

[dhermes]

Current plan of attack:

• Remove --plugin=protoc-gen-grpc=$(GRPC_PLUGIN) from "helper" modules that don't use gRPC (e.g. google.bigtable.v1.bigtable_data) (this way sub-packages that don't need gRPC can use them) (Updating Makefile to only use gRPC when needed. #1354)
• Ditch shared generated modules for those included with protobuf and googleapis-common-protos (Use googleapis common protos #1353) Move shared (i.e. non-Bigtable specific) modules from gcloud.bigtable._generated to gcloud._generated (this way all sub-packages can have access to shared protobuf message classes) and then subsequently update the bigtable imports to accommodate this change
• Pull out _get_pb_property_value in gcloud.bigtable.cluster into core (gcloud._helpers) and separate out a _has_field piece since pb_message.HasField doesn't work on non-message fields in proto3 (Moving _get_pb_property_value from bigtable into core. #1329)
• Update Makefile to reflect changes above (i.e. make sure it can be run without changing the code) (See Bringing auto-gen import re-writing up to date with files. #1316)
• Update Makefile to automatically add the _foo.proto files to the repo (See Making pb2 auto-gen script also copy over .proto files. #1317)
• Fold Batch.add_auto_id_entity into Batch.put (the auto_id_entity mutation is removed in favor of just using insert with a partial key) (see Removing datastore Batch.add_auto_id_entity. #1296)
• Loosen Batch implementation's reliance on Mutation message class (it is used in a LookupRequest and their combined structure changes in v1beta3) (See Renaming datastore Batch.mutation to mutations. #1306, Puting helpers in datastore Batch for getting new mutations. #1319, Using protobuf CommitRequest in datastore Connection.commit. #1341)
• Remove use of serializable in Client.transaction and Transaction (it is no longer an option in v1beta3, so we will curtail it's use in v1beta2 before the switch) (See Removing serializable option from datastore Transaction. #1294)
• Manually partition the _datastore_v1_pb2.py module into 3 modules that just import portions of the namespace. These modules (datastore_pb2.py, entity_pb2.py, query_pb2.py) will be according to the membership in the new proto definitions for v1beta3. (See Replacing datastore pb uses with entity shim. #1297, Replacing datastore pb uses with query shim. #1299, Replacing datastore pb uses with "datastore" shim. #1301, Creating _generated pb directory for datastore #1328)
• Switch over imports from directly using _datastore_v1_pb2.py to using our "shim" imports to mock the structure of v1beta3. (See Replacing datastore pb uses with entity shim. #1297, Replacing datastore pb uses with query shim. #1299, Replacing datastore pb uses with "datastore" shim. #1301, Creating _generated pb directory for datastore #1328)
• Make sure API_BASE_URL for the Connection class is used (rather than a parent's version) (See Explicitly using API_BASE_URL from current connection in datastore. #1293)
• Change Client.dataset_id to Client.project (in advance of the rename to PartitionId.project_id in v1beta3) (Replace dataset id with project in datastore #1330)
• Change the return type of Connection.commit to be a tuple of index_updates and mutation_results (they were previously on the same result object but are being split apart in v1beta3) (See Reducing datastore commit reliance on structure of response. #1314)
• Replace uses of HasField for non-message values with _has_field (see above) (Moving _get_pb_property_value from bigtable into core. #1329)
• Remove v1beta2 generated pb2 file and old .proto definition (can also remove _datastore_v1_pb2.py from the pylintrc_default ignored file) and delete our "shim" modules (Upgrading Makefile to generate datastore v1beta3. #1355, Upgrading Makefile to generate datastore v1beta3. #1428)
• Update Makefile to incorporate protobuf definitions for v1beta3 (I did this in a side-project, Use googleapis common protos #1353, Updating Makefile to only use gRPC when needed. #1354, Upgrading Makefile to generate datastore v1beta3. #1355, Upgrading Makefile to generate datastore v1beta3. #1428)
• Rewrite our "shim" imports to use the actual datastore._generated files for v1beta3.
• Switch API_BASE_URL from https://www.googleapis.com to https://datastore.googleapis.com and drop 'https://www.googleapis.com/auth/userinfo.email from the scope list (Updating datastore URI template for v1beta3. #1339, Updating datastore URI template for v1beta3. #1406)
• Accommodate renames / retyped / removed
  • CommitRequest.mutation --> CommitRequest.mutations (Updating CommitRequest, Mutation and helpers for v1beta3. #1461)
  • LookupRequest.key --> LookupRequest.keys (Handling datastore renames key -> keys #1358, Handling datastore renames key -> keys #1456)
  • AllocateIdsResponse.key --> AllocateIdsResponse.keys (Handling datastore renames key -> keys #1358, Handling datastore renames key -> keys #1456)
  • AllocateIdsRequest.key --> AllocateIdsRequest.keys (Handling datastore renames key -> keys #1358, Handling datastore renames key -> keys #1456)
  • Key.path_element --> Key.path (Renaming path_element->path in Key. #1360, Renaming path_element->path in Key. #1457)
  • Entity.property --> Entity.properties (Upgrading Entity.property to properties map in datastore. #1458)
  • Query.group_by --> Query.distinct_on (Handling datastore renames on Query and QueryResultBatch. #1357, Handling datastore renames on Query and QueryResultBatch. #1455)
  • Query.limit --> Query.limit.value (Handling datastore renames on Query and QueryResultBatch. #1357, Handling datastore renames on Query and QueryResultBatch. #1455)
  • QueryResultBatch.entity_result --> QueryResultBatch.entity_results (Handling datastore renames on Query and QueryResultBatch. #1357, Handling datastore renames on Query and QueryResultBatch. #1455)
  • PartitionId.namespace --> PartitionId.namespace_id (Handle datastore renames on PartitionId #1359, Handle datastore renames on PartitionId #1452)
  • PartitionId.dataset_id --> PartitionId.project_id (Handle datastore renames on PartitionId #1359, Handle datastore renames on PartitionId #1452)
  • Value.indexed --> Value.exclude_from_indexes (Rename Value.indexed->exclude_from_indexes. #1365, Rename Value.indexed->exclude_from_indexes. #1453)
  • Value.list_value --> Value.array_value (Upgrading list_value -> array_value for v1beta3. #1460)
  • Value.null_value added (Adding support for null and geo point values in v1beta3. #1464)
  • Value.timestamp_microseconds_value --> Value.timestamp_value (with type change) (Moving _pb_timestamp_to_datetime into core. #1361, Upgrading timestamp_microseconds_value to timestamp_value. #1459)
  • Value.geo_point_value added (Adding support for null and geo point values in v1beta3. #1464)
  • Value.blob_key_value removed
  • CompositeFilter.filter --> CompositeFilter.filters (Handling datastore renames on CompositeFilter and PropertyFilter. #1356, Handling datastore renames on CompositeFilter and PropertyFilter. #1454)
  • CompositeFilter.operation --> CompositeFilter.op (Handling datastore renames on CompositeFilter and PropertyFilter. #1356, Handling datastore renames on CompositeFilter and PropertyFilter. #1454)
  • PropertyFilter.operation --> PropertyFilter.op (Handling datastore renames on CompositeFilter and PropertyFilter. #1356, Handling datastore renames on CompositeFilter and PropertyFilter. #1454)
  • CompositeFilter.AND --> CompositeFilter.OPERATOR_UNSPECIFIED (as default value) (Handling datastore renames on CompositeFilter and PropertyFilter. #1356, Handling datastore renames on CompositeFilter and PropertyFilter. #1454)
  • ReadOptions.DEFAULT --> ReadOptions.READ_CONSISTENCY_UNSPECIFIED (as default value) (Handling datastore renames on CompositeFilter and PropertyFilter. #1356, Handling datastore renames on CompositeFilter and PropertyFilter. #1454)
• Deal with the fact that Entity.property is not a map called Entity.properties (somewhat different semantics) (Adding helpers for interacting with properties in Entity protobuf. #1340)
• Tear out dataset (project) prefix code (Removing hacks that avoid using project ID in key protos. #1466)
• Re-vamp environment variable usage (dataset_id no longer needed?) (Removing custom dataset ID environment variable. #1465)
• Remove use of isolation_level in Connection.begin_transaction (Removing use of isolation level in datastore. #1343, Removing use of isolation level in datastore. #1407)

New Features:

• Add GqlQuery support (Missing GQL support #304)
• QueryResultBatch.skipped_cursor added (only set when skipped_results != 0)
• EntityResult.cursor added (set by the backend when the entity result is part of a RunQueryResponse.batch.entity_results response)
• Parse protobuf errors

[pcostell]

All those project IDs have already been whitelisted for the API.

Just double checking that this is a typo in the issue, but the correct url is:

https://datastore.googleapis.com/v1beta3/projects/omega-moonlight-697:runQuery

[dhermes]

D'oh! Thought I changed it! Thanks for the keen eye

[dhermes]

Culprit found, the constructor was relying on the connection base class (non-datastore specific)

from gcloud import connection
...
        if api_base_url is None:
            api_base_url = os.getenv(GCD_HOST,
                                     connection.API_BASE_URL)

[dhermes]

For posterity, here is the "retired" original description:

---

pcostell I believe I am "done" updating for v1beta3 but am getting 404s when running the system tests with both

https://www.googleapis.com/v1beta3/projects/omega-moonlight-697:runQuery
https://www.googleapis.com/v1beta3/projects/s~omega-moonlight-697:runQuery

I think it requires a whitelist at this point to use v1beta3? I seem to remember a whitelist discussion but can't find any evidence of this.

Project IDs:

omega-moonlight-697
precise-truck-742
citric-celerity-697

/cc tseaver

[dhermes]

FYI @jgeewax @tseaver I just did a merge of master in the datastore-v1beta3 branch. I don't know of a better place to surface such comments. Just wanted to keep them in sync / catch any merge conflicts before the happen. Merged perfectly with no issues!

[dhermes]

Another FYI, just did another merge. Only one merge conflict in system_tests/run_system_test.py (due to Bigtable modifying REQUIREMENTS and datastore-v1beta3 just removing it as a variable). Merge conflict easily resolved and merge completed.

[navreetaft]

Will this update change the user facing API for gcloud-python when interacting with datastore?

Also, will this help with ndb being implemented in gcloud-python? If so, any idea on an ETA for that? Also, @dhermes commented that

ndb saves take about 1/5 the amount of time the gcloud saves do because the RPCs happen directly in GAE instead of via the Cloud Datastore API

Will this performance gap be addressed by #557?

[dhermes]

@navreetaft Essentially yes. The only interface change is Query.group_by to Query.distinct_on (to match the changed name in the protobuf). We also add support for geo points and for explicit null values stored and remove support for blob key values. We also plan to add support for GqlQuery.

You can check it out right now at the datastore-v1beta3 branch. It is 100% working (system tests even pass against the new version).

[dhermes]

As for the performance gap, that is in reference to code running within App Engine.

[navreetaft]

I see, so the performance gap is due to ndb being able to talk directly to DataStore servers, whereas gcloud-python will have to go through some REST gateway (an additional hop)? Just want to make sure I understand.

[dhermes]

Yup. The ndb library (inside GAE) talks to Google services via RPCs (remote-procedure calls) while gcloud-python (at least when I ran that test in November 2014) was using standard HTTPS requests to just call the datastore API.

[dhermes]

@pcostell May have some thoughts / comments here

[pcostell]

Yes, right now ndb inside GAE will still be faster than gcloud-python due to communication via RPC instead of HTTPS requests. gcloud-python on v1beta3 will be faster from GAE because v1beta3 is more performant than v1beta2, however it likely still won't be as fast as ndb in GAE for the same reasons above.

[dhermes]

Another FYI, just did another merge. Only merge conflict was the delete of the demos.

/cc @pcostell

[dhermes]

Another FYI, just did another merge but rebased by mistake, d'oh!

@pcostell Both of my projects got 400's when running the system tests against v1beta3

[pcostell]

@dhermes My guess this is related to some permissions changes -- now you have to explicitly enable the API (instead of just being part of the whitelist).

You can either login to your app with an @google.com account to do this (look for Cloud Datastore API - NEW in the API manager), or add me as an owner and I can do it for you.

[pcostell]

@dhermes Are there any steps that need to happen before we can merge the v1beta3 branch into master and push to pypi?

[dhermes]

1. Merge the datastore-v1beta3 branch into master
2. Send a PR to update the version in setup.py
3. Push a new tag with the version (Travis pushes to PyPI on tag builds)

[pcostell]

@dhermes We are ready for you to merge this in and push to pypi.

[dhermes]

w00t.

[dhermes]

Did my final merge of master into datastore-v1beta3. Only conflict was in run_system_tests.py (as predicted).

[tseaver]

@pcostell I can't seem to enable the v1beta3 API. I have added you as an owner to my project: can you figure out what I screwed up?

[pcostell]

@tseaver There is an (unfortunately timed) issue with the API service causing this issue. In particular if you disable Datastore v1beta2 (and potentially other APIs), you get left in state where you can't enable APIs. I'll follow up when I get more details.

[tseaver]

@pcostell thanks for looking into it.

[pcostell]

googleapis/google-cloud-datastore#97

[tseaver]

@pcostell Thank you -- whatever you did for the user in that issue seems to have worked for me, too.