Land Bigtable v2 #1932

tseaver · 2016-06-29T02:02:31Z

This is the monster. See #1850 (comment) for the PRs which landed on the bigtable-v2 branch.

Pass the 'PROTOC_CMD' and 'GRPC_PLUGIN' values through from make's environment to the scripts used to pick apart GRPC-specific code.

Put them in a separate '_generated_v2' subdir, to ease migration. Attempt to factor generation process for clarity (toward #1482).

It was never actually implemented on the back-end in V1, and has been dropped altogether in V2.

Addresses: #1908 (comment)

Generate code for Bigtable v2 protobufs

Processes the 'ReadRowsResponse.CellChunk' state machine. Add tests based on JSON acceptance tests: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/master/bigtable-client-core/src/test/resources/com/google/cloud/bigtable/grpc/scanner/v2/read-rows-acceptance-test.json.

Drop 'Table.rename'.

Indicate their V1 source in their names. Prepratory to converting to V2 equivalents.

…logic Add 'ReadRowsResponseV2' wrapper

…ints Alias Bigtable V1 imports / factories / entry point constants.

Note that {Create,Update,Delete}ColumnFamily messages all collapse to ModifyColumnFamilies.

Update bigtable.column_family to use V2 protos

Assert that the state is 'ROW_IN_PROGRESS', and check that the completed rows match the expected results.

Verify that completed, non-error rows match expected results after an invalid chunk testcase.

…ror-cases Clarify handling of V2 'ReadRowsResponse' error cases

Folds new ReadRowsResponse logic from #1907, #1915 into table row handling.

Convert non-instance-admin protos to Bigtable V2

In V2, those operations are on the instance.

Support V2 instance admin

…tos. Closes #1928. This is a better hack for #1482, but we still really want #1384.

…1 protos. This is a better hack for #1482, but we still really want #1384.

daspecster · 2016-06-29T17:58:28Z

@tseaver, ok sounds like a plan! Thanks!

tseaver · 2016-06-29T17:59:23Z

Just FTR, I was able to run the bigtable system tests under Python 3.4:

$ tox -re system-tests3 --notest
$ .tox/system-tests3/bin/pip install grpcio 
$ .tox/system-tests3/bin/python system_tests/run_system_test.py --package=bigtable
D0629 13:53:51.485668817   28141 ev_posix.c:101]             Using polling engine: poll
test_read_row (bigtable.TestDataAPI) ... ok
test_read_rows (bigtable.TestDataAPI) ... ok
test_read_with_label_applied (bigtable.TestDataAPI) ... ok
test_create_instance (bigtable.TestInstanceAdminAPI) ... ok
test_list_instances (bigtable.TestInstanceAdminAPI) ... ok
test_reload (bigtable.TestInstanceAdminAPI) ... ok
test_update (bigtable.TestInstanceAdminAPI) ... ok
test_create_column_family (bigtable.TestTableAdminAPI) ... ok
test_create_table (bigtable.TestTableAdminAPI) ... ok
test_delete_column_family (bigtable.TestTableAdminAPI) ... ok
test_list_tables (bigtable.TestTableAdminAPI) ... ok
test_update_column_family (bigtable.TestTableAdminAPI) ... ok
(grpcio shutdown noise elided)

----------------------------------------------------------------------
Ran 12 tests in 29.129s

OK

The bigtable-happybase system tests aren't yet Py3k-clean (they use native string literals for row keys, etc.)

mbrukman · 2016-06-29T18:57:53Z

What's the status of this PR? @daspecster, are you reviewing it?

daspecster · 2016-06-29T19:07:25Z

@mbrukman, I am looking it over.
I'm sure we need more eyes than mine on it :)

docs/bigtable-instance-api.rst

+
+.. code:: python
+
+    intances = client.list_intances()


lesv · 2016-06-29T19:17:34Z

@tswast PTAL

lesv · 2016-06-29T19:18:44Z

@jonparrott PTAL - could you review this PR - ideally, we'd like to get it published today as part of Bigtable GA.

gcloud/bigtable/client.py

@@ -18,8 +18,8 @@

 In the hierarchy of API concepts

-* a :class:`Client` owns a :class:`.Cluster`
-* a :class:`.Cluster` owns a :class:`Table <gcloud.bigtable.table.Table>`
+* a :class:`Client` owns a :class:`.Instance`


tswast · 2016-06-29T21:23:38Z

Changes LGTM. I tested with this branch on my Hello World samples, and both ran fine. Thanks!

tseaver · 2016-06-29T21:32:32Z

@dhermes any objections to a merge?

lesv · 2016-06-29T22:22:24Z

@tseaver - We've had two pythonistas review. dhermes is away. It's ok to merge.

lesv · 2016-06-29T22:30:09Z

@tseaver Can you let us know the PyPi status -- (ie submit and let us know)

tseaver · 2016-06-29T22:40:29Z

@lesv 0.17.0 is tagged. Travis will push the release when the tag build is finished.

lesv · 2016-06-29T22:52:22Z

@tseaver What about docs?

tseaver · 2016-06-29T23:00:26Z

@lesv RTD already shows the 0.17.0 docs. The github.io docs get updated by the same Travis job that makes the PyPI release.

mbrukman · 2016-06-29T23:04:18Z

RTD docs on instances have several misspellings: it says intances instead of instances (missing s) which I thought we addressed during code review. One of the more visible ones is right in the section header.

mbrukman · 2016-06-29T23:06:35Z

Also, RTD Bigtable usage docs talk only about clusters, does not mention instances at all.

There's also a possible typo or syntax error on the instance API page, where it says:

[...] object will be returned by create() <gcloud.bigtable.instance.Instance.create>`().

tseaver · 2016-06-29T23:18:10Z

PyPI: https://pypi.python.org/pypi/gcloud/0.17.0
github.io: http://googlecloudplatform.github.io/gcloud-python/stable/bigtable-instance-api.html

mbrukman · 2016-06-29T23:20:53Z

Thank you, @tseaver!! Much appreciated.

tseaver added 30 commits June 24, 2016 15:59

Split 'Makefile' into datastore and bigtable portions.

a43b2eb

Pass the 'PROTOC_CMD' and 'GRPC_PLUGIN' values through from make's environment to the scripts used to pick apart GRPC-specific code.

Generate Bigtable V2 protos.

0f4a064

Put them in a separate '_generated_v2' subdir, to ease migration. Attempt to factor generation process for clarity (toward #1482).

Exclude new '_generated_v2' files from coverage.

9d7d8af

Drop 'Table.rename'.

9a2d313

It was never actually implemented on the back-end in V1, and has been dropped altogether in V2.

Drop overlooked docs / system test usage of 'Table.rename'.

f5b1719

Addresses: #1908 (comment)

Merge pull request #1903 from tseaver/bigtable-v2-regenerate_protobufs

e570b03

Generate code for Bigtable v2 protobufs

Parse JSON acceptance tests.

16b04c9

Merge pull request #1908 from tseaver/bigtable-v2-drop_table_rename

35103f9

Drop 'Table.rename'.

Alias Bigtable V1 imports / factories / entry point constants.

47b9246

Indicate their V1 source in their names. Prepratory to converting to V2 equivalents.

Merge pull request #1907 from tseaver/bigtable-v2-read_rows_response_…

377f0da

…logic Add 'ReadRowsResponseV2' wrapper

Merge pull request #1912 from tseaver/bigtable-v2-alias_imports_endpo…

cdbc47c

…ints Alias Bigtable V1 imports / factories / entry point constants.

Factor out generating GCRule-related protobufs.

3c533e0

Update bigtable.column_family to use V2 protos.

7e9c61d

Note that {Create,Update,Delete}ColumnFamily messages all collapse to ModifyColumnFamilies.

Merge pull request #1913 from tseaver/bigtable-v2-column_family_crud

ae3d818

Update bigtable.column_family to use V2 protos

Clarify: testcases leaving incomplete final rows do not raise.

5d179bc

Assert that the state is 'ROW_IN_PROGRESS', and check that the completed rows match the expected results.

Distinguish response-level errors from invalid chunks.

6b2f07f

Verify that completed, non-error rows match expected results after an invalid chunk testcase.

Merge pull request #1915 from tseaver/bigtable-v2-readrowsresponse-er…

fcc3751

…ror-cases Clarify handling of V2 'ReadRowsResponse' error cases

Factor out generating RowFilter-related protobufs.

8916b6c

Convert to non-instance-admin Bigtable V2 protos.

1b0fb53

Folds new ReadRowsResponse logic from #1907, #1915 into table row handling.

Merge pull request #1919 from tseaver/bigtable-v2-convert_imports

3b48e0f

Convert non-instance-admin protos to Bigtable V2

Drop 'Cluster.table' and 'Cluster.list_tables'.

4ed0b8f

In V2, those operations are on the instance.

In V2, tables hold a ref. to their instance, not a cluster.

7cbb215

Add V2 Instance type.

498271e

Convert instance admin to V2 protobufs.

e144b09

Adjust docs for V2 instance admin patterns.

9880a42

Merge pull request #1923 from tseaver/bigtable-v2-instance_admin

90cd2e7

Support V2 instance admin

Use 'grpcio-tools' in a virtualenv to generate latest Bigtable V2 pro…

aa95442

…tos. Closes #1928. This is a better hack for #1482, but we still really want #1384.

Remove duplicated line.

d97d030

Use 'grpcio-tools' in a virtualenv to generate from latest Bigtable V…

4a223b7

…1 protos. This is a better hack for #1482, but we still really want #1384.

daspecster reviewed Jun 29, 2016
View reviewed changes

docs/bigtable-instance-api.rst

.. code:: python

intances = client.list_intances()

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

tswast mentioned this pull request Jun 29, 2016

Update Bigtable samples to v2. GoogleCloudPlatform/python-docs-samples#392

Merged

tseaver mentioned this pull request Jun 29, 2016

Prepare 0.17.0 release. #1940

Merged

tswast reviewed Jun 29, 2016
View reviewed changes

tseaver merged commit 1a2fa6c into master Jun 29, 2016

tseaver deleted the bigtable-v2 branch June 29, 2016 22:25

mbrukman mentioned this pull request Jun 29, 2016

Fixed spelling and formatting. #1945

Merged

tseaver mentioned this pull request Jun 30, 2016

Bigtable V2 docs fixes lost in merge #1948

Merged

theacodes unassigned daspecster and dhermes Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Land Bigtable v2 #1932

Land Bigtable v2 #1932

tseaver commented Jun 29, 2016

daspecster commented Jun 29, 2016

tseaver commented Jun 29, 2016

mbrukman commented Jun 29, 2016

daspecster commented Jun 29, 2016

This comment was marked as spam.

This comment was marked as spam.

lesv commented Jun 29, 2016

lesv commented Jun 29, 2016

This comment was marked as spam.

This comment was marked as spam.

tswast commented Jun 29, 2016

tseaver commented Jun 29, 2016

lesv commented Jun 29, 2016

lesv commented Jun 29, 2016

tseaver commented Jun 29, 2016 •

edited

Loading

lesv commented Jun 29, 2016

tseaver commented Jun 29, 2016

mbrukman commented Jun 29, 2016

mbrukman commented Jun 29, 2016 •

edited

Loading

tseaver commented Jun 29, 2016

mbrukman commented Jun 29, 2016

Land Bigtable v2 #1932

Land Bigtable v2 #1932

Conversation

tseaver commented Jun 29, 2016

daspecster commented Jun 29, 2016

tseaver commented Jun 29, 2016

mbrukman commented Jun 29, 2016

daspecster commented Jun 29, 2016

This comment was marked as spam.

This comment was marked as spam.

lesv commented Jun 29, 2016

lesv commented Jun 29, 2016

This comment was marked as spam.

This comment was marked as spam.

tswast commented Jun 29, 2016

tseaver commented Jun 29, 2016

lesv commented Jun 29, 2016

lesv commented Jun 29, 2016

tseaver commented Jun 29, 2016 • edited Loading

lesv commented Jun 29, 2016

tseaver commented Jun 29, 2016

mbrukman commented Jun 29, 2016

mbrukman commented Jun 29, 2016 • edited Loading

tseaver commented Jun 29, 2016

mbrukman commented Jun 29, 2016

tseaver commented Jun 29, 2016 •

edited

Loading

mbrukman commented Jun 29, 2016 •

edited

Loading