Add GeneratedClassRowTypeConstraint #22679

TheNeuralBit · 2022-08-10T23:11:35Z

A RowTypeConstraint that wraps a generated NamedTuple type cannot be pickled, because the generated type cannot be pickled. This PR adds a specialization, GeneratedClassRowTypeConstraint, with a custom __reduce__ that avoids pickling the user type.

Extracted from #22575

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…nstraint

codecov · 2022-08-10T23:32:22Z

Codecov Report

Merging #22679 (59686fe) into master (c9c57a7) will decrease coverage by 0.00%.
The diff coverage is 97.29%.

@@            Coverage Diff             @@
##           master   #22679      +/-   ##
==========================================
- Coverage   74.19%   74.19%   -0.01%     
==========================================
  Files         708      709       +1     
  Lines       93465    93498      +33     
==========================================
+ Hits        69347    69367      +20     
- Misses      22843    22856      +13     
  Partials     1275     1275

Flag	Coverage Δ
python	`83.58% <97.29%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ks/python/apache_beam/typehints/schema_registry.py	`95.23% <95.23%> (ø)`
sdks/python/apache_beam/typehints/schemas.py	`94.17% <97.14%> (-0.15%)`	⬇️
sdks/python/apache_beam/coders/row_coder.py	`94.54% <100.00%> (+0.05%)`	⬆️
sdks/python/apache_beam/transforms/core.py	`92.94% <100.00%> (ø)`
sdks/python/apache_beam/typehints/row_type.py	`97.01% <100.00%> (+0.46%)`	⬆️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (-2.44%)`	⬇️
sdks/python/apache_beam/utils/subprocess_server.py	`56.54% <0.00%> (-2.20%)`	⬇️
...python/apache_beam/runners/worker/worker_status.py	`77.53% <0.00%> (-2.18%)`	⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py	`88.62% <0.00%> (-0.32%)`	⬇️
...eam/runners/interactive/interactive_environment.py	`91.71% <0.00%> (-0.31%)`	⬇️
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2022-08-11T01:34:16Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

TheNeuralBit · 2022-08-11T15:51:03Z

R: @yeandy

Since valentyn is out

github-actions · 2022-08-11T15:54:12Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

yeandy

LGTM. Probably need to run formatter. Also seeing some Dataframe schema test errors, but didn't look too much into that.

yeandy · 2022-08-11T17:49:36Z

sdks/python/apache_beam/typehints/row_type.py

+ schema_options: Optional[Sequence[Tuple[str, Any]]] = None,
+ field_options: Optional[Dict[str, Sequence[Tuple[str, Any]]]] = None,
+ schema_registry: SchemaTypeRegistry = None,
+ ) -> RowTypeConstraint:


Suggested change

) -> RowTypeConstraint:

) -> GeneratedClassRowTypeConstraint:

I actually prefer to use the base-class here. GeneratedClassRowTypeConstraint can be an implementation detail.

yeandy · 2022-08-11T21:52:50Z

sdks/python/apache_beam/typehints/schemas_test.py

+ row_proto = schema_pb2.FieldType(
+ row_type=schema_pb2.RowType(


Why does schema_pb2.RowType have to be wrapped by schema_pb2.FieldType?

Looked again, looks like top level of protos are schema_pb2.FieldType I believe

yeandy · 2022-08-11T22:13:50Z

sdks/python/apache_beam/typehints/row_type.py

+ self._fields,
+ self._schema_id,
+ self._schema_options,
+ self._field_options))


Nice. Do you it would be clear to explicitly add None for the 5th item (schema_registry)? I was initially looking for the 5 args for from_fields, but only saw 4 😄 . I'm perfectly ok with the current implementation though.

Good idea, thanks

sdks/python/apache_beam/typehints/row_type.py

yeandy · 2022-08-11T22:17:35Z

sdks/python/apache_beam/typehints/row_type.py

+class GeneratedClassRowTypeConstraint(RowTypeConstraint):
+ """Specialization of RowTypeConstraint which relies on a generated user_type.
+
+ Since the generated user_type cannot be pickled, we supply a custom __reduce__


the generated user_type cannot be pickled

Can you please explain why this is the case?

I actually don't fully understand it, but it's been a consistent issue with the Schema code. Each pickle library (built-in, dill, cloudpickle) fails for a different reason. I filed #22714 to track this, and added a (skipped) test.

sdks/python/apache_beam/typehints/row_type.py

Co-authored-by: Andy Ye <andyye333@gmail.com>

chamikaramj · 2022-08-16T21:31:11Z

Probably this resulted in x-lang test suite failures here: #22748

…duced in apache#22679

* Add (failing) pickling tests * Add GeneratedClassRowTypeConstraint, plumb options * Add top-level option conversion functions * Refactor NamedTuple generation, always create GeneratedClassRowTypeConstraint * Move registry to apache_beam.typehints.schema_registry * yapf,lint * fixup! Move registry to apache_beam.typehints.schema_registry * Apply suggestions from code review Co-authored-by: Andy Ye <andyye333@gmail.com> * Add None SchemaRegistry * Add skipped test for pickling generated type Co-authored-by: Andy Ye <andyye333@gmail.com>

…duced in apache#22679

TheNeuralBit added 4 commits August 10, 2022 16:06

Add (failing) pickling tests

71992e7

Add GeneratedClassRowTypeConstraint, plumb options

b1e49ae

Add top-level option conversion functions

6ebbbf6

Refactor NamedTuple generation, always create GeneratedClassRowTypeCo…

bb17e7a

…nstraint

github-actions bot added the python label Aug 10, 2022

TheNeuralBit mentioned this pull request Aug 10, 2022

Add BatchConverter implementations for pandas types, use Batched DoFns in DataFrame convert utilities #22575

Merged

TheNeuralBit added 3 commits August 10, 2022 16:49

Move registry to apache_beam.typehints.schema_registry

d4e7cfb

yapf,lint

8529cb4

fixup! Move registry to apache_beam.typehints.schema_registry

f60fbab

github-actions bot added the Next Action: Reviewers label Aug 11, 2022

yeandy approved these changes Aug 11, 2022

View reviewed changes

TheNeuralBit and others added 2 commits August 12, 2022 09:27

Apply suggestions from code review

0f20d6f

Co-authored-by: Andy Ye <andyye333@gmail.com>

Add None SchemaRegistry

2ee9040

TheNeuralBit mentioned this pull request Aug 12, 2022

[Bug]: Python schema generated types cannot be pickled #22714

Open

Add skipped test for pickling generated type

59686fe

TheNeuralBit force-pushed the generated-class-rowtypeconstraint branch from e3e71cc to 59686fe Compare August 12, 2022 17:02

TheNeuralBit merged commit 3173b50 into apache:master Aug 12, 2022

chamikaramj mentioned this pull request Aug 16, 2022

[Bug]: ValidateRunnerXlangTest$SingleInputOutputTest.test failing for various x-lang test suites #22748

Open

TheNeuralBit mentioned this pull request Aug 25, 2022

[Bug]: Type inference failing for Python SDK with External transforms and beam.Row #22854

Closed

Abacn mentioned this pull request Aug 26, 2022

Support Timestamp type in xlang JDBC Read and Write #22561

Merged

4 tasks

Abacn added a commit to Abacn/beam that referenced this pull request Aug 26, 2022

Fix logical type with same language type gets completely hidden intro…

b0484e7

…duced in apache#22679

dedocibula pushed a commit to dedocibula/beam that referenced this pull request Sep 15, 2022

Fix logical type with same language type gets completely hidden intro…

675ce34

…duced in apache#22679

kkdoon pushed a commit to twitter-forks/beam that referenced this pull request Sep 29, 2022

Fix logical type with same language type gets completely hidden intro…

12b9b6a

…duced in apache#22679

cushon pushed a commit to cushon/beam that referenced this pull request Oct 17, 2022

Fix logical type with same language type gets completely hidden intro…

4da6064

…duced in apache#22679

Abacn mentioned this pull request Aug 25, 2023

[Bug]: Reading from BigQuery provides inconsistent schemas #28151

Open

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GeneratedClassRowTypeConstraint #22679

Add GeneratedClassRowTypeConstraint #22679

TheNeuralBit commented Aug 10, 2022

codecov bot commented Aug 10, 2022 •

edited

Loading

github-actions bot commented Aug 11, 2022

TheNeuralBit commented Aug 11, 2022

github-actions bot commented Aug 11, 2022

yeandy left a comment

yeandy Aug 11, 2022

TheNeuralBit Aug 12, 2022

yeandy Aug 11, 2022

yeandy Aug 11, 2022

yeandy Aug 11, 2022

TheNeuralBit Aug 12, 2022

yeandy Aug 11, 2022

TheNeuralBit Aug 12, 2022

chamikaramj commented Aug 16, 2022

	) -> RowTypeConstraint:
	) -> GeneratedClassRowTypeConstraint:

		row_proto = schema_pb2.FieldType(
		row_type=schema_pb2.RowType(

Add GeneratedClassRowTypeConstraint #22679

Add GeneratedClassRowTypeConstraint #22679

Conversation

TheNeuralBit commented Aug 10, 2022

GitHub Actions Tests Status (on master branch)

codecov bot commented Aug 10, 2022 • edited Loading

Codecov Report

github-actions bot commented Aug 11, 2022

TheNeuralBit commented Aug 11, 2022

github-actions bot commented Aug 11, 2022

yeandy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chamikaramj commented Aug 16, 2022

codecov bot commented Aug 10, 2022 •

edited

Loading